Booservers - All about dedicated servers

4 Primary Areas For Tuning Your Server

Thu, 04 Sep 2008 14:57:07 +0000

I just thought I would add my two cents in for everyone. I posted an old howto with some sysctl.conf, but I think that was when I was running on my previous server using and older version of redhat. So here's a fresh howto that is a little more complete.

These configs are based on my server specs, which is a Dual 2.0GHz Xeon with 2GB of RAM running RedHat Enterprise. Depending on your server's RAM you might have to reduce some of the settings, which I'll try make notes with each section.

First, is the /etc/sysctl.conf file. Most people overlook tweaking these settings, always thinking it is a mysql or apache problem. You can get a tremendous boost in throughput by adjusting these settings. These are the settings I use on my server, and have come about by constantly adjusting and monitoring performance, and this is what works best for me, your mileage may vary based on server specs and traffic. I suggest finding some guides and reading up about what each seting does before you make changes. (Note: most out there are pretty dated unfortunatly). Also, some people out there like to have tcp_window_scaling, sack, fack, etc, turned off, but I leave them on. I guess it is just a personal preference thing. So don't complain, but feel free to leave your comments, testing, and results.

/etc/sysctl.conf

CODE

# Kernel sysctl configuration file for Red Hat Enterprise Linux

# Controls IP packet forwarding

net.ipv4.ip_forward = 0

# Controls source route verification

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.all.rp_filter = 1

# Disables IP source routing

net.ipv4.conf.default.accept_source_route = 0

net.ipv4.conf.all.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.

# Useful for debugging multi-threaded applications.

kernel.core_uses_pid = 1

# Increase maximum amount of memory allocated to shm

# Only uncomment if needed!

# kernel.shmmax = 67108864

# Disable ICMP Redirect Acceptance

net.ipv4.conf.default.accept_redirects = 0

net.ipv4.conf.all.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets

net.ipv4.conf.default.log_martians = 1

net.ipv4.conf.all.log_martians = 1

# Decrease the time default value for tcp_fin_timeout connection

net.ipv4.tcp_fin_timeout = 25

# Decrease the time default value for tcp_keepalive_time connection

net.ipv4.tcp_keepalive_time = 1200

# Turn on the tcp_window_scaling

net.ipv4.tcp_window_scaling = 1

# Turn on the tcp_sack

net.ipv4.tcp_sack = 1

# tcp_fack should be on because of sack

net.ipv4.tcp_fack = 1

# Turn on the tcp_timestamps

net.ipv4.tcp_timestamps = 1

# Enable TCP SYN Cookie Protection

net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request

net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection

net.ipv4.icmp_ignore_bogus_error_responses = 1

# Make more local ports available

# net.ipv4.ip_local_port_range = 1024 65000

# Set TCP Re-Ordering value in kernel to '5'

net.ipv4.tcp_reordering = 5

# Lower syn retry rates

net.ipv4.tcp_synack_retries = 2

net.ipv4.tcp_syn_retries = 3

# Set Max SYN Backlog to '2048'

net.ipv4.tcp_max_syn_backlog = 2048

# Various Settings

net.core.netdev_max_backlog = 1024

# Increase the maximum number of skb-heads to be cached

net.core.hot_list_length = 256

# Increase the tcp-time-wait buckets pool size

net.ipv4.tcp_max_tw_buckets = 360000

# This will increase the amount of memory available for socket input/output queues

net.core.rmem_default = 65535

net.core.rmem_max = 8388608

net.ipv4.tcp_rmem = 4096 87380 8388608

net.core.wmem_default = 65535

net.core.wmem_max = 8388608

net.ipv4.tcp_wmem = 4096 65535 8388608

net.ipv4.tcp_mem = 8388608 8388608 8388608

net.core.optmem_max = 40960

After you make the changes to the file, you can make them effective immediately by typing in /sbin/sysctl -p

Also, you will need to issue /sbin/sysctl -w net.ipv4.route.flush=1 to flush the routing table to make some of these changes happen instantly.

Here's some URLs with useful info, benchmarks, etc... (I believe one was posted from someone below)
http://www.aarnet.edu.au/engineering/netwo.../mtu/local.html
http://sverre.home.cern.ch/sverre/TenGBE_w...er_04232003.pdf
http://www.hep.ucl.ac.uk/~ytl/tcpip/linux/...en/datatag-tcp/
http://www-didc.lbl.gov/TCP-tuning/TCP-tuning.html
http://ipsysctl-tutorial.frozentux.net/chu...html/index.html

-------------------------------------------

Second is the MySQL /etc/my.cnf settings file. A lot of people just leave this file with its default settings until they notice problems with their server performance. Please note that I'm not including the datadir or socket settings since those can vary based on your server setup. Also I'm only including the base [mysqld] section and not any of the [safe_mysqld], [mysqldump], or [myisamchk] sections.

Also, update your MySQL to the latest version, if you are still running 3.x you should be dragged out into the street and beaten with a stick, seriously. Just download the MySQL RPMs from the MySQL website, it takes 30 seconds to upgrade. They usually release a new version every month. Be aware of the difference between 4.0.x and 4.1.x (or higher).

CODE

[mysqld]

connect_timeout=15

interactive_timeout=100

join_buffer_size=1M

key_buffer=256M

max_allowed_packet=16M

max_connections=500

max_connect_errors=10

myisam_sort_buffer_size=64M

read_buffer_size=2M

read_rnd_buffer_size=2M

sort_buffer_size=2M

table_cache=1024

thread_cache_size=100

thread_concurrency=4

wait_timeout=300

query_cache_size=128M

query_cache_limit=1M

query_cache_type=1

skip-innodb

For people with a single CPU be sure to set thread_concurrency to 2 (4 is for Dual CPUs). People with 1GB of RAM, you might want to consider lowering the key_buffer to 64M and the myisam_sort_buffer_size to 32M. This really just depends on how much free memory your system has during peak traffic hours. If you increase these too much and your system runs out of physical RAM and starts swapping to disk, your system is going to eat it hard.

For more information about Mysqld variables, please read the following articles as they explain all the settings in-depth and how to fine-tune them: Article 1 and Article 2 and Article 3

-------------------------------------------

Third is Apache. Some people run 1.x, and some run 2.x, me personally I run 2.x because of the better performance. But some people are tied to the older version because of other software packages.

The first thing to do if you are running 1.x is to get mod_gzip and use it. If you are running 2.x then use mod_deflate (it is included). This compresses all your HTML/TXT/XML data before it is sent, saving you bandwidth, and faster load times for your users.

If you are serving up pages + images then you *probably* want to set your keepalive to on, and have your settings something like this:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 10

By setting the KeepAliveTimeout low you won't have all those lingering connections. You can probably set it even lower if you like.

If you are only serving up html (or php or whatever) pages, and using another web server for your images (like tux). Then you probably want to set your KeepAlive to Off since the user will only be requesting 1 file at a time.

Most people have the bad habit of instantly increasing their MaxClients to 256. This can be BAD if you don't take into account memory availability. You need to determine how much memory you have free, how much each apache process consumes, then do the math to figure out what you can safely set the MaxClients to. If you exceed your physical memory then once again the server will swap to the HD and the server will take a dive in performance.

Comment out / remove and Dynamic Shared Object (DSO) modules that you do not use! There are a ton loaded by default, most which you will never use. I commented out 20+ personally! Read the apache documentation on what each one does, the apache docs are very detailed.

If possible, set the AllowOverride option to None. This prevents apache from checking for the .htaccess file in every directory whenever a request is made. However if you use .htaccess files then you have to leave the setting there, but if you can limit it down to certain directories, then do it.

Mask your Apache version by using the following settings:
ServerSignature Off
ServerTokens ProductOnly

That's just good practice, you can also hide your PHP info by setting expose_php = Off in your /etc/php.ini file.

Fourth is PHP. One thing to do is use a program like eAccelerator which caches pre-compiled versions of your php files to help reduce overhead and increase performance. It is a free download from sourceforge, but it will require a little know-ho on your part to install. There are plenty of other guides on how to install this. It is very simple and quick.

A lot of people use the redhat PHP RPMs, which can be quite bloated. My libphp4.so module is only 2.07MB in size. (I don't remember what the default redhat one is, but I'm willing to bet it is larger). Also Redhat never seems to keep up to date with the latest PHP (or MySQL) version, I always recommend updating as soon as a new release is published.

Here's my configure line. There's a lot of settings you may not use, and they could be ones that you use that I don't. You can view your current configure line via the phpinfo() function. These include all the big things such as GD, XML, SHM, etc.. Some people maybe want to enable a certain memory-limit to prevent PHP from eating too much memory per process.

Also, I don't use mm simply because I found it would crash apache on an almost daily basis. I had problems with session storage, and also it would not restart after rotating logs...

CODE

./configure

--prefix=/usr

--exec-prefix=/usr

--bindir=/usr/bin

--sbindir=/usr/sbin

--sysconfdir=/etc

--datadir=/usr/share

--includedir=/usr/include

--libdir=/usr/lib

--libexecdir=/usr/libexec

--localstatedir=/var

--sharedstatedir=/usr/com

--mandir=/usr/share/man

--infodir=/usr/share/info

--disable-cgi

--disable-debug

--disable-rpath

--disable-memory-limit

--disable-ipv6

--disable-safe-mode

--enable-pic

--enable-discard-path

--enable-inline-optimization

--enable-gd-native-ttf

--enable-gd-imgstrttf

--enable-magic-quotes

--enable-sysvsem

--enable-sysvshm

--enable-sysvmsg

--enable-shmop

--enable-track-vars

--enable-exif

--enable-wddx

--enable-bcmath

--enable-calendar

--enable-ftp

--enable-inline-optimization

--with-apxs2=/usr/sbin/apxs

--with-mysql=/usr

--with-pear

--with-config-file-path=/etc

--with-exec-dir=/usr/bin

--with-gd

--with-png-dir=/usr

--with-jpeg-dir=/usr

--with-freetype-dir=/usr

--with-gettext

--with-openssl

--with-regex

--with-ttf=/usr

--with-expat-dir=/usr

--with-dom=/usr

--with-dom-xslt=/usr

--with-dom-exslt=/usr

--with-iconv

--with-db4=/usr

--with-gdbm=/usr

--with-zlib=/usr

--with-zlib-dir=/usr

--with-xmlrpc

--with-xml

--with-bz2=/usr

--with-cdb

--enable-mbstring

-------------------------------------------

When compiling programs (like PHP, eaccelerator, etc..), you can fine-tune some of your compile-options to enhance performance for your CPU's capabilities (and remove excess stuff like debug info)

As mentioned before, I run dual xeon's (P4's for all practical purposes). If you are using a different CPU then you might have to go look up the proper flags at the GCC website.

Before compiling a program, you can set the following flags:

CODE

export CFLAGS="-O3 -pipe -mcpu=pentium4 -march=pentium4 -fomit-frame-pointer"

export CXXFLAGS="${CFLAGS}"

export CHOST="i686-pc-linux-gnu"

export MAKEOPTS="-j2"

export LDFLAGS="-Wl,-O1"

These flags are considered "stable" and should enhance performance a little for software that you compile with these options. There are tons of other flags, however some reduce precision for certain math (which can cause problems in certain software) and others may reduce stabililty.

-------------------------------------------

I guess that's about it... Use the information at your own risk. Hopefully it will help some people out, or at least point them in the right direction.

Please don't post questions that are like: "here's my config, can you optimize it for X server?". I don't check these forums that often, so I probably won't reply to your question.

Server tuning is more of an art than just entering X setting to Y number. Before making changes, keep your old configs. Also get a monitoring program so you can graph out your server load and other vitals. That way you can see before & after results. Also, if you run a forum, let your users know that you are going to make changes, and get their feedback on response time and such from them.

Enjoy.

Source: http://forums.theplanet.com/index.php?showtopic=48880

Easy CURL 7.10.4 w/SSL

Thu, 15 Dec 2005 21:35:04 +0000

We've put together these RPM's for libcurl 7.10.2 with SSL support. They upgrade over the stock RH ones. Installation is via rpm -Uvh

Applicability (RH 7.2 / 7.3 required) :

Webppliance 3.1
Webppliance Pro 3.5
cPanel 6
Plesk 5.0

These are also Modernbill compatible on all above panel types.

Downloadables:

curl-7.10.4-1.i386.rpm
curl-devel-7.10.4-1.i386.rpm

MD5 Checksums

37a04f7db8c34fddbb309d067d16a453 curl-7.10.4-1.i386.rpm
e09253b7c227573f4e5b10d3987f433b curl-devel-7.10.4-1.i386.rpm

Credits: http://www.cheetaweb.com/

APF Firewall 0.9.4-7

Thu, 15 Dec 2005 21:32:07 +0000

Just thought I'd update the howto's for APF.

Type ifconfig

Find out if it�s using eth0 or eth1.

Usually its eth0 but if its not, change it in conf.apf or you�ll be completely blocking the server from access

wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz

tar -xvzf apf-current.tar.gz
cd apf*
./install.sh
pico -w /etc/apf/conf.apf

RESV_DNS="1"

All SYSCTL options should be set to 1 EXCEPT for
SYSCTL_OVERFLOW="0"
SYSCTL_SYNCOOKIES="0"

USE_DS="1"
USE_AD="1"

FOR PLESK:

IG_TCP_CPORTS="20,21,22,25,53,80,110,143,443,465,993,995,8443"
IG_UDP_CPORTS="37,53,873"

EGF="1"
EG_TCP_CPORTS="20,21,22,25,53,37,43,80,113,443,465,873"
EG_UDP_CPORTS="53,873"

For CPANEL:

IG_TCP_CPORTS="20,21,22,25,53,80,110,143,443,993,995,2082,2083,20 86,2087,2095,2096"
IG_UDP_CPORTS="37,53,873"

EGF="1"
EG_TCP_CPORTS="20,21,22,25,37,53,43,80,113,443,465,873,2087,2089"
EG_UDP_CPORTS="53,873"

apf �s to start firewall.

If you are not kicked out of SSH and you can type commands after it loads, that means it�s installed correctly.

pico -w /etc/apf/ad/conf.antidos

LP_KLOG="1"

USR_ALERT="1"
USER = �root�
ARIN_ALERT="1"

pico -w /etc/apf/conf.apf

change DEVM to 0

apf -r

Get FTP running on your P-NET (PNET) storage server

Thu, 15 Dec 2005 20:43:11 +0000

Telnet/SSH into your P-NET server as root (or use su).

Use the following command:
/sbin/service vsftpd start

This will start the ftp server.

That's it!

If you are going to use cPanel's backup feature you should probably create a separate user account.

Here's now:
/usr/sbin/useradd username
passwd username

The last command will ask you to set a password and confirm it.

Check server hardware

Thu, 15 Dec 2005 20:41:09 +0000

There are a lot of things that may cause a server to crash, this guide is going to primarily look at the hardware side of crashing. There are many things that might be causing the server to crash from a software standpoint such a process that runs out of control or uses too many resources. There are a fwe things that might be going wrong with a server. Normally the component that goes wrong is the hard drive, simply because it is use so much and is a moving part. The RAM on a server will occasionally go but this is more common when the server is moved around or the RAM moved because it has a chance of being statically shocked. On the less common side of things you could have the CPU, powersupply, ethernet card, or motherboard going out.

A few important things first:

Does the server crash on a regular basis at the same time? If so look at the cronjobs on the server. If you are running RHEL and cPanel you should disable the auditd because it may be causing the crashes.

Check your /var/log/messages file for the time before the crash and when the server boots up because you will sometimes be able to see an error. If something like the ethernet card goes down you should also see it because the log files will still be writing, you just will not be able to access the server physically.

Make sure your kernel and other software is at the latest version. There have been a few kernel versions with bad drivers that can cause stability problems.

The easiest thing to look at it smartctl which is a command line tool that reads the status information from the drive directly. Almost modern drive will have the SMART capability which can be probed.

# smartctl -H /dev/sda

If that shows nothing bad take a look at

# smartctl -a /dev/sda

which will show any errors that the disk has encountered. A few errors are ok and normal but if you have more then just a few then it is definatly something to check out. Now since smartctl is a computer program there is a chance that the diagnostics are not correct. Because of this the next best thing to do is to run a disk check. Badblocks is a common tool used to check the disk for errors. To run it go ahead and run badblocks:

#badblocks -v -v /dev/sda

This disk check is going to take a few minutes and will also be constantly using your drive during this time. If your server is very busy you may have trouble with other services running in the background that need to access the disks. If you still think that the disk is bad i would suggest running badblocks 3-5 times. It is possible that the disk drive is just starting to go bad and it might not detect any problems on the first run. If smartctl and badblocks both show nothing chances are the disk driver is just fine.

The next thing to check out on the server is the RAM. To do this we will use memtester which is a very simple program that simply uses the RAM continously and runs it though a series of tests in order to detect any problems with it. First download and install memtester:

cd /usr/local/src
wget http://pyropus.ca/software/memteste...er-4.0.5.tar.gz
tar -zxf memtester-4.0.5.tar.gz
cd memtester-4.0.5
make

Now that it has been compiled we are going to have to run it. NOTE when you run memtester it is going to use all of the RAM and your server load will probably jump pretty high. First make sure of how much RAM you have in the system via

free -m

If for some reason this amount is not what you expect the RAM may in fact be bad. It would be in your best interest to have it checked out to make sure why the correct amount is not present, it might simply be because of an error at the datacenter. Below is what you would run if you have 1024 or 1Gb of ram and have it run 5 seperate times to hopefully catch and problems.

./memtester 1024 5

Since the server is using some of the RAM you are going to get an error like the below:
got 460MB (483188736 bytes), trying mlock ...too many pages, reducing...

Though it is best to close as much as possible before the test it is usually not required. Chances are if the server is using the RAM then it is probably good, of course if nothing else can be found it might be worth closing everything and going back and running memtester again. If all 5 tests come back with no problems chances are that the RAM is ok.

Ok that covers most of the stuff that normally goes wrong. The next thing we can try is simply to try to overload the server and see what happens. The program that we will use is simply called stress. It attempt to overload the cpu and hard drive all at once. First go ahead and download and compile the program to run the tests:

cd /usr/local/src
wget http://weather.ou.edu/~apw/projects...s-0.18.4.tar.gz
tar -zxf stress-0.18.4.tar.gz
cd stress-0.18.4
./configure
make; make install

Now we are going to actually run it. Try using the following command to see what happens. This is more or less going to be a last ditch effort to try and get the server to crash by simulating it being very busy. Since this test is relatively short you may try running it a few times to see if you can get it to crash. If the server does crash it does not necessarily mean that there is a hardware problem but it says something is wrong. It is probably best to switch out the hardware if possible if it crashes running the test assuming as I put above you are running the latest drivers and kernels. Here is the command:

stress --cpu 8 --io 4 --vm 2 --vm-bytes 256M --timeout 60s

Credits: John W

Install HotSaNIC (updated)

Thu, 15 Dec 2005 20:26:37 +0000

HotSaNIC is a very nice tool which combines all sorts of very important system graphs into a simple and easy to understand webpage. It allows the admin to take a quick glance at the graphs to see what may or may not be working with the system. With the new APPS graph it is also possible to watch as more processes are started which can be helpful in tracking down why the server load is increasing. For instance if you see a huge load spike but you see that the number of exim processes has gone up significantly at the same time you can start to investigate.

First, we will need to install RRDtool

-----command-----
cd /usr/local/src/
wget http://people.ee.ethz.ch/~oetiker/w.../rrdtool.tar.gz
tar -zxf rrdtool.tar.gz
cd rrdtool*
./configure
make
make install
make site-perl-install
cd ..
rm -fr rrdtool*

-----command-----

Now we will install and configure HotSaNIC

-----command-----
cd /usr/local
wget http://internap.dl.sourceforge.net/...-0.5.0-pre5.tgz
tar -zxf hotsanic-0.5.0-pre5.tgz
rm -rf hotsanic-0.5.0-pre5.tgz
cd HotSaNIC
./setup.pl

-----command-----

You will be asked a number of questions during the install script.

Module found: APCUSV Do you want to use this module?
Choose 'n'

Module found: APPS Do you want to use this module?
Choose 'y'