Booservers - All about dedicated servers

Set up Tripwire on RedHat

Thu, 15 Dec 2005 21:45:38 +0000

Install tripwire

It's installed by default, but if it isn't, grab your RedHat disks, and install it:

---------------------

# rpm -ihv /mnt/cdrom/RedHat/RPMS/tripwire-2.3.1-5.i386.rpm
# /etc/tripwire/twinstall.sh

---------------------

Clean the policy file
Go over to /etc/tripwire, and clean out the policy file with this handy script:

---------------------

# cat > ./cleantw.pl
#!/usr/bin/perl

while (<>){
#look at the line, and check for a line that can be
# Construed as a file name
CASE:{
( m|(^s*)(/[/w._-]+)(s+->.*)| ) and do {
print $1;
print "#" unless (-e $2);
print "$2$3n";
last;
};

print $_;
}
};
^D
#

---------------------

Now clean out the crap in the sample policy file they have.

---------------------

# perl cleantw.pl < twpol.txt > cleanedpol.txt

---------------------

Edit the file. You'll need to change the hostname that is defined somewhere around the 65th line to match what comes out of the 'hostname' command. You'll also need to change some of the parameters for things that are monitored (Log rotating changes inodes for a given filename).

---------------------

/var/run -> $(SEC_INVARIANT)
/var/log -> $(SEC_INVARIANT)
/var/tty[1-8] -> $(SEC_INVARIANT)

---------------------

Finishing up

---------------------

# /usr/sbin/twadmin -m P cleanedpol.txt
# /usr/sbin/tripwire --init

---------------------

Credits: http://www.afonsoconsulting.com/

Easy CURL 7.10.4 w/SSL

Thu, 15 Dec 2005 21:35:04 +0000

We've put together these RPM's for libcurl 7.10.2 with SSL support. They upgrade over the stock RH ones. Installation is via rpm -Uvh

Applicability (RH 7.2 / 7.3 required) :

Webppliance 3.1
Webppliance Pro 3.5
cPanel 6
Plesk 5.0

These are also Modernbill compatible on all above panel types.

Downloadables:

curl-7.10.4-1.i386.rpm
curl-devel-7.10.4-1.i386.rpm

MD5 Checksums

37a04f7db8c34fddbb309d067d16a453 curl-7.10.4-1.i386.rpm
e09253b7c227573f4e5b10d3987f433b curl-devel-7.10.4-1.i386.rpm

Credits: http://www.cheetaweb.com/

Check server hardware

Thu, 15 Dec 2005 20:41:09 +0000

There are a lot of things that may cause a server to crash, this guide is going to primarily look at the hardware side of crashing. There are many things that might be causing the server to crash from a software standpoint such a process that runs out of control or uses too many resources. There are a fwe things that might be going wrong with a server. Normally the component that goes wrong is the hard drive, simply because it is use so much and is a moving part. The RAM on a server will occasionally go but this is more common when the server is moved around or the RAM moved because it has a chance of being statically shocked. On the less common side of things you could have the CPU, powersupply, ethernet card, or motherboard going out.

A few important things first:

Does the server crash on a regular basis at the same time? If so look at the cronjobs on the server. If you are running RHEL and cPanel you should disable the auditd because it may be causing the crashes.

Check your /var/log/messages file for the time before the crash and when the server boots up because you will sometimes be able to see an error. If something like the ethernet card goes down you should also see it because the log files will still be writing, you just will not be able to access the server physically.

Make sure your kernel and other software is at the latest version. There have been a few kernel versions with bad drivers that can cause stability problems.

The easiest thing to look at it smartctl which is a command line tool that reads the status information from the drive directly. Almost modern drive will have the SMART capability which can be probed.

# smartctl -H /dev/sda

If that shows nothing bad take a look at

# smartctl -a /dev/sda

which will show any errors that the disk has encountered. A few errors are ok and normal but if you have more then just a few then it is definatly something to check out. Now since smartctl is a computer program there is a chance that the diagnostics are not correct. Because of this the next best thing to do is to run a disk check. Badblocks is a common tool used to check the disk for errors. To run it go ahead and run badblocks:

#badblocks -v -v /dev/sda

This disk check is going to take a few minutes and will also be constantly using your drive during this time. If your server is very busy you may have trouble with other services running in the background that need to access the disks. If you still think that the disk is bad i would suggest running badblocks 3-5 times. It is possible that the disk drive is just starting to go bad and it might not detect any problems on the first run. If smartctl and badblocks both show nothing chances are the disk driver is just fine.

The next thing to check out on the server is the RAM. To do this we will use memtester which is a very simple program that simply uses the RAM continously and runs it though a series of tests in order to detect any problems with it. First download and install memtester:

cd /usr/local/src
wget http://pyropus.ca/software/memteste...er-4.0.5.tar.gz
tar -zxf memtester-4.0.5.tar.gz
cd memtester-4.0.5
make

Now that it has been compiled we are going to have to run it. NOTE when you run memtester it is going to use all of the RAM and your server load will probably jump pretty high. First make sure of how much RAM you have in the system via

free -m

If for some reason this amount is not what you expect the RAM may in fact be bad. It would be in your best interest to have it checked out to make sure why the correct amount is not present, it might simply be because of an error at the datacenter. Below is what you would run if you have 1024 or 1Gb of ram and have it run 5 seperate times to hopefully catch and problems.

./memtester 1024 5

Since the server is using some of the RAM you are going to get an error like the below:
got 460MB (483188736 bytes), trying mlock ...too many pages, reducing...

Though it is best to close as much as possible before the test it is usually not required. Chances are if the server is using the RAM then it is probably good, of course if nothing else can be found it might be worth closing everything and going back and running memtester again. If all 5 tests come back with no problems chances are that the RAM is ok.

Ok that covers most of the stuff that normally goes wrong. The next thing we can try is simply to try to overload the server and see what happens. The program that we will use is simply called stress. It attempt to overload the cpu and hard drive all at once. First go ahead and download and compile the program to run the tests:

cd /usr/local/src
wget http://weather.ou.edu/~apw/projects...s-0.18.4.tar.gz
tar -zxf stress-0.18.4.tar.gz
cd stress-0.18.4
./configure
make; make install

Now we are going to actually run it. Try using the following command to see what happens. This is more or less going to be a last ditch effort to try and get the server to crash by simulating it being very busy. Since this test is relatively short you may try running it a few times to see if you can get it to crash. If the server does crash it does not necessarily mean that there is a hardware problem but it says something is wrong. It is probably best to switch out the hardware if possible if it crashes running the test assuming as I put above you are running the latest drivers and kernels. Here is the command:

stress --cpu 8 --io 4 --vm 2 --vm-bytes 256M --timeout 60s

Credits: John W

setup remote serial console to display grub menu and boot process

Thu, 15 Dec 2005 20:32:59 +0000

this HOWTO will show you how to setup your redhat linux server (and probably a few others) to do the following:

* send the grub menu to the remote serial console (so you can select the kernel to boot from)

* send boot process info to the remote serial console (like what you see on your monitor when you boot up at home... you use linux at home, right?)

why does this matter? for me, this all equates to less stress during kernel upgrades (being able to see any problems, choose different kernels, change kernel boot parameters from the grub menu), and less stress when things go awry somewhere on a box and i have to use the master reboot switch and sit around twiddling my thumbs wondering if the system really is rebooting.

WARNING: if you aren't 100% sure of what you're doing, DO NOT do any of the following on a live/production server. in other words, if you are new to this stuff, only try this on your brand new box you just ordered just in case you mess it up so bad you have to get it re-imaged.

if you aren't sure what the remote console is or how to use it, please see this HOWTO: http://forum.ev1servers.net/showthread.php?t=52961

a thread discussing some uses for the remote console (and the people there probably could have made use of this tutorial) can be found here: http://forum.ev1servers.net/showthread.php?t=51199

using this tutorial requires grub to be installed as your server's bootloader (some basic instructions below). it's probably very possible to get the boot process info sent to the serial console even if you're using lilo, but i don't know how to do it with lilo and have no desire to google that info as i don't like or use lilo (maybe someone else will post info on how to do this with lilo).

this tutorial also makes a few extra assumptions, like that the serial port to use is ttyS0, but i doubt that won't be the case for anyone.

enjoy and let me know if you have any comments/additions/complaints.

on to the tutorial:

MAKE SURE GRUB IS YOUR BOOTLOADER

please note that what i have here does not replace outside knowledge of grub/bootloaders. again, if you aren't sure what you are doing, don't try any of this on a production server. i am far from a guru. don't put to much trust in my geek advice (my political views, however, should be taken very seriously =).

if you aren't sure if grub is your bootloader (default ev1 images at the time of writing have both grub and lilo installed [that is, the programs are present] but lilo is the bootloader installed on the MBR and so is the one in action). if you aren't sure which is on the MBR and you have grub installed, you can use this command (the # denotes the shell. don't type the #):

---------------------

# grubby --bootloader-probe

---------------------

that will output either "lilo" or "grub", telling you which is on the MBR.

so, this isn't a tutorial for installing grub, but i have found with fresh ev1 RHEL servers you can just run:

---------------------

# grub-install /dev/device_name_of_first_disk

---------------------

where device_name_of_first_disk is likely either hda or sda, depending on the type of drives you have. this will (hopefully) install grub on the MBR (and so lilo won't be there anymore). if you have just installed grub on the MBR and haven't rebooted yet, you should do that now just so you know that is working before you proceed with other changes (that is, so if there are problems you know at what point things got messed up).

SETUP GRUB AND THE BOOT PROCESS TO GO TO THE SERIAL CONSOLE

now, assuming you have grub installed, it's time to proceed with setting up the grub menu and the boot process to be sent to the serial console (i believe it will also still allow someone at the terminal to view the grub menu).

1) edit /etc/sysconfig/kudzu and make sure the value for SAFE is yes (change it if needed):

---------------------

SAFE=yes

---------------------

if you do not do this then the serial connection will be reset by kudzu during the boot process.

2) backup /boot/grub/grub.conf

3) edit /boot/grub/grub.conf and add the following two lines to the general config (not to any specific kernel):

---------------------

serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=5 console serial

---------------------

now append this to any kernel lines who you want to see the output of in the console when you boot with those kernels:

---------------------

console=ttyS0,9600n8 console=tty0

---------------------

your resulting grub.conf might look something like this:

---------------------

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/hda3
# initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=10
fallback=2
splashimage=(hd0,0)/grub/splash.xpm.gz
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=5 console serial

title Red Hat Enterprise Linux ES (2.4.21-32.0.1.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.4.21-32.0.1.ELsmp ro root=LABEL=/ console=ttyS0,9600n8 console=tty0
initrd /initrd-2.4.21-32.0.1.ELsmp.img
title Red Hat Enterprise Linux ES (2.4.21-32.0.1.EL)
root (hd0,0)
kernel /vmlinuz-2.4.21-32.0.1.EL ro root=LABEL=/
initrd /initrd-2.4.21-32.0.1.EL.img
title Red Hat Enterprise Linux ES (2.4.21-27.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.4.21-27.ELsmp ro root=LABEL=/
initrd /initrd-2.4.21-27.ELsmp.img
title Red Hat Enterprise Linux ES-up (2.4.21-27.EL)
root (hd0,0)
kernel /vmlinuz-2.4.21-27.EL ro root=LABEL=/
initrd /initrd-2.4.21-27.EL.img

---------------------

if you aren't sure about editing the grub.conf file, just note that if your system already booted fine from grub and you are just trying to get the serial console to show boot info, then you don't want to make any changes whatsoever to any of the vmlinuz or initrd filenames that are referenced. the only change you'll make to any of those kernels lines is to append the string i mentioned above to the end of the kernel line, as you can see in the above example.

note that in my example i've only changed one kernel to send boot info to the console. there's no reason for that other than a) i was lazy, b) i wanted to leave some untouched in case i messed things up adding this bit to the kernel line, and c) i don't boot from the other kernels. also, as you should be able to see the grub boot menu from the console now, you could likely boot your system fine even if you messed up the individual kernel configurations in grub.conf by using grub's handy ability to let you declare kernel boot parameters from the grub menu (for info on how to do that, seek out some grub tutorials).

that's it. now save your grub.conf, open a connection to the serial console (in a new window, if you want) and reboot. if all is goes well, you shouldn't be disconnected even because of a reboot. when the box starts back up you will see "press any key to continue" a few times. if you press a key right then you will get the grub menu. if you don't, a few seconds later (possibly after grub has shown the menu to the terminal?) your server will boot from the default kernel and you will start to see the boot info.

CREDITS

http://www.commondreams.org

yum upgrade; redhat 9 -> CentOS 3.1

Thu, 15 Dec 2005 20:02:35 +0000

Here are the steps i took to update a Redhat 9 box running on a minimal install ( no X, etc ) as suitable for a server to CentOS 3.1 ( which is a fork of RedHat Enterprise Linux v3 )..

I took a fresh install of redhat 9, slapped it on a test box then updated it with up2date. I then installed yum for redhat 9 which can be found here:
Yum Download

Now, you might want to rebuild the rpm database, just incase, thats up to you:
# rpm -vv --rebuilddb

I then modified /etc/yum.conf to look like this:

Code

---------------------------------

[main]
exclude=httpd* mysql* php* perl*
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=redhat-release
installonlypkgs=kernel kernel-smp kernel-hugemem kernel-enterprise kernel-debug kernel-unsupported kernel-smp-unsupported kernel-hugemem-unsupported
tolerant=1
exactarch=1

[base]
name=CentOS-$releasever - Base
baseurl=http://mirror.pacific.net.au/linux/cAos/centos-3/3.1/os/i386/
gpgcheck=1

#released updates
[update]
name=CentOS-$releasever - Updates
baseurl=http://mirror.pacific.net.au/linux/cAos/centos-3/3.1/updates/i386/
gpgcheck=1

#packages used/produced in the build but not released
[addons]
name=CentOS-$releasever - Addons
baseurl=http://mirror.pacific.net.au/linux/cAos/centos-3/3.1/addons/i386/
gpgcheck=1

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=http://mirror.pacific.net.au/linux/cAos/centos-3/3.1/extras/i386/
gpgcheck=1

---------------------------------

Change "http://mirror.pacific.net.au" to a mirror near you, a list can be found here:
CentOS Mirrors

After that you need to import the GPG key for the centos RPMs. Issue the following command:
# rpm --import http://mirror.centos.org/centos/3.1/i386/RPM-GPG-KEY-CentOS-3

Nows the time you want to check you have made appropriate backups, etc ( i hope you all test this out before rolling production servers! ).

Next step is to run the following command:
# yum upgrade

This will download the rpm headers and determine what needs to be done, play carefull attention of course here and in all parts if this process for errors. Yum will then ask you if you are sure you want to go ahead, if no errors are present, press "y" then "enter".

This process will take a while, depending on your connection and the gear your installing it on of course. Go brew a fresh pot of coffee, order a pizza, grab a movie or GET BACK TO WORK YOU!!!!

After thats done, check for errors, make sure the new kernel is installed by issueing:
# rpm -qa|grep kernel

You should be presented with a kernel versioned like this: 2.4.21-15.EL.c0 as well as your other kernels previously installed. Check your boot loader to make sure the new kernel is default.

Finally run:
# yum update

Just to ensure you have everything you need.

Now your set to reboot ( hopefully ):
shutdown -rf now