Helm’s Technology Blog

Those of you with Exim will sometimes needs to do some work on it – here’s a list of the commands which I find most useful:

If you want to see what exim is doing right now :

“exiwhat”

if you think you have queue problems then you can print a countof messages in the queue this will often be quite high – so it’s worth running it a few times over a couple of hours to see if you have a queue problem:

“exim -bpc”

If you want more information about what is in the queue then the following command will give you the count, the colume, oldest, newest, domain to be sent to and total summaries.  The oldest and volume parts are often the most informative in this to help you see problem domains.

“exim -bp | exiqsumm”

Below are some commands to help you manage the queue – especially useful if you want to try to purge it or take action.

To tell exim to try to deliver things in the queue according to the rules for a queue run

“exim -q -v”

If you are blocked up with external mail you can tell exim to run the queue to deliver local mail only – this may help you get things such as notifications etc.. if they are delayed by other mail in the queue

“exim -ql -v”

If you have some problems with older mail (maybe spam related) – you can use the following to delete mail that over 7 days old (the time given is in seconds so just take one day 86400 and multiply it by 7)

“exiqgrep -o 604800 -i | xargs exim -Mrm”

There are many many other commands and other examples – some sites have longer lists, but these are all the commands I have needed (or variations thereof) for administrating my servers over the last 9 years.

Another of the most useful tools in server administration is the Crontab – you use this to schedule jobs (programs) to run at regular intervals.

These tasks can be  varied, from restarting servers, services – creating files, changing permissions, runing batch programs, deleting files etc…

Of course sometimes crontab is not the best solution as it can have several problems – in which case other choices are available – including writing your own – however we will cover that another time.

First a coouple of facts about cron:

1. Cron has evolved a bit since inception but the basic principles are the same – old server admins are as good as new ones with this tool!
2. Cron will run new jobs at most once a minute – that’s the smallest resolution you can have.
3. Old systems used to wake up every minute and check for new work – the same as a simple replacement service might do – however this didn’t scale well when cpu power was low but users high on old mainframe systems – new systems still only execute once a minute.
4. the cron service loads up any relavent cron files (which exist in users home directories or in /etc/cron.d/) and saves the information so it only needs to run when there is really jobs to run – when you edit the crontab it reloads it’s list so it doesn’t need to keep checking for changes – it’s important to edit it right!
5. Even if a previous program hasn’t finished – cron can run it again or run more jobs – this can bring down badly configured systems! so be careful how you use it.

Generally to edit the crontab – which is where you would add things to run type in “crontab -e” – this will open it in your system editor (vim is my choice, but others can be used).

you will probably see some content – similar to this:

1 0 * * *  shutdown -r now

This means that at one minute past midnight the server will reboot (execture the immediate shutdown and restart command with no delay) each day.  The * are wildcards saying that the instruction should run no matter the criteria of that column (if it’s all * then it will run each minute).

The fields are the following:   I recommend pasting this in the top of your crontab file – the # at the start mean it will ignore the entries.

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  *

Remember if the program has output you can store it in a file the same as you can on the command line – just type ” > /var/log/logfile”

in the entry after the command and it will log output to the file in question.

Grep is a tool that is used often by system administrators – including myself, however I often have to double check the syntax online – as a result this post is to help me find out how and you of course.

first I recommend to change to the directory that you think or know the file containing the text is in:

• To find the string michael in any file type “grep michael *         #this will return a the files if any are found within the directory
• To find the string michael in any file and search recusively in directories under where you are type  ”grep -R michael *”

Grep is one of the best tools available to an administrator especially for checking log files out and finding regular patterns.

I realised after my previous post I realised that it would be worthwhile talking about adding a new partitition.

The first thing you need – or I assume – is that you have a separate hard-drive which is not active (mounted) or can be unmounts using the standard umount command.

We assume the device is sdc  (it could be sdb, sda, hda1 etc..).

1. Ensure the drive is unmounted – turn off any swap already on it with swapoff (or if it’s clean of swap partitions then you can ignore swapoff)
2. type in “parted /dev/sdc”  This loads the parted prompt which allows you to manage the disk.
3. enter “print” to view details of any existing partitions and free space.
4. once you have decided the size you need (at minimum I would recommend match your RAM) type “mkpartfs part-type linux-swap start end”  Where for the start and end match the start point available in the print command and ending a number x megabtes above i.e. if the start point is 1024 then the end point would be 5120 for a four gigabyte partition (4096)
5. once done type quit to exit parted.  The new partition will have been given a number e.g. sdc2 (or higher if others exist)
6. all that needs doing is to format the partition “mkswap /dev/sdc”
7. to enable it’s the same process as for the file – “swapon /dev/sdc”

If you want to make it work each boot – then edit the fstab using

vi /etc/fstab

add to the bottom: (press insert to enter edit mode)

/dev/sdc2 swap swap defaults 0 0

then press escape and shift-Z shift-Z

as before you can check to see if it’s been added by using

/proc/swaps

We had a problem on one of the servers which I help administer.  Every now and again RAM usage would max out and SWAP would then be used – this would sometimes max out and bye bye server, normally this happened for only a few seconds and all was fine – however if it lasted more then when SWAP ran out – the server ground to a halt and needed a manual restart.

The server in question is quite busy – but only had 512Meg of RAM (this has since been rectified) historically as it wasn’t that busy at first several years ago has slowly grown as websites on it have grown in popularity and databases have grown.

So an immediate fix was required – it had a swap partition – couldn’t create a new one, it would mean downtime as we are using the hard-drives!

So added a swap file – here’s how to do it:

1. Change to a directory (on a partition) where you have lots of space – e.g. /home  ”cd /home”
2. Create a file swapfile using dev null – the following command will create a 4meg file:  ”dd if=/dev/zero of=swapfile bs=1024 count=4194304″
   This creates an empty file called “swapfile” with a block size of 1024bytes and creates 419304 (to get the size do 4194304 / 1024 then divide by 1024 to get the size in gigabytes).
3. chmod it so it can be used “chmod 600 swapfile”
4. format the file to a swapfile type so the system can use it “mkswap swapfile”
5. In order to use it – turn it on “swapon swapfile”

At the end of this if yo do the following “cat /proc/swaps” to see your swapspace.  It’s now larger by a lot and hey presto.  Of course a long-term fix has to be more ram if possible or tweak your setups! (or a new server!)

This doesn’t get setup to activate on boot – mainly because I didn’t need this – a temporary fix and therefore didn’t want todo this as the RAM would be upgraded soon, but if you need more memory – and don’t mind it being slow for stability then this is a possible solution.

I recently had a little problem when deleting a store of spam e-mails off one of my personal servers.

Basically entering the standard death defying command of “rm -Rfv *” while in a directory (this will delete everything recursively – have fun), the system responded “Argument list too long” – cue the first “huh?” moment – I’ve used the command for years and years, never had a problem.

I did “man rm” to check I hadn’t had a “moment” and forgotten how to use it – nope.. . A little trawling came up with several people with the same problem but no really practical solutions.  The problem occurs when you have too many files for the built in systems used by the rm command in the kernel – these are pretty old so have some limits that you don’t expect! – you can do this:

rm -Rfv a* then rm -Rfv A* –  these deletes all files starting with a or A etc.. – very long and rather silly – so after thinking and thinking and muttering an old command came to mind  using a combination of find and xargs you can pipe a file on at a time so here we go – my favoured solution:

find . -name * | xargs rm -fv

This works fine – I suspect it will have a limit somewhere, but my server hasn’t hit it yet!

An incident earlier this week has made me review the backup systems in place for my wifes company (www.ihelm-enterprises.co.uk).

One of our clients doesn’t exclusively use us and so had a problem when another host lost a server and had no backups – the client also doesn’t backup frequently and had hoped that the server hosts would have some form of backup – but of course they didn’t.

We have always been a bit unusual in that we have routinely done backups – however we don’t do them every day and they do frequently fail – as we have a mixture of servers we had a rather un-orthodox way of doing them.

Server 1 performed its backup by transferring to Server2
Server 2 then transferred server 1 and server 2’s backup to the next server etc… until the final resting place was a repository where only the final server could get teh files – this was backed up by a 3rd party regularly.

However after hearing of our clients problems I thought I would try our system to see if we could easily recover – and found that
a) our repository was full and I hadn’t had any notification of this.
b) geting the files off was extremely difficult as some of the files are very large and the server will access to the repository didn’t have a lot of space…
c)  Some of the servers have not been notifying me of messages encountered during the backup so no idea of the files were complete or not.

So decided to re-think this a bit.

Now our servers transmit to our office computers – one of which is on 24hrs – this then uses windows file sharing (despite it being a linux computer ) to copy the files over to a new NAS which is located somewhere the office.

We have already seen some benefits from this – we can easily see what files are on the NAS & restoring is simple, also each server has a set time to transfer its files – so any problems we only need to look at that server and re-run specific backups.

The bottom line is – if you do provide backups or just to backup your own work – do check your backup system every now and again – ours had been in place over a year after the previous model was un-doable with the adsl we had at the time – but since then the cheapness of Network Attached Storage devices has meant we can affordably implement this as well.

Backup – Backup – Backup – Check – Review – Backup – Backup – Backup – Check – Review, this is now my new mantra!