Helm’s Technology Blog

Those of you with Exim will sometimes needs to do some work on it – here’s a list of the commands which I find most useful:

If you want to see what exim is doing right now :

“exiwhat”

if you think you have queue problems then you can print a countof messages in the queue this will often be quite high – so it’s worth running it a few times over a couple of hours to see if you have a queue problem:

“exim -bpc”

If you want more information about what is in the queue then the following command will give you the count, the colume, oldest, newest, domain to be sent to and total summaries.  The oldest and volume parts are often the most informative in this to help you see problem domains.

“exim -bp | exiqsumm”

Below are some commands to help you manage the queue – especially useful if you want to try to purge it or take action.

To tell exim to try to deliver things in the queue according to the rules for a queue run

“exim -q -v”

If you are blocked up with external mail you can tell exim to run the queue to deliver local mail only – this may help you get things such as notifications etc.. if they are delayed by other mail in the queue

“exim -ql -v”

If you have some problems with older mail (maybe spam related) – you can use the following to delete mail that over 7 days old (the time given is in seconds so just take one day 86400 and multiply it by 7)

“exiqgrep -o 604800 -i | xargs exim -Mrm”

There are many many other commands and other examples – some sites have longer lists, but these are all the commands I have needed (or variations thereof) for administrating my servers over the last 9 years.

Another of the most useful tools in server administration is the Crontab – you use this to schedule jobs (programs) to run at regular intervals.

These tasks can be  varied, from restarting servers, services – creating files, changing permissions, runing batch programs, deleting files etc…

Of course sometimes crontab is not the best solution as it can have several problems – in which case other choices are available – including writing your own – however we will cover that another time.

First a coouple of facts about cron:

1. Cron has evolved a bit since inception but the basic principles are the same – old server admins are as good as new ones with this tool!
2. Cron will run new jobs at most once a minute – that’s the smallest resolution you can have.
3. Old systems used to wake up every minute and check for new work – the same as a simple replacement service might do – however this didn’t scale well when cpu power was low but users high on old mainframe systems – new systems still only execute once a minute.
4. the cron service loads up any relavent cron files (which exist in users home directories or in /etc/cron.d/) and saves the information so it only needs to run when there is really jobs to run – when you edit the crontab it reloads it’s list so it doesn’t need to keep checking for changes – it’s important to edit it right!
5. Even if a previous program hasn’t finished – cron can run it again or run more jobs – this can bring down badly configured systems! so be careful how you use it.

Generally to edit the crontab – which is where you would add things to run type in “crontab -e” – this will open it in your system editor (vim is my choice, but others can be used).

you will probably see some content – similar to this:

1 0 * * *  shutdown -r now

This means that at one minute past midnight the server will reboot (execture the immediate shutdown and restart command with no delay) each day.  The * are wildcards saying that the instruction should run no matter the criteria of that column (if it’s all * then it will run each minute).

The fields are the following:   I recommend pasting this in the top of your crontab file – the # at the start mean it will ignore the entries.

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  *

Remember if the program has output you can store it in a file the same as you can on the command line – just type ” > /var/log/logfile”

in the entry after the command and it will log output to the file in question.

Grep is a tool that is used often by system administrators – including myself, however I often have to double check the syntax online – as a result this post is to help me find out how and you of course.

first I recommend to change to the directory that you think or know the file containing the text is in:

• To find the string michael in any file type “grep michael *         #this will return a the files if any are found within the directory
• To find the string michael in any file and search recusively in directories under where you are type  ”grep -R michael *”

Grep is one of the best tools available to an administrator especially for checking log files out and finding regular patterns.

I realised after my previous post I realised that it would be worthwhile talking about adding a new partitition.

The first thing you need – or I assume – is that you have a separate hard-drive which is not active (mounted) or can be unmounts using the standard umount command.

We assume the device is sdc  (it could be sdb, sda, hda1 etc..).

1. Ensure the drive is unmounted – turn off any swap already on it with swapoff (or if it’s clean of swap partitions then you can ignore swapoff)
2. type in “parted /dev/sdc”  This loads the parted prompt which allows you to manage the disk.
3. enter “print” to view details of any existing partitions and free space.
4. once you have decided the size you need (at minimum I would recommend match your RAM) type “mkpartfs part-type linux-swap start end”  Where for the start and end match the start point available in the print command and ending a number x megabtes above i.e. if the start point is 1024 then the end point would be 5120 for a four gigabyte partition (4096)
5. once done type quit to exit parted.  The new partition will have been given a number e.g. sdc2 (or higher if others exist)
6. all that needs doing is to format the partition “mkswap /dev/sdc”
7. to enable it’s the same process as for the file – “swapon /dev/sdc”

If you want to make it work each boot – then edit the fstab using

vi /etc/fstab

add to the bottom: (press insert to enter edit mode)

/dev/sdc2 swap swap defaults 0 0

then press escape and shift-Z shift-Z

as before you can check to see if it’s been added by using

/proc/swaps

We had a problem on one of the servers which I help administer.  Every now and again RAM usage would max out and SWAP would then be used – this would sometimes max out and bye bye server, normally this happened for only a few seconds and all was fine – however if it lasted more then when SWAP ran out – the server ground to a halt and needed a manual restart.

The server in question is quite busy – but only had 512Meg of RAM (this has since been rectified) historically as it wasn’t that busy at first several years ago has slowly grown as websites on it have grown in popularity and databases have grown.

So an immediate fix was required – it had a swap partition – couldn’t create a new one, it would mean downtime as we are using the hard-drives!

So added a swap file – here’s how to do it:

1. Change to a directory (on a partition) where you have lots of space – e.g. /home  ”cd /home”
2. Create a file swapfile using dev null – the following command will create a 4meg file:  ”dd if=/dev/zero of=swapfile bs=1024 count=4194304″
   This creates an empty file called “swapfile” with a block size of 1024bytes and creates 419304 (to get the size do 4194304 / 1024 then divide by 1024 to get the size in gigabytes).
3. chmod it so it can be used “chmod 600 swapfile”
4. format the file to a swapfile type so the system can use it “mkswap swapfile”
5. In order to use it – turn it on “swapon swapfile”

At the end of this if yo do the following “cat /proc/swaps” to see your swapspace.  It’s now larger by a lot and hey presto.  Of course a long-term fix has to be more ram if possible or tweak your setups! (or a new server!)

This doesn’t get setup to activate on boot – mainly because I didn’t need this – a temporary fix and therefore didn’t want todo this as the RAM would be upgraded soon, but if you need more memory – and don’t mind it being slow for stability then this is a possible solution.

Kick-starting an Iphone app review cycle, first up is one of the most useful app’s I have for server monitoring.

R-U-ON http://www.r-u-on.com/ is a great service, which has a free level and a paid.  Personally I used the free service for several years, now I use the paid service.

The Iphone app works with both absolutely fine.  Basically you create an account and then download a small program called an “agent”, this is set running or installed on your computer or server – these agents then connect to the r-u-on servers and send information.

You can get information about several things, most useful to myself is disk space, virtual memory usage, ram usage, cpu and load stats.  If any of these exceed limits set (you can customise them).  Critical failures can be notified by text message (theres a fee involved for this) which is absolutely essential.

The IPhone app simply connects to the r-u-on account and shows a summary, of your alarms (server’s which have exceeded a notification threshold – see the graphic below for this), or general server status – which hopefully is all green and show’s that all your servers are running well.  You can also view your trouble tickets, personally I’ve never done this and have no real idea what it is.

Screenshot of the alarm screen.

The only thing missing from the app is the ability for it to run in the background, though of course the Apple SDK does forbid this – bah! however it would be great if it could or some kind of background notification system used (which I believe may be available in OS 3.0 but not 100% sure).

I recently had a little problem when deleting a store of spam e-mails off one of my personal servers.

Basically entering the standard death defying command of “rm -Rfv *” while in a directory (this will delete everything recursively – have fun), the system responded “Argument list too long” – cue the first “huh?” moment – I’ve used the command for years and years, never had a problem.

I did “man rm” to check I hadn’t had a “moment” and forgotten how to use it – nope.. . A little trawling came up with several people with the same problem but no really practical solutions.  The problem occurs when you have too many files for the built in systems used by the rm command in the kernel – these are pretty old so have some limits that you don’t expect! – you can do this:

rm -Rfv a* then rm -Rfv A* –  these deletes all files starting with a or A etc.. – very long and rather silly – so after thinking and thinking and muttering an old command came to mind  using a combination of find and xargs you can pipe a file on at a time so here we go – my favoured solution:

find . -name * | xargs rm -fv

This works fine – I suspect it will have a limit somewhere, but my server hasn’t hit it yet!

After several years of running different solutions on different servers I have found that ASSP is one of the best anti-spam solutions around.
However – after some time it still struggles and gets weaker, can anyone recommend any other systems that can work ALONGSIDE it (not replace).

An incident earlier this week has made me review the backup systems in place for my wifes company (www.ihelm-enterprises.co.uk).

One of our clients doesn’t exclusively use us and so had a problem when another host lost a server and had no backups – the client also doesn’t backup frequently and had hoped that the server hosts would have some form of backup – but of course they didn’t.

We have always been a bit unusual in that we have routinely done backups – however we don’t do them every day and they do frequently fail – as we have a mixture of servers we had a rather un-orthodox way of doing them.

Server 1 performed its backup by transferring to Server2
Server 2 then transferred server 1 and server 2’s backup to the next server etc… until the final resting place was a repository where only the final server could get teh files – this was backed up by a 3rd party regularly.

However after hearing of our clients problems I thought I would try our system to see if we could easily recover – and found that
a) our repository was full and I hadn’t had any notification of this.
b) geting the files off was extremely difficult as some of the files are very large and the server will access to the repository didn’t have a lot of space…
c)  Some of the servers have not been notifying me of messages encountered during the backup so no idea of the files were complete or not.

So decided to re-think this a bit.

Now our servers transmit to our office computers – one of which is on 24hrs – this then uses windows file sharing (despite it being a linux computer ) to copy the files over to a new NAS which is located somewhere the office.

We have already seen some benefits from this – we can easily see what files are on the NAS & restoring is simple, also each server has a set time to transfer its files – so any problems we only need to look at that server and re-run specific backups.

The bottom line is – if you do provide backups or just to backup your own work – do check your backup system every now and again – ours had been in place over a year after the previous model was un-doable with the adsl we had at the time – but since then the cheapness of Network Attached Storage devices has meant we can affordably implement this as well.

Backup – Backup – Backup – Check – Review – Backup – Backup – Backup – Check – Review, this is now my new mantra!

Ok – I use 3 distinctly different operating systems routinely.

I have windows XP – this is on my laptop and also have a copy on my desktop – though this rarely runs it now.

I have ubuntu (kubuntu) on my desktop – this is nearly always on and is great.

I then have a Mac in the office at work – running Mac OS X.

generally apart from keyboard issues (as i don’t have a Mac Keyboard so sometimes the mappings go wonky) – the Mac is fine – no probs, but in order to access SSH the Mac just doesn’t have anything to touch Putty.

Putty is a very well known and very popular SSH client – I’ve been using it since around 2003 – maybe earlier.  It exists for windows and this appears to be where it is primarily developed for – and it’s great – never seen a better SSH client – does what’s needed – nice and clean, lets you save session data etc…

It exists for Linux – so I have in it my ubuntu install – don’t recall how I got it – but it’s pretty much like the windows client – few little quirks but it’s putty and i can use it.
But when it comes to the Mac – I’ve hunted – searched – tried someones build – didn’t work – got the source – can’t compile it!  Why does no real equivalent of Putty exist  - theres a few that pretend to be – buth when you are administrating multiple servers they just don’t quite cut it!

The problem is it’s not just Putty – theres a handful of programs that seem to be available on “free” operating systems like ubuntu which help people who have come from windows – and who – to be honest – like to use a graphical interface sometimes – which just aren’t available on the Mac – I am sure this works the other way – problem is the Apple want the Mac to be more mainstream and whilst they are making progress – it’s just not there yet…

P.s. I am sure I will rant about how useless windows is – and ubuntu at some point – in fact I feel a Ubuntu rant coming along soon…..