SMTP Error: SMTP error: Connection failed: Failed to connect socket: Connection timed out.

After reading a few things  I turned on the SMTP tweak..  things appeared to work for a few hours, then it started failing again.

Searching online said to trying telnet on the command line – so did that, connected to port 25 – all fine… hmm odd – tried webmail again – failed…..

After a more indepth search I found a forum which gave me a hint – apparently other people have had the same problem when using the firewall for cpanel – CSF.  CSF is definately one of the best firewalls around for linux and works really well with Cpanel – however every now and again the amount of options can cause you to miss out something obvious:

# If SMTP_BLOCK is enabled but you want to allow local connections to port 25
# on the server (e.g. for webmail or web scripts) then enable this option to
# allow outgoing SMTP connections to 127.0.0.1
SMTP_ALLOWLOCAL = 0

The SMTP_ALLOWLOCAL set to 0 stops webmail etc… not sure when this appeared (the server has been working fine for about 4 months) but change it to 1 – restart CSF and there we go it’s working fine

You’ve logged in as root to check – and yup you still can’t write any files – even a “touch test.txt” fails with a message saying can’t write file – disk or partition full?

You could be out of inodes! Linux stores file address information in inodes – and if you have a lot of very small files (or quite an oddly setup filesystem) you could be out of inodes…

How do you find out though?

just type in

“df -i” – same as the normal diskfree space command except it tells you inodes, if you have none left (or very few) you may not be able to create a new file (or edit it if your editor creates temporary lockfiles).

In a future blog I’ll go through how to resize the number of inodes available (primarily aimed at your temporary dev/shm space as this can suffer from the inode problem if you allocate more to it from a small size – the number of inodes is not increased so you can run out).

umount tmpfs

The above command will ask the OS to unmount the filesystem identified by tmpfs in the fstab file.

This may return an error if you are actively using it – i.e. you have an ssh session open and are in the filesystem or open files exist. – Exit any of these and try again

mount tmpfs

The above command will attempt to re-mount the drive.

When would you use this?  Normally if you have done some dynamic re-sizing of a partition  - for example the tmpfs partition – see post:

http://www.ihelm.org.uk/2010/04/resizing-ram-disk-in-linux-devshm/

for more info on how to do that.

The Cpanel system has a feature to migrate accounts from another server – be it Cpanel or another webhosting panel – this often works, recently though I’ve had a problem with using it for Cpanel to Cpanel where it would connect but fail to transfer the backup file.

Fortunately it’s really easy to use the commandline with cpanel and you can do it manually (actually it’s easier than the wizard!)

On the server you are migrating from just type (while logged in as root!)

/scripts/pkgacct username

This will create a tar.gz in the /home directory on the server – just copy this to the new server (it could take a while – it has everything from the account you are copying).

On the destination server just type in

/scripts/restorepkg username

This creates (or overwrites the account!) on the server – and your done!.  But what if you have a dedicated IP (due to SSL?)

Easy just use this instead

/scripts/restorepkg –ip=y username

This gives it the next free dedicated IP address (make sure you have one free!)

And that’s it – it is honestly easier than the wizard which requires you to add more info to establish the transfer link.

One thing to note is that you should ensure that the versions of cpanel are as near as you can get – otherwise it might fail or do things a little weird.

If you want to see what exim is doing right now :

“exiwhat”

if you think you have queue problems then you can print a countof messages in the queue this will often be quite high – so it’s worth running it a few times over a couple of hours to see if you have a queue problem:

“exim -bpc”

If you want more information about what is in the queue then the following command will give you the count, the colume, oldest, newest, domain to be sent to and total summaries.  The oldest and volume parts are often the most informative in this to help you see problem domains.

“exim -bp | exiqsumm”

Below are some commands to help you manage the queue – especially useful if you want to try to purge it or take action.

To tell exim to try to deliver things in the queue according to the rules for a queue run

“exim -q -v”

If you are blocked up with external mail you can tell exim to run the queue to deliver local mail only – this may help you get things such as notifications etc.. if they are delayed by other mail in the queue

“exim -ql -v”

If you have some problems with older mail (maybe spam related) – you can use the following to delete mail that over 7 days old (the time given is in seconds so just take one day 86400 and multiply it by 7)

“exiqgrep -o 604800 -i | xargs exim -Mrm”

There are many many other commands and other examples – some sites have longer lists, but these are all the commands I have needed (or variations thereof) for administrating my servers over the last 9 years.

These tasks can be  varied, from restarting servers, services – creating files, changing permissions, runing batch programs, deleting files etc…

Of course sometimes crontab is not the best solution as it can have several problems – in which case other choices are available – including writing your own – however we will cover that another time.

First a coouple of facts about cron:

1. Cron has evolved a bit since inception but the basic principles are the same – old server admins are as good as new ones with this tool!
2. Cron will run new jobs at most once a minute – that’s the smallest resolution you can have.
3. Old systems used to wake up every minute and check for new work – the same as a simple replacement service might do – however this didn’t scale well when cpu power was low but users high on old mainframe systems – new systems still only execute once a minute.
4. the cron service loads up any relavent cron files (which exist in users home directories or in /etc/cron.d/) and saves the information so it only needs to run when there is really jobs to run – when you edit the crontab it reloads it’s list so it doesn’t need to keep checking for changes – it’s important to edit it right!
5. Even if a previous program hasn’t finished – cron can run it again or run more jobs – this can bring down badly configured systems! so be careful how you use it.

Generally to edit the crontab – which is where you would add things to run type in “crontab -e” – this will open it in your system editor (vim is my choice, but others can be used).

you will probably see some content – similar to this:

1 0 * * *  shutdown -r now

This means that at one minute past midnight the server will reboot (execture the immediate shutdown and restart command with no delay) each day.  The * are wildcards saying that the instruction should run no matter the criteria of that column (if it’s all * then it will run each minute).

The fields are the following:   I recommend pasting this in the top of your crontab file – the # at the start mean it will ignore the entries.

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  *

Remember if the program has output you can store it in a file the same as you can on the command line – just type ” > /var/log/logfile”

in the entry after the command and it will log output to the file in question.

first I recommend to change to the directory that you think or know the file containing the text is in:

• To find the string michael in any file type “grep michael *         #this will return a the files if any are found within the directory
• To find the string michael in any file and search recusively in directories under where you are type  ”grep -R michael *”

Grep is one of the best tools available to an administrator especially for checking log files out and finding regular patterns.

The first thing you need – or I assume – is that you have a separate hard-drive which is not active (mounted) or can be unmounts using the standard umount command.

We assume the device is sdc  (it could be sdb, sda, hda1 etc..).

1. Ensure the drive is unmounted – turn off any swap already on it with swapoff (or if it’s clean of swap partitions then you can ignore swapoff)
2. type in “parted /dev/sdc”  This loads the parted prompt which allows you to manage the disk.
3. enter “print” to view details of any existing partitions and free space.
4. once you have decided the size you need (at minimum I would recommend match your RAM) type “mkpartfs part-type linux-swap start end”  Where for the start and end match the start point available in the print command and ending a number x megabtes above i.e. if the start point is 1024 then the end point would be 5120 for a four gigabyte partition (4096)
5. once done type quit to exit parted.  The new partition will have been given a number e.g. sdc2 (or higher if others exist)
6. all that needs doing is to format the partition “mkswap /dev/sdc”
7. to enable it’s the same process as for the file – “swapon /dev/sdc”

If you want to make it work each boot – then edit the fstab using

vi /etc/fstab

add to the bottom: (press insert to enter edit mode)

/dev/sdc2 swap swap defaults 0 0

then press escape and shift-Z shift-Z

as before you can check to see if it’s been added by using

/proc/swaps

The server in question is quite busy – but only had 512Meg of RAM (this has since been rectified) historically as it wasn’t that busy at first several years ago has slowly grown as websites on it have grown in popularity and databases have grown.

So an immediate fix was required – it had a swap partition – couldn’t create a new one, it would mean downtime as we are using the hard-drives!

So added a swap file – here’s how to do it:

1. Change to a directory (on a partition) where you have lots of space – e.g. /home  ”cd /home”
2. Create a file swapfile using dev null – the following command will create a 4meg file:  ”dd if=/dev/zero of=swapfile bs=1024 count=4194304″
   This creates an empty file called “swapfile” with a block size of 1024bytes and creates 419304 (to get the size do 4194304 / 1024 then divide by 1024 to get the size in gigabytes).
3. chmod it so it can be used “chmod 600 swapfile”
4. format the file to a swapfile type so the system can use it “mkswap swapfile”
5. In order to use it – turn it on “swapon swapfile”

At the end of this if yo do the following “cat /proc/swaps” to see your swapspace.  It’s now larger by a lot and hey presto.  Of course a long-term fix has to be more ram if possible or tweak your setups! (or a new server!)

This doesn’t get setup to activate on boot – mainly because I didn’t need this – a temporary fix and therefore didn’t want todo this as the RAM would be upgraded soon, but if you need more memory – and don’t mind it being slow for stability then this is a possible solution.

R-U-ON http://www.r-u-on.com/ is a great service, which has a free level and a paid.  Personally I used the free service for several years, now I use the paid service.

The Iphone app works with both absolutely fine.  Basically you create an account and then download a small program called an “agent”, this is set running or installed on your computer or server – these agents then connect to the r-u-on servers and send information.

You can get information about several things, most useful to myself is disk space, virtual memory usage, ram usage, cpu and load stats.  If any of these exceed limits set (you can customise them).  Critical failures can be notified by text message (theres a fee involved for this) which is absolutely essential.

The IPhone app simply connects to the r-u-on account and shows a summary, of your alarms (server’s which have exceeded a notification threshold – see the graphic below for this), or general server status – which hopefully is all green and show’s that all your servers are running well.  You can also view your trouble tickets, personally I’ve never done this and have no real idea what it is.

Screenshot of the alarm screen.

The only thing missing from the app is the ability for it to run in the background, though of course the Apple SDK does forbid this – bah! however it would be great if it could or some kind of background notification system used (which I believe may be available in OS 3.0 but not 100% sure).