ssh -l some_user -i ~/.ssh/other_key.pub example.com

After having a few servers and starting to lose the commands in my history, I figured there must be a better way to keep track of this. Turns out there is (surprise surprise). Inside your home directory, there’s a .ssh folder where you can drop a config file. So, to recreate the above example you would add the following section:

Host example.com
    User some_user
    IdentityFile ~/.sh/other_key.pub

and now I can connect with

ssh example.com

There is a whack of documentation about how this can make your life easier. Check out the man pages

man ssh_config

Fearing a faulty motherboard, or inproper CPU installation, I decided to watch what was happening to the temperature in the BIOS. The temp was staying quite low ( < 30 C for both the CPU and motherboard) but, the system would still lock up… even in the BIOS screen. At this point I had detached everything except for the video card, 1 stick of RAM, and the hard drive and was still getting the same problem.

As a last ditch attempt, I swapped out the final stick of RAM for one of the other ones in the pile of components sitting beside the case. And what do you know, it worked!

Now, the only conclusion I can come to is that the RAM was overheating after about 2 minutes or so of use, and then completely ceasing to function. The bad memory addresses must have been quite close the beginning since even in the BIOS screen (where I’m assuming it doesn’t do all that much with RAM) would lock up.

Has any one else ever experienced something like this? For the record it was a Corsair 2GB stick.

sudo apt-get install network-manager-pptp pptp-linux

Did the trick and now it was just a matter of entering the server information, username, password. At this point I was still unable to connect to the network. I double and triple checked my info and it was all right. It turned out I had to enable Point to Point Encryption (makes sense) in the advanced section.

Now, I have a nice little ‘locked’ icon on my network status bar.

Who knew it would be that easy?

Apache Benchmark can be a helpful tool to determine response times based on various traffic patterns. It lets you determine how many requests per second your server should be able to handle, and how long each visitor will have to wait to receive a response.

The syntax is pretty straight forward:

ab -c 10 -n 1000 http://yourdomain.com/

(note the trailing / if you are wanting to test your main document). The -c tells ab to make 10 concurrent requests at a time and -n tells it to make 1000 requests total. This will create output similar to

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking yourdomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
Server Software:        Apache/2.2.9
Server Hostname:        yourdomain.com
Server Port:            80
 
Document Path:          /
Document Length:        240 bytes
 
Concurrency Level:      10
Time taken for tests:   20.824 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      565000 bytes
HTML transferred:       240000 bytes
Requests per second:    48.02 [#/sec] (mean)
Time per request:       208.238 [ms] (mean)
Time per request:       20.824 [ms] (mean, across all concurrent requests)
Transfer rate:          26.50 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       59  120 376.0     68    3093
Processing:    61   80  44.0     70     466
Waiting:       61   80  44.0     70     466
Total:        121  200 377.8    144    3165
 
Percentage of the requests served within a certain time (ms)
50%    144
66%    151
75%    156
80%    162
90%    176
95%    203
98%    520
99%   3132
100%   3165 (longest request)

From this we can see that the server tested was able to hand approximately 48 requests per second and was able to process 1000 requests in just under 21 seconds. Whether or not these are acceptable values completely depends on the requirements of your project and application.

You are also able to send in post and cookie data (check the man pages for instructions on how to do that) if you would like to benchmark an authentication process or some form processing.

I’ve found ab to be  a useful tool to determine which pages on a site are taking too long to server / bogging down the server. It’s certainly not the end-all / be-all but can get you pointed in the right direction. After collecting some baseline data though ab, it may give you some ideas as to what code to optimize or where to implement caching on your server.

Remember, be kind and only stress test your own servers. Other people may not take kindly to you hammering their server just to see how they hold up.

edit

Just a quick note about dynamic urls. You can use these if you put single quotes around the entire url. For example:

ab -c 5 -n 2000 ‘http://yourdomain.com/index.php?id=27&action=edit’

Rsnapshot is a series of scripts and commands which can automate the process of backing up your files to a remote location, as well as keeping a incremental copy of any changes. I have it set to keep 7 daily copies, 4 weekly copies (on Saturdays) and 6 monthly copies. Now this may sound like it will use up a great deal of disk space, but rsnapshot makes clever use of hard links, which means it only needs to store copies of files that have changed since the last backup as well as aslight overhead. In my case, this means that for every 100 megs that is backed up, on average only 1 extra meg is stored per copy. So, if I had 1GB of data and 17 old versions (7 days + 4 weeks + 6 months) it would require approximately 1.2GB of physical disk space.

Installing rsnapshot using Linux is pretty straight forward. You’ll need rsync installed, and ssh access to the remote server. In Debian based distributions, a simple

sudo apt-get install rsnapshot

will install it for you on the client side. If you are using Fedora, or CentOS there is an rpm on the rsnapshot download page as well as gzip for any other distribution.

Once that step is complete, it’s time to configure it. The configuration file by default is read from /etc/rsnapshot.conf. There is an important message at the very top of this file that I managed to miss. Note that the file requires trailing slashes on any directories you specify (ie: /home/danklassen/) and it uses tabs and not spaces. Failing to follow these two simple standards can lead to untold amounts of grief (not that I would know of course).

Before going much further, it would be good to set up ssh keys between your client and server. If you’re not sure how to do this, I conveninently have a post about that titled Creating SSH Keys. This will keep you from having to type in a password each time rsnapshot tries to connect to the remote server.

Since we are going to be backing up a remote server, we’ll want to double check that cmd_ssh line is uncommented and points to where where the ssh command resides on your system (double check with ‘which ssh’ if you are unsure). Also, we’re using linux, so we will want to uncomment the cmp_cp line as well. Since linux’s cp command has a few extra features built it, this will allow rsnapshot to take care of normal files and special files all in one pass which should be quicker.

I found I had to play with the rsync_long_args line a little bit to get things to work as expected. What I ended up with is:

rsync_long_args --delete --numeric-ids --delete-excluded

Which will delete any files that have been removed on the server, use numeric ids for user / group permissions, and delete any files that are in the excluded list. If you have any other fancy parameters you want to pass along to rsync, this would be the place to enter them.

Now the important part: setting what to back up, and how often. Near the top of the config file are the lines:

interval    hourly    6
interval    daily    7
interval    weekly    4
interval    monthly    6

These lines will define how many copies to store of each ‘interval’. If you want, you can add ‘interval    yearly    5′ in there as well. You will need to set up cron jobs for each of these intervals.

The backup points section defines what to backup. An example for a remote server would be something like:

backup    user@example.com:/home/danklassen/    danklassen/

What this will do is ssh over to example.com with the username ‘user’ and back up the /home/danklassen/ directory into a folder called danklassen/ in the snapshot_root of the client computer. Again, you’ll want ssh keys working first so that you can do this without having to enter a password.

The last step once you have gotten everything working is to cron your scripts because who are we kidding, you won’t remember to run them yourself. A quick crontab -e and enter something like

00 * * * * rsnapshot hourly
30 23 * * * rsnapshot daily
30 23 * * 6 rsnapshot weekly
30 23 1 * * rsnapshot monthly

I hope this gets you on the right path. If you’re looking for any more details, the rsnapshot site has some example setups and typing ‘man rsnapshot’ will explain every setting for you. Now when your client calls and tells you they deleted their precious data a few days ago, you’ll be able to go back in time and save the day!

Coming up, I’ll show you how to do a similar setup for your mysql databases.

Enter virtualization. There are a number of options to create a virtual computer on your host system. The most popular hypervisors are VMware and Xen. What these programs will let you do is create a completely separate installation of an operating system inside your current one. This let me create the clean environment I needed for this project.

VM’s can come in handy in quite a few different situations. There’s the previously mentioned fresh development environment. Also, you can install windows so that you can test sites in Internet Explorer 6 and 7 to make sure things render properly for your clients if you’re doing web development. Also, if you want to test out the latest and greatest release of your favourite distribution, but don’t want to commit to a beta release or something, you can test to your heart’s content in a VM. If you are hosting sites, a VM is a much more cost-effective way to give a client a dedicated environment as opposed to having dedicated hardware.

In my case, I’m running Ubuntu, but needed a CentOS based server to test this code on. To do this in VMware, it was as simple as clicking file -> new virtual machine and a wizard will walk you through the rest of the steps. Also available are pre-built ‘appliances’ which are ready-to-go environments that you need to boot up. (check out the vmware appliance listings).

Initially getting VMware to run was a little bit more involved however. First hit their site and download the latest version from http://www.vmware.com/download/server/ (1.06 at time of writing). While it’s downloading, register for a serial number (yes, it’s free) so that you have that part ready to go when needed. After the file has downloaded, extract it, and run the installer as root with the command

sudo ./vmware-install.pl

This script will run through the entire setup with you. I was able to accept the defaults in most cases, but instead of using eth0 for networking, I switched to wlan0 since I’m on wireless. During the process you may need to compile support for your kernel (don’t worry, it takes care of everything for you. Just make sure you have the kernel headers installed through apt or yum).

After the script finished running and I entered my serial, I was in business, or so I thought. For some reason, running vmware brought up nothing but the following output

~$ vmware
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/libcairo.so.2)
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/libcairo.so.2)
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/libcairo.so.2)
/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1: version `GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)

A bit of googling brought up the solution:

sudo mkdir /usr/lib/vmware/lib/bak
sudo mv /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1 /usr/lib/vmware/lib/bak/.

to move the offending library. After that everything worked great!

After creating the VM, I bumped up the allocated ram from the default 256 up to 512. To get networking working, I switched from Bridged networking to Custom. (do an ifconfig on your host OS to figure out which specific virtual network to use. In my case it was /dev/vmnet8).

The thought of doing this coding project entirely in vi wasn’t too exciting, so I needed a way to access the code that is on the VM from my host OS so I could use Eclipse to do the editing. The easiest way to do this that I found was to mount a folder over ssh using ssh-fuse. (sudo apt-get install sshfs). Then, create a mount point and mount (swap in your VM’s IP / username where necessary).

sudo mkdir /media/vm_mount
sudo sshfs user@192.168.46.128:/var/www /media/vm_mount

next I symbolically linked it to my Eclipse workspace with

ln -s /media/vm_mount ~/workspace/vm_code

And there you have it. Now I can edit the code as if it was local, yet it is residing on a completely separate operating system. Sheer development bliss.

Now, as you are browsing the web sometimes you may notice that it can take quite a while for the page to initiate a connection with the remote server. This can sometimes be caused by a slow response from a dns server so it takes a while for your browser to figure out who to ask for content. Imagine you’re running a server and you are sending out a few thousand emails an hour, as well as pulling in content from multiple sources, and doing reverse lookups on numerous IP addresses. Any slight delay will add up to huge latency over time, and DNS lookups can be quite costly (time-wise).

OpenDNS is a free service that has a number of servers around the world. They’ve created an amazingly reliable (and quick) network of DNS servers that are free for anyone to use. To set this up on any linux box:

sudo vi /etc/resolv.conf

and enter the lines:

nameserver: 208.67.222.222
nameserver: 208.67.220.220

And that’s it. No really, that is all you need to do.

As an added bonus, they provide you with an easy cache check to see what a domain (or sub-domain) currently resolves to. If you have recently updated, you can force OpenDNS to pull down the latest zone file with your changes, something that can take hours, or even days on other name servers. One thing to realize though, is that once you start using OpenDNS, even non-existent domains and subdomains will resolve to an IP, usually 208.x.x.x, but is something to be aware of. If you don’t know how this could effect you, don’t worry about it.

For home/corporate users, they have phishing philtering, adult site blocking, customizable domain black/white listing, etc. The way they seems to be supporting their financial costs is through serving ads when you type in a domain that doesn’t exist (ie: for some reason, google hasn’t seen fit to set up danklassen.google.com)

Hey, if they can handle 6.3 Billion (yes, that’s a B at the front) dns requests every day, I’m pretty sure they can handle your needs.

We use Pingdom to monitor our web, dns and email servers. Pingdom is a relatively inexpensive service that will ping your server on a regular basis from multiple locations around the world and time the responses. It will then create some nice pretty graphs reporting your uptime. If there should ever happen to be some down time (that never happens does it?), it can notify a list of people via email or sms. The main downside to pingdom is that it is only a reactive service. By the time it sends out an email, the server is down.

A more ideal solution is to have a pro-active monitoring system. For this we use a OSS solution named Monit. It can be configured to trigger actions when certain limits are met. For example, if Apache is using up >= 75% of your system’s memory, Monit can trigger a restart of httpd. Or, if you volume is >= 95% full, it can send out a notification email to an admin to take appropriate actions. Check out their samples and documentation. It’s a pretty powerful system that can help prevent a complete server crash. One thing we have noticed how ever… if you intentionally bring down apache for maintenance and Monit is checking for a live instance of the webserver, be sure to kill monit first. Otherwise, it will unexpectedly restart apache causing potential issues.

Since I thought only the source was available for the linux distribution on one of the servers I manage I had to compile it. Afterwards, I did find a .rpm though but hey, compiling has never killed anyone right? Thankfully it was a pretty straight forward

yum install byacc flex gcc
wget http://www.tildeslash.com/monit/dist/monit-4.10.1.tar.gz
md5sum monit-4.10.1.tar.gz
tar -zxvf monit-4-10.1.tar.gz
cd monit-4.10.1
./configure
make
make install

Then edit /etc/monitrc to add some checks into it. Change this to whatever will make sense for you. Feel free to take a look at some of the configuration samples on the Monit Documentation page.

set daemon  60 #number of seconds between checks

#where and how to log stuff
set logfile syslog facility log_daemon

#outgoing mail settings
set mail-format { from: you_server@address.here
 subject: Server monit alert -- $SERVICE $EVENT
}

#email addresses to notify
set alert email@1.com
set alert email@2.com

#overall system health checks
  check system server_name-memory_cpu
    if loadavg (1min) > 5 for 3 cycles then alert
    if loadavg (5min) > 2 for 3 cycles then alert
    if memory usage > 90% then alert
    if cpu usage (user) > 95% then alert
    if cpu usage (system) > 95% then alert
    if cpu usage (wait) > 95% then alert

#apache web server checks
  check process apache with pidfile /var/run/httpd.pid
    start program = "/etc/init.d/httpd start"
    stop program  = "/etc/init.d/httpd stop"
    #if cpu > 60% for 2 cycles then alert
    #if cpu > 80% for 5 cycles then restart
    if totalmem > 512.0 MB for 5 cycles then alert
    if children > 400 then restart
    if failed host www.yourdomain.com port 80 protocol http
       and request "/index.html"
       for 2 cycles then alert

#drive space checks
check device datafs_main with path /dev/sda1
if space usage > 90% for 5 times within 15 cycles then alert
if inode usage > 80% then alert

then double check things will work with

monit validate

If you get the thumbs up, finally start the daemon with

monit

It should send out an email to your contact address letting you know that it has started. Now any time something starts to go awry on your server, you’ll know ahead of time.

It’s great to have a combination of monitoring solutions in place. Monit is good for notifying you before serious server issues happen. The main downside is, if sendmail dies on your system, there will be no way for it to send out it’s notifications. This is where having another solution such as Pingdom as a backup is very handy.

Has anyone else found a better solution for monitoring multiple remote servers? Something that will be pro-active and still work if the server completely fails? Feel free to comment and let me know.

UPDATE:

The developers of Monit were very kind and pointed out a couple of things I missed. To prevent monit from restarting apache (during maintenance, or whatever) use the command:

monit unmonitor apache

and then when you’re ready to go again,

monit monitor apache

Also, monit has a built in event queue (for the situation where sendmail dies for example) as well as options for backup smtp servers. To set these up, use the following lines in your config file:

set eventqueue basedir /var/monit slots 100
set mailserver smtp1.foo.bar, smtp2.foo.bar

Thanks again Monit Team!

Thankfully KeePass (or KeepassX on Linux or Mac OS-X) can help out where my memory lacks. KeePass will provide you with a secure place to store all of you passwords in an encrypted format. From the screenshot you can get a rough idea as to what the interface looks like. When you highlight any of the entries, you can just Ctrl+C to copy the password and paste it into whatever application is asking for it. For websites, there’s even an ‘auto-type’ feature. To use this, you first go to the site you want to log in to, click in the username box, and then open KeePass and click on the entry for the site. Then press Ctrl+V and it will type in your username / password for you. Slick eh?

I’ve also been asked about how to generate secure passwords. If you are going to use something like KeePass, I would recommend using it’s built in password generator tool (take a look to the left). It gives you a few options to configure, and then spits out a nice random password for you. For those of you using Ubuntu (or any variant of linux for that matter) you can use the command line tool pwgen (click to install in Ubuntu or ‘apt-get install pwgen’). After installing run something like:

pwgen -B -N 5 -1

To get a list of passwords similar to:

eiyah7Ei
Aeh3Ooxo
jaW9ahFi
rohxiJ7z
uth9ZieY

The -B parameter tells it to not use ambiguous characters (is it a 0 or an O? l or I?). -N 5 creates 5 for you to choose from and -1 puts them each on their own line to make it easier to pick.

If you don’t have access to either of those tools, or you need something that you’ll have a better shot of remembering, try basing the password off of a phrase or something you’ll remember, and add in a couple twists. Let’s try one here… off the top of my head, the first phrase that came to mind was “an apple a day keeps the doctor away” and the password I came up with is “4pP13/d=!phD”. (4pp13 kinda loos like apple, /d is short for per day, =! is ‘not equals’ in programming, and phD is a doctor.. apple/day equals no doctor). Okay, that may not be the best example, but you get the idea right?

If you are needing to create a password for a remote server, I would recommend looking into ssh keys. Used well they can provide better security and less hassle for you.

Anyone else have any password strategies they’d like to share? Please, please don’t say “my favorite pet’s name” or “my birthday and last 4 digits of my phone number”…