Mastering rsync and Bash to Backup Your Linux Desktop or Server

Run a Linux-based Web server? If so, your backup possibilities have increased dramatically. Depending on whether you are copying to or grabbing from the server, there are two applications I love to use: rsync, as you have likely already guessed, and LFTP, a feature-rich command-line FTP application.

Let’s delve into our LFTP usage first, since our rsync techniques will require a little more time. First things first: Make sure lftp is installed by typing ‘lftp’ in a terminal. If it’s not, install it either from your distro’s repository, or via the official source code.

Suppose you are the head of a popular open-source project and would like to upload nightly builds of your source code. The easiest way would be first creating an LFTP script, followed by a shell script and adding it to your crontab. Let’s tackle the first challenge first.

If you have a directory full of nightly builds (which could be built with our scripts on the next page), you can use LFTP to automate the process of uploading them. First, you would want to create a simple script that LFTP understands, which are thankfully similar to shell scripts. I highly recommend saving this under a root account because your password will be clear as day otherwise (unless you happen to live in a fog-belt, whereas that would be a bad example).

lftp open -u username,password -p 21 domain.com ;
lftp mput -O public_html/nightly /home/techgage/projects/appname/nightly/*.tar.gz

Running this script with lftp -f lftp_script would proceed to upload all *.tar.gz files found in that directory. If the tarball exists on the server already, it will not be overwritten; that way, it will only upload whatever is new, which is likely what you want. On the server-side, you could create a script which would symlink the new file to say, latest.tar.gz, so that people who grab that file always know they are getting the latest build.

Want more than just one file uploaded? How about an entire folder? Let’s move onto the mirror command with our sample script:

open -u username,password -p 21 domain.com ;
mirror -R /home/techgage/projects/appname public_html/appname

This script will upload all the files found under your /appname folder, including subfolders. That way, if you have a mounted folder on your local machine that has all of your nightly builds, hash files, readmes or anything else, then the same will be found on the server. To reverse the process (mirror a remote folder to a local one), remove the -R toggle and switch the directories around:

open -u username,password -p 21 domain.com ;
mirror public_html/appname /home/techgage/projects/appname

LFTP is a great program for simple file exchanges, and is perfect for uploading to shared hosting or a server that has SSH disabled (which in turn = no rsync).

Using rsync Without Passwords

If you run a website or have files on a server that need to be updated often, then rsync can become your best friend, very quickly. If you rsync your site on a regular basis, it means you always have a recent copy stored on your local machine’s hard drive, NAS or external storage. So, if something breaks, you will be able to quickly restore the file that’s gone awry.

Using rsync manually is simple… just type in the command and the password, then wait. However, if you want to make nightly backups or sync multiple times a day, it can become an arduous chore. That’s why you will be interested in setting up rsync keys and automating the entire process.

To sync everything without incident, it’s recommended you do this as root. However, if all of the files you require are from a user account only (eg. /home/username), you should be 100% fine as long as that same user exists locally (else you will lose the original file owner). In my case, I want to sync more files than what’s found in /home, so I first enter su mode and then the SSH user folder: cd /root/.ssh (or /home/username/.ssh for a regular user). Once in there, you will need to create an identification key; when prompted for a passphrase, hit enter twice without entering any characters.

localhost .ssh # ssh-keygen -t dsa -b 1024 -f rsync-key
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in rsync-key.
Your public key has been saved in rsync-key.pub.
The key fingerprint is:
0e:2d:a9:fc:34:66:86:3c:42:43:b4:d2:8e:3c:50:52 root@localhost

Once the process is finished, it will create rsync-key and rsync-key.pub. You will need to duplicate this rsync-key.pub to your server’s /root/.ssh folder, to keep things simple. You can do this a variety of ways: Copy over via SSH, Copy/Paste the entire contents of the file, use SCP… it’s up to you.

At this point, you are almost finished. Making sure that the file is on your server, SSH in and enter the same folder as on your local machine: cd /root/.ssh. In order to gain access to the server without a password, you will need to copy your rsync credentials to an authorized_keys file. Regardless of whether it is there already, add the contents of your rsync-key.pub and chmod it before you finish up.

[root@fakeserver .ssh]# cat rsync-key.pub >> authorized_keys
[root@fakeserver .ssh]# chmod 644 authorized_keys

That step was the last one, so you can now log out and back in to your local machine. From here, log back into the server you just logged out of, hopefully password free:

localhost techgage # ssh -i /root/.ssh/rsync-key domain.com
Last login: Mon Aug 20 21:06:09 2007 from xxx000000000000.cpe.cable.domain.com
[root@fakeserver ~]#

If that last step proved successful, you are ready to rsync without the hassle of a password, meaning you can set up a new entry in your crontab to automate your backups! To properly rsync with your server, use these steps:

rsync –av -e “ssh -i rsync-key” domain.com:/home/username/projects /home/techgage/projects

Now that you are all set, let’s jump into our script-writing and the best part, crontabs.