Tar, Zip, and Rsync #2

Rsync:
According to wikipedia, rsync was first announced in June of 1996.
See link: http://en.wikipedia.org/wiki/Rsync
In its basic form, rsync is used to copy files from one folder to another folder and maintain a synchronized copy of the files, when activated.
The folders can be on the same computer, different computers on the same network, or on another network.  It uses a delta algorithm to compare new files to old and only transfer enough bytes to compare, make changes, and delete when necessary.  There are many options when using rsync.  Read the man pages, read the internet forums, and do what works best for you.

I use ssh and rsync to maintain a copy of about 100GB worth of important data that changes on a weekly basis.  Rsync is loaded on the source and destination machines.  I ssh between the machines and initiate rsync on the destination machine.  Bandwidth is not an issue for me as they are both on a gigabit router so I do not use compression.  IF you are going across the internet, use compression to more efficiently use bandwidth.

My rsync command looks something like this:
rsync -r root@<server>:/folder1/folder2/folder3/ /home/Rsync/folder1/

From the destination server, the command starts rsync, login as root on the source server,
and begins rsync on the source server.  (You should have rsync loaded on both servers)
The "r" says recursive. (get all the underlying folders and files).
The first path ending in folder3/ is my source directory. 
This is followed by a space and the path to my destination folder.
I end each path in a "slash" but the syntax is up to you.

I don't save the owners/rights because of the type of data I am saving. 
If you are backing up user folders, you should save owners/rights to simplify any restore you might need to make.

I have just briefly touched on rsync and how it works for me.  Study it, read the man pages, read the forums on the net, and let it work for you.

NOTE:
This post was later than I expected.  We had record 100 year snowfall and low temperatures in my area of Oklahoma and several things were put on hold for a few days.  Through the use of linux and ssh, we were able to maintain servers, computer support, backups, etc., from our homes, as the power and internet stayed up.

Have fun computing and until next time,
Jim

    

Tar, Zip, and Rsync #1 Linux

A little history or TAR and Zip.  TAR, or tar, actually is a Tape ARchive program.  For those of you younger than 45 or so, we actually backed up our data on tape drives; serial and slow.  If you go back far enough, the tapes were reel-to-reel and we had to specify the tape length, starting point, compression (if any), etc.   When used with zip, TAR becomes a serial backup program with compression.  On today's computers and media, this process is many times faster than the old tape drives and is still very reliable.  Zip basically is a compression program that works well.  Most of us have used Zip in either Windows or Unix/Linux based backups. 

There are many websites with information on tar zip backups.  Used with ssh,
you can safely backup across networks, the internet, etc.  Bandwidth becomes the slowdown obstacle.  Personally, I use tar zip and ssh to make those daily backups of small databases and other stuff.  I use rsync for the really big stuff.
More on rsync next week.

OK.  For the actual command line using ssh, I do the following:

tar -cvzpf - /folder to backup/* | ssh jim@BackupStorageServer1 "cat > /destination folder/DestinationFileName.$1.tar.gz"

Read the man pages on tar and you will understand the -cvzpf.
I usually put the date in the $1 place.  
Practice extracting a file in a location on your computer that is not vital. 
That is all for now.  Next time: rsync. 
Have fun computing.
Jim

Panic, Paranoia, and Planning

I said I was going to write about tar, zip, and rsync, but I decided this needed to go first.  I used to ask my clients, "how long can you afford to be down?"
They always said never.   Wrong answer.  The truth is, everyone can afford to be down for a given period of time, if you know and plan ahead for that specific time.
Whether it is 1 minute, 1 hour, or 1 day, all network/sysadmins plan for a down time, sooner or later, for a specific server or group of servers, or applications.
Backups are for those unplanned times when hardware fails, or a worm attacks the system, or data gets accidentally overwritten. 

Backups are an insurance policy.  You only need it when things go wrong.
Very large corporations use multiple server farms, (cloud computing), to store and backup data.  Most of us use multiple servers, RAID systems, off site data mirroring, and/or other stuff.  Each one of these is a form of backup.

So, let us start with a common sense approach. 
How much gross income did the company make in 2010?
There are approximately 250 work days in a year. 
If the gross was $250000, then the average is $1000 per day gross income.
If the gross was $1 million, then the average is $4000 per day gross income.
If it cost you about $4000 per day for your business server to be down, then you can easily justify a $4000 backup plan.   

So, let' start at the basic hardware level.
Does the company lose time and money if a certain hard drive fails?
If the answer is yes, then mirror, stripe, etc .....  that drive.
If the drive is just a convenient temporary bucket for non-critical data,
the answer might be no.  Just keep a spare drive on hand.


What about at the next level: whole servers.
If your server mainboard fails unexpectedly, how much would it cost you in down time? (Real dollars!)
Maybe it is time to mirror that server with another complete server, on site or off site. Or, will just a copy of all critical data updated every 24 hours to a secondary server  keep you going? 


What about a router failure, web page server, or email system?
What if your UPS battery fails, causes a short in the system, and downs everything attached to it? (Rare, but I have experienced it.)

And, I have seen new hardware fail.  Just because the box is new does not guarantee 100% success.  


Write down (on paper and in red ink) the time lost in hours and days, and the cost in real dollars.  Be logical and think it through.  This will help you make your decision.

Much better to plan now than to be in the middle of a panic attack because a mission-critical server is down and there are no spare parts. 
Plan for the best and worst case scenarios, and sleep well.

Jim

Save Your Thoughts

Next posts: backups in Linux.  Tar, zip, rsync, and more.

Open Source -- Why It Works

Wikipedia defines open source as "practices in production and development that promote access to the end product's source materials." 
"http://en.wikipedia.org/wiki/Open_source"

I like that definition.  Notice that it says "practices that promote".  Open source is not an accident, but a decision.  Open source is a concept put into practice. 

It is a great concept AND practice.  When companies lock down their code, and sometimes there are legitimate reasons to do so, they are restricting the development and debugging of that code, to their code writers.
Open source takes advantage of good code writers all over the world.
Code and programs can be tweaked, altered, and configured to run better under certain situations, or overall.  Literally, hundreds and thousands of people contribute to open source code.  The internet, email, web pages, and blogs, made this possible.  It is a natural development in the sharing of information.  

So, open source is here to stay.  Take advantage of it, contribute to it; either in good code, good testing, buying products, or by donating money to those legitimate web sites and companies that you feel are doing a good job.  We all benefit.

Linux Layers

This post is a back to the basics for understanding how Linux works.  For those of you just now migrating from Windows to Linux, this should help you understand the difference in concepts in the two operating systems.

Several years ago, Bill Gates, with the help of some other people, wrote DOS; Disk Operating System.  It was command line only, no graphics.  Then Windows came along.  It was a layer of graphic programming that "sat" on top of DOS.
Windows could not function without the bottom layer of DOS.  It was like 2 layers of a cake.  DOS on bottom, Windows on top.  Over the years,  the separation between layers has become fuzzy and the Windows part includes everything but the basic commands.  You can still run command line programs in Windows, if you know what you are doing. 
The top layer is the software, like MS Office, that interacts with the Windows program, and helps the user to be productive.

With Linux, the cake has more layers and they are distinctly separate, so far.
The bottom layer is Linux.  It will run by itself with no need for other layers.
It is command line only.  Some people build servers this way to cut down on the number of extra programs running in the background.
The next layer is the X-windows graphical interface.  It contains graphical programming to interface between a GUI (graphical user interface) desktop view and commands, and the underlying Linux operating system.
The next layer is the Desktop Window Manager.  It interacts with the X-windows system graphical interface, has the icons, mouse effects, screen savers, colors, etc.  Typically, it is KDE, Gnome, Xfce4, WindowMaker, Fluxbox, Ice, etc.      
The top layer is the software that will run on almost any Linux Desktop Window Manager.  Typically it is OpenOffice, Firefox, GIMP, etc.

I hope that helps you understand some of the basic differences in the two operating systems.

Next time, what is open source, and why it works.  

Have a good day.
Jim

Reading Material

This will be a short but important post.  It is almost impossible to know every command, shortcut, and trick in any operating system and Linux is no different.  I recommend keeping a reference book handy on your desk.
The first book I go to is:

The Linux Pocket Guide by Daniel J. Barrett.
Essential Commands
Published by O'Reilly.
ISBN: 978-0-596-00628-0

It is written to cover Fedora Linux, but the information is valuable for all distros.  You should be able to find it online or at your local book store.

Also,
Learn to use "man" pages in Linux. 
At the command prompt, type:  man ls <enter>
You will see a manual for the ls (list) command.
This works with most commands in linux.

Read some each day and enjoy being part of the world-wide open source community.
Jim