I am Joe Blog’s Left Ventricle

Fri April 29, 2005

Post: rsync howto rsync network backup tips

Finally got back to this. Firstly, I have to say I still think tapes have a place in backups, but more as the weekly/monthly archive than the primary method of backing stuff up.

Naturally, I can’t find a good howto that’s specific to the kind of rsync backups I use (server, no encryption, no ssh)

But this one sets up a server very simply:

http://transamrit.net/docs/rsync/

Additionally:
* Yes, the rsync server wont function without the correct permissions on the secrets file
* To backup TO an rsync server, obviously you change read only = no
* To synchronise a volume to a server I use rsync -av –delete /path/ |username@server::modulename(usually the name of the server you’re backing up)
* To synchronise from a server (servers set up on machines to be backed up, this helps to centrally manage a few of these):
   rsync -av –delete username@server::modulename /destination_path/
* To script this activity, the following options come in handy:
–password-file=file (with password to use in it, must have similar brutal permissions if I recall)
–bwlimit=X (kilobytes per second) handy if you’re rynchronising a couple of gig over an xDSL link during the day and don’t have QOS.
–stats produces some useful output for your logs
–exclude-from=FILE can be handy if you’re backing up / and don’t want to include the CEO’s mp3 collection or the backups folder
–max-delete=NUM the one time this would have come in handy, I really wished I’d used it. Basically if you have an insane problem (disk not mounted), this helps prevent rsync helpfully synchronise a blank volume to your backup machine

Performance issues:
* Using rsync, roughly 500GB of user files (400ish users) seems to move about 5GB of data a day. That’s about 7 hours over a 2Meg xDSL link. (slightly less as rsync also compresses data in transit). These are from memory, I don’t post from work so don’t have the exact data, or the pretty graph I made.
* Our mail server backups (Groupwise) are also a couple of gig per day.
* Doesn’t work well on MS-SQL backup files, sends entire file every time (I think MS do something really creative with the backup file that makes it impossible to track what’s changed
* Works very well on VMware virtual disk files (which must be either unmounted or a snapshot), including virtual disks that CONTAIN live MS-SQL databases
* 400GB of files seems to consume about 1Gig of RAM while rsyncing (again, from memory, YMMV). Going heavily into swap is unlikely to effect performance if you’re doing it over a slow network link anyway.

Storage requirements:
* The last 1TB array I built using commodity hardware cost about $3k. (and took me about eight hours of fighting with RAID controller drivers, but that is another story
* I have a quote for a Promise 15drive iSCSI unit with 3.2TB of SATA drives (WD Raid Edition, 1M hours MTBF, 7200rpm SATA) for about $12k.

Cool stuff:
* Any linux machine whose data has been synchronised to another linux server can have its *functionality* brought up on the backup server, in the event of a disaster. I have previously managed to merge two file servers and a web server using this method. As such, your local backup box can become a handy Business Continuity server. Any Virtual machine… even easier. If you’re doing the same thing to an offsite machine, well, there’s your DR done.
* It’s really quite handy to have Yesterday’s Data on disk all the time. Cuts down the number of restores we have to do.

Regards,
James

Tue April 26, 2005

Bah

I have become totally cynical. Again. The last time, I was 16 years old and had nothing to defend myself against the world with but what genetics and some really lame experiences had provided.

Now I’ve got the genetics minus a small drugs and alcohol levy, a library of personal philosophy, some worse experiences, and a packet of steamed vegetables.

Life outside of work, through a series of convoluted maneuvers, has converted itself into a conveyor belt of Sardonic Comment & Wisecrack Ammunition, class X. I just got back from a holiday in Apollo Bay, that included several little car trips along the Great Ocean Road (not to be confused with the Mediocre Ocean Road), and my first ever sighting of the majestic Twelve Apostles. Meh-or-fucking-rama. Apollo Bay this time of year is a soggy coastal bog somehow mounted on a hill, covered in salt-spray and for some reason crammed to the eyesockets with a music festival.

The most amusement I gleaned from the whole affair (aside from the book I’m reading) was in noting that the Information Centre at the twelve apostles contained no information whatsoever, and was in fact a repository of Art. What I wanted was some geological data about the layers clearly visible in these massive natural monoliths. How old are the fuckers? What epoch was that big grey layer? Nothing. No information whatsoever. Thousands of foreign tourists whose impression of Australia is just that little bit dumber now though. Maybe its all written on the wall in the toilets? I didn’t check.

Cluster One - Pink Floyd. When’s the last time a modern artist brought out a purely instrumental piece? When did an artist have the humility to do so? When’s the last time a new piece of music made your spine tingle?

What am I doing out here in the middle of the desert? Chasing some girl? Chasing some dream? Trying to save money to save the world? This world, that I hold in contempt? Where are my friends? They’re back where I left them, destroying themselves with alcohol and drugs or getting married and getting divorced in the same year. Why not? Why not get it over with? Where are my new friends? I’ve been here a year, how many friends have I made? Who is my best friend? I’ve had a best friend since I was fucking 12, who is my best friend now? People aren’t looking for friends in this town. They like to have 800 acquaintances. A friend is someone who, when you want to get drunk, you think of getting drunk with them. I don’t know anyone who gets drunk here. They go out, they have a couple of drinks, have a reserved conversation that rarely touches on anything they really care about, and go home to their wives.

The question is not “Why should we go to war?”, the question is “Now that we have the technology and wisdom of centuries of warfare, how do we end the killing for good?”. The question is not “What do we do about the Environment?”, the question is “Now that we have taken control of the global environment, what outcomes are we seeking from change?”. The question is not “Should we let Refugees stay here”, the question is “How can we use people and technology to fill up all our empty spaces?”

What kind of ass-backwards fucknut can crank out that paragraph in five minutes but can’t handle chit-chat at a party? Like minded people? Where the fuck will I find people with a mind like that? Why would anyone want to? Bah!