[Eug-lug] rsync problems

[Ben Barrett]

Rsync --partial still needs to iterate all the files/paths you specify, of
course, so it will still use some system resources... I'd suggest doing some
system load-testing (for rsync differences), and also gaining a sense of how
much file change happens ("ls -lt"?) as well as how much actually matters.
For instance, in the common case of a webapp, would it be beneficial to
ignore the app's temp files, or crucial to include them?  Will you be
wanting DB snapshots to accompany filesystem snapshots -- is a goal to be
able to restore a complete system to a particular point in time?  A common &
desirable use-case is to pull out a single file, so consider the overhead of
un-tarballing if you go that route.  In most business applications, storage
space is cheap and time is costly; while in most home and some hobbyist
environments, time is cheaper than storage|bandwidth|processing, although
the new-product push wants to sell us our time back, it seems :)

Ben

PS - It might matter quite a bit about system load, during the rsync,
depending on the remote host's situation (virtual host/slice on
cluster/colo'd iron/etc).  Consider nicing the rsync since it can run
slowly... also consider using unionfs, for a variety of reasons, and whether
you'll want to lock or snapshot the rsyncing portion of the FS from changes
for the duration of the backup -- might not matter, but OTOH your backups
could end up fairly useless, depending...


On Mon, Mar 3, 2008 at 10:41 AM, Rob Hudson <rob at cogit8.org> wrote:

> On 3/3/08, Ben Barrett <stircrazyben at gmail.com> wrote:
> > Are you using the "--partial" option for rsync?
>
> No, but I just read the man page on it.
>
> >  Sticking with rsync for now... do you have any reasons other that
> >  network timeouts to desire a different solution?
>
> No... if it would work, this would be the solution we're looking for.
>
> >  The best approach I've seen using rsync for backups uses a wrapper to
> >  setup a lock-file of sorts,
> >  kicked off [repeatedly] by a cron job... the wrapper script will not
> >  try to do 2 rsyncs at once, in short.
>
> Hmmm, with partial it seems I could set up an hourly cron job (or so)
> rather than a big nightly.  If things are pulled down little by
> little, that might reduce the chance for a bad network disconnects.
> And if it did disconnect, it would just pick up where it left off last
> time.
>
> My other idea was to use tar and do a big monthly tar, then do nightly
> tars of the differences (archival tar?) and scp those down.  I'm not
> sure which might be preferred.
>
> Thanks,
> Rob
> _______________________________________________
> EUGLUG mailing list
> euglug at euglug.org
> http://www.euglug.org/mailman/listinfo/euglug
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://euglug.org/pipermail/euglug/attachments/20080303/2edaae66/attac=
hment-0001.htm