Re: Zimbra Backup Strategies

[John Fulton <fultonj@lafayette.edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim Ross wrote:
> We ... have approximately 30,000 accounts with an estimated storage
> of 8-10 TB of storage. One of our big concerns is if we will be
> able to back up all this data to SAN storage and then to tape for
> offsite storage on a weekly basis.

Lafayette has 4,500 accounts with 3TB of space reserved on our SAN
(Fibre and SATA via HSM). However, we have allocated a second 3TB of
space on our DR SAN and additional space for backups. (DR != backup).

> The biggest problem we see with the Zimbra
> backup is that the database is stored as thousands of small files,
> which take much longer to move than one or several zipped or tarred
> files.

I'm assuming you are talking about backing up what's already in
Zimbra's backup directory which by default is /opt/zimbra/backup.

For those who don't know, Zimbra's internal storage is complicated:

:BEGIN storage complication rant:
All of your mail metadata is stored in MySQL so backing up the mail
dir files directly won't do. E.g. when you open your Zimbra inbox
you see an overview of messages containing the subject, sender, date
etc. All of that is pure SQL. It didn't come from parsing your mail
dir files it's more like a "SELECT subject, sender, date ... FROM
table WHERE username=...". It's not until you click a message that it
displays the contents of one particular mail dir file. There's also
the calendar, the indexed documents etc.

Luckily Zimbra has abstracted all of this away for you so you don't
really have to worry about it. By default Zimbra automatically does
full backups (including LDAP) weekly and incremental backups (user
data diffs) nightly and it retains data for one month. If you have
the space for it I recommend leaving this setting. Otherwise you can
update it with zmschedulebackup. Remember you'll be using zmrestore
or the GUI to do any restores since restoring a user's mail by
updating SQL and moving mail dir files around is error prone. All of
these files are in /opt/zimbra/backup.
:END storage complication rant:

> With our current setup, the fastest transfer rate we could get would
> be approximately 2 Gigabit/sec from our Zimbra servers to the SAN
> storage.

If you can afford it you could upgrade your fabric switches. AFAICT:

* 2 Gbps FC can queue 256 I/O commands/second
* 4 Gbps FC can queue 2048 I/O commands/second

I'm using a 4G Cisco MDS-9124 with a cx3-20.

> We are thinking that we would probably do a full backup once a week
> with incrementals for the rest of the week.

You could use zmschedulebackup to update to this. I imagine that you
are looking to backup your backups already in /opt/zimbra/backup. If
you have /opt/zimbra/backup mounted to a separate partition you could
umount and clone that partition. EMC SAN Copy uses differentials so
you could probably do this in a reasonable amount of time with your
vendor's cloning product. If you're going to backup this directory to
tape you'll have the tape as your bottleneck.

> What backup strategies/hardware are others on this list using?

We're not yet in production but our DR plan is:

Our Zimbra store server mounts /opt/zimbra (and /opt/zimbra/backup) on
our SAN. Then the /opt/zimbra is synced to a LUN on a second SAN with
a second store server ready to mount it in the event of DR. The sync
can be done with EMC SAN Copy but we're hoping to use Zimbra 5.5's
server-to-server sync instead.

Server-to-server sync will be a nice new feature since it will use the
same intelligence Zimbra uses to route mail in a multi-server install
as it does to move a copy to DR server. I prefer this option to EMC
SAN copy since as far as I know ext3 doesn't write your data to the
disk directly; it buffers it until later and then writes it with the
sync system call. So, block level snapshots of your disk can be
incomplete. To get around this each SAN Copy session is initiated via
Replication Manager which automatically quiesces I/O for an instant to
start the session. This sounds very difficult to test if it's working
correctly (since any test may not involve the block that's being
written to). It also requires a license so using something the new
Zimbra has built-in which is less complicated will be better.

Note how our backup plan is different:

/opt/zimbra/backup is for user error recovery. For DR we'll use our
clone instead (though we'll have the backup if the clone fails). By
user error recovery I mean being told something along the lines of "I
deleted a message by accident, can you get it for me?" For these
backups we will have /opt/zimbra/backup mounted in a second data
center and running on RAID'd SATA. If we're happy with our server
sync we'll probably mount /opt/zimbra/backup from our DR server and
run our backups on that server. This way an account on the live
server won't have to go into maintenance mode for the backup and I
won't have to waste clock cycles on the DR system.

John
- --
John Fulton, Linux Administrator, Lafayette College ITS
610-330-5650, Pardee Hall Room 11, Easton PA 18042-1775
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFICQ03NZmEpbCkXmERAoVvAKCvdyj8T5YHv4eZm0uQ8656tf9rKgCgpM3B
snduDwqxGJoC1888f+0Lesg=
=yT4X
-----END PGP SIGNATURE-----