[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Mail.app DoS
BTW: We made this configuration change last night and so far have not seen the IMAP frenzy, but it's pretty early to tell.
Pam
----- "pam buffington" <pam.buffington@oit.gatech.edu> wrote:
> From: "pam buffington" <pam.buffington@oit.gatech.edu>
> To: "David N. Blank-Edelman" <dnb@ccs.neu.edu>
> Cc: "Helpful Zimbra Admins" <zimbra-hied-admins@sfu.ca>
> Sent: Wednesday, May 5, 2010 1:52:03 PM GMT -05:00 US/Canada Eastern
> Subject: Re: Mail.app DoS
>
> We have had a very similar issue, and have been working with Zimbra on
> it. (Doug might respond in a bit with more information) What is your
> maxmessagesize? and what is your zimbraMessageCacheSize ? Try
> increasing the zimbraMessageCacheSize to 100MB.
>
>
> *Excerpt of a message*
> When the Imap client issues N partial fetch commands for the same
> message, we expect the server parses the same message only once(The #
> of N is decided by the IMAP client 's buffer)
>
> Now the stack trace from GA confirmed that the same message was
> actually parsed again and again instead of just being parsed once as
> we expected. This is root cause of the high CPU usage.
> a) between 16:00 ~ 17:00, ImapServer-690982 consumes 261493 CPU ticks
>
> b) between 16:00 ~ 17:00, ImapServer-690982 issues 1500 partial FETCH
> cmds for part 1 of the MIME message 29042.part 1 is about 47MB.
> c) between 16:00 ~ 17:00, The continuous thread dumps showed that the
> ImapServer-690982 was constantly doing parsing when we expect only one
> parsing for the very first fetch command of mime message 29042.
> you can find log in qa24
> imap command trace:
> /opt/support/gt/0503/ImapServer-690982.log
> imap thread dumps stack trace:
> /opt/support/gt/0503/ImapServer-690982.stacktrace.log
> d) GC was very active when ImapServer-690982 was doing partial fetch
> of that big mime message. it's possible that the MimeMessage was
> quickly purged from MessageCache right after it's parsed due to the
> small Message cache size and large amount of new parsed messages. When
> server receives the subsequent partial command for the same
> MimeMessage, it has to reparse because it fails to find that message
> in Message Cache. It could happen when the server is very busy and the
> size of the targeted mime message for partial fetch is very big.
> zimbraMessageCacheSize: 20971520
> zimbraMailDiskStreamingThreshold: 1048576
>
> zmailbox_mailbox_cache_size=1
>
>
> Action Item: Thom asked Eng for next step. Increasing
> zimbraMessageCacheSize to 300MB is recommended [wiki recommends
> 100MB, but server-team agrees to use 300MB:
> http://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments]
>
>
>
> **
>
> ----- "David N. Blank-Edelman" <dnb@ccs.neu.edu> wrote:
>
> > From: "David N. Blank-Edelman" <dnb@ccs.neu.edu>
> > To: "Helpful Zimbra Admins" <zimbra-hied-admins@sfu.ca>
> > Sent: Wednesday, May 5, 2010 1:17:07 PM GMT -05:00 US/Canada
> Eastern
> > Subject: Mail.app DoS
> >
> > Hi-
> > Given the preponderance of macs in edu, I'm curious if other
> sites
> > have seen this particular phenomena. I'm about to burn a support
> > ticket on this because it is getting a bit troublesome.
> >
> > We're seeing an unpleasant situation where by if a Mail.app user
> > deletes a bunch of messages and then empties their trash folder,
> > sometimes Mail.app gets into this tight loop with out Zimbra 5.x
> > server. Mail.app at that point starts to send commands at a rate of
> N
> > per second when N is a surprisingly large number. Zimbra, in turn,
> > logs tens of thousands of lines that look like this:
> >
> > > 2010-05-04 11:41:35,306 INFO [ImapSSLServer-634329]
> > [name=user@zimbra;mid=419;ip=ip;] imap - S: 291726.299 BAD parse
> > error: invalid message sequence number: 1:*
> > > 2010-05-04 11:41:35,362 INFO [ImapSSLServer-634329]
> > [name=user@zimbra;mid=419;ip=ip;] imap - S: 291727.299 BAD parse
> > error: invalid message sequence number: 1:*
> > > 2010-05-04 11:41:35,432 INFO [ImapSSLServer-634329]
> > [name=user@zimbra;mid=419;ip=ip;] imap - S: 291729.299 BAD parse
> > error: invalid message sequence number: 2:*
> >
> > while this is going on. As you'd expect, this drives the load up on
> > the server significantly and only abates when the bad connection
> goes
> > away. I haven't yet stuck a sniffer on the line to catch the
> precise
> > command it is sending. If mail.app is restarted or the connection
> > forcefully broken, the mail client becomes sane again.
> >
> > I haven't found much in the forums (we added a message to this
> active
> > thread
> >
> http://www.zimbra.com/forums/administrators/32576-mailbox-log-full-invalid-message-sequence-number-errors.html#post182064).
> >
> > So, anyone else having this fun, and were you able to do anything
> > about it?
> >
> > -- dNb
>
> --
> Pam Buffington, Department Manager II - EAS
> Office of Information Technology/A&I :: Georgia Institute of
> Technology
> pam.buffington@oit.gatech.edu :: (404) 385-7518
> https://mail.gatech.edu/home/pc21?fmt=freebusy (my calendar
> availability)
--
Pam Buffington, Department Manager II - EAS
Office of Information Technology/A&I :: Georgia Institute of Technology
pam.buffington@oit.gatech.edu :: (404) 385-7518
https://mail.gatech.edu/home/pc21?fmt=freebusy (my calendar availability)