Re: Zimbra v6 - slow autocomplete

[Tim Ross <tross@calpoly.edu>]

Steve,

We have had a number of complaints about that here at Cal Poly also (one from our campus Provost just this week), but had not discovered the root cause.  Thanks for the run down.  I voted for the bug.  I agree whole-heartedly that a re-index should not be triggered by an autocomplete call.

Just an fyi, we are running on ZCS 6.0.7.1 NE.

Tim Ross
Zimbra Application Administrator
Collaboration Support Group
Cal Poly State University, San Luis Obispo


----- Original Message -----
From: "Steve Hillman" <hillman@sfu.ca>
To: "zimbra-hied-admins" <zimbra-hied-admins@sfu.ca>
Sent: Friday, October 15, 2010 2:59:12 PM
Subject: Zimbra v6 - slow autocomplete

Since there has been some discussion on this list about what to expect regarding performance when upgrading to Zimbra 6, I thought I'd share a story about an issue we encountered. The executive summary is that autocomplete speed is adversely affected when a user has messages waiting to be indexed. The BugID to work around this is here https://bugzilla.zimbra.com/show_bug.cgi?id=46963 but has been targeted for Zimbra 7, which for us is at least a year away.   Read on for the technical explanation

In Zimbra 5, contacts were loaded into the browser at login time. When an autocomplete operation was performed, the search was done by javascript in the browser using the loaded contacts. If less than 20 matches were found, and if the GAL was enabled, a SOAP call was made to the server to do Autocomplete using the GAL. This resulted in the initial autocomplete being very fast, with the GAL results (if any) loading afterward - typically a second later.

In Zimbra 6, autocomplete has been completely redesigned. Now, the contacts aren't loaded into the browser. *All* autocomplete operations do a SOAP call to the server. The server searches your contacts for matches and, if appropriate and configured, also queries the GAL. But there's a problem: From my trace-level debugging, it appears that the autocomplete mechanism uses the Lucene Index files to do a quick search (the same ones that index messages and most other blobs), but those index files can potentially be out of date if messages have arrived recently and you have batch indexing enabled (recommended for large sites and I think the default in Zimbra 6). Right now, if the index files are detected as being out of date, an indexing operation is kicked off and any messages that haven't yet been indexed are done on the spot. The problem is, this all happens while the user is waiting for the autocomplete operation to complete. Depending on how many messages there are, how busy the server is, and how busy the index disks are, this indexing can take anywhere from sub-seconds to 20-30 seconds. All while the user waits.

This is most noticeable with power users who get a lot of mail -- I see it all the time. Light-duty users may never notice it.

A workaround could be to add code on the server to the autocomplete that simply skips the reindex operation. Since the vast majority of the time, it's only message blobs waiting to be indexed, this should very rarely affect the autocomplete results. Unfortunately, the fix for this issue has been pushed to Zimbra 7. For us, that means a year of living with the annoying delays.

If you've already upgraded to Zimbra 6, I'm curious whether you've noticed these delays. If you have screaming fast index volumes or lightly loaded mailbox servers, perhaps you don't notice it. If you do notice it, feel free to vote on the bug and add a comment that it's a QA issue for you. It would be great to get a hotfix into 6.0.9 if possible

-- 
Steve Hillman                                IT Architect
hillman@sfu.ca                               IT Infrastructure
778-782-3960                                 Simon Fraser University