Thomas Jefferson National Accelerator Facility - A DEPARTMENT OF ENERGY FACILITY
|
|
E-mail System Gets Overhauled |
 Andy Kowalski, head of the IT Division’s Computing and Networking Infrastructure group, and Paul Letta, CNI e-mail upgrade effort coordinator, (left to right) stop for a moment in the Computer Center after getting the new e-mail system up and running.
There wasn't a lot of love for Jefferson Lab's e-mail system on Valentine's Day. The servers that handle the Lab's electronic missives were overburdened, severing computer users' digital link to the outside world and the office next door. But by noon the following day, a more stable and faster electronic message delivery was in service. The dramatic turnaround was the result of months of planning and preparation. But with the Valentine's Day e-mail collapse, a command decision was made to immediately execute those plans a month early. Paul Letta, Unix systems manager in the IT Division's Computing and Networking Infrastructure (CNI) group and e-mail upgrade effort coordinator, says the old system used a 20-year-old method of storing data. In that system, each user's messages were stored as a single text file. "Every time you got a message or every time you clicked on a message to read, the mail servers would have to go and reread that entire file," Letta says. "Now, if you only had a reasonable-sized inbox or a reasonable number of messages, that's not that big a deal. But we have some account holders at the Lab whose text file is almost two gigabytes, 26 million lines." The new e-mail server software, an open-source package called dovecot, handles files differently. "As dovecot reads the e-mail folders, it creates index files. The next time you want to read a message, it first looks in an index file. It then goes directly to that point in the folder," Letta explains. Fast Facts Andy Kowalski, who heads the CNI group, says the switch to dovecot has made a huge difference in the amount of work the e-mail servers have to perform. "Before dovecot, the overnight load on the e-mail system was triple what it is now when everybody's here at work. We had all these desktops sitting around overnight doing nothing but saying: Any new mail? And this would happen over and over and over," he says. "The old system wasn't a bad system, it had just become inadequate to deal with the amount of data being sent through our e-mail system." Before the switch, for example, the e-mail system was transferring up to 100 Megabytes per second over the network, whereas Hall B sends a data stream of up to 25 Megabytes per second when writing experimental data to magnetic tape. Since the switch, the e-mail system transfer rate almost never exceeds 10 Megabytes per second. The switch to dovecot wasn't the only change made in the e-mail system. The CNI group formed a committee of computer experts to tackle the increasing demands of e-mail in mid-2007. Their plan included repurposing four other computers to replace the older servers; requiring that all e-mail clients be configured to access e-mail using the same method (a common protocol called IMAP); centralizing all computer users' e-mail inboxes; and finally, switching over to the dovecot software. Many of the changes had already been completed in preparation for the switch to dovecot, and a dovecot test server was undergoing extensive testing. The plan to release dovecot in the spring was on track. "We thought that deploying dovecot in March/April was actually going to give us enough time to do this before people really noticed the strain on the e-mail system," Kowalski notes.  This graph depicts the load on the e-mail servers with the old system (weeks 05 and 06) compared to the new system (weeks 07 and 08).
When the e-mail servers began buckling under the strain, the decision was made to switch over to dovecot immediately, before testing was completed. The switch was made the morning of Feb. 15, a Friday. Letta and Kowalski say the switch to dovecot is only an intermediate step in handling e-mail. More changes are needed to ensure e-mail stability. In addition, the CNI group will continue to investigate alternatives in meeting the ever-increasing demand on the e-mail servers. "Many other organizations have already switched over to backend databases to store e-mail. And that would be a giant leap for us. It would also be a costly leap (in the range of $100,000-250,000), and we're not going to do that right now. Paul found a low-cost, intermediate solution," Kowalski says. To help matters, Letta and Kowalski say that there are several things that individual computer users can do to alleviate the stress placed on e-mail servers. These include: - Reduce the size of your inbox, by either deleting or moving mail to individual file folders, especially those items that contain attachments. By Kandice Carter |