July 25, 2011

During the Fall of 2010, there were lots of discussions about what was going to happen in the SAD in 2011. SAD is a local acronym, S cheduled A ccelerator D own. Steadily, we began to realize that the breadth of the related work being planned was enormous. It went way beyond the usual accelerator maintenance. Of particular, but not exclusive, importance was to be the work associated with the 12 GeV project. For example, the civil construction required a breakthrough from the external Hall D beamline into the accelerator tunnel proper. It also involved the refurbishment and modification of arc magnets to provide capability up to 12 GeV.

You could also worry that one way of expressing this program was to say that the intent was to “… get in the tunnel, rip out the magnets, refurbish them and throw them back in and hope that the machine would still work.” Clearly, such an approach would be unacceptable; among other things, it would not provide a good basis for planned running of experiments in 2012. But the example is a particularly clear example of the potential interference we faced between two aspects of the work being planned.

Other obvious interfaces concerned the Technology Engineering Development Facility (TEDF) project. Constructing the new building and the extension of the Test Lab could lead to interference with the superconducting radiofrequency work, including that for the 12 GeV Upgrade Project. There would be removal of some apparatus from Hall C which might affect the important Q-weak experiment. Other nominally isolated work would be going on inside the other experimental halls, installation of new experiments in Hall A, maintenance work on Q-weak in Hall C, maintenance and HDice work in Hall B. But even nominally isolated work is connected to the real world through limits on resources, cash and effort and skills. In the end, we realized that almost every piece of the lab was implicated, with the possible exception of theory.

To handle this, we defined a project and, of course, an acronym, the 6MSD. Yes, the “Six Month ShutDown!” One of the things that the lab has learned to do pretty well is project management, and many of these jobs had well-developed project schedules, some loaded with resources. We also know how to conduct operations with the accelerators and the experiments; that is what we do most of the time. So, when we appointed Fulvia Pilat as the “Czarina” for the 6MSD, she realized that her job was to pull together a lot of existing plans, to understand the interfaces in whatever parameter, alignment technician to electrical engineer to mechanical tech, and to generate some coherence. Over the course of a fairly intense six-month planning period, this was achieved and on May 15 we started the 6MSD.

Over the course of the first 10 weeks, progress has been amazing. Many teams, such as that responsible for magnet refurbishment, are a 75:25 split of new people and old hands. The way the new team members have been integrated is impressive. The magnet team has done so well that we have been able, after due deliberation and an understood change control process, to expand the scope of work by 50 percent while retaining the original schedule.

So, it’s all a triumph? No, not quite. Unfortunately we are concerned that the numbers of first aid cases have undergone an uptick in the past few weeks. Thus far the numbers of more serious incidents is quite limited, but we are concerned. A week ago, after discussion among various components of management, we concluded that we needed to raise the flag. So, on Thursday, July 21 we met in All Hands gatherings to talk about things. Fulvia Pilat showed the phenomenal progress we have made, Mary Logue got to show the evidence that we need to tighten up our safety act. A photo of a gas bottle, unsecured in a pickup truck, with regulator still in place and the tailgate down, raised a stir. The stories were clear. We know better. Most of the time, we do better. Perhaps we feel the need to rush? There is no need: a) we should never sacrifice safety to speed, and b) we are hitting the schedule in almost every area of work.

So, to repeat, the message is clear: “Let’s not snatch defeat from the jaws of victory!”