Privacy and Security Notice

OPERATIONS ORIENTATION(http://jlab.org/~erb/ops_orientation.html)

OPERATIONS ORIENTATION
(http://jlab.org/~erb/ops_orientation.html)

Rev 4; 10/17/02
Michele Joyce; erb@jlab.org

Introduction

The Operations Orientation Program is designed to help you, the experimenter, become familiar with the standard practices used to run the accelerator. This document introduces some of the standard procedures used to deliver usable beam and also explains some of the terminology that is unique to Jefferson Lab. Our goal is simple: facilitate communication between the Operations Department and the Experimental Users so that we can achieve our common goal, world-class physics. Ultimately, better communication will make your job--and our job--easier.

An Accelerator Overview

The CEBAF accelerator is a superconducting radio frequency (RF) electron accelerator that was commissioned during the early 1990s and produced the first experimental beam in October of 1994.

 

Date

Milestone

1976

Facility envisioned and requested by physicists as a necessary tool to answer emerging questions about the quark structure of matter

1980

Initial design developed

1983

The SURA proposal was selected by the Department of Energy after competition with Massachusetts Institute of Technology, University of Illinois, Argonne National Laboratory, and the National Bureau of Standards

1984

The Newport News site was selected and the initial federal funding was received for research, development and design

1985

Superconducting electron accelerating technology adopted

1987

Accelerator construction underway

1990

Prototype injector installed in tunnel

1990-1993

Accelerator commissioning with beam

1994

First physics experiments started

1995

Design energy of 4 GeV reached

1997

Simultaneous 5-pass, 4-GeV, 3-beam separation to all three Halls

1999

Machine energy at 5.5 GeV with 3-hall operation

The accelerator uses a state-of-the-art photocathode gun system that is capable of delivering beams of high polarization and high current to Hall A and Hall C while maintaining high polarization low current beam delivery to Hall B. An RF chopping system operating at 499 MHz is used to develop a 3-beam 1497 MHz bunch train at 100 keV. The beam is then longitudinally compressed in the bunching section to provide 2 picosecond bunches, which are then accelerated to just over 1% of the total machine energy in the remaining injector section. The beam polarization, optics and energy are verified in the injector matching region prior to injection into the main machine. The beam from the injector is accelerated through a unique recirculating beamline that looks something like a "racetrack", with two linear accelerators joined by two 180° arcs with a radius of 80 meters. Twenty cryomodules, each containing eight superconducting niobium cavities, line the two linear accelerators. Liquid helium, produced at the Lab's Central Helium Liquefier (CHL), keeps the accelerating cavities superconducting at a temperature of 2 Kelvin. The linac energies are each set identically and the RF cavities are phased to provide maximum acceleration. Subsequent passes through the accelerator are phased to maximum energy gain by adjusting the length of travel in the dogleg section of the preceding arc. Quadrupole and dipole magnets in the tunnel steer and focus the beam as it passes through each arc. More than 2,200 magnets are necessary to keep the beam on a precise path and tightly focused. Beam is directed into a hall's transport channel using magnetic or RF extraction. The RF scheme uses 499 MHz cavities, which kick every third bunch out of the machine. The accelerator can deliver the first four passes to one hall only. The fifth pass can be sent to all three halls simultaneously.

Accelerator Operations Directives (AOD)

The Accelerator Operations Directives is a comprehensive, detailed document that describes how business is conducted in the MCC Control Room. The major sections of the document are Program Control, Accelerator Operations, and Maintenance and Tracking. The document defines how we do business at the lab and is an invaluable source of information pertaining to accelerator operations. We will not go into that same level of detail in this Operations Orientation; however, it will be useful for you to know that such a document exists and is available on the web at: http://opweb/internal/ops/opsdocs/accel_ops_directives.pdf .

MCC Shift Staffing

An accelerator day (24 hours) begins at 2300 and ends at 2259. This period is separated into three distinct shifts:

Each MCC Control Room crew consists of a Crew Chief (the senior member of the team) and two or three operators. Shifts for the operators and the Crew Chiefs are staggered. The Operations crew comes in an hour later than the Crew Chiefs (0000, 0800, 1600). When a new crew comes in, it is necessary for the outgoing team to "turnover" the machine. "Turnover" or "passdown" involves explaining the current status of the accelerator as well as events that have happened during the last shift. Typically, this information exchange lasts approximately 15 to 20 minutes, during which requests by the halls may be postponed.

Communicating with the MCC

Generally speaking, routine communication between the halls and the MCC during a shift is via telephone, and you will speak with one of the MCC operators who will then coordinate with the Crew Chief to fulfill your request. Use one of the following telephone extensions to call the MCC Control Room

Scheduling and Overall Program Coordination

Program Deputy

The overall accelerator running schedule is controlled on a shift-by-shift basis by the Program Deputy (PD). During his/her two-week period of service, the PD is responsible for the accelerator program for all shifts. The day-by-day long-term schedule for running is developed well in advance by the Nuclear Physics Experiment Scheduling Committee (a copy of this schedule is posted on the web at http://www.jlab.org/exp_prog/experiment_schedule/ ). There is also a detailed memo from the scheduling committee that explains many considerations for upcoming running (this is very useful information; see http://www.jlab.org/exp_prog/experiment_schedule/2002/pub_Oct11/footnotes.html ).

Through meetings with the experimenters, program updates, and shift plans, the PD further refines the long-term accelerator schedule, developing it into a shift-by-shift plan of coordinated activity. The following meetings are a critical part of coordinating the accelerator schedule.

  • DAILY 0745 MEETING WITH EXPERIMENT RUN COORDINATORS: This meeting is held with the Experiment Run Coordinator for each experiment that is scheduled to receive beam that day or the following day. It serves as a routine means of exchanging information concerning the operating schedule, planned activities for the next 24 hours, accelerator performance, or any other concerns that affect the scheduled program. The meeting generally occurs at 0745 in the MCC Conference Room on weekdays, and on weekends at 0830 as necessary.
  • DAILY SUMMARY MEETING: Here, the PD presents the Crew Chief Shift Summaries for the previous 24 hours (or weekend), develops any necessary short-term schedule modifications, and assures that the appropriate laboratory resources are utilized as necessary to keep the accelerator program on schedule. This meeting occurs every weekday morning at 0800 in the MCC Conference Room.
  • WEEKLY ACCELERATOR PLANNING MEETING: At this meeting, the PD defines the short-term operating schedule for the upcoming two-week period. The schedule is posted on the MCC Conference Room "White Board" and kept as up to date as possible at all times. This information is also routinely transcribed and posted to the World Wide Web at http://opweb/internal/ops/whiteboard . This meeting is held on Wednesdays at 1330 in the MCC Conference Room.

The PD integrates the information obtained at these meetings into a detailed running schedule and passes it on to the Accelerator Operators via PD Shift Plans ( http://opweb/internal/ops/pd_logs/current_logs/ ). At the beginning of their shift, the Operations crew reads the eight-hour shift plan, which provides detailed instructions for operations during the shift. Any changes to the PD Shift Plan (with the exception of necessary, unanticipated accesses) must be approved by the Program Deputy. The PD cell phone number is: 876-7997.

Hall Liaisons

Each Experimental Hall has an operator or Crew Chief assigned as a Hall Liaison. The Experiment Liaison helps facilitate information exchange between the experimenter(s) and the MCC Operations Group, both in advance of and during actual experiment scheduled to be performed. Since each experiment generally has unique operating requirements and associated personnel who may not be familiar with normal operating procedures, the Hall Liaison serves as a valuable link to help identify problems before they occur and educate both the experimenter and fellow operators.

Accelerator (Machine) States

The safety interlocks of the Personnel Safety System (PSS) determine the access status of the tunnel and the halls and permits or inhibit operation of the electron gun, high power magnets, and RF acceleration systems.

Setting Up Beam to the Halls

Sweeps

Most work done in the halls is performed with the hall in "Restricted Access". Before relocking the hall for beam delivery, the hall must be thoroughly searched (called a "sweep") to ensure that all personnel have exited. Announcements will be made 15 minutes and 5 minutes prior to the sweep to allow personnel ample time to leave the hall. The sweep is performed by two people: one a trained sweeper who actually performs the sweep, and a second "untrained" person to act as a guard so that people cannot reenter the area. It is very helpful to the MCC if the hall provides a person to act as the guard. Freeing up the second operator will help you get beam back faster.

During a sweep, the operator searchs the hall in a pre-determined pattern while arming the run/safe boxes. If any of the detector huts (in Halls A and C) have been opened, they will be searched and closed at this time. The run/safe boxes in the detector huts disarm themselves if the door to the hut has been opened. The huts will, therefore, need to be searched, and the boxes re-armed after each entry.

Establishing Beam to Hall A and/or Hall C

When the operators initially set up beam to Halls A or C, it is necessary for the orbit to be checked with "PULSED BEAM" (also referred to as "TUNE BEAM.") This beam has a duty factor of 1.5%. This means that 8 mA of pulsed beam is equivalent to:

Low-current pulsed beam will generally not affect hall equipment, except for possibly a polarized target (if that is the case, the potential for damage should be communicated to the Hall Liaison and Program Deputy well in advance of initial running). The MCC Control Room will always notify the hall when pulsed beam is about to be run for initial steer up or to check the orbit to the hall.

Establishing Beam to Hall B

When beam is initially being established to Hall B, the orbit at the beginning of the Hall B line is set up using pulsed beam to the Tagger Dump. After tuning beam in this region, the hall performs a harp swipe with low current CW to check the profile of the beam. Once the profile is deemed acceptable, the hall switches to straight-ahead running to the Faraday Cup (that is if the experiment calls for electron as opposed to photon running), and the Tagger Magnet is degaussed and turned off. Degaussing the magnet takes about 15 minutes. Before sending initial beam to the Hall B Faraday Cup, the target solenoid and the raster must be OFF, and the cryotarget ladder must be removed or set to an empty cell. The beamline from the Tagger Magnet to the Faraday Cup is steered with small amounts of CW beam. The hall will then be asked to turn on the target solenoid and torus and insert the target. If the drift chambers remain on without tripping and the downstream PMT counts are acceptable, beam has been successfully established to the Faraday Cup. Minor steering to the desired positions on the nA BPMs (2C22A, 2C24A, and 2H01) can usually be accomplished during the physics run.

The nA BPMs (nano-Amp Beam Position Monitors) are beam position diagnostics designed specifically to work with the low beam currents used in the Hall B line. Keep in mind that these monitors do not accurately measure the beam position at currents of less than 1 nA. When the beam current drops below the 1 nA threshold, even though the beam has not physically changed position, it will appear to have on the BPM readbacks.

Beam Delivery Interruptions

Once beam has been established for Physics running, the operators will do their best to continuously deliver beam. During running, the halls should notify the MCC Control Room of any configuration changes that may alter the beam orbit, such as changing the state of magnets, radiators, rasters, target cells etc. Interruption of beam delivery to a hall can be for a variety of reasons, either planned or unplanned.

Unplanned Beam Interruptions

When the beam trips off (i.e., it is not turned off deliberately), an alarm sounds in the MCC Control Room, alerting the operators to the trip (these are referred to as Fast Shutdown [FSD] trips).

  • MOMENTARY FAULTS: The beam may trip because of a simple RF fault. These types of trips can be reset in a matter of seconds. RF faults become increasingly frequent as energy and current are increased.
  • EXTENDED FAULTS: A more serious problem may, of course, take longer to correct. If the operators suspect that the problem will take longer than five minutes to fix, they will call each hall and provide an initial update on the problem and then provide periodic updates thereafter. Please remember that these first five minutes are crucial to the Operations Crew. During this time they are trying to determine the extent and cause of the failed beam delivery and get beam back to you as quickly as possible.

Planned Beam Interruptions

  • AN INDIVIDUAL HALL REQUEST: A hall can, at any time, request that the beam to their hall be turned off. This will not affect beam delivery to the other halls.
  • A "PRIORITY HALL" REQUEST: For each day, the long-term schedule identifies a specific hall as the "Priority Hall". The Priority Hall has the primary voice in decisions concerning beam quality and/or changes in the accelerator operating conditions (this does not include changes in beam energy that would affect planned running in the other halls without their consent). The "Priority Hall" has the right to do the following:
  • Require an immediate retune of the accelerator when beam quality is not acceptable.
  • Insist that an energy change occur as scheduled.
  • For more information on Priority Hall status, please refer to the Accelerator Schedule Memorandum posted on the World Wide Web at: http://www.jlab.org/exp_prog/experiment_schedule/2002/pub_Oct11/footnotes.html
  • SCHEDULED TEST PLANS: On occasion, beam time may be required to test program-critical hardware or software in the accelerator. These activities are planned in advance and executed according to written test plans. Each test plan follows a standard format and must be preapproved by the PD and MCC Operations. Every effort is made to minimize interruption to scheduled beam delivery.
  • MACHINE DEVELOPMENT: The "Machine Development" time is used to prepare for new experiment setups such as polarized beam nonstandard energies, and three-beam operation.
  • HOT CHECKOUT : Dedicated time during which all accelerator systems are recovered, exercised, and made ready for beam operation prior to actually attempting to restore beam.
  • RESTORE: The period assigned to tune the beam to meet specifications for its next scheduled use.

 

The accelerator schedule allots shifts as necessary for each of these activities.

Operational Restrictions

Each hall has certain administrative operating restrictions that limit specific operating parameters. These restrictions are spelled out in a web-accessible document know as the Operational Restrictions ( http://opweb.acc.jlab.org/internal/ops/ops_webpage/op_limits/oplimits.html . These restrictions include, but are not limited to the following:

  • Energy restrictions
  • Current limits
  • Power limitations for dumps
  • Beam raster information for targets and dumps

Any time a hall requests a setting change for one of the above parameters (not a change to the actual limit, just a setting adjustment), the Crew Chief is responsible for making sure that the request falls within the limits of the Operational Restrictions. A list at the top of the Operational Restrictions document shows who has the authority to make a change to any of the limits. Some require run coordinator approval, others hall coordinator approval, and still others can only be changed by the Accelerator Division (i.e., dump power limits).

Making a Controlled Access Into a Hall

At times, it may be necessary for hall personnel to enter a hall to investigate or repair a problem. If this is the case, the hall notifies the MCC Control Room, so that preparations can be made to make the hall safe for entry. This process takes about 10 minutes . Prior to entry, a radiation survey must be performed (this requirement is a little different for Hall B) and then specific access procedures must be followed. These steps are detailed below.

Radiation Surveys in the Halls

Halls A and C: To ensure personnel safety, radiation surveys must be performed by RadCon personnel or a qualified Assigned Radiation Monitor (ARM) before entering Hall A or Hall C. At the discretion of the person surveying the hall, hall personnel may be allowed into the hall under escort. If this is the case, the hall personnel must remain with the escort and must not enter any area that has not been surveyed.

Hall B: Because Hall B runs such low current and induces very little radiation, they utilize a "Rapid Access System" when making Controlled accesses. This system allows Hall B personnel to check the radiation levels inside the hall from the hall's entrance. A magenta beacon is located outside of the Hall B door. If the beacon is not flashing but turns on when the test button is pressed, it is safe to enter the hall. If the beacon is flashing and does not turn off after a few minutes or fails its function test, a full survey of the hall by RadCon or an ARM is required. During a Rapid Access, certain areas of the hall must not be entered. These areas are clearly posted, and if there is a need to enter, a survey of the area must be performed. Similarly, areas on or around the beampipe must be surveyed prior to beginning any work.

The Controlled-Access Procedure

After the hall has been surveyed (or the "Rapid Access System" has been tested in Hall B) and RadCon (or the ARM) has deemed the hall safe, personnel are free to enter the hall by following the Controlled Access procedure. During a Controlled Access, the doors remain locked (using magnetic locks) and entry is controlled remotely by the Safety System Operator (SSO) in the MCC Control Room. This allows the SSO to account for each person entering or exiting the hall. For this reason and to check for proper training, the following procedure must be followed when entering or exiting a hall under Controlled Access.

ENTERING THE HALL

  1. Proceed to the hall entrance and, using the phone located at the entrance, call ext. 7050. The SSO will unlock the outer door to the hall.
  2. Enter the access room. The first person (or a member of a group) to enter the hall must take the Master Key out of its slot and put it in the empty slot below. The SSO must release the Master Key before it can be removed. This will release the other keys.
  3. Each person entering the hall must take a key (there are 10 keys) and keep it with them at all times.
  4. Call ext. 7050. Each person must give the SSO their full name (including spelling), the key number in their possession, show their radiation badge (TLD) (the camera is located in the access room), and indicate whether or not their ODH training is up to date. If a person's ODH training is not up to date, they can be escorted into the hall--one untrained person to each trained escort.
  5. After collecting the needed information from each person entering the hall, the SSO will open the inner door to the hall

EXITING THE HALL

  1. To leave, proceed to the hall exit door and call ext. 7050. The SSO will open the inner door.
  2. Enter the vestibule. Each person must reinsert their key into the appropriate slot. If a person (or group) is the last to leave the hall, the Master Key must be reinserted into its slot above the other keys.
  3. Call ext. 7050. Provide the SSO with the name of each person leaving the hall.
  4. The SSO will open the outer door.

If a detector hut door in either Hall A or Hall C has been opened during the Controlled Access, the MCC Control Room must be informed so that they can re-arm the hut's run/safe box and shut the hut door.

Once the Controlled Access has been completed, the hall should inform the MCC Control Room and request a return to Beam Permit. The operators will then take the hall to Power Permit, remove the hall beam stopper, turn on the hall magnetic dipole, and cycle the dipole through hysteresis. This entire process takes about 15 minutes. The operators will then send pulsed beam to verify that the orbit has not changed.

Emergency Response

In the event of an emergency in one of the halls or a counting house, the MCC Control Room should be contacted immediately. The Crew Chief will immediately dispatch two responders to investigate the problem and, if necessary, call 911. Instructions for responding to specific types of emergencies can be found on the information card attached to each telephone.

Screens and Logs

Each hall has a customized MEDM information screen that shows general accelerator operating parameters and appropriate hall-specific parameters. These screens are designed to the individual hall's specifications, information can be added by contacting the Operation's Staff. This screen should be displayed on a terminal in the hall control room whenever beam delivery is in progress.

General Accelerator Parameters

The accelerator parameters that are generally useful to the halls include the following:

  • The names of the on-duty Crew Chief and operators
  • The current PSS status of each hall
  • The energies and currents that all halls are receiving
  • An FSD trip indicator that turns RED when the beam has tripped off (it coincides with an alarm that is received in the MCC Control Room).

Hall A-Specific Parameters

Parameters that are specific to Hall A are as follows:

  • Beam position on target
  • Fast raster settings
  • Fast feedback status
  • Ion chamber readings and mask status
  • Compton chicane BPMs. If beam is being run through a BPM, the presence of beam (i.e., beam position) is indicated by a blue bar.
  • Moller target position

Hall B-Specific Parameters

Parameters that are specific to Hall B are as follows:

  • Upstream and downstream PMT counts
  • 1 nA BPM data, including X and Y positions (in mm) and intensity (in nA)
  • Tagger dump PMTs

Hall C-Specific Parameters

Parameters that are specific to Hall C are as follows:

  • Beam position on target
  • Fast and slow raster settings
  • Ion chamber readbacks

MCC Electronic Log Book

The MCC Control Room Electronic Logbook (ELOG) can be accessed via the web at http://opsrv.acc.jlab.org/CSUEApps/elog02/elog.php . Each hall also has its own logbook; however, only the Hall B and Hall C logbooks are accessible to the MCC operators. It is helpful to the operators if when you post important information (test results, etc.) to your individual logbook, you also post that same information to the MCC Control Room electronic logbook as well as to the individual hall logbook.

Time Accounting

Keeping track of how much useful beam is delivered to experiments (i.e., "time accounting") is especially important here at Jefferson Lab. The Lab's contract with the Department of Energy is a "performance based" contract, where success is gauged at least in part on how much beam is delivered to experiments. Thus, careful attention is given to monitoring how the beam is used. Each of the experimental halls plays an important role in the time-accounting process.

Keeping Track of ABU (Available Beam in Use)

The most important concern for the halls is keeping track of ABU (i.e., when the beam is being used to perform a part of the hall's scheduled program). It is important to note that there are occasionally times when ABU is not just the delivery of CW beam to a target. An energy measurement or current calibration may need to be performed, and that time should also be counted as ABU.

It is also important to understand the time spans used to record ABU. The Control Room Crew Chief's shift begins and ends an hour earlier than the rest of the crew. For example:

  • Swing Shift Crew Chief hours: 1500-2300
  • Swing Shift Operator hours: 1600-0000.

Time accounting for accelerator shifts is based on the CREW CHIEF's hours. Therefore, at the end of the shift, when the Crew Chief calls the halls and asks for the "Available Beam in Use" (ABU) hours for the shift, the Crew Chief needs to know the number of hours that acceptable beam was used during the Crew Chief's shift. In the above example for Swing shift, that would be from 1500-2300.

Up Time

Up time is just a little bit different than ABU. Up time is actually measured automatically by a program called "BOOM" which counts Up time whenever it sees CW beam on a hall-specific beam-current monitor (BCM). BOOM feeds this number directly into the Crew Chief shift log, where it is only edited by the Crew Chief if it is blatantly wrong.

Useful Information Sources

Title

Description

Source

Long Term Accel. Schedule

A day-by-day long-term schedule of accelerator running developed by the Nuclear Physics Experiment Scheduling Committee.

http://www.jlab.org/exp_prog/experiment_schedule/

Long Term Accel. Scheduling Memo

A detailed memo from the scheduling committee that explains many considerations for upcoming running.

http://www.jlab.org/exp_prog/experiment_schedule/2002/pub_Oct11/footnotes.html

Short Term Accel. Schedule ("whiteboard")

A web-accessible version of the short term (shift-by-shift) schedule posted on the MCC conference room whiteboard. Updated daily.

http://opweb/internal/ops/whiteboard

Program Deputy (PD) Shift Plans

The PD's shift-by-shift instructions for running the accelerator. Provided to the MCC crew for each shift.

http://opweb/internal/ops/pd_logs/current_logs/

Elog

The MCC Operations official electronic logbook (includes a web interface to make an entry). Also includes links to lots of other useful stuff.

http://opsrv.acc.jlab.org/CSUEApps/elog02/elog.php

8 a.m. Meeting Notes ("Benesch Report")

A subscribeable email summary of the daily 8 a.m. meeting held in the MCC conference room. Summarizes machine operations in an occasionally irreverent fashion. Also available as a web-accessible archive (which also includes subscription instructions).

Email Group: machine-ops

 

Web Archive:

http://www.jlab.org/ccc/mail_archives/ACCELERATOR/machine-ops/?intralab

Weekly Performance Summaries

A web-accessible version of the information that is presented at the Weekly Accel. Scheduling Meeting (Wednesdays at 1330 in the MCC conference room). Includes PD Weekly Summary, Lost Time Report, and Tune Time Report.

http://opweb/internal/ops/weekly_perf_summaries/

Machine Status

An eight hour snapshot of beam activity in the accelerator.

http://opweb/external/operations/overview/cebaf.html

MCC Operations Documentation

An index of operating procedures and other information used in the MCC to perform specific tasks.

http://opsntsrv.acc.jlab.org/ops_docs/MCC_web_interface

Accelerator Operations Directives (AOD)

A comprehensive directive that describes and governs how MCC Operations "does business."

http://opweb/internal/ops/opsdocs/accel_ops_directives.pdf

CEBAF-Specific Acronym List

An informal list of common acronyms used at JLab and what they stand for (BOOM, SWIS, BLM, etc.).

http://opweb.acc.jlab.org/internal/OPS/acronym.html

Operational Restrictions

A list of restrictions on parameters such as current, power, and energy. Also refers to personnel able to change these limits.

http://opweb.acc.jlab.org/internal/ops/ops_webpage/op_limits/oplimits.html

 

Useful Telephone Numbers