Problem Reports  & Solutions & Hot tos & Stuff

Qweak target: Resetting the horizontal motion after it has slipped.
Qweak target: Amp Fault/Stuck lifter:
Qweak target: Pump/Fan Reset (LH2 recirculation pump):
Qweak target: Turbopump reset
Qweak target: IOC reboot
Qweak target: Target computer (poltarcc) crashes
Qweak target: Cooldown tricks
Qweak target: Encoder Table (Last updated Feb 10, 2011)
Qweak target: Reload the alarm limits
Qweak target: Grab the screens for an hclog
Qweak target: Move to a different workspace

G0 target: Opening the JT actually closes it
G0 target: Site-wide Power Failure
G0 target: typical warmup
G0 target: typical cooldown , here too
G0 target: target scan
G0 target: Changing the alarm limit files
G0 target: Tach problems
G0 target: MedmQ
G0 target: New HPH PS pin assignments, and problem (and fix) with power calculation
G0 target: Mystery pressure drop when going to C or out-of-beam position
G0 target: ITCs hang, heater goes beserk
G0 traget: Coolant xfer line insulating vacuum
G0 target: Gas target running
G0 target: House helium leak
G0 target: xfer line IV
G0 target: No Striptool, readout errors stuck on

Any tgt: Frozen computer (eg jeffylab)? Then restart X by typing `control-alt-backspace' , then type `startx'.
Sp03 tgt: GRS notes on the cooldown procedure
SP03 tgt: GRS notes on the warmup procedure
SP03 tgt: GRS notes on the ALH files & Checklist

SP03 tgt: Can't operate JT valve
SP03 tgt: Yellow alarm handler light won't go away.
SP03 tgt: Can't start alarm handler. Can't start the stripcharts.
SP03 tgt: IOC won't reboot.
SP03 tgt: Cryostack lifter does not work.
SP03 tgt: JT PID for parking the target on autopilot:
SP03 tgt: Shorted high power heater

SM04 tgt: Archiver stopped.
SM04 tgt: Scattering chamber vacuum.
SM04 tgt:

Any tgt: Terminal Bell volume
Any tgt: Automatic Paging
Any tgt: Flatline Alarm
Any tgt: Beam Current & Raster Alarms
Any tgt: Turbopump or vacuum problems
Any tgt: Power cycle the fans

FAQs:

  Encoder table: The encoder table for all the target positions is here (Feb 10, 2011).

  Switch workspace on poltarcc: Use cntrl-alt-rightarrow to move to the workspace to the right.

  Screen grab for the hclog on poltarcc: Use the printscreen button on the keyboard to grab the picture of both screens (striptools & main GUI). If you just want to grab a single screen, press alt-printscreen. Then save the images, and make an hclog from the browser. Attach the file(s) you just saved.

  Amp fault/stuck lifter: Hit the big red panic button on the left side of the U-shaped console in the counting room to power cycle the 6961 controller. Then pull it back out! You may want to look at the controller with the TV to insure it is back on, and out of fault condition. Then go "HOME" on the Main Gui target selection. If you look at the lights above the 6961 controller (with the TV camera), when not moving the target the 2 middle lights ("no fault", and "power cycle") should be lit green.

  Reset the LH2 recirculation pump/fan: Bring up the pump GUI from the Main GUI. Hit "reset", that usually does the trick. If not try "clear" and "Update". If that fails try entering a new fan frequency (~ 3* what you want in Hz for the tach) and try stopping and restarting. Look at the TV camera to see the tach in the hall to see if it is starting. Note that it may take a few seconds to see it start to move after a successful action. If all else fails hit the big red pump panic button on the center section of the U-shaped console in the counting room (and then pull it back out). Then go back and try to start it the usual ways. For a summary of the instructions, see http://www.jlab.org/~smithg/target/Qweak/IFTHELH2PUMPTRIPS.pdf.

  Vacuum Problem: reset turbos or open turbo gate valve: See target web page. The downstream beamline turbopump controller and CCG controller are in the main detector hut, beam left side, near the back at ankle height on the FW platform. You can see it as you enter the MD hut.

The turbopump controllers for the scattering chamber, surge tank, and (for now) the 15K supply xfer line are located just behind the standard pivot gas panel in a blue rack about head height. Walk straight ahead as you enter the hall and you will run into this gas panel, and the blue rack just behind it is where these controllers are. The CCG controllers for these are in the array of other CCG controllers just to the left of the target computer in the target section of the blue racks in the hall, just to the right as you enter the hall.

If the turbo controller has tripped, you can see it on the Hall C/Qweak vacuum GUI from edmmonticello. In the hall, on an escorted (quick) access, you will see a fault light on the controller is lit. Press "Stop", wait a few minutes for it to spin down, then hit "Start". The lights will cycle up the diagonal until all are lit. This will take 5 minutes or so. Then they will start  un-lighting until only the bottom load light is left lit.

Finally, there is a gate valve between each turbopump and the space it pumps on. A common control is used for the 3 on the target chamber, it is on the 1st floor of the blue platform, essentially right above the 3 turbopump controllers. Red is closed, if it is open there are no lights cuz the open light is burned out. SOmetimes this GV closes, and you may need to reopen it by hand in the hall. Just to the left of this valve you will also find the backing pump and blower shared by all 3 of the turbos.

  Cooldown tricks: Open MV28 and MV2, leave MV3 closed. Close MV28, MV2 (and MV3) after cooldown, and watch out you will get 1 kW extra on the heater when you do that. Use 5-10 g/s 4K, 10-15 g/s 15K on cold return, 8 g/s while on WR. Close 4K bypass when 4K return temp at ESR begins to drop. Run pump at 30 Hz early to get some head earlier. Phone numbers: Errol 7058.

  IOC reboot: Press reset button the the IOC, which is in the electronics room next to the counting room. You'll see it as you pass through the star trek doors immediately on your left, at head height. Carefully poke your finger under the IOC ribbon cable and press the "reset" button. Use the pot in the counting room to control the auxiliary heater power supply and supply heat to the heater in the target during the reboot. You have to switch this on, and off again when you are done. You may wish to switch off the main heater while you do this. Keep an eye on the target temps just to the left of this pot. These temps will tell you what is going on while the IOC is down. They don't go through a computer to get to those red LED displays.

  Target controls computer crash recovery: If the target controls computer goes down for whatever reason, slide over to the cdaql6 terminal. Login as poltar@cdaql6 with the usual un/pw. Then type "cd Desktop/Screeens". Then type "./tgtgui". Now you can monitor and control the target from cdaql6 while you take your time resetting the target controls computer, poltarcc. Note you can also do this on poltarch, the target computer in the hall., by ssh'ing to it from anywhere.

  Reload the alarm limits: Move to Desktop/Screens/ALH. If you need to change an existing alarm limit, just emacs the text file alarmlim-coldH2.csh which just consists of a bunch of caputs. Then get out and go back a level to Desktop/Screens. First type "source setup" to establish the EPICS addresses you need. Then type "source ./ALH/alarmlim-coldH2.csh". To add new alarms, emacs the file qweak.alhConfig in the same directory. Parrot the syntax used for the other alarms. See hclog 227303.

 Horizontal Motion reset after it has slipped:
1. Click "Horiz. Motion." Use this gui to move the target horizontally by putting step numbers in the "Absolute Position:" box.

2. Have the qwkphytronXpert.adl gui opened. It should look like this.
And the qwkphytronParams.adl should look like this.

3. Keep moving to negative step numbers until LS- is hit, write down the "Actual Steps:"
Also Xpot should show about 4.3-4.5 

4. Go to the Xpert gui and in the command line PC? put PC-66628 (no spaces) and hit ENTER. Then put the command PC? and hit ENTER. The read-back in "Actual Steps:" box should change to -66628 (this resets the step counter to the calibrated value for the LS-).

5. Click Reset button and then click "Lim SW reset" button. It shoudl clear "Limit switch -" red light.

6. At this stage you can either move it by hand or from the main gui to check it out.

7. Once you moved it away from LS- and still see an Error red light click Status, if it has a red light for "Limit switch error" then click "Reset" and "Lim SW reset" again. This should clear it.

The difference between actual steps at 3. and 66628 divided by 1600 is how much it slipped in mm.

The LH2 Xpot is 48.50 - 48.60


Archiver stuck:  Check status at http://jeffylab:4812/   Go to Groups, Target.
 To restart archiver, on jeffylab, ~/Archives,  type ./start_archiver  (that stops & starts it). There is also a ./stop_archiver, by the way.

How to print or make the Checklist: cd $GUI, ./Make_Checklist.bash
To print: rm ~/Checklist,      ./Make_Checklist.bash > ~/Checklist,       pdq -Pchchp2 ~/Checklist

Can't operate JT valve: Usually because the step size is zero. One operator had problems entering a step size because they had forgotten that you have to double-click in the step-size field, enter the step size, then hit enter all while the mouse is still in that step-size field.

Stuck alarm handler: Sometimes after an alarm, you can't clear the light in the alarm handler. Just stop and restart the alarm handler from the main gui.

Can't start alarm handler and/or stripcharts: Usually because someone killed the xterm window from which the main GUI was launched. When this happens, kill the main gui and everything else, bring up a new xterm, restart the main gui and everything else. Note that this xterminal has to stay alive in order to receive diagnostic messages. If it is dismissed, things won't work or won't start like the alarm handler and the stripcharts.

IOC won't reboot: Ususally because the network is down. In this case, closely monitor the TV temps on the ITCs and adjust the auxillary heat load pots by hand to keep the temperatures near their goal values. Retry the IOC reboot every 5 or 10 minutes until the network is back and the reboot succeeds. Can also be the optilogic box behind the furthest left blue racks in the electronics room. Go thru the startrek doors, turn left, walk to the wall. Go behind the racks. In the middle rack, about chest high, is some black box with the word optilogic on it. Pull the power cable to power cycle it and you ought to be back in business.

Cryostack lifter doesn't work: Usually this is because someone has pressed (accidentally) one of the several emergency buttons which kill the power to the lifter. See for example this hclog entry.

 JT PID: This JT pid was tried out over the 2003 open house. It adjusts one of the JTs to keep the return temperature at 30K. This way the targets can all be left safely as gas, not liquid. The pid parameters are given in hclog 60105.

Shorted HPH: Symptoms of the short we had in the HPH can be found in hclog entries 58828 , 58857 ,

Changing default alarm limits: In /group/gzero/g0targ/bin/alarmlim-cold-H2.csh, edit the caputs to reflect the new alarm limits you want to overide the existing defaults. As g0targ@gzerol2.acc.jlab.org.

Tach Problems: In the main gui "general" button, under "tachometer expert". After a power cycle it may come back in local instead of remote mode. You can reset it from this GUI.

Or, you can filter out the noise that is there on the readout sometimes. Note that an IOC reboot can sometimes reduce the noise. But if it persists, the options are:
The smoothing factor is a configurable value between 0 and 1 were 0 means no smoothing and 1 is ultimate smoothing (in fact, the data value will never change)..default will be .95

After a reboot, sometimes the tach readback is so noisy even the filtering is useless. Another reboot will solve this problem. Don't ask me why, but this ALWAYS works!

:MedmQ Starts the main G0 target GUI without all the strip charts and without the alarm handler. Useful when running/monitoring from home, for example.

New G0 HPH PS: The current reaback is flakey. So new variables ghptrue_ca and ghtot_ca were made based on V^2/R instead of V*I. But the main GUI still shows ghptrue and ghtot based on V*I for now, because it is hoped the curr3ent readback problem can be fixed in May.

Terminal Bell Volume: From an xterm on the target computer, use the xset command:
To set bell volume, pitch and duration:
         xset b [vol [pitch [dur]]]          b on

b       The  b  option  controls bell volume, pitch and duration.  This
               option accepts up to three numerical  parameters,  a  preceding
               dash(-),  or  a  'on/off' flag.  If no parameters are given, or
               the 'on' flag is used, the system defaults will  be  used.   If
               the  dash  or 'off' are given, the bell will be turned off.  If
               only one numerical parameter is given, the bell volume will  be
               set  to  that value, as a percentage of its maximum.  Likewise,
               the second numerical parameter specifies  the  bell  pitch,  in
               hertz, and the third numerical parameter specifies the duration
               in milliseconds.  Note that not all hardware can vary the  bell
               characteristics.   The X server will set the characteristics of
               the bell as closely as it can to the user's specifications.

Transfer Line IV: Look at this link. It describes usual vacuum (a few mTorr) for the 15/20K coolant lines from the C-can to the target.

GUI_Cleanup: We had a weird error. Stipcharts would not come up after a reboot. There were white alarms on everything that were latched on but which were neither active nor clearable. But everthing was being read out just fine. Solution was to kill -9 all processes listed by the "ps -a" command.

Saw made a script called "gui_cleanup" that should kill all medm, StripTool and alh processes running under the g0targ account.  It should be enough to type "gui_cleanup".  The script is in /group/gzero/g0targ/bin which is in the path.

Automatic Paging :
April 25, 2003

Steve has successfully set up the alarm paging system.
To turn it on, as cvxwrks@jeffylab goto /home/cvxwrks/PAGER and type
./enable_target_paging. Likewise, to disable it type
./disable_target_paging. These little scripts are:

more ./enable_target_paging
     #!/bin/sh
     cd ~/PAGER/oncall_pager
     cp targetdaq.alhConfig pager.alhConfig
     killall alh.new

So the enabler copies the file you edit (targetdaq.alhConfig) into the
file that is looked at by the alarm handler (pager.alhConfig). Make
any changes in alarms, etc. to targetdaq.alhConfig (not to
pager.alhConfig). Or set up new scripts with different alarm limits
(for parking, for example) and cp them into pager.alhConfig.

This file consists of entries like this:
CHANNEL TARGET hcITC502_3_Sensor_1_R
$ALARMCOUNTFILTER 999999999 10
$STATCOMMAND HIHI pageOnCall.pl -target -grp "TARGET" -sig "hcITC502_3_Sensor_1_R" -m "hcITC502_
3_Sensor_1_R HIHI"
$STATCOMMAND LOLO pageOnCall.pl -target -grp "TARGET" -sig "hcITC502_3_Sensor_1_R" -m "hcITC502_
3_Sensor_1_R LOLO"

The 10 on the second line says the alarm condition has to have been on
for 10 seconds in order to generate a page. The actual alarm limit
value is that already in the IOC, defined by the loaded
alarmlimits.bash file in
/u/group/poltar/ctarg/Screens/ALH/alarmlimits.bash.

The kill command kills the pager alarm handler. It is restarted
automatically by cron. To see that it's in the cron table do a
crontab -l.

Phone numbers for paging are defined in /home/cvxwrks/PAGER/oncall_pager/pageOnCall.pl

Flatline Alarm:
Steve W. set up a nice alarm that goes off if the temps have flatlined for 10 successive reads. It runs on cdaql5 under cvxwrks in the file checkbeamAndHVAndTarg.bash in ~/users/willy. See hclog 126947.

In spring 2008 we modified this and put it in a cron job. See hclog  155145.

Beamcurrent and Raster Alarms:
There is now a GUI which assigns the maximum beam current for each target we have. The same GUI also allows you to specify whether the raster has to be on or not for each target. This GUI can be found at /group/poltar/ctarg/Screens/BDS_maxcurr.adl and you open it from medm. You can type in the desired beam current maximum for each target. Enter a raster value of 1000 for those targets you want the raster to be on for. And a zero for those targets you don't require a raster for. This GUI is only used to generate alarms through our EPICS alarm handler in the counting house. It doesn't make an FSD or anything like that, just a beep in the counting house. The alarms themselves in the alarm handler are hcMaxBcmCurrentAlarm and hcRasterMaskAlarm. These are logical values, they are 0 when the respective test is OK, and 1 when the test has failed (ie when the beam current is higher than specified in the BDS_maxcurr.adl table for the target currently in the beam).

The values used in BDS_maxcurr.adl during spring 2008 are shown here.

To reload all these limits along with all the other alarm limits, click on the "alarm handler" button in the main GUI, then choose the option "Reload Alarm Limits". This will load all the values in the file Gep07.alarms in the /Screens directory.

Turbopump:
The normal vacuum in the scattering chamber is in the 10^-7 Torr range when the target is cold and cryopumping. The gate valve interlock which closes the gate valves on both sides of the SC is set to 5 x 10^-5 Torr- we obviously have to stay well below this pressure. We don't start a cooldown of the target unless we are well into the 10^-6 range (with the target warm).

If you suspect the scattering chamber turbopump has tripped off, then you have to go into the hall to check. The turbo controller for the turbo on the scattering chamber is located in a blue rack, just behind the target gas panel, behind the right side of the gas panel. You can't miss it, it's about face height, a black box with a bunch of white stripes running diagonally along the front panel. Normally, the green power and normal lights are on.There should only be one or two of the lowest of the diagonal leds lit, indicating the load is light. If the "failure" light is lit you have to restart it. Press stop, then start. If that doesn't work, power cycle it and then turn it on and press start. The load will run up to the top of the diagonal stripes, then back down over a couple minutes as the turbopump spins up. The manual is here (second half is in English).

It can be that the gate valve between the turbo and the scattering chamber has closed. That valve control is on the 1st floor of the blue platform (right above your head if you are standing in front of the turbocontroller) about halfway along the long side closest to the HMS. Just to the left of the 4He panel. The turbo's backing pump is located right there as well.

There are two cold cathode gauges right next to the turbocontroller. One reads the vacuum in the girder just upstream of the target. The other reads the vacuum in the scattering chamber itself.

Power cycling the Fans:
Before you power cycle the fans, you should try to restart the fan from the GUI. It's finnicky. Try to clear, reset, off and on. Try it several times. Wait a few seconds between operations. Try a few permutations on the buttons. Try starting it at 30 Hz (12%) instead of 60 Hz (25%). Try rebooting the IOC and then retrying to start from the GUI.

If none of that works, then you will have to power cycle the fans. First, note the status lights in ALL 3 fan controller GUIs. Having that info in hclog will help us diagnose the proble later on and should be useful in devising methods to improve reliability of the fan controllers. Then ask MCC for an ARM, escorted access. Do not wait for a sweep, you need to get the fans back on asap.

Go into Hall C with the ARM. As soon as you're in the hall, turn right and walk to the row of blue racks just upstream of the green shielding wall (where the SOS power supplies are). At the nearest end of those blue racks are some cotton coats that look like they just were delivered by the dry cleaners. Put one on, this is your PPE for this activity. There should be safety glasses in the vest pocket. Put those on too. Then walk to almost the end of the racks. In the blue rack that contains the 3 fan controllers and the three tachs (waist height) you will find the circuit breaker for the fans. It is mounted on the side of that rack and it's best to access it from behind the rack. It's on the right side of the rack, about chest high, and is the size of a couple of shoe boxes. It's grey with a big handle on it. Throw the switch on this 480 V circuit breaker off. Wait 30 seconds, Then turn it back on. Then try to restart the fan from the GUI.