FPP Debugging Guide: How to Solve Likely Hardware Problems One Might Encounter During a Run

• Overview and Safety Issues
• Typical Problems - Not Readout
  
• Readout Problems
• References
• FPP People

Overview

The Hall A Focal Plane Polarimeter was built largely by Rutgers and William & Mary, and installed in Hall A in 1996. The system consists of four wire chambers that track particles going into and out of a remotely configurable carbon analyzer - for high energies it has been reconfigured into a dual analyzer system. The signals are discriminated at the chambers; reduced ECL logical signals (to prevent oscillations) are sent to remote level shifters, where they are converted back to normal ECL levels to be sent to TDCs. To reduce the number of electronics channels, 8 channels are multiplexed into a single electronics channel at the chamber readout boards, by using different width signals and reading out both leading and trailing edges of the pulses at the TDCs.

The absolute best way to solve hardware problems is to call in an expert, but perhaps this is not possible. This document then will give you some clues
that might allow you to identify the nature of the problem and even fix it. In general, only experts should work at / on the chambers, but others
can check parts of the systems away from the chambers.

For debugging purposes, one needs to know what the inputs and outputs of the system are. These include the following:

• An Ar-CO2 gas system: controls in the gas shed, and controls and readback to the left as you enter the detector stack lower level
  
• Discriminator threshold voltages: four supplies located in the upper level of the detector stack, on the lower shelf at the back.
  
• High current +/- 5 V supplies: 9 big black boxes, 6 on the back of the stack, 2 in the upper level of the detector stack, on the upper shelf at the back, and one at the bottom of the level shifter crate.
  
• High voltage, low current supplies: The LeCroy box in the top position of the rearmost electronics rack on the upper level of the detector stack - the FPP uses +HV modules in slots 7 and 8.
• Carbon analyzer controls and read backs: the analyzer is in the middle of the stack, with control electonics and power supplies mounted on the large angle side of the analyzer frame.
  
• The readout electronics system for the chamber signals. This includes readout boards at the chambers, ~42 level shifter boards in the rack to the right as you enter the detector stack lower level, and 7 LeCroy multihit TDCs in the upper FASTBUS crate, located in the rack to the left as you enter the detector stack lower level.

Work at the chambers should be done with extreme care. Access to the chambers is limited. Some parts of the chambers cannot be reached at all when the chambers are installed in the stack, and most parts that can be reached can only be reached when using the ladders in the detector hut. Working from ladders is inherently dangerous. The chambers are delicate. In particular, the straws on the front chambers are exposed and can easily be damaged. The straws on the rear chambers are protected from minor accidents by ~14-mil thick sheets of carbon fiber.

For debugging purposes, it is often necessary to leave LV power on. The LV supplies are only 5V, but up to 50 A. If shorted to ground by a conductor, the large current can heat and vaporize the conductor. The discriminator supplies are low current, ~1 A, and not so dangerous. Problems are easily
avoided by not wearing metal objects, by carefully carrying only a limited number of tools, and BY TURNING OFF THE LV SUPPLIES IF POSSIBLE.
If you need to open a LV supply box, it is absolutely required to turn the box off, and unplug the power cord.

For debugging purposes, it is often necessary to leave HV on. Standard operating HV for the chambers is 1775 V with Ar-CO2 gas. Given the current limit of the supplies, about 100 µA, and the capacitance of the chambers, the result of accidently shorting the HV to ground with a finger is a sensation similar to the feeling of a bee sting. This is mainly dangerous as one is typically working on the ladders with delicate equipment, and the reaction to the surprising "bee sting" can easily lead to hurting oneself and damaging equipment. If it is desired to check that HV is actually present at the boards, reduce the HV to ~100 V and use a DVM carefully. It is generally a good idea if you plan to work anywhere near the front of the cards at the chamber to reduce the HV levels to ~100 V, if not turning them off entirely.

Typical Problems - Not Readout

• If the Ar-CO2 gas supply goes off, it can take ~1 day to purge the chambers thoroughly. If the system goes off, there should be a gas system alarm. Questions about or problems with the gas system should be discussed with Jack Segal. The gas system is controlled by a mixer in the gas shed. There are five additional flow controls in the detector hut, each of which is marked to indicate standard setting. The gas flow can be read out on the Hall A General Tools screen; the meters can be viewed directly in the lower level of the detector stack. The meters are sensitive; if tapped some of them go into a mode in which they read a small negative number. Try gently moving the misbehaving one on the back of the panel.
• Occasionally there have been problems with the gas distribution systems at the chambers, particularly the small (~ 2 mm diameter) gas tubes that go into the end of the straws. Fixing a tube that has slipped out is done by sticking it gently several mm back into the gas cap. This should be done carefully as one might pull the gas cap off, or disconnect wires from boards to straws. Given the large number of tubes, it is essentially impossible to notice that a tube is out, unless one knows from data where the problem starts, to within a few straws. Recall it might be at either end of the straw.
  
• The wire chambers require a threshold voltage to set the discriminator level. The threshold is set and monitored on the FPP threshold GUI. If needed, it can also be checked directly at the power supplies on the upper level fo the stack, or on the buses on the chambers. Over the years there have been few problems with the threshold voltages.
  
• The wire chamber readout cards, and the level shifter cards, require high current +/- 5 V power supplies. There are 9 big black boxes in the stack, 6 at the lower back end of the stack, 2 on the upper platform, and 1 at the bottom of the level shifter crate. All are turned on by a button on the front panel, have fuses on the back panel, and are monitored by the FPP LV GUI. The most common problem with the LV supplies is with the connector at the back of the power boxes. A common problem, especially when resetting up the FPP, is that the connection at the back of the box is not good, or the cable get pushed slightly, and no longer makes contact with the leads in the box. On the GUI you will see one or two of the monitored LV from a box now reads a few tenths of a V. If you can figure out what cable was moved, you can try to adjust it at the back of the box. This is a difficult procedure - it is hard to adjust the cables on any of the 6 boxes at the back of the stack without bumping into the others. Further, it is possible that the ``tongues'' in the connectors have slipped off, and one will need to try to pull / push the tongue back into position - if this needs to be done in the box, disconnect the power cords first! Finally, for debugging purposes, note that the LV power is sent on a pair of wires ``floating'', ground is determined at the chambers. Note that the chambers have a couple of -5V supplies for each +5 V supply, so depending on which supplies are not powering a chamber, you can lose from a fraction of it to all of it.
  
• The main problem over time with HV supplies has been with individual straws having problems, sparking when HV is applied, tripping the HV channel. If this happens, it is necessary to figure out which wire is causing the problem. Each HV master controls several slave channels, and each slave channel powers several stacks of boards. An expert is needed to alternately disconnect stacks of boards at the chambers, then to disconnect individual wires within the stacks to identify the problem wire. During the experiment, the problem wire is best left disconnected at the HV end. The process is sped up if only the affected slave channel is turned on and off. Since it is fastest to disconnect groups of wires to identify the problem (16 = 2⁴), one needs to carefully check afterward that the wires are reconnected in the correct order, and not crossed - this is often difficult as the access to and view of the chamber is limited.
• When HV trips, often the slave channels still are at several hundred volts, and they can be much different from each other. Usually, the master needs to be slightly below 100 V to turn a slave channel back on, and then the slave channel at several hundred Volts trips again when turned back on. The quickest procedure is often the following. If there are any nontripped slave channels, use the master to lower the voltage to around 80 - 90 V. Turn these slaves off. Turn the other slaves on one at a time, lowering their HV to 80 - 90 V. When all are below 100 V, you should be able to turn all back on, and then (relatively quickly) turn the voltage back up to operating voltage. In normal operations, there is no reason for the chamber to trip, so you should restore voltage in several steps. The shift crew can attempt to ascertain which slave channel is the problem, while awaiting an expert to debug the problem in the slave channel section.
  
• The carbon door controls have been reliable over the years. If there is any problem, Mark Jones developed the control GUI, and Ron Gilman and Ron Ransome are probably the only other people around who remember building the system. The most obvious problem is that you have not turned on the 12 V supply, so the read back is actually off and shows all doors closed. Read and follow the directions on the GUI. The doors take about 3 minutes to open or close. If you are convinced they are not working, go to the detector stack, visually determine their position, and make sure the power supplies are plugged in and turned on.
  

Readout problems

Most typically, the HV, LV, and gas systems all appear to be fine, but some number of wires are not seen in the software analysis. The number of wires not seen is a clue to where the problem is. It is possible for an elecronics / readout board to go bad, but this has been rare. It is most likely that either a cable or connector is bad, or at least poorly plugged in. We have been fortunate in that no boards have gone bad since several were replaced during the initial installation / checkout in late 1996. Furthermore, I make the assumption that noone has introduced software problems. That is, it is clear that there are no signals on the TDC channel corresponding to the missing wire(s). I also assume that you have already checked that the LV, threshhold LV, HV, and gas system GUIs and all appears to be okay, so it is most likely a readout problem.

• If a single wire is missing: About 10 wires or so out of the ~5100 in the system have been disconnected. The likeliest way for a new problem with a single wire to develop is if someone has worked in that area of the chamber, and accidently left a wire disconnected from the HV or readout boards. Missing single wires should generally be left for experts to fix. Occasionally we have seen that the demultiplexing spectrum shows 8 peaks, but shifted so that the first peak has a 15 ns width, rather than 25 ns. Usually there is a bad connection somewhere - either cable connectors are not well inserted or perhaps one needs to be recrimped; the signal shape gets degraded and the width changes in an odd, but repeatable way. While it is best to locate the bad connection and fix it, it is certainly possible for the short term to just change the software demultiplexing cuts.
  
• If a group of 8 or 16 wires is missing: 8 wires are multiplexed into a single electronics channel, while 16 wires are handled together on the HV and readout boards at the chamber. The most common problem is the telephone jack readout connector at the back of the readout board ineeds to be unplugged, recrimped, and plugged back in. (There is a white plastic crimp tool in the red common tool box on the floor of the Hall, or ask Jack Segal.) Also bring a DVM and check the access points on the board to see if there are +/-5 V and threshold voltages - occasionally a fuse is blown and needs to be replaced. Finally, turn HV down to 100 V and check that HV is present at the straws. If everything seems fine at the board, and someone has worked on the cables, they might have plugged the 16 channel / 34 pin cables in backwards - this connects ground to channel 1 and channel 1 to ground, and generally makes signals on the cable  have funny widths. The signals coming from the chamber to the level shifter can be plugged into an ECL/NIM/ECL converter above the level shifters and looked at with a scope. (Note that the cables coming from the chambers to the level shifters have different conventions. The front chambers have to correct convention that the brown wire is #1, The rear chambers have the incorrect convention - also adopted by Gen - that the purple wire is #1. All cables from the level shifters to the TDCs have the same convention.)
  
• If a stack of boards - the same 16 wires on adjacent U or V or X wires of a chamber - are missing: on the front chambers, the LV power to each stack shares a common fuse, and perhaps the fuse needs to be replaced. (On the rear chambers, each board in the stack is separately fused.) The HV boards share a common HV supply, and perhaps the cable connection to the HV boards is bad. Also, some of the HV slaves supply only a single stack of boards, so perhaps the slave driving those boards is off.
• If 128 wires are missing, usually from 2 or 3 sections of each of the adjacent planes on a chamber, it is likely a problem with the 16 channel / 34 pin readout cable. Perhaps it was not reconnected in debugging or setting up the chamber, or it has been pulled on a little and come loose. You can check that there are signals to the level shifter board on the cable using the ECL/NIM/ECL converter and a scope. You can check that there is an output from the level shifter using the spare twisted pair cable that is usually left with the ECL/NIM/ECL converter. You can check that there are signals to the TDCs by carefully removing the cable at the TDC input and bringing it back across to the ECL/NIM/ECL converter.
• If 768 wires are missing, corresponding to all 96 channels of one TDC module - not all channels are used on the last TDC module - it is possible that something is wrong with the TDC.
• If ~1/3 -> 2/3 of a chamber goes out, it is likely that a single -5 V supply, or a box of two -5 V supplies, is gone. Check the LV readback. Also, check that the HV on a chamber has not tripped. Depending on whether one loses a single slave channel, or an entire master, you can lose signals from a small section of the chamber or the entire chamber.
  
• If an entire chamber is missing, particularly if both rear chambers are almost entirely missing: perhaps the protons are being ranged out in the analyzer, and you need a thinner analyzer. In this case there will still be several Hz of cosmic rays.
  
• If all the chambers are missing: the most likely single hardware problem to turn off all the chambers is that the level shifter power supply is turned off. The supply sits below the level shifters, and is monitored on the FPP LV GUI.
  

• If a single wire is hot: this probably indicates that the wire if failing. It has stretched, or the soldering of the wire into the pin is failing. It is loose enough that there are mini voltage fluctuations on the wire that are interpreted as signals. It is not possible to raise the threshold on a single wire. If it is too hot, in effect killing the other wires in the group of 8, it needs to be disconnected at the chamber, preferably at the HV end.
  
• If a group of 128 wires are hot: Most likely the cable has been plugged into the TDCs offset to the side, so it is only on a single pin. It is hard to do this at the level shifter, due to the connectors. Note that is it relatively easy to see if one cable is off to the side of the others at the TDCs, so it is easier to plug all 6 cables, with 96 channels and 768 wires, off to the side consistently wrong.
  

References

Please see R Gilman's JLab FPP home page for more information than is shown here. In particular the links to the descriptions of the hardware and the debugging page might be helpful.

Doing the debugging described above requires knowing where things are.

• General hardware locations are given in the Overview section of this document.
• To bring up the HV, LV, threshold, and carbon door GUIs: If you are not at the hacsbc2 terminal in the counting house, log into hacsbc2 as hacuser and type hlamain. From the hlamin menu, all GUIs are available under FPP; they are also available from the FPP option in the Hall A General Tools screen.
• Here are links to various documentation detailling the readout setup, what is on cables, where boards are on chambers, etc.
  
• TDC Map: use detmap.config
  Warning: the TDC slots given in all the chamber maps refer to the pre-LEDEX setup. The post GEn reinstallation shifted the slots by 2, as noted in detmap.config.
• Level Shifter Map
• Chamber 1 and 2 Cable Maps in gif format: 1, 2, 3, 4, 5 or the entire set as a single postscript
  
• Chamber 3 Cable Map and Chamber 4 Cable Map, ordered by readout cable
• Chamber 3 Cable Map and Chamber 4 Cable Map, ordered by wires within each plane
• Drawings indicating where readout modules are physically located on Chambers 1 and 2, 3, and 4
• Mark Jones has a list of dead / inefficient wires from the NDelta experiment in 2000, for chambers 1, 2, 3, and 4
• There is an old FPP debugging document, http://www.jlab.org/~gilman/fpp-debug.html. This document details a number of the problems we had in the installation period with the FPP, many of which have not occurred in the almost 10 years since.
  

FPP People

In case of problems with the FPP, one should contact the people mentioned above for specific details. Experienced people generally on-site include:

• Ron Gilman, x7011, gilman@jlab.org (built system)
  
• Mark Jones, jones@jlab.org (built system)
  
• Xiaodong Jiang, jiang@jlab.org (worked on system many times)
  
• Jack Segal, segal@jlab.org (did not build chambers, but knows about several aspects of the system)
  
• Ed Brash, brash@jlab.org (did the original software setup for the system, but usually has not worked with the hardware)