Find attached a note "EPICS Tasks and Questions for Hall B." Chip
asked me to write down our needs and here they are as I see them. The
idea is that this will help us plan areas where ACC can give us a
hand.
This is literally a first draft and there are some glaring
omissions. (I gotta get a real channel count for the HV.) Let me know
about the things I have left out or misstated.
--
Mark M. Ito, Thomas Jefferson National Accelerator Facility
12000 Jefferson Ave., Mail Stop 12H, Newport News, VA 23606
Email: marki@cebaf.gov, Office: (757)269-7175, Pager: (757)680-7175
^ ^
Note: new area code and prefix | |
------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------------------------------------------------------
EPICS Tasks and Questions for Hall B
Mark Ito
September 12, 1996
I. Introduction
This note presents a quasi-prioritized summary of EPICS tasks for Hall
B. They are presented in three sections (1) items that must be
complete for the December '96 run, (2) items for the February '97 run,
and (3) longer-term projects.
There are four major areas that we are concentrating on for EPICS
control in the intermediate term: LeCroy System 1450 high voltage (HV)
control, beamline control and monitoring, the drift chamber (DC) gas
system, and monitoring for the cyrogenic target. Also, there are
tasks that have to do with the overall framework for EPICS development
and operations; both the hardware and software infrastructure. I will
refer to these as "EPICS system" tasks.
Each of these areas have distinct features. The cryogenic target
monitoring will not be needed until February, is being developed by
our collaborators at Saclay and so far they have only requested an
RS232 port from us. I am taking this as a good sign, and will not
mention this effort again in this note. The DC gas system is rather
well in hand at this point and so will not receive much attention
here. The beamline projects involve very few electronic channels and
can use standard tools and records and therefore appear to have
straightforward solutions. The HV control software is complicated, and
none of the authors are present on-site. There appear to be myriad
problems, and often the seriousness/scope of a given problem is not
understood by on-site Hall B personnel (e. g., me).
In addition I have listed a set of questions/problems. The order in
which they have to be addressed, indeed whether they need to be
addressed at all, will in many cases depend on decisions that have yet
to be made about hardware or software approaches. The idea here is to
put them all on the table.
At present the Hall B personnel we have to devote to EPICS development
consists of two people: Sebastien Fabbro, a University of South
Carolina student working towards his master's degree, and the author,
a Jefferson Lab staff physicist. Sebastien has primary responsibility
for the beamline control and has worked extensively on the HV
control. He will be leaving the lab sometime in November (of 96!). I
have worked with the HV and have been worrying about EPICS system
issues. We have also been working closely with Kathy McCormick, an
ODU PhD student, who has done much of the development of the HV
control system. Her time is, appropriately, more and more being
devoted to Hall-A-specific projects.
II. Requirements for the December run
A. HV
The broad requirements for the HV control system are sketched in
Ref. 1. Although this document addresses requirements for the wire
chamber high voltage cards (LeCroy 1469's) the requirements for
photomultiplier (PMT) cards are not significantly different at this
level of detail. These are functional requirements and they all must
be in place to operate the detector at all.
The low-level software we are using was written by Thierry Auger and
Marc Swynghedauw under the supervision of Dave Barker. The main
documents for their work are their master's theses, Refs. 2 and 3.
Prominent features of the system are a new HV EPICS record with
multiple VAL fields to accommodate the multiple constants associated
with each HV channel, and a set of asynchronous tasks that form an
interface between record support and the arcnet driver software.
The main problem areas that need to be solved for December are:
o Download of hardware constants
Presently, if one tries download all of the system constants by
repeated calls to channel access, the database will not update some
key values, most notably the measured voltage of a channel. Also error
messages appear on the IOC console. This probably has to do with the
fact that the software does not wait for a response from a mainframe
(the mainframe responds to every command) before going on to issuing
the next command. Some element of data is getting lost or the desired
logical execution thread is getting broken in a flood of channel
access requests.
o Concerted ramping of DC HV
As described in Ref. 1, sense, field and guard wires in a chamber need
to be ramped in concert to keep the E field configuration in the
chamber within well-understood and therefore safe (for the chamber)
regions. Neither the algorithm nor the EPICS control paradigm for
doing this has been decided on. It's not a precision-type problem, but
will take some careful thought in consultation with the chamber
experts.
o Integration with the Alarm Handler
The record support generates alarm conditions. No work has been done
to transmit these alarms to the alh in a systematic way.
o User interface
We are in the process of developing a C program that generates medm
screens using Jeff Karn's package. This appears to be on track and
should be ready for December. As a backup, we have a functional
FORTRAN program that invokes channel access and does all I/O with
ascii characters to/from a terminal session. Both approaches read the
channel-by-channel initialization information from an ascii file, and
can also control channels individually.
B. Beamline
The beamline tasks for December are listed in the following table in
decreasing priority (low number => high priority). Low-level software
in the DONE column means that no new record, device,... support needs
to be developed.
DEVICE PRIORITY DONE TO DO COMMENTS
----------------------------------------------------------------------------
Mini-torus magnet 1 Software spec. SNL program SNL in progress
Algorithm design GUI Dynapower DC power
Calibration supply
----------------------------------------------------------------------------
Tagger magnet 2 Software spec. Algorithm design Inverpower DC power
SNL program supply
GUI Similar to
Calibration mini-torus
----------------------------------------------------------------------------
Solid Target 3 Low-level software GUI OMS Steppermotors
Calibration
----------------------------------------------------------------------------
Radiator 4 Low-level software GUI OMS Steppermotor
(long stroke harp) Database design Linear encoder
Calibration
----------------------------------------------------------------------------
Primary collimator 5 GUI Hardware box OMS steppermotors
Low-level software Calibration VME-44 E
----------------------------------------------------------------------------
Secondary collimator 6 Low-level software GUI OMS steppermotors
Hardware box VME-44 E
Calibration Similar to primary
collimator
----------------------------------------------------------------------------
C. DC gas control
o Integration with alarm handler
o Backup and restore scheme
D. EPICS system
o Stable development environment
The directory structure that we inherited has a single copy of base
and extensions as well as a low-level application tree and a high
level application tree. The application trees work just fine, though
the structure is a bit complicated (e. g. 4 levels of links to get to
default.dctsdr). The flaw in this scheme (and this is well known) is
that changes made in the base areas (e. g. device support, record
support, ascii files, etc.) by an individual developer essentially
break all the databases built by all other developers if the change
involves default.dctsdr or could introduce bugs in the objects used by
all developers, if for example, a bad initialization routine is
created. No backup version exist and so there is no quick retreat
route. CVS control doesn't help here because (a) checking out the
entire EPICS base or even pieces of it is too unwieldy and (b) we did
not receive enough training to even allow us to realize (a) until
rather late in the game.
I have developed an alternate structure for our local contributions
and modifications to base that allows use of a vanilla version of an
EPICS release. Appendix A is a "getting started guide" for this
structure. It avoids the problems described above by using CVS to
manage a much smaller, set of source and data files and by allowing
each developer to have a private copy of default.dctsdr and of the
base objects. As a side benefit it facilitates (NOT accomplishes)
upgrades to new versions of base since the local code is stored
completely independently of the official EPICS release files. Much of
this song and dance will be obviated by R3.13, but we are not going to
run with R3.13 in December. My present plan is to use this
structure. I would be happy to abandon it if a better way to manage
the system can be provided.
At a minimum, we need a version of base and extensions that we can
build from scratch on our own, at our convenience. This is an
invaluable learning and debugging tool for us: we can make scratch
copies of the code to write out intermediate results and find bugs in
our own software that only manifest themselves in the standard EPICS
routines. We need to be able to do this without disturbing the main
development tree for others. (Note that the directory structure I have
developed also addresses this issue.)
III. Requirements for the February run
A. HV
o GUI
The ascii-based interface described above is pretty clunky. We really
should have a GUI by February.
o New requirements
The December run will probably generate a raft of new items for
improvement of HV control. It would be wise allocate manpower for
these tasks even though they cannot be identified at present.
B. Beamline
These systems are need for the February run.
DEVICE PRIORITY DONE TO DO COMMENTS
----------------------------------------------------------------------------
Pair Spectrometer 1 Low-level software GUI OMS Steppermotors
Converter Calibration
----------------------------------------------------------------------------
Beam profile system 2 Low-level software GUI Joerger VME scaler
Calibration
----------------------------------------------------------------------------
C. DC gas
No new items.
D. EPICS system
o Migration to Sun Ultra Spark
Hall B has a data acquisition system based on Sun's. The center piece
of the data acquisition computing is a symmetric multiprocessor, but
in addition we have purchased four Ultra Sparks. The plan was to use
one of these Ultra's for slow controls, principally EPICS. I have been
trying to port EPICS to the Sun, but so far have not been
successful. In general, the Sun provides a nicer development
environment than the HP (faster CPU, exclusive use of resources, lots
of disk space, good network connection to the rest of the experiment,
modern debugging tools, nice color monitor, etc.) and my preference
is to not only run the OPI software there, but also to use it for the
IOC based stuff. We should do this by February.
IV. Longer-range goals
A. HV
o A new approach?
The present HV software is not very well understood by Hall B
personnel. As such, we will have difficulty maintaining it. One
solution is to go to an entirely new, simpler scheme. If the approach
chosen is the one Bonnie Madre is using for LeCroy 1454 control, then
our Accelerator Division colleagues could maintain the code for us.
Issues involved in this decision are:
- Memory limits
- Complications due to large channel count
- Need for arcnet driver code
B. Beamline
No new items
C. DC gas
No new items
D. EPICS system
o R3.13
It's supposed to be the solution to a lot of our problems. We should
at least try to prove that it is not, and use it if it is. This switch
might be easier for us that for ACC because of our much smaller number
of applications.
V. Questions/Problems
This section is written in laundry-list mode.
A. HV
Errors in initialization. These messages appear on the IOC console for
a *successful* IOC initialization:
...
Missing record support entry table : _palRSET
0x3a183c (tShell): iocInit: Record Support Failed during Initialization
...
Missing device support entry table : devAiXy560
0x25727c (Tx): arcnet node id read from the board itself = 5
...
interrupt: int_handler() : Reconfiguration interrupt!!
...
Missing device support entry table : devOms58
Missing device support entry table : devOms58
0x3a183c (tShell): iocInit: Device Support Failed during Initialization
...
We have to get rid of these apparently extraneous error messages.
Why does initialization take so long? Is it really necessary to verify
that the files conform to the hardware? Is it even desirable?
The code in the original directories, epics_physics/hallb/..., has not
all been committed. We don't have a complete CVS record of the Hall B
base. Some developers have been using CVS, some not.
Should we change the software architecture to conform to the
call-and-response mode that was intended by the LeCroy engineers?
Should we implement a downloading scheme that interleaves commands to
different mainframes? This a factor of n improvement in speed, where n
is the number of mainframes on the network.
How does the system behave with multiple mainframes. Got two to work
once. Can we do it again?
Card-by-card rather than channel-by-channel initialization routine
needed. Thierry gave us a skeleton program to start with here. Speeds
up a download by an order of magnitude.
There is a rumor that the HV mainframe status code does not update
properly. Is it true? Can it be fixed?
Right now we can't build extensions from scratch with a working
db2database on either HP or Sun. On Sun it gives a segmentation fault,
on the HP's the database spews errors on initialization and when the
periodic task queries the mainframe
The LeCroy mainframes have a DUMP command. Is this be easier to use
than BURT?
The mainframes have a built-in memory with battery backup. The values
have been seen to survive unplugging and transport across the lab
site. Can we exploit this feature to avoid time-consuming
initialization procedure? For example, DUMP the values and only
download those that need to be changed.
We need some low-level tools to troubleshoot the hardware and
communication problems. Specifically a program to "single-step" the
arcnet, and one to "single-step" the mainframe interactively.
We need a low-level routine to flush the message pipe in the mainframe
for reset-everything-out-of-desperation situations.
We need an understanding of the following arcnet details:
- How are interrupts used/handled?
- Is the double buffering in the interface cards being exploited?
- Are we generating reconfiguration interrupts? If so are they
necessary?
All of the alarm fields in the HV record come to life as invalid. Is
this good? Bad?
We have never been able to successfully run the 1450 software on an
MV162 without hardware floating point support. What is our problem?
There will be an effort to use a LeCroy 1440 in the radiative phi
decay experiment which sits in the alcove in Hall B. Dave Barker wrote
a tcl-based gui which controls the same low-level routines that are
resident in our library. Where is this thing? Does it work?
B. Beamline?
C. DC gas?
D. EPICS system
Our standard version of the base is linked such that channel access
occasionally complains about a missing shared library. No one dares
try to rebuild the system, so we live with these error messages. We
don't know if this is serious or not.
We are using VxWorks 5.2 on the HP. Only 5.3 is presently available on
the Sun. So as things stand, migrating to the Sun involves a VxWorks
version change. The directory structure of these two versions is
different. Is that a problem for the EPICS build?
Are we using the correct board configurations for our IOC's. How do we
find out?
There are some packages that are referred to in extensions that are
not presently installed on the sun, for example, interviews, imake,
and the dp library. Are these easy to get? Are they necessary?
(Probably not in many cases.)
Should we buy a capfast license for the Sun?
Can we provide a package so that non-EPICS applications can generate
alarm events and therefore take advantage of the features of the alh?
Do we need a system to remotely reboot the IOC's independent of the
state of the IOC or the network? A mechanical system? What does ACC do
about this?
What should our attitude be towards the error logging facility? Do we
need it?
A quote from Jim Kowalkowski: "Why are you using db2database? Just
use dbLoadRecords() and load the ascii ".db" file directly into the
IOC. Be sure to load default.dctsdr first using dbLoad()." I know
this is not done at Jeff Lab. Can we start?
Where is the tar file for R3.12.0Beta8_1? This is the version Hall B
is using. Why are we using this version?
VI. Conclusions
There are a lot of things to do. A lot of my time has been devoted to
learning enough about EPICS to understand these issues. After
discussions with Chip Watson, who encouraged me to write this, I am
quite confident that we can muster the internal (Hall B) and external
(Accelerator Division) resources to achieve these goals.
Appendix A
My note on getting started with baseB
------------------------------------------------------------
------------------------------------------------------------
Slow Ones,
I have created a set of new directories in the Physics EPICS tree on
the cebafb/e/h cluster. These contain all of the *.c and *.h files
that we (Hall B) have developed. They were all copied from the
$EPICS/base/src/dev, /drv, /rec and $EPICS/base/include
directories. The original files are undisturbed, in their original
locations. The new directories just contain copies at the
moment. They are maintained using cvs.
I would like people to try using the new structure. It is described in
detail below. As partially mentioned above, all of the things in the
old tree will work as they did before (makefiles, applications, IOC
load script, etc.). The new stuff is presently sitting off to the
side. If we decide to go as a group to the new scheme, or something
else similar, then the relevant files can be deleted from the old tree
and be used exclusively from the new tree.
The motivation for making this new directory tree is to allow
concurrent development of our software by multiple individuals. Up to
this point we have been using the current directory structure, not as
intended, but as one large repository, with all of us changing or
rebuilding the same copies of the core software potentially and
actually affecting the behavior of others' programs. In addition,
speaking for myself, I have been very reluctant to jump into the
production source area and play with the code, trying new things,
since everybody else would then have to deal with my bastardized
versions of the code while I was playing and count on me to restore
all files back to their original condition after I was done. I think
most of us would agree that this is an unacceptable way to work.
The original intention (as Johnny described it to me) was for our
local code to reside in the relevant individualized application
directory. The problem with this is that many of our applications are
sharing the same set of basic record and device support, and it is
desirable that there be one authoritative copy of this code, rather
that separate ones sitting in a bunch of different application
directories.
So please try it out. Let me know what you think.
Mark M. Ito, Thomas Jefferson National Accelerator Facility
12000 Jefferson Ave., Mail Stop 12H, Newport News, VA 23606
Email: marki@cebaf.gov, Office: (804)249-7175, Pager: (804)680-7175
-------------------------------------------------------------------
To get a private version of the Hall B EPICS core software:
> setup epics_physics/hallb # Our standard setup.
> cd local_dir # where local_dir is an arbitrary directory
# of your choosing. (You'll need ~1.5 MB)
> cvs checkout baseB # Get the current version of the software
# from cvs. The directory baseB will appear
# in your local_dir.
> cd baseB # Go to the new directory.
> install_baseB # Runs a cshell script that will install
# versions of the Hall B device support and
# record support. Messages that contain
# "Error 0" are OK.
The procedure presently creates the following four files:
local_dir/baseB/bin/mv162/devSupB
local_dir/baseB/bin/mv162/recSupB
local_dir/baseB/bin/mv167/devSupB
local_dir/baseB/bin/mv167/recSupB
I'm still working on the driver support stuff.
To use these files, in your IOC load script instead of the standard:
ld < iocCore
ld < drvSup
ld < recSup
ld < devSup
ld < seq
do the following (for example, with a 167 target):
ld < iocCore
ld < drvSup
ld < recSup
ld < local_dir/baseB/bin/mv167/recSupB
ld < devSup
ld < local_dir/baseB/bin/mv167/devSupB
ld < seq
Presently this will essentially overwrite the versions of the routines
that are already present in the standard record and device
support. Since right now the routines are the same, doing this should
make no difference to your application.
If for example you would like to modify the routine devAiLecroy.c in
our device support:
> cd local_dir/baseB/src/dev # go to the source directory
> emacs devAiLecroy.c # make your changes with the editor
> make TARGET=mv167 # rebuild the library
A new version of local_dir/baseB/bin/mv167/devSupB will be created
with your changes reflected. If you like your changes and would like
to incorporate them into the Hall B cvs repository then:
> cd local_dir/baseB/src/dev # same directory as above
> cvs commit -m "I changed it to make it better" devAiLecroy.c
The -m option is a required comment. If you want to make a longer
comment, leave out the -m and cvs will fire up an editor to let you
input your comment. Now all others that checkout baseB will get your
new version of the routine. And we know when the version changed. And
we know who did it. And we know why. And we can get back to the old
version if we want to.
Note the following:
o Until the "cvs commit" step in the above procedures you are free to
hack around to your heart's content without affecting anyone else.
o The directory structure under baseB is exactly the same as you find
under $EPICS/base.
o If you want to use the Hall B software without modification, then use
the files in the tree $EPICS/baseB (i. e. local_dir = $EPICS). These
should reflect a built version of the current cvs repository. You
should not modify files here. Rather make your own copy and play with
that.
o There is more to CVS than is mentioned here obviously. See the
document in:
$EPICS_PHYSICS/DOC/cvs.ps
Of particular interest is the "cvs release" command that will let you
see which files of your local copy you have changed, or whether they
have become "obsolete" due to someone else making a change in the
repository. "cvs diff" will let you see the differences between your
local copy and the one in the repository. Also "cvs add" will allow
you to add a new file or directory to the repository.
o My final goal is for us never to go into $EPICS/base or
$EPICS/extensions to change things and just let the Accelerator
Division maintain those directories, give us updates, new versions,
etc. If we have local versions of things that are in there, we can
maintain them separately on our own tree.
o I need feedback on whether I have identified the Hall B routines and
header files correctly, or if I have left any out. Your comments here
would be appreciated.
------------------------------------------------------------
------------------------------------------------------------
References
1. Mark Ito. "Drift Chamber High Voltage Control Specifications."
http://www.science.urich.edu/~vineyard/research/controls/DCHV_spec.ps
2. Thierry Auger. "Elements of EPICS and the High Voltage Control
System for the CLAS Detector."
http://www.science.urich.edu/~vineyard/research/controls/1458/LeCroy1458_auger.ps
3. Marc Swynghedauw. "EPICS Development: High Voltage Slow Controls for
the CLAS Detector at CEBAF." http://www.science.urich.edu/~vineyard/research/controls/1458/LeCroy1458_swynghed.ps