Thomas Jefferson National Accelerator Facility - A DEPARTMENT OF ENERGY FACILITY
GlueX Computing Effort Gets Nod from NSF
To help improve our understanding of the universe, more than 185,000 computer owners in 202 countries have volunteered their computers' spare time to the Einstein@home project. The project splits huge amounts of data from giant telescope arrays into small chunks that home computers can analyze. This allows researchers to look for wrinkles in space known as gravitational waves. Now a Jefferson Lab-based group is taking this idea, called "distributed computing," and applying it to meet the demands of particle physics.The Gluonic Excitations Experiment (GlueX), set to run in Jefferson Lab's Hall D after the 12 GeV Upgrade, is expected to generate huge amounts of data – about 2 Gigabytes per second. Scientists will need distributed computing to make sense of it all. But instead of ordinary computers – they’re planning to split the analysis among cluster-style supercomputers....... more
I grew up in Yuba City, Calif., which for various reasons, had been rated among the worst places to live in the country. It wasn’t so bad for me; one of the best things that happened to me there was meeting the girl who would later become my wife, Barbara..... more
Born in the former Yugoslavia, Marija Raskovic's work on her doctoral degree at Old Dominion University and her participation at Jefferson Lab were born of her desire to pursue cutting-edge research opportunities. By the time she graduated from high school, she had her sights set on becoming an architect...... more
There wasn't a lot of love for Jefferson Lab's e-mail system on Valentine's Day.The servers that handle the Lab's electronic missives were overburdened, severing computer users' digital link to the outside world and the office next door. But by noon the following day, a more stable and faster electronic message delivery was in service..... more
A grant from the National Science Foundation is helping researchers to gear up for GlueX data analysis, long before the experiment even begins....
To help improve our understanding of the universe, more than 185,000 computer owners in 202 countries have volunteered their computers' spare time to the Einstein@home project. The project splits huge amounts of data from giant telescope arrays into small chunks that home computers can analyze. This allows researchers to look for wrinkles in space known as gravitational waves. Now a Jefferson Lab-based group is taking this idea, called "distributed computing," and applying it to meet the demands of particle physics.
The Gluonic Excitations Experiment (GlueX), set to run in Jefferson Lab's Hall D after the 12 GeV Upgrade, is expected to generate huge amounts of data – about 2 Gigabytes per second. Scientists will need distributed computing to make sense of it all. But instead of ordinary computers – they’re planning to split the analysis among cluster-style supercomputers.
Several of the universities collaborating on the GlueX experiment have received a grant from the National Science Foundation to develop and test what would be the next generation of data analysis tools. Once complete, scientists would be able to distribute processing simultaneously over a cross-country network of cluster computers.
Indiana University and co-principal investigator on the grant."NSF is funding three-quarters of a million dollars to engage people in an interesting computing project that's going to push the edge of data analysis and help us do better science in the future," says Matthew Shepherd, an assistant professor at
"It's a proposal funded through the NSF program known as Physics at the Information Frontier. It funds projects that are at the intersection of science and computing. Our proposal is to help develop an algorithm for data analysis on data that will come as part of the 12 GeV Upgrade at JLab," he says.
Scientists in the GlueX collaboration will study "glue". Not Elmer's or Loctite – but the glue that binds the foundations of matter that make up our universe. This glue binds together the bits of matter that build the protons and neutrons in every atom.
It’s not possible to scoop a glue sample out of a proton and plop it down on a microscope for examination. But scientists think they can study this glue by producing a new type of particle called an exotic meson. This particle has the same building blocks as a proton: quarks. Welding the two quarks together in an exotic meson is a "flux tube" of glue – the kind the scientists want to study.
Unfortunately, signatures of these exotic mesons will be buried in a mountain of data from other particles that are also likely to be produced in the GlueX experiment.
"If you were to do this type of analysis on your personal computer, there's no way that you could do this in any reasonable amount of time," Shepherd explains. "An analysis the Indiana University group did about two years ago consumed 10-to-20,000 hours of computing time. And for GlueX, we're going to have data sets that are maybe 10 or 100 times that. This presents a challenge for analyzing the signatures of exotic mesons that we are tackling now, five years before the data arrives. With these NSF funds, we will make great advances in the speed and efficiency of the data analysis. "
The proposal funds three postdoctoral research associates to work with scientists at Indiana University, Carnegie Mellon, the University of Connecticut and Old Dominion University. These physicists are developing data analysis tools.
The tools are for open access amplitude analysis on a grid, a fancy term that means the tools will, simply put, allow researchers to use supercomputers connected via the Internet to analyze particle physics data in concert.
Ultimately, Shepherd would like to make it possible for any nuclear physicist or graduate student to use the tools developed for GlueX to analyze data, with minor tweaks, for almost any particle physics experiment.
"What we really want are off-the-shelf tools that anyone can take and use right away, where we've solved all the hard computing problems. You can think about physics and not about programming and interfacing multiple computers and networking and all that sort of stuff," he says.
By Kandice Carter
I grew up in Yuba City, Calif., which for various reasons, had been rated among the worst places to live in the country. It wasn’t so bad for me; one of the best things that happened to me there was meeting the girl who would later become my wife, Barbara.
My dad was a field scientist for the Department of Agriculture and my mom was always entertained by science, so I grew up with a sort of "Gee whiz!" attitude toward it. We had a small farm with cows and horses. Although I didn't exactly grow up doing morning chores, there is, indeed, a photo of my best man and me mucking out the horse stalls on the morning of my wedding.
I went to the University of Houston and ended up majoring in physics. Coming from small-town Yuba City, it took me about a year to adjust to the larger, more cosmopolitan Houston. I had the same experience of adjusting when I went from Houston to Carnegie Mellon for my graduate work, although I really liked the snow a lot. I initially found my grad school classes quite difficult; I'd been able to sail through my undergraduate classes without trying too hard. I'd been fortunate enough to get a position with the nuclear physics group at Carnegie Mellon, working at Brookhaven, during the summer before I started classes. I think of that as the period when I was "learning to learn physics." I’ve always found experimental work to be fun.
I did my thesis work at CERN, finished the analysis at Carnegie Mellon, and then came to Jefferson Lab for my postdoc work in parity violation. I like the combination of topics this field addresses, including topics in QCD. The dynamics of the strong force seems, to me, to be an under-appreciated topic in fundamental physics.
That also coincided with when I was doing my application for the DOE Outstanding Junior Investigator Award. Carsten was born a few weeks early and had a medical problem that meant he had to be transferred from Williamsburg to Norfolk General. Although he wasn't in any immediate danger, it was certainly stressful for us. I finished writing the grant proposal at the hospital and got it in just under the wire. I certainly learned a lesson about pushing deadlines with a baby on the way.
Receiving this award in my first year at UVa was a huge boost for me. Since there is no "umbrella" grant and we are all funded by our own grants, this really let me get on with my program much more quickly. I was able to recruit a student to start research right away and to get on with assembling a research lab at the university to support Jefferson Lab research. As a junior faculty member, this sort of award helps make a positive first impression among my colleagues, and it certainly feels good to receive such a competitive award.
Although my initial and continuing attraction to physics was born of that "Gee whiz!" factor, like all physicists who have foregone the big bucks of industry for an academic career, I believe in the value of the work. I think that subatomic physics isn't just an expression of humanity's natural curiosity and fascination with shiny things, it is important in an enduring sense. Fundamental science contributes to the greater progress of humanity, and I like to think that what I do at Jefferson Lab contributes to that.
As told to Judi Tull
As a high school student, she dreamed of becoming an architect. But by the time she graduated in 1992, her country was undergoing rapid political and economic changes.
"My parents couldn't afford to send me on to architecture school," she recalled, "so I sat down and thought 'What am I good at?'"
The answer turned out to be easy. She had always excelled in math and science, having taken coursework in high school that Americans would consider the equivalent of advanced placement classes. Once she reset her goal to science, she was accepted into the University of Belgrade, an instiution she chose because of its excellent reputation in physical chemistry, which she had decided to pursue.
After receiving her bachelor's degree, she was hired on as a teaching assistant while working toward her master's. The economic situation in Serbia had been difficult during this period and hadn't changed by the time she defended her master's thesis in plasma science. It was hard to find funding for experimental research in fundamental science. Fortunately, she met ODU Prof. Svetozar Popovic at a conference. They shared an interest in plasma science, as well as their Serbian heritage. He asked if she would be interested in coming to the United States for her Ph.D.
During her first year at ODU, Raskovic joined the atomic physics research group led by Popovic and Prof. Lepsha Vuskovic, who encouraged her to pursue a project at Jefferson Lab in collaboration with Larry Phillips and Anne-Marie Valente-Feliciano. The goal of her work is to investigate plasma etching for surface preparation on superconducting radiofrequency cavities.
"Since accelerator performance depends directly on the physical and chemical characteristics of the superconducting radiofrequency cavity surface, the preparation of the cavity walls has been one of the major challenges in SRF accelerator technology," she explained. "Constant research and development efforts are devoted to developing surface preparation processes that will decrease roughness without introducing impurities, like hydrogen or oxygen, into the niobium surface. Plasma based surface modification provides an excellent opportunity to achieve these goals."
Raskovic splits her time between JLab and ODU, where she is part of an effort to pursue research in accelerator science. Her project, which spans plasma, atomic and accelerator physics, is the first of its kind at ODU and is supported by the Department of Energy through a grant to the university.
"I am very happy to be involved in such a multidisciplinary project – a project that is researching and solving a real accelerator physics problem. I had to learn a lot, but that is what being a scientist is all about: You never stop learning."
When she came to the U.S., her first stop was to visit a friend in New York City. It resonated immediately with her experiences of living in the capital city of Belgrade. Her arrival in Norfolk, where she now lives near ODU, was something of a shock.
"At first I kept asking 'Where is the public transportation?'" she recalled with a laugh.
She recently married and her husband, Hector Garcia, works at the Virginia Modeling, Analysis and Simulation Center in Suffolk. Together, they enjoy ballroom dancing, and, in what little spare time she has, Raskovic enjoys crafting.
"I made our ornaments for this past Christmas," she said. "I like to see something made by my own hands. It's a good stress reliever. On the days when my experiments don't go well, I need to see that I really can produce something good."
Despite her initial culture shock, she has come to enjoy this area, especially the international community of students and scientists she works with.
"We understand each other, we are all dealing with many of the same things, personally and professionally," she said.
By Judi Tull
There wasn't a lot of love for Jefferson Lab's e-mail system on Valentine's Day.
The servers that handle the Lab's electronic missives were overburdened, severing computer users' digital link to the outside world and the office next door. But by noon the following day, a more stable and faster electronic message delivery was in service.
The dramatic turnaround was the result of months of planning and preparation. But with the Valentine's Day e-mail collapse, a command decision was made to immediately execute those plans a month early.
Paul Letta, Unix systems manager in the IT Division's Computing and Networking Infrastructure (CNI) group and e-mail upgrade effort coordinator, says the old system used a 20-year-old method of storing data. In that system, each user's messages were stored as a single text file.
"Every time you got a message or every time you clicked on a message to read, the mail servers would have to go and reread that entire file," Letta says. "Now, if you only had a reasonable-sized inbox or a reasonable number of messages, that's not that big a deal. But we have some account holders at the Lab whose text file is almost two gigabytes, 26 million lines."
The new e-mail server software, an open-source package called dovecot, handles files differently.
"As dovecot reads the e-mail folders, it creates index files. The next time you want to read a message, it first looks in an index file. It then goes directly to that point in the folder," Letta explains.
Andy Kowalski, who heads the CNI group, says the switch to dovecot has made a huge difference in the amount of work the e-mail servers have to perform.
"Before dovecot, the overnight load on the e-mail system was triple what it is now when everybody's here at work. We had all these desktops sitting around overnight doing nothing but saying: Any new mail? And this would happen over and over and over," he says. "The old system wasn't a bad system, it had just become inadequate to deal with the amount of data being sent through our e-mail system."
Before the switch, for example, the e-mail system was transferring up to 100 Megabytes per second over the network, whereas Hall B sends a data stream of up to 25 Megabytes per second when writing experimental data to magnetic tape. Since the switch, the e-mail system transfer rate almost never exceeds 10 Megabytes per second.
The switch to dovecot wasn't the only change made in the e-mail system. The CNI group formed a committee of computer experts to tackle the increasing demands of e-mail in mid-2007. Their plan included repurposing four other computers to replace the older servers; requiring that all e-mail clients be configured to access e-mail using the same method (a common protocol called IMAP); centralizing all computer users' e-mail inboxes; and finally, switching over to the dovecot software.
Many of the changes had already been completed in preparation for the switch to dovecot, and a dovecot test server was undergoing extensive testing. The plan to release dovecot in the spring was on track.
"We thought that deploying dovecot in March/April was actually going to give us enough time to do this before people really noticed the strain on the e-mail system," Kowalski notes.
When the e-mail servers began buckling under the strain, the decision was made to switch over to dovecot immediately, before testing was completed. The switch was made the morning of Feb. 15, a Friday.
Letta and Kowalski say the switch to dovecot is only an intermediate step in handling e-mail. More changes are needed to ensure e-mail stability. In addition, the CNI group will continue to investigate alternatives in meeting the ever-increasing demand on the e-mail servers.
"Many other organizations have already switched over to backend databases to store e-mail. And that would be a giant leap for us. It would also be a costly leap (in the range of $100,000-250,000), and we're not going to do that right now. Paul found a low-cost, intermediate solution," Kowalski says.
To help matters, Letta and Kowalski say that there are several things that individual computer users can do to alleviate the stress placed on e-mail servers. These include:
- Reduce the size of your inbox, by either deleting or moving mail to individual file folders, especially those items that contain attachments.
By Kandice Carter
Below the Fold
Becoming Enthusiastic About Math and Science – BEAMS – Jefferson Lab's long-running math and science enrichment program needs volunteers to assist with, student activities conducted at the Lab during normal business hours.
BEAMS supports Newport News inner-city public school students as they progress from sixth through eighth grade. Nearly 1,500 students and their teachers visit JLab between two and five days each school year to participate in science and math activities conducted by JLab scientists, engineers, technicians and administrators. Volunteers spend about 75 minutes in a BEAMS classroom (VARC 72A & 72B), either leading an activity or assisting the students as they carry out an activity.
The latest round of BEAMS visits began in February and will run through May. Usually two classes attend at a time and participate in two to four separate activities during each day-long visit. Volunteers are needed to lead or help with the following activities:
• Go-Far Cars: Determine how the height of an inclined ramp (potential energy) impacts the distance a model car travels.
"JLab staff, students and users are critical to the success of all of the Lab's science education programs, especially BEAMS," said Christine Wheeler, Science Education administrator. "Our BEAMS volunteers have been exposing young students to the applications, importance, and excitement of mathematics and scientific principles for more than 16 years. Students get to learn about you and your job and you get to have fun working with lots of smiling faces. Everybody wins!"
"We'd love to have you join us: for the first time, the hundredth time, or for the first time in a long time," she added.
"Want to help, but not sure how? That's what we're here for," Wheeler explained. "We need volunteers to help students during the activities and we need volunteers to lead activities. Science Education staff will provide the training you need. If you want to observe a classroom activity to see if it's something you want to do, we can schedule that."
Research that will cast a new spin on particles inside the atomic nucleus has earned a researcher based at Temple University the first-ever Jefferson Science Associates Postdoctoral Research Fellowship at Jefferson Lab.
Brad Sawatzky, a Canadian who earned bachelor's and master's degrees at the University of Saskatchewan and a Ph.D. at the University of Virginia, will spend the fellowship's $10,000 research grant to purchase and assemble components of a flash-ADC-based data acquisition system. This cutting-edge electronics system, which is still in development, may allow researchers to record five times more data than they can currently capture.
Sawatzky plans to test the system in the upcoming Spin Asymmetries on the Nucleon Experiment that is scheduled to run in JLab's Experimental Hall C and the d2n experiment scheduled for Experimental Hall A. Both experiments aim to exploit a property of protons and neutrons, their spin, to determine how the constituent quarks and gluons are distributed inside. To do that, Sawatzky and his colleagues will control the orientation of the spin of both the electrons used to probe the protons and neutrons and the spin of the protons and neutrons themselves.
The JSA Postdoctoral Research Fellowship honors nuclear physics postdoctoral candidates who have a record of accomplishment in physics and whose planned research will have a high impact on the physics program at JLab. The fellow is expected to be a future leader in the Lab's research fields. Fellows receive a $10,000 research grant for JLab-related research. The fellowship was initiated by representatives of the roughly 1,200 scientists who conduct research at JLab, the Users Group Board of Directors.
"The Users Group Board was very happy this year to work on a new program sponsored by Jefferson Science Associates, JLab's management contractor, which allows us to recognize and reward our finest postdocs," said Ron Gilman, board chairman and a Rutgers University professor.
"We had a number of excellent fellowship candidates, which made the process of choosing the winner both more difficult and longer than I expected. Brad Sawatzky's research proposal was chosen because he has been outstanding in all aspects of experimental physics."
"I really enjoy working with the great group of people at JLab, and I try hard to support the collaboration and develop the physics. It's just very nice to have that effort be recognized," Sawatzky said of winning the inaugural fellowship.
Editor's note: This story first ran on JLab's main webpage earlier this year.
When Sue Parham underwent tests prior to undergoing surgery for breast cancer, she had no idea how much her life was about to change.
Parham's surgeon had sent her to the hospital for a new procedure designed to provide better images of a small tumor in her left breast. To get those better images, Parham underwent a BSGI scan, short for Breast-Specific Gamma Imaging, at Legacy Good Samaritan Hospital in Portland, Ore.
Based on gamma imaging technology developed by Jefferson Lab's Radiation Detector and Medical Imaging Group, BSGI is a non-invasive breast imaging procedure that captures the cellular function of breast tissue, complementing mammography in helping to resolve difficult to interpret cases.
"As the fifth patient to use the Breast-Specific Gamma Imaging at Portland's Good Samaritan Hospital, I could tell they were still working with the technology. But that didn't explain what was to follow," the 49-year-old Parham recalled of the procedure she underwent a year ago.
The BSGI found "a slow-growing, potentially deadly tumor in my right breast that had reached 3 centimeters in size that had gone unnoticed by the mammogram," Parham said. And it was then discovered that the cancer had spread to the lymph nodes in her right arm.
"My diagnosis moved from straightforward to complex, my treatment from routine to aggressive and my life from normal to anything but," Parham explained.
"For me, the past year has been the most challenging of my life. I've endured a double-mastectomy, removal of the lymph nodes under my right arm, removal of both ovaries, reconstructive breast surgery, four-and-a-half months of chemotherapy, and six-and-a-half weeks of radiation therapy. But, most important, I am alive. I am a living, breathing testament to the early detection made possible by BSGI."
The BSGI machine used to diagnose Parham's cancer was manufactured by Dilon Technologies, a Newport News company. Dilon licensed its high-resolution gamma imaging technology from Jefferson Lab after recognizing its potential for use in a marketable compact dedicated breast-imaging camera.
JLab's Radiation Detector and Medical Imaging Group began exploring the use of nuclear physics detector technology for medical imaging applications in 1995, starting with two types of breast imagers. Members of the group are experts in radiation detector technology, which is used extensively at JLab to conduct basic nuclear physics research. In the biomedical field, the group has been involved in the development of high-resolution gamma and positron imagers and in novel multimodality (including X-ray and optical) imaging concepts.
"It's a never-ending challenge and desire for us to take our technical expertise and make a difference in people's lives. BSGI is the first of what we anticipate will be more instruments we hope to develop and help bring to market," says Stan Majewski, head of the Radiation Detector and Medical Imaging Group.
BSGI is designed for situations where mammography is inconclusive and further evaluation is needed, especially when patients have dense breast tissue, implants, multiple suspicious lesions or clusters of microcalcifications, palpable lesions not detected by mammography or ultrasound, post-surgical or post-therapeutic mass, or if they have been taking Hormone Replacement Therapy.
The link is that both cancer and breast tissue are dense in nature, and the more dense the breast tissue, the more likely that a cancer could be missed in evaluation, especially with imaging tests that look at structure and not metabolic function, such as mammography and ultrasound.
In the Newport News area, BSGI procedures are available at Riverside Diagnostic and Breast Imaging Center and the Dorothy G. Hoefer Comprehensive Breast Center at Sentara Port Warwick. Dr. Curtis Stoldt, director of Riverside Diagnostic and Breast Imaging, says he first heard about BSGI in 2005 at a conference and immediately began working to secure a BSGI machine for his hospital.
"It took three years but it is online and we are doing 6-8 patients a week, and that number is increasing quickly," Stoldt says. "BSGI images are simpler and easier for the patient and technologist. The studies are much simpler to read and compare with mammograms," Stoldt adds. "There are fewer studies with questionable or indeterminate findings, and my overall confidence is greater.
Editor's note: This story first ran on JLab's main webpage earlier this year.
During February, Brookhaven National Lab highlighted in its newsletter and on its webpage substantial energy and money savings at the Relativistic Heavy Ion Collider after implementing portions of a process invented by Jefferson Lab Cryogenics Group staff. The process has been saving BNL approximately $50,000 per week in electric costs for running RHIC. All told, because of using this technology, called the "Ganni Cycle," as part of a three-phase upgrade to the RHIC cryogenic system, the electricity needed to operate the system has dropped from 9.2 megawatts (MW) in 2002 to 5 MW currently, saving the Laboratory as much as $1.5 million per year during a typical 30-week annual operating schedule for RHIC.
RHIC's cryogenic plant – the largest in the world – cools the collider's 1,740 superconducting magnets to -456 degrees Fahrenheit, the temperature at which there is no resistance to electricity. The cooling cycle begins by compressing room-temperature helium to high pressure. This high-pressure helium is then expanded through turbines to make it progressively colder until the liquid helium temperature of -456 degrees Fahrenheit is reached. This liquid helium is then supplied to the collider ring where it boils off to cool the superconducting magnets. The vapor created is returned to the plant to begin the cycle again.
The final phase of the cryogenic upgrade was completed in 2006, three years after the Ganni Cycle was first implemented. With the help of funding from the Department of Energy's Energy Management Program and in consultation with JLab's Venkatarao Ganni, the primary inventor of the Ganni Cycle, piping and valves in the cryogenic plant were reconfigured. Also, the method used to send helium flow through the RHIC magnets was modified to eliminate the use of liquid-helium circulating pumps, more efficient heat exchangers were installed, and a seventh turbine was added to the existing six in RHIC's refrigeration system.
These upgrades further reduced the helium flow-rate and the compression ratio needed by the refrigerator, thus decreasing electrical power consumption and component wear. The Ganni Cycle had reduced electric power consumption for cooling RHIC from 9.2 MW to 7.2 MW in 2003, and the completion of the upgrade brought about another 2.2 MW reduction.
The Ganni Cycle improved the cryogenic system's reliability, availability, stability, and efficiency. The technology effectively reconfigured the RHIC refrigeration compute – control system to match the amount of helium being compressed with the heat load from the RHIC magnets.
"The Ganni system increases plant capacity very efficiently when we need it and decreases it when we don't need it, all automatically via computer," Than said.Roberto Than explained that RHIC's cryogenic plant was originally built for ISABELLE, a collider that was to be constructed in the RHIC tunnel but was cancelled in 1983. The plant has three times the cooling capacity necessary for RHIC. Thus, in order to run efficiently, the plant needed to be operated at a lower capacity. It had 15 MW of compressor power available, but it started operations requiring 9.2 MW of compressor power to cool RHIC. Modifications, such as the Ganni Cycle, made during the three-phase upgrade of the plant have helped to reduce the power required to 5 MW. (Reprinted from the Feb. 8, 2008, Brookhaven Bulletin).
The On Target newsletter is published monthly by the Thomas Jefferson National Accelerator Facility (Jefferson Lab), a nuclear physics research laboratory in Newport News, Virginia, operated by Jefferson Science Associates, LLC, for the U.S. Department of Energy's Office of Science. Possible news items and ideas for future stories may be emailed to firstname.lastname@example.org, or sent to the Jefferson Lab Public Affairs Office, Suite 15, 12000 Jefferson Avenue, Newport News, VA 23606