The Scientific Computing group is working with JLab's Theory group to deploy a sequence of high performance clusters in support of Lattice Quantum ChromoDynamics (LQCD). LQCD is the numerical approach to solving QCD, the fundamental theory of quarks and gluons and their interactions. This computationally demanding suite of applications are key to understanding the experimental program of the laboratory.
Deployed Clusters
 |
12k ClusterThe 2012 cluster consists of 42 nodes of dual opto-core Sandy Bridge E5-2650 CPUs running at 2.00 GHz. Each node contains 128 GB of host memory and four passively-cooled NVIDIA Tesla K20m GPUs. |
 |
12s ClusterThe 2012 cluster consists of 212 nodes of dual octo-core Sandy Bridge E5-2650 CPUs running at 2.00 GHz. Each node is equipped with 32 GB DDR3 ECC memory clocked at 1600 MHz and a 500 GB local hard drive. In addition, these nodes are connected via QDR InifiniBand networks providing 40 Gb/s bandwidth. |
 |
10g ClusterThe 2010 cluster consists of 50 nodes of dual quad-core Westmere (E5630) CPUs running at 2.53 GHz. Each node is equipped with 48 GB DDR3 ECC memory clocked at 1331 MHz. Among these 50 nodes, there are 32 nodes each of which is equipped with 4 Nvidia Tesla C2050 GPUs. Each C2050 GPU consists 480 processing cores and 3 GB GDDR5 memory. Each C2050 can deliver 144 GB/s memory bandwidth and 1.03 Tflops of single precision performance. These 32 nodes are connected via QDR InifiniBand networks providing 40 Gb/s bandwidth. Each of the remaining nodes consists of 4 Nvidia GTX 480 GPUs. Each of these GPUs provides 480 processing cores and 1.6 GB GDDR5 memory. Every node provides cuda 3.0 development toolkit, cuda 195.36.24 driver, OFED 1.4.2 InfiniBand Development library and message passing libraries such as mvapich and OpenMPI. Currently, each node runs CentOS 5.3 with gcc 4.1.2 and Linux Kernel 2.6.18. |
 |
10q ClusterThe 2010 cluster consists of 224 nodes of dual quad-core Westmere (E5630) CPUs running at 2.53 GHz. Two nodes are hosted inside one 2U box. Each node is equipped with 48 GB DDR3 ECC memory clocked at 1331 MHz and 500 GB local hard drive. In addition, these nodes are connected via QDR InifiniBand networks providing 40 Gb/s bandwidth. Every node provides OFED 1.4.2 InfiniBand Development library and message passing libraries such as mvapich and OpenMPI. Currently, each node runs CentOS 5.3 with gcc 4.1.2 and Linux Kernel 2.6.18. |
 |
9g ClusterThe 2009 cluster consists of 65 nodes of dual quad-core Nahalem (E5530) CPUs running at 2.4 GHz. Each node is equipped with 48 GB DDR3 ECC memory clocked at 1331 MHz. There are 4 GPUs installed on every nodes. Among these GPUs, there are 96 Nvidia GTX 480 GPUs with Fermi architecture. The rest of 156 GTX 285 GPUs utilize older GT 200 architecture. Every node provides cuda 3.0 development toolkit, cuda 195.36.24 driver. Currently, each node runs CentOS 5.3 with gcc 4.1.2 and Linux Kernel 2.6.18. |
 |
9q ClusterThe 2009 cluster consists of 320 nodes of dual quad core Nehalem CPUs connected via a QDR InifiniBand switched network. Each node has two processors running at 2.4 GHz, 24 GB DDR3-1333 memory and 500 GB SATA disk. In addition, each node has an InfiniBand HCA adapter that provides 40 Gb/s bandwidth. |
Decommissioned Clusters
 |
7n ClusterThe 2007 cluster consists of 396 nodes of AMD Opteron (quad-core) CPUs connected via DDR InifiniBand switched networks. Each node has two processors running at 1.9 GHz, 8 GB DDR2 memory and an 80 GB SATA disk. In addition, each node has a PCI-EXPRESS (16x) slot for an InfiniBand HCA adapter that provides 20 Gb/s bandwidth. |
 |
6n ClusterThis cluster deployed in 2006 consists of 280 nodes of Intel Pentium D (dual-core) CPUs connected via InifiniBand switched networks. Each node has a single processor running at 3.0GHz with 800 MHz front side bus, 1 GB memory and 80 GB SATA disk. In addition, each node has a PCI-EXPRESS (4x) slot for an InfiniBand HCA adapter that provides 10 Gb/s bandwidth. |
 |
4g ClusterThe 2004 cluster consisted of 384 nodes arranged as a 6x8x2^3 mesh (torus). These nodes were interconnected using 3 dual GigE cards plus one half of the dual gigE NIC on the motherboard (the other half is used for file services. This 5D wiring could be configured as various configurations of a 3D torus (4D and 5D running was possible in principle, but was less efficient so not used). Nodes were single processor 2.8 GHz Xeon, 800 MHz front side bus, 512 MB memory, and 36 GB disk. This cluster achieved approximately 0.7 teraflops sustained. Message passing on this novel architecture was done using an application optimized library, QMP, for which implementations will also target other custom LQCD machines. The lowest levels of the communications stack were implemented using a VIA driver, and for multiple link transfers VIA data rates approaching 500 MB/sec/node were achieved. |
 |
3g ClusterThe 2003 deployed cluster of 256 nodes each interconnected using 3 dual GigE cards (one per dimension, view of wiring) and one on-board link, an approach which delivered high aggregate bandwidth while avoiding the expense of a high performance switch. Nodes were single processor 2.67 GHz Pentium 4 Xeon with 256 MByte of memory. The cluster achieved 0.4 TeraFlops on LQCD applications. Additional information can be found in an Intel case study (pdf file). |
 |
2m ClusterThe oldest cluster, now decommissioned, was installed in 2002. The cluster contained 128 nodes of single processor 2.0 GHz Xeons connected by a Myrinet network. It achieved 1/8 TeraFlops of performance (Linpack), and contained 65 GBytes of total physical memory, and delivered 270MB/s simultaneous node-to-node aggregate network bandwidth. All nodes ran Red Hat Linux 7.3 with kernel 2.4-18. |
