Notes on Configuring and Porting SZIN

Table Of Contents

1. Introduction
2. The current SZIN configuration and build system
1. Configuration script variables
2. The SZIN source files
3. Porting SZIN
1. Miscellaneous Porting Issues (not answered)
2. Porting SZIN to the T3E without assembler
1. Single PE version
2. Multiple PE version
4. Site Customisation (hacking the configure script)
1. Case Study 1: The Glasgow Alpha Cluster
2. Case Study 2: The EPCC Sun Cluster
1. Non GNU versions of popular tools
2. The mitsmp target
3. The gmitsmp target
4. The jlabsmp target
5. Miscellany

This is a set of notes I drew up while compiling SZIN on various platforms, from Linux, and the T3E, to Sun and Alpha clusters.

The current SZIN configuration and build system

Currently in order to build SZIN one has to first configure the system, using the supplied configure perl script, with the desired MACHINE_TYPE variable set which sets the concrete machine type. This script performs the following functions:

1. creates a file called Makefile.${MACHINE_TYPE} for the current machine:
   

   This file contains makefile variables relating to directories, the invocation for GNU m4 (this is needed for the build). It also contains a bunch of Machine specific variables.

2. runs make in the makestuff subdirectory of the toplevel.
3. builds a thread library for SMP systems in the other_libs/smp_lib/pt-lib if this is required for a target
4. builds the libqcd if required. -- This currently is not implemented

The variables set by the configure script, in two steps. In the first step, variables that are predominantly system independent are set. We list them below:

TOPDIR

topdir is the absolute path szin source tree location

LIBDIR

parent directory of library build

MAKDIR

where make thingies like hashsub and splitsub are kept

EXTDIR

parent directory of the external lib files (MACHINE_TYPEdependent

OUTDIR

where to dump the m4 processed C files

ROOTDIR

where the root {or true} main programs are kept

INCDIR

location of various include files. Note some are copied to this directory.

OTHERDIR

where some other libraries can be found

ARCH_TYPE

This sets the type of architecture the target system has. By this we mean parallel architecture, not CPU architecture, such as scalar, vector etc. The complete list of architectures are listed in the official SZIN manual. We list them here for reference:

scalar

Outer grid loop/no inner grid. Basically a workstation.

vector

No outer grid loop/inner grid loop. Vector machine

mixed

Both outer and inner grid loop. Example is CM-5 or possibly a CRAY

cm

No outer grid. Inner grid is via PARIS calls for CM-2. -- This is now probably deprecated

dsp

Scalar structure with outer grid divided among other processors (should be called parscalar) -- This may fit most MIMD machines with SPMD programming models

smp

Scalar structure using multi processes for smp.

tsmp

Scalar structure using threads for smp.

parsmp

Scalar structure using multi processes for smp and outer grid also divided among other processors.

partsmp

Scalar structure using threads for smp and outer grid also divided among other processors.

ROOT_PREFIX

This variable is used to decide which main subroutine to use with the program, and presumably which memory allocator to use also. The relevant main programs live in the ${ROOTDIR} directory. (This is usually: ${TOPDIR}/root).

• stand -- (standard) for scalar and vector architectures
• dsp -- (qcdsp) for a qcdsp architecture
• smp -- for the smp and parsmp architecture
• tsmp -- for the tsmp and partsmp

CC

The C Compiler

CFLAGS

The flags for the C compiler (not including the include path)

XFLAGS

Flags for the C compiler used when making the external library. Appears to be the same as the CFLAGS variable

SPECIAL_CC

Compiler for the SPECIAL_FILES

needed for a particular target. This is often the same as the normal CC but not always. For example on the jlabsmp architecture, the ccc compiler is used as CC but cc is used for SPECIAL_CC.

SPECIAL_CFILES

These are the flags for the SPECIAL_CC compiler. They often contain extra includes etc to deal with communications.

CC_SUFFIX

The standard suffix for programs processed by the C compiler -- usually .c

ASM_SUFFIX

The standard suffix for assembler files. This can change quite a lot depending on architecture. Common ones are .as, .s, .asm, .S

OBJ_SUFFIX

The standard suffix for object files. This is .o for everything except the QCDSP.

AR_SUFFIX

The standard suffix for archive files. Usually .a.

C_FLAG

The compiler flag/switch that produces object files from source files. Typically this is -g on UNIX like systems.

SUPPORT_LIB

The name of the library into which the object files contained by SUPPORT_FILES macro are collected into. This is always set to szin. This is also used to construct the names of the SZIN libraries.

LFLAGS

Flags passed to the linker.

LIBS

List of libraries to link in to executables, for example -lm to link in the math library on most systems.

AR

The name of the archiver program for creating libraries.

ARFLAGS

The flags to the archiver program

RANLIB

The name of the ranlib tool used to add tables of contents to archive files

PRETTY

The name of a preprocessing tool in the makestuff directory.

SMPDIR

The directory containing SMP related code. Only set for SMP targets

MPI

The root directory of an MPI installation. This is not exactly unambiguous and there are separate notes on using the system with MPI. Only set on certain targets

GM

The root directory of a GM (Myrinet) installation.

In the third step a few more variables get set. First, the $INCDIR variable is added on to all the variants of the compiler flags ( CFLAGS, XFLAGS and SPECIAL_CFLAGS. The machine character gets decided this is usually a single character identifying the MACHINE_TYPE that gets prepended to C files generated for that particular architecture. In any case these variables and dependencies are then autogenerated, and need not concern us necessarily for porting purposes.

A quick glance through the Makefile in the $TOPDIR/lib directory should reveal that there are three types of Make variables that contain lists of source files:

SUPPORT_FILES

These are SZIN files that code the various physics routines.

SPECIAL_FILES

These are files that deal with system specific issues such as byte ordering or communication. These files may need a different compilation procedure (different flags etc) than the SUPPORT FILES. These are the files for which the SPECIAL_CC and the SPECIAL_CFLAGS are used.

TYPES_MACROS

These files typically have the suffix .mh and are files that contain the macro definitions that implement the SZIN primitives

Target and Architecture Independent

The files in this layer are common to all targets. These files are held in the variables SUPPORT_FILES SPECIAL_FILES and TYPES_MACROS respectively

Architecture Specific

These are files which are specific to a given architecture (not a single target), such as scalar or vector. The variables collecting these files are held in the variables {arch}_FILES and {arch}_TYPES_MACROS, where instead of {arch} one should use the relevant architecture. The variable {arch}_FILES typically contains SUPPORT_FILES.

Target Specific

These are files that make up the source files for any given specific target. They include the high level SUPPORT_FILES, the architecture specific files and a few extra things, such as optimised assembler. The files are held in the macros {target}_FILES, {target}_SPECIAL_FILES and {target}_TYPES_MACROS following the three kinds of files available in the target and architecture independent layer. Here {target} can be any machine type eg cibm qnxsmp etc.

In the end the final list of files to be compiled are those held in the target specific makefile variables (as these generally subsume and extend the lower levels.

From the foregoing, it can be seen that there are essentially the following stages in porting the SZIN system, at least on the vanilla level (without assembler optimisations).

• Think of a name for your target (I will generically use newtarg)
• Decide which architecture it best fits into I will refer to this as arch.
• Add a section to the configure script, into the target dependent part where you can set up compiler options etc (see the meanings of the flags earlier)
• Create makefile variables: newtarg_FILES, newtarg_SPECIAL_FILES and newtarg_TYPES_MACROS appropriately. Generally one can get away with setting:
  • newtarg_FILES := $(arch_FILES)
  • newtarg_SPECIAL_FILES := $(SPECIAL_FILES)$
  • newtarg_TYPES_MACROS := $(arc_TYPES_MACROS) newtarg_specific.mh
  Here newtarg_specific.mh is a file you have to write see later. If you are using a standard communications harness such as MPI, you may need to add parmpi.c to the list of SPECIAL_FILES too.
• Write the newtarg_specific.mh file.

There is of course more to porting the code to making it compile and link (although that is usually a good start). In the case of a parallel code like SZIN the following are just some of the issues :

Communication:

Different parallel architectures offer differnt ways of performing communications with varying degree of efficiency. While MPI and SMP targets are supported generically, (but see caveats about hacking the configuration scripts for your local site later on) there is not currently a clean interface, where the communications routines of a new architecture (say QCDOC) could be cleanly inserted. Perhaps with the advent of the SciDAC efforts, this situation will become a lot cleaner in the future (ie: one can build SciDAC compliant comms libraries on all architectures, and SZIN could use the SciDAC interface.

I/O:

File IO, and also terminal IO is an important issue in parallel programs. For a lot of the cluster systems, life is relatively easy, as each node has access to an IO stream, from say a local disk. This is not guaranteed for machines like the QCDSP and the QCDOC, and has to be fixed up with various broadcasts/gathers or rotations of data around the processor grid. Some of this can be found in the file dspsun_specific.mh in the macros directory, but a cleaner interface is needed.

High Performance Assembler Code

I currently am having difficulty, finding the interface layer, for introducing new high performance assembler routines. This will be of greatest importance when compiling for QCDOC. For the QCDSP target I can see that assembly routines are declared in the dspsun_specific.mh file. (I can also see hooks for profiling the code - but I havent yet worked out exactly how these work.)

Cross Compilation

Currently two compilers are supported in the build system, (CC, and SPECIAL_CC). As far as I can work out, CC is used to build most of the object files, while SPECIAL_CC is used to build various special files ($SPECIAL_FILES Makefile). In the current build system there is a source file called ch-table.c which is created from the file mk-table.c. The procedure compiles mk-table.c into an executable, which creates ch-table.c upon being run. This sort of thing wouldn't work well when cross compilation is being performed, as the cross compiler doesn't produce executables which can necessarily be run on the host platform. (How is this sidestepped for the QCDSP? -- perhaps it isn't ?). In any case, there may be a call for introducing at some point a different compiler, giving for example the following set: (CC, SPECIAL_CC, HOSTCC) for compiling auxiliary programs on the front end.

In this example we will port the SZIN system to the Cray T3E. Our approach is a two step one, first port to a single processor machine (architecture of type scalar) and then to a multiprocessor machine using MPI. This latter architecture is basically like the QCDSP architecture (should really be called parscalar).

elsif ($MACHINE_TYPE eq "t3escalar")
{
print CONF << 'EOF2';
#+
# Cray T3E single processor
#-
ARCH_TYPE = scalar
ROOT_PREFIX = stand

CC = cc
CFLAGS =
XFLAGS = $(CFLAGS)
SPECIAL_CC = $(CC)
SPECIAL_CFLAGS =
CC_SUFFIX = .c
ASM_SUFFIX = .s
OBJ_SUFFIX = .o
AR_SUFFIX = .a
C_FLAG = -c
SUPPORT_LIB = szin
LFLAGS =
LIBS = -lm
M4 := m4
AR = ar
ARFLAGS = rv
RANLIB = ranlib
PRETTY = pretty

EOF2
}

and we added the following lines to the /lib/Makefile:
#+
# Files for the scalar Cray T3E
#-
t3escalar_FILES := $(scalar_FILES)
t3escalar_SPECIAL_FILES := $(SPECIAL_FILES)
t3escalar_TYPES_MACROS := $(scalar_TYPES_MACROS) t3escalar_specific.mh

We created a t3escalar_specific.mh file with the following contents:
/* -- start M4 file: t3escalar_specific -- */

divert(-1)
/*
* T3ESCALAR_SPECIFIC
*
* This file contains all the macro definitions for constructs specific to
* Cray T3E on a single processor
*
*/
/* Pragma code hooks */
define(`DISJOINT_PRAGMA_CODE',`')

divert(0)dnl
/* -- end M4 file: t3escalar_specific -- */

With these changes, I could perform a build of the library on the Edinburgh T3E.

Multiple Processor version

• The machine is soon to be decommissioned
• The build had to be done by batch queue due to EPCC's amusing environment

One may be able to get the jlabsmp version going though, but with caveats about porting the Terminal and File I/O.

Occasionally, one would desire to build for a system, for which code and targets already exist, but setup differences exist between the distributed version of SZIN and the local machine. Chief culprits in this matter are compilers, compiler flags and the nature and location of standard libraries such as MPI.

The previous description of the configure environment should allow one to hack the makefile effectively. Here we outline 2 particular case studies. Compiling szin on the Glasgow alpha cluster, and compiling it on the EPCC Sun Cluster.

The EPCC Sun Cluster consists of an 8 processor Sun HPC 3500 UltraSPARC II based system. This is the front end to two Sunfire 6800 systems. The Sunfire 6800 systems each contain 24 processors and are SMP systems. This system fits three supported MACHINE_TYPEs:

• mitsmp - SPARC Solaris 2.x using ultra sparc, multiprocess SMP single box
• gmitsmp - SPARC Solaris 2.x using ultra sparc, multiprocess SMP (double precision), single box
• Jlabsmp - SPARC Solaris 2.x using ultra sparc, multiprocess SMP, with MPI communication between boxes.

It may be an EPCCism, but the standard make program on the Sun Cluster is Sun's own version. SZIN uses a lot of features of the GNU make, which is available on the cluster. In particular, the configuration script configure invokes make with the -C flag, which seems not to be supported by Sun's version of make. This requires hacking the configure script, which calls make. (The same effect may be achieved by playing with the path, or aliasing make to gmake. In my instance, to be sure of success, I just replaced all invocations of make in the configure script with gmake. I have had similar problems with m4 but only for the mitsmp target. See below.

The mitsmp Target

This target needs to build some auxiliary SMP libraries. This gave a difficulty, because the makefile in the relevant directory $(TOPDIR)/other_libs/smp_lib/pt-lib uses Solaris' m4 as opposed to the GNU m4. In my attempt to configure for the mitsmp, target, the m4 claimed to run out of buffer. Whilst hacking the configure script in the toplevel directory does not fix this problem it is possible to fix by either:

• aliasing m4 to be gm4 using facilities of the user shell.
• editing the non-user serviceable parts of the Makefile in $(TOPDIR)/other_libs/smp_lib/pt-lib. The section to change is 3 lines below the line: ifeq '$(TARGET)' 'ultra'. Replace:
  M4 = m4
  with
  M4 = gm4

After this change, configuration proceeded satisfactorily, the required SMP library was built, and the rest of the system compiled without further intervention. This hack should also fix the gmitsmp target.

The gmitsmp Target

See the previous section: The mitsmp target. The gmitsmp target is identical to it, but using double precision.

The main difference I can see here between the distributed version and the EPCC cluster, is that the distributed version uses MPICH, whereas on the EPCC cluster Sun MPI or LAM MPI is used. We consider here the case of the Sun MPI.

In fact the situation is straigthforward, we need merely to change the compiler information. In the section after elsif ($MACHINE_TYPE eq "Jlabsmp"):

• The MPI variable is superfluous, as the mpcc compiler takes care of everything MPI related. Replace:
  MPI = ./mpi
  with
  MPI =
  
• The compiler is mpcc instead of mpicc and is in the standard PATH. Replace:
  CC = $(MPI)/bin/mpicc
  with
  CC = mpcc
• The MPI libraries are in the link path. We are not using MPICH but SUN MPI. Replace:
  LIBS= -L$(MPI)/lib -lmpich -lm
  with
  LIBS= -lmpi -lm

All in all this has covered most main issues that I found in configuring and compiling (and potentially 'porting') szin between various computer systems. I list here some general problems that are not addressed above:

T3E Problem: DOS newlines

The T3E C-compile seemed to get confused by the DOS like way of ending files. (With Carriage return and newline characters). These have probably slipped into the source from those authors who use Windows 2000 and its various derivatives. On a machine with dos2unix a convenient way to sidestep this problem is to use the command:

find . -name "*" -exec dos2unix \{\} \{\} \;

Unfortunately the T3E doesn't have dos2unix which meant I couldn't compile the code as checked out directly from the repository on the T3E. I had to move it to a machine with dos2unix first, then convert it and then to move it to the T3E.