Jlab LQCD cache

Jlab LQCD Cache Disk Pool
(Last modified date: January 23, 2007)
The cache disk pool is made of several large cache disks (/cache1, /cache5, and /cache6), which are mounted as /cache on interactive machine qcdi01, qcdi02, qcdi03 for cluster users to use. An attempt is made to evenly balance the data on these disks by organizing the data into different projects, with path names /cache/<project-name> (where the project name generally contains a collaboration name) and then distributing these projects onto the multiple servers. So far there are directories created for six projects. User must send a request through CCPR system in order to add a new project at the top level. Anyone in the lattice group can create a subdirectory under these existing project directories.

For users wishing to do some development work outside of formal projects, there are directories /cache/users/<username> for each user of the system. Please send a request through CCPR system to set up a work space under/cache/users/ area.

The cache disk is managed by cache manager software with pre-defined backup policy and deletion policy. Cache Disk Usage and Management describes the the cache disk management policy in detail. Please read this document before using any of this disk space. Files can disappear without any backup if a user doesn't understand the policy, as the cache system supports the notion of transient files at any point in the directory tree -- files that can be deleted when the disk is full.

The /cache/<project> directories have a default policy which will make all newly discovered files permanent. The /cache/users/<username> directories have a default policy which will make all newly discovered files volatile. User can change the default policy of any directory by create a new .srm-default-policy policy file in that directory. Please reference this page for the sample policy file.

User quotas are set up on all these disks. Please only keep data files needed in the near future on the disk. When any one of the disks is full, the least recently used files will be removed from disk automatically by the cache manager to free up space. The cache manager daemon will ensure that the files with permanent storage type are copied into the silo system before removing them from the disk. We also provide a srmPin utility for users to mark files as in use ("pin") for a period of time to ensure they stay on disk for a specified period of time.

Each user has a limited amount of pin quota, and it is much smaller than the disk quota. In this way, there will always be files that can be removed to free up disk space. In other words, only a small proportion of files can be pinned by each user at the same time. A user can pin his own files or files owned by others. One file can be pinned by several different users. When a user pins a file owned by another, it only guarantees it will not be removed by the clean up daemon, not the owner of the file. If a pinned file is deleted by the owner, all the pin records to this file will be removed and the pin quota will be returned to the user(s).

Please remember to give every file a unique name (path+filename). If a file is put into the silo and then deleted from disk, this path name should not be used again except to stage the original file back from the silo. If you create a different file with same path name, it will be deleted without backup since the cache manager will treat it as if it is already in the silo.

Users can use any Unix tool to manage their files (scp, cp, mv, rm. etc.) in the cache disk pool. Besides these Unix commands, the cache manager software provides additional utilities, srmQuota, srmPin, srmPinStatus, srmPut, srmGet, srmPermanent, srmLs, srmRequest and more. The following table summarizes the usage of these utilities.

Utility	Argument	Description	Option
srmQuota	(none)	lists user's pin quota	-p (including pinned file)
srmPinStatus	path1, path2, ...	lists given file(s) pin status	-l (to obtain detailed information)
srmPin	path1, path2, ...	pin a given file or files in a given directory	-r (recursive) -t (life time) number of days
srmGet	path1, path2, ...	get file(s) or files in given directories from the silo	-r (recursive)
srmPut	path1, path2, ...	add file(s) or files in given directories into the silo	-r (recursive) -d (delete) after put into silo
srmPermanent /srmVolatile	path1, path2, ...	set storage type of file(s) or files in given directories to permanent/volatile	-r (recursive)
srmRequest	id1, id2, ...	obtain status information of given request	-
srmPendingRequest	(none)	obtain status information of all pending and active requests submitted by this user	-
srmLs	(none)	lists all files in the silo but not in disk owned by this user	-l (long detailed list) -h (display the usage)
srmLs	path	lists the contents of a given directory	-l (long detailed list) -r (recursively) --cache (only file in the cached disk) --ncache(only file not in the cached disk) --pin (only pinned file) --unpin (only unpinned file) --silo (only file in the silo) --nsilo (only file not in the silo) --perm (only permanent file) --volatile (only volatile file)

Cache disk directory layout

Cache disk file backup policy

Files will be backup into silo if it is in permanent space and older than the backup threshold (operational parameter, current is 7 days). Files are queued for writing on a least recently used basis.
Due to the limitation of our silo system, only files with size between 1MB to 2 GB will be added to the tape storage system. Files outside this range will be copied to a different disk for backup.

Cache disk deletion policy

Unpined large files (size > 1 MB) (already exist in the silo if within permanent space OR files within volatile space).
Files in above group with the least recently used files will be deleted first.

Cache manager Utilities: (located in /usr/local/bin directory on all interactive nodes).

srmQuota: Check the user's pin quota, including total quota, used quota and available quota. It takes no argument with only one option [-p]. Use [-p] option to list all files pinned by this user. pinQuota only lists current user's pin quota. To check other's pin quota or to request an increase in quota, please contact Jlab hpc group.
srmPinStatus: Returns the files pin status. Prints out the pin count and pin expiration date. It takes a list of file or directory paths with only one option [l]. Use [-l] option to list more information. Wild cards in the file name are not supported.
srmLs: If no argument is given, lists all files in the silo but not on disk, owned by this user. Otherwise lists the contents of a given directory. There are six options to used. Use [-h] or [--help] to display the usage and exit. Use [-l] to show long list including group and modification time. [-r] to list the entire contents of given directory recursively. Use [--cache] to list only the file in the cache disk, [--ncache] to list only the file not in the cache disk, [--silo] to list only the file in the silo, [--nsilo] to list only the file not in the silo, [--pin] to list only the file pinned, [--unpin] to list only the file not in the silo, [--perm] to list only the permanet file, [--volatile] to list only the volatile file and [--pin] to list only the pinned files. Options [--cache], [--ncache],[--silo], [--nsilo], [--pin], [--unpin], [--perm], and [--volatile] can be used together. Any combinations of the options is equivalent to an "AND" operation. For examples, with [--ncache] and [--silo] options only the files in silo and in not in cache disk will be listed. Some combinations, such as [--perm] and [--volatile], will always return a empty set.

srmPin: Marks a file in use so it will not be deleted from cache disk. It takes a list of file paths or directories you wish to pin. If a file with permanent storage type has been deleted from the cache disk, srmPin utility will stage it back to cache disk at the same location and mark it in use. In this case the operation (stage file from silo) is queued and a job id will be returned to user. Use the srmRequest utility with this id to check the status of the stage request. Use [-r] option if the given path is a directory. Use [-t] option to specify the number of days the file should be pinned (life time). If option [-t] has not been used, the default life time 30 days will be given. Once a file has reached its pin life time, the pin will be removed and the pin quota will be returned to the user automatically. When the file is no long needed on the disk cache, the user can run srmPin -t0 file_path or srmUnpin file_path to release this pin quota.

Wild cards in file names is supported. When using the wild card, it must in the file name not in the path directory. (note: use escape '\' character before wild card '*'). Use path_to_dir/\* to srmPin/srmUnpin all files in a given directory, not including the files in the subdirectories. (-r option will include the files in subdirectories).

%srmPin /cache2/LHPC/NF0/aniso/test/foo

%srmPin /cache2/LHPC/NF0/aniso/

%srmPin /cache1/LHPC/NF0/iso/11p5_16_32_dbw2/cfgs/dbw2_16_32_11p5.cfg1\*0

%srmPin /cache1/LHPC/NF0/iso/11p5\*dbw2/cfgs/dbw2_16_32_11p5.cfg1\*

~~%srmPin /cache1/LHPC/NF\*/is0/11p5_16_32_dbw2/cfgs/dbw2_16_32_11p5.cfg105~~

srmGet: To get file(s) from silo. It takes one or more argument(s) - array of the full path of a file or a directory. Usually it will take a few hours (even a few days) to get file(s) from the silo. srmGet utility will print a request id on STDOUT. Users can check the request status using srmRequest utility. srmGet has only one option [-r] (recursive). Use [-r] option to recursively adds the entire file in a given directory into silo. Wild cards in the file name is supported. When using a wild card, it must in the file name not in the path directory (note: use escape '\' character before wild card '*').
srmPut: To add file(s) into the silo. It takes one or more argument(s) - array of the full path of a file or a directory. Usually it needs a few hours to a few days to add file(s) into the silo. srmPut utility will print a request id on STDOUT. Users can check the request status using srmRequest utility with correspond request id. srmPut has two options [-r] (recursive) and [-d] (delete). Use [-r] option to recursively add all files in a given directory into the silo. Use [-d] option to tell the cacheManager daemon to delete the file after it is added to the silo. Wild cards in the file name is supported. When using a wild card, it must in the file name not in the path directory (note: use escape '\' character before wild card '*').
srmPermanent/srmVolatile: Set files' storage type to permanent/volatile. It takes one or more arggument(s) - array of the full path of a file or a directory. After a file is set to permanent, it will be add into silo once it is older than 7 days. srmVolatile will fail if that file is already in silo. srmPermanent/srmVolatile has one option [-r] (recursive). Use [-r] option to recursively set storage type of all files in a given directory. Only the owner of the file can change its storage type. If the user is not the owner of the file, these commands will do nothing and exists.
srmRequest: To obtain status information of given request(s). It takes one or more argument(s) - array of the request id. The information such as request type, request state, submit time, and files included in the request will be printed on stdout if given request id is valid.
srmPendingRequest: To obtain all pending and active request(s) submitted by this user or a given user. It takes no argument if the user name is as same as this user, or one argument - user name. The information such as request type, request state, submit time, and files included in the request will be printed on stdout if there is any pending request.