JASMine,
the software that manages the tape silo and the disk caches,
was upgraded over the holiday weekend. Overall the upgrade was a success.
There were, however, some problems with the disk cache this week that
resulted from the JASMine upgrades. The problems were reported/discovered
mid-day on Tuesday and resolved by 2:30pm Thursday.
The problems that users noticed were that certain cache servers were not
responding in a timely manner (or at all), that the newly cached files
were not staying on the cache disks long enough to use before they were
deleted, and the jcache requests took more time than usual to complete.
The cause of the problem seems to have come from the fact that the new
jcache command no longer checked the user's group membership to see
if they were in the clas group. So clas users not using the '-g clas'
option
to jcache were caching files to the default cache disks. This space was
not large enough (400GB) to hold all the requests and files were being
deleted almost as soon as they appeared. In addition, the 2 default cache
disk file servers were being over loaded by all the files being copied
to
them from the 13 data movers.
To resolve the problem the group check was added back to the jcache
command. Jcaches from users in the clas group that do not make use of
the '-g <group>' option are once again going to the clas cache disks
(2+TB).
For a long term solution we will make the '-g <group>' option mandatory
in a future version.
It is also worth noting that over 6TB of data have been successfully moved
to/from tape over the past 24 hours and that there were over 10,000 files,
representing 10TB worth of data, in the JASMine file request queue at one
time.