Reading and Writing data from the grid
This assumes you have access to the grid (certificate, ui and everything). If not, rather check the explanation page first.
Reading Data
There are (at least) two ways to discover the already stored files:
via the lcg-ls command, which can take different types of protocols and doors etc. An example would be:
lcg-ls srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/
- Easier to use is the logical file catalog. You need to initialise a shell (choose your flavour) variable with the right one:
export LFC_HOST=`lcg-infosites --vo ilc lfc`
And then use the type of command that follows below.
Top level in the lfc is:
lfc-ls -l /grid/ilc/tpc
Four years are present (always with the subdirectory t24/):
lfc-ls -l /grid/ilc/tpc/2008/t24 lfc-ls -l /grid/ilc/tpc/2009/t24 lfc-ls -l /grid/ilc/tpc/2010/t24 lfc-ls -l /grid/ilc/tpc/2011/t24
2008
lfc-ls -l /grid/ilc/tpc/2008/t24/micromegas
2009
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/december lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/december lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/december lfc-ls /grid/ilc/tpc/2009/t24/bgem/ lfc-ls /grid/ilc/tpc/2009/t24/bgem/lcio/sept **EMPTY** lfc-ls /grid/ilc/tpc/2009/t24/bgem/misc/sept **EMPTY** lfc-ls /grid/ilc/tpc/2009/t24/bgem/rawdata/sept lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/ lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio/july **EMPTY** lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping/july **EMPTY** lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata/july
2010:
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/sept **EMPTY** lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/sept lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/sept **EMPTY**
2011:
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio/june lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata/june lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc/june
Copying files to your local computer
Simply use the lcg-copy command once you know the filename. An example that won't work is (the file doesn't exist, replace to your liking):
lcg-cp -v --vo ilc lfn:/grid/ilc/tpc/2008/tb-desy/testrun1/file1 file:$PWD/file1
Writing data to the grid storage
Please be aware that you know what you are doing! Please test at a safe location before testing with the real paths!
Writing privileges need and additional pre-requisite: you need a valid grid certificate with the sufficient rights to write to the right storage. To my understanding you need to have the role /ilc/tpc/production; check with the VOMS server (with an imported, valid certificate in your browser) at: https://grid-voms.desy.de:8443/voms/ilc
If you don't have it yet, but would like to write data simply request the sufficient rights at the same place. Unfortunately this is a two-step procedure, since you first need to be a member of /ilc/tpc before you can request the additional /production role. Usually the admins are very quick if your name and affiliation are known. If you are impatient, you could try to send a friendly mail to speed things up.
Initialisation is similar to the usual setup, but roles are acquired only when the requested:
- Obtain proxy with write rights:
voms-proxy-init -debug -voms ilc:/ilc/tpc/Role=production
- (optional: see if it worked; especially with the role)
voms-proxy-info -debug -all
- To make life easier, use the logical file catalog (see also above about reading the data from the grid).
export LFC_HOST=`lcg-infosites --vo ilc lfc`
- If your directory doesn't yet exist in the logical file catalog, you have to create it. According to the current scheme, this would read like:
lfc-mkdir -p /grid/ilc/tpc/$YEAR/t24/$MODULETYPE/$FILETYPE/$MONTH
The current scheme, as evolved over time is as follows:YEAR
[2008|2009|2010|2011|etc.]
MODULETYPE
[jgem|bgem|micromegas|desygem|...]
FILETYPE
[lcio|rawdata|misc|...]
MONTH
[january|february|march|april|may|june|july|etc.]
Suppose that the file you want to write to grid storage is called foobar.slcio (in your local directory), then the right command is lcg-cr (copy and register) -- which takes three arguments and at least one option. An example would be (on a single line, the slashes indicate line breaks):
lcg-cr --verbose --vo ilc -l lfn:/grid/ilc/tpc/2014/t24/test/foobar.slcio -d srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/2014/t24/test/foobar.slcio file:$PWD/foobar.slcio
In a more formal way:lcg-cr --verbose --vo ilc -l lfn:${LFN} -d srm://dcache-se-desy.desy.de/${DEST} file:${SOURCE}The options are:--verbose creates verbose output
--vo sets the VO explicitly
-l specifies the logical filename ${LFN}
-d specifies the destination ${DEST} (in this case via the dcache door at DESY)
file: specifies the proctocol (none in this case, meaning POSIX file access) and the filename ${SOURCE} (including the path!)
If the copy command works you'll see a file transfer progress bar and get a summary in the end. Please check the file catalogue if the file is really there (see file reading) and possibly also check a file out.
Good luck!