= Reading and Writing data from the grid = This assumes you have access to the grid (certificate, ui and everything). If not, rather check the [[BasicGridInfo|explanation page]] first. = Reading Data = There are (at least) two ways to discover the already stored files: 1. via the '''lcg-ls''' command, which can take different types of protocols and doors etc. An example would be: {{{ lcg-ls srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/ }}} 2. Easier to use is the logical file catalog. You need to initialise a shell (choose your flavour) variable with the right one: {{{ export LFC_HOST=`lcg-infosites --vo ilc lfc` }}} And then use the type of command that follows below. <
> Top level in the lfc is: {{{ lfc-ls -l /grid/ilc/tpc }}} Four years are present (always with the subdirectory t24/): {{{ lfc-ls -l /grid/ilc/tpc/2008/t24 lfc-ls -l /grid/ilc/tpc/2009/t24 lfc-ls -l /grid/ilc/tpc/2010/t24 lfc-ls -l /grid/ilc/tpc/2011/t24 }}} '''2008''' {{{ lfc-ls -l /grid/ilc/tpc/2008/t24/micromegas }}} '''2009''' {{{ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/december lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/december lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/ lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/june lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/august lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/november lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/december lfc-ls /grid/ilc/tpc/2009/t24/bgem/ lfc-ls /grid/ilc/tpc/2009/t24/bgem/lcio/sept **EMPTY** lfc-ls /grid/ilc/tpc/2009/t24/bgem/misc/sept **EMPTY** lfc-ls /grid/ilc/tpc/2009/t24/bgem/rawdata/sept lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/ lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio/july **EMPTY** lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping/july **EMPTY** lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata/july }}} '''2010:''' {{{ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/sept **EMPTY** lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/sept lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/ lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/sept **EMPTY** }}} '''2011:''' {{{ lfc-ls -l /grid/ilc/tpc/2011/t24/desygem lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio/june lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata/june lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc/june }}} = Copying files to your local computer = Simply use the lcg-copy command once you know the filename. An example that ''won't'' work is (the file doesn't exist, replace to your liking): {{{ lcg-cp -v --vo ilc lfn:/grid/ilc/tpc/2008/tb-desy/testrun1/file1 file:$PWD/file1 }}} = Writing data to the grid storage = '''Please be aware that you know what you are doing! Please test at a safe location before testing with the real paths!''' <
> Writing privileges need and additional pre-requisite: you need a valid grid certificate with the sufficient rights to write to the right storage. To my understanding you need to have the role '''/ilc/tpc/production'''; check with the VOMS server (with an imported, valid certificate in your browser) at: [[https://grid-voms.desy.de:8443/voms/ilc]] <
> If you don't have it yet, but would like to write data simply request the sufficient rights at the same place. Unfortunately this is a two-step procedure, since you first need to be a member of '''/ilc/tpc''' before you can request the additional '''/production''' role. Usually the admins are very quick if your name and affiliation are known. If you are impatient, you could try to send a friendly mail to speed things up. Initialisation is similar to the usual setup, but roles are acquired only when the requested: 1. Obtain proxy with write rights: {{{ voms-proxy-init -debug -voms ilc:/ilc/tpc/Role=production }}} 2. (optional: see if it worked; especially with the role) {{{ voms-proxy-info -debug -all }}} 3. To make life easier, use the logical file catalog (see also above about reading the data from the grid). {{{ export LFC_HOST=`lcg-infosites --vo ilc lfc` }}} 4. If your directory doesn't yet exist in the logical file catalog, you have to create it. According to the current scheme, this would read like: {{{ lfc-mkdir -p /grid/ilc/tpc/$YEAR/t24/$MODULETYPE/$FILETYPE/$MONTH }}} The current scheme, as evolved over time is as follows: || YEAR || [2008|2009|2010|2011|etc.] || || MODULETYPE || [jgem|bgem|micromegas|desygem|...] || || FILETYPE || [lcio|rawdata|misc|...] || || MONTH || [january|february|march|april|may|june|july|etc.] || 5. Suppose that the file you want to write to grid storage is called foobar.slcio (in your local directory), then the right command is '''lcg-cr''' (copy and register) -- which takes three arguments and at least one option. An example would be (on a single line, the slashes indicate line breaks): {{{ lcg-cr --verbose --vo ilc -l lfn:/grid/ilc/tpc/2014/t24/test/foobar.slcio -d srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/2014/t24/test/foobar.slcio file:$PWD/foobar.slcio }}} In a more formal way: {{{ lcg-cr --verbose --vo ilc -l lfn:${LFN} -d srm://dcache-se-desy.desy.de/${DEST} file:${SOURCE} }}} The options are: * {{{--verbose}}} creates verbose output * {{{--vo}}} sets the VO explicitly * {{{-l}}} specifies the logical filename ${LFN} * {{{-d}}} specifies the destination ${DEST} (in this case via the dcache door at DESY) * {{{file:}}} specifies the proctocol (none in this case, meaning POSIX file access) and the filename ${SOURCE} (including the path!) If the copy command works you'll see a file transfer progress bar and get a summary in the end. Please check the file catalogue if the file is really there (see file reading) and possibly also check a file out. Good luck!