Contents

  1. Introduction
  2. PROOF at the NAF
  3. How to connect to the PROOF cluster and manage PROOF sessions
  4. How to run a PROOF-based analysis
    1. The TSelector
    2. Running on D3PD files
    3. Running on AOD files
  5. Links

Introduction

The PROOF (Parallel ROOT Facility) library is designed to perform parallelized ROOT-based analyses with a possibly heterogeneous cluster of computers. It allows an interactive utilization. A simple PROOF cluster is composed of one master node and several worker nodes. The parallelization is intended to be transparent for the user.

The service "xrootd" is used for the communications. The PROOF master and workers thus refers to an xrootd service running on a given node.

PROOF at the NAF

A PROOF cluster can be created on demand by the user at the NAF, using interactive SGE batch jobs which will start the worker daemons. This method has originally been developed for the CMS experiment, and has been adapted for ATLAS. It relies on a single script ("proofcluster.pl"). Information on the original CMS script can be found here:

https://twiki.cern.ch/twiki/bin/view/CMS/HamburgWikiComputingNAFPROOF

The "proofcluster.pl" script

Please use the version available at this path: 

~calfayan/scripts/proofcluster.pl

PROOF requires ROOT. If the environment variable "ROOTSYS" is not set, please enable a recent ROOT version: 

ini ROOT52200

How to connect to the PROOF cluster and manage PROOF sessions

First start a PROOF cluster with the command: 

~calfayan/scripts/proofcluster.pl start

The class TProof manages a PROOF session. A PROOF cluster is specified by a user login name, the name of the PROOF master node, and its according xrootd port: 

<user_login_name>@<PROOF_master_name>:<port>

When starting a PROOF cluster with the default settings at the NAF, the PROOF master will be your login node. To connect to the PROOF cluster and create a PROOF session, one can execute the following: 

root -l
TProof* proof = TProof::Open("<user_login_name>@localhost:<port>")

or, alternatively, create a PROOF manager (TProofMgr) and then create a session: 

root -l
TProofMgr* mgr = TProofMgr::Create("<user_login_name>@localhost:<port>") ;
TProof* proof = mgr->CreateSession() ;

The use of "proofcluster.pl start" will generate in addition the ROOT script "connectproof.C". This file contains the correct host name and port number for the PROOF master. One can start a PROOF session by simply executing this script: 

root -l /scratch/current/atlas/`whoami`/proofcluster/connectproof.C  # creates a PROOF session (gProof)

Then, if you want to modify the current session:

It is also possible to create a PROOF manager from a TProof instance: 

TProofMgr* mgr = gProof->GetManager()

Then:

How to run a PROOF-based analysis

This section assumes you already have a running PROOF cluster associated with your user name and with the PROOF master being your login node.

The TSelector

The ROOT class TSelector provides a framework for data analysis by managing the initialization, the event processing, and the termination. An analysis code inheriting from the TSelector has to be used when processing data with PROOF.

The documentation relative to the development of a TSelector based analysis can be found at this URL:

http://root.cern.ch/drupal/content/developing-tselector

In the TSelector class, the methods SlaveBegin() and SlaveTerminate() are executed on each slave, before and after the event processing, respectively. The method Process() is executed for each event, on each worker, and does not load the event by default, i.e., one has to call GetEntry(entry_index) or load exclusively the entries of the desired branches (to increase performance).

Running on D3PD files

Some D3PD files (ROOT-readable files for ATLAS analysis) are available at this path: 

/scratch/current/atlas/calfayan/ADT09/D3PD

These D3PD files have been generated via TauDPDMaker (using v14.2.25 AODs), and embody the analysis tree "ControlSample0".

Simple example

How to manage output objects (histos, trees, ...) in a PROOF analysis

The following is an example. Such objects could be declared as attributes of the analysis class, be instanciated in SlaveBegin(), and filled in Process(). To be able to retrieve them after processing, they could be 'booked' in SlaveBegin() as output objects.

In ControlSample0.h: 

#include "TH1F.h"

class ControlSample0 : public TSelector {
 // ...
 TH1F* htest ;
 // ...
} ;

In ControlSample0.C: 

#include <iostream>
using namespace std ;

void ControlSample0::SlaveBegin(TTree * /*tree*/) {

  // Instanciate objects

  htest = new TH1F("htest", "htest", 100, 0, 100000) ;

  // Book all objects defined in current TDirectory

  TList* obj_list = (TList*) gDirectory->GetList() ;
  TIter next_object((TList*) obj_list) ;
  TObject* obj ;

  cout << "-- Booking objects:" << endl;
  while ((obj = next_object())) {
    TString objname = obj->GetName() ;
    cout << " " << objname << endl ;
    fOutput->Add(obj) ;
  }

}

Bool_t ControlSample0::Process(Long64_t entry) {

  // Load entry
  int nb = GetEntry(entry) ;

  htest->Fill(nb) ;

}

To retrieve the objects after processing (TProof::Process() is over) and store them within a root file, it is then possible to use the following code: 

    // Define output file
    TFile* output_file = new TFile("output.root", "recreate") ;

    // Retrieve objects                                                         
    TList* list = gProof->GetOutputList() ;
    TIter next_object((TList*) list);
    TObject* obj ;
    cout << "-- Retrieved objects:" << endl ;
    output_file->cd() ;
    while ((obj = next_object())) { TString objname = obj->GetName() ; cout << " " << objname << endl ; obj->Write() ; }

    // Write output file
    output_file->Write() ;

With the preceding method, problems may occur if your output is too large. To cope with this issue, please refer to this documentation: http://root.cern.ch/drupal/content/handling-large-outputs-root-files

How to write the log file of the master and each worker to one text file

In a ROOT session, afer having processed your analysis with PROOF: 

    // get proof manager (if not already available)
    TProofMgr* mgr = gProof->GetManager() ;

    // get proof logs                                                           
    TProofLog *log = mgr->GetSessionLogs() ;

    // save log                                                             
    int flag = log->Save("*", "./log_all-workers.txt") ;

Running on AOD files

To be able to run on AOD files with ROOT, the Athena package AthenaROOTAccess can be utilized. In order to use it together with PROOF, the package ara_analysis has been written in order to simplify the process and enable either Python or compiled C++ analysis loops. It is available (and already compiled) at: 

/scratch/current/atlas/calfayan/ADT09/proof/ara_proof.tar.gz

To try it, we will use Athena v15.5.0. Please set up the Athena release without the local flag: 

mkdir -p ~/atlas/testarea/15.5.0
cd ~/atlas/testarea/15.5.0
source ~/cmthome/setup.sh -tag=15.5.0

Please ensure you use a PROOF cluster that has been set up with the ROOT version of your current Athena framework: 

~calfayan/scripts/proofcluster.pl stop
~calfayan/scripts/proofcluster.pl start

Then, install ara_proof: 

cd ~/atlas/testarea/15.5.0/
tar xfz /scratch/current/atlas/calfayan/ADT09/proof/ara_proof.tar.gz
cd ara_proof/ara_analysis/cmt
cmt config
source setup.sh
#make # only if you want to recompile it
cd ../scripts
sh generate_profiles.sh

The package ara_analysis contains the following classes:

To test the framework, you can use the script "start_ara_proof.C", which launches the analysis: 

mkdir -p ~/atlas/testarea/15.5.0/run
cd ~/atlas/testarea/15.5.0/run
cp /scratch/current/atlas/calfayan/ADT09/proof/start_ara_proof.C .
root -l start_ara_proof.C

Per default, the script "start_ara_proof.C" uses "ara_selector_cpp". To use the Python version of the event loop, please use: 

gProof->Process(set, "ara_selector_py") ;

Per default, the script "start_ara_proof.C" provides the PROOF instance with the local path to the libraries of ara_analysis. Since all workers are able to see the your login node, it is possible to avoid sending the complete package to the workers (the directory "ara_proof/ara_analysis/python/" is still uploaded to avoid hardcoding absolute paths in the ara_analysis package). It is also possible to send the complete package to the workers and compile it automatically on each of them, but this takes more time. To do so, please change "start_ara_proof.C" as follows: 

//gSystem->Exec("tar cfzh ara_proof.par ../ara_proof/ara_analysis/python") ;
gSystem->Exec("tar cfzh ara_proof.par --exclude=../ara_proof/ara_analysis/i686-slc4-gcc34-opt/* ../ara_proof/") ;

For more information on how to upload and enable additional software with PROOF, please refer to the following documentation: http://root.cern.ch/drupal/content/preparing-uploading-and-enabling-additional-software

Links

ATLAS: WorkBook/NAF/ADT09PROOF (last edited 2009-09-23 10:58:19 by philippecalfayan)