Developing MarlinTPC

Software repository

A subversion server is hosted by DESY IT. It was used for (almost) all ilcsoft projects, including MarlinTPC. Except MarlinTPC, most of the ilc projects are now hosted on GitHub

The web address of the server is https://svnsrv.desy.de/

As the websvn interface was disabled end of 2019, only the pure file tree can be browsed: https://svnsrv.desy.de/public/marlintpc/ Some of the links below therefore may not work anymore, in the web browser. Also the websvn features are no longer available (diff, revision browsing, log messages, ...).

SVN usage via command line or graphical tools should not be affected.

There is a graphical workaround using SmartSVN.

Read-only access

If you want to contribute to the development of MarlinTPC, or simply want to change some code according to your needs, read-only access is not enough. If you are interested see HowToDownloadMarlinTPC

User registration

To have write access with the possible authentication methods, you need first to be registered with the svn server. This is described on https://svnsrv.desy.de/access.html. You simply use the so-called "admin web interface". The recommended authentication method is with a kerberos ticket, then the basic account. Certificate based authentication has some issues on some systems and is therefore discouraged to use, but you can try if it works.

For the kerberos and the certificate authentication, you can register yourself under the addresses:

using kerberos auth     https://svnsrv.desy.de/admin/desy (with valid DESY.DE or IFH.DE ticket)
using auth with certificate     https://svnsrv.desy.de/admin/svn (valid certificate needed)

Basic authentication requires an already registered user, which needs to be created for you. Please ask one of the MarlinTPC svn administrators to register you. After you are registered, you can also access the web admin interface:

using basic authentication      https://svnsrv.desy.de/admin/basic (only after registering a basic account)

Once you are known to the server, you can register for the project "marlintpc" (in the "join project" part). Then one of the admins of the repository will grant you the access.

Working with the repository

After having completed your registration to the SVN server and the MarlinTPC project as a developer, you can start working with the MarlinTPC repository. In the following sections the basic working cycle is described with examples. As most developers are working at DESY or on the NAF, kerberos authentication was chosen for the examples.

Different addresses in svn commands

Depending on your authentication method you will need to specify a different base address for the repository in the svn commands, described below.

You can check the base address to use for an existing local working copy of the trunk or a branch with the command:

svn info

The trunk/branch model

The trunk is the current stable development version. Everyone that is developing some code (including bugfixes, beautification, typos, documentation, etc.) is strongly encouraged to commit the developments to the trunk.

Please make sure that your code is running, test as well as you can.

The minimum requirement is that the code compiles without error messages and warnings.

Creating your personal branch

It is recommended that you create your own development branches within the repository. Within a branch you develop everything, test, commit changes to. This can be achieved by copying the current trunk (or a tagged version) using the following syntax (please specify a log message with the "-m" switch, stating which revision you copied):

svn copy https://svnsrv.desy.de/desy/marlintpc/trunk https://svnsrv.desy.de/desy/marlintpc/branches/MYUSERNAME -m"copied revision 1234 to create my own branch"

MYUSERNAME should be replaced with your favourite user name, e.g. your login-name. If you don't use Kerberos authentication, you will have to adapt the svn URL (see section above).

Each directory in "branches" should start as a copy of the trunk. Please do NOT create sub-structures:

+-tags
+-trunk
+-branches
  +-DirectoryOfD  // should have been created by copying trunk
  | +-TrunkCopy1OfD  // wrong place, put one level higher
  | +-TrunkCopy2OfD  // wrong place, put one level higher
  | +-UnrelatedDataOfD  // wrong place, no relation to trunk
  |
  +-TrunkCopyOfA  // OK
  +-TrunkCopyOfB  // OK
  +-TrunkCopyOfC  // OK

SVN will interpret these directories with sub-structures as completely changed versions of the trunk. Any merging can still be done in SVN by so-called subdirectory merges, but they cause problems in maintaining a clean revision history. Especially third-party tools are easily confused by this.

If you already created such a sub-structured directory in the past, please do NOT try to repair it.

Working with your branch

After creating your development branch, you can check it out, i.e. make a local copy of it:

svn checkout https://svnsrv.desy.de/desy/marlintpc/branches/MYUSERNAME ./MYUSERNAME

"./MYUSERNAME" will create the directory "MYUSERNAME" in your current directory and place the checked out files in it. You can freely choose this path to meet your needs. You work on this copy by editing, adding, deleting files and directories. After a set of work is done or at the end of a day, you commit your changes to the repository. A description how such a basic work cycle is done can be found in the SVN documentation:

Basic Work Cycle

In case you are working on a branch together with other people, they may have committed changes to the branch. You should regularly update your working copy with these changes:

svn update

Merging trunk changes into your branch

As the trunk progresses, your initial copy of the trunk will become outdated. To keep in sync with the recent developments, you have to merge the changes that happened between the time you created your branch (or the last time you merged) and the current version.

The command you need to invoke in your clean working copy of your branch is

svn merge https://svnsrv.desy.de/desy/marlintpc/trunk

Note Change: With the server running a newer SVN version now, you do not have to supply the revision numbers of the creation of your branch or the last merge to the current one (but it's still good to write them in the commit message to keep track yourself), as it was necessary in pre SVN v1.5 versions, where you would have had to write: "svn merge -r1234:1802 https://svnsrv.desy.de/desy/marlintpc/trunk" since SVN couldn't keep track of it itself.

A clean working copy means no local edits (e.g. a recent version that exists also in the repository, in your branch), otherwise the merging can become a nightmare. Only this way no mixing of your local changes and the merges can happen. Also - whenever something went wrong you can always revert the changes!

What happens is in principle a diff is created between your working copy and the specified revisions of the trunk, and then applied to your working copy. This has several possibilities to create a conflict, that can not be resolved automatically. You have to solve these conflicts by hand, please refer to the documentation below.

After you resolved all the conflicts and made sure that your working copy is actually again in working condition, you simply commit the changes. While its not necessary, its' a good idea to include the revision number that you have merged (maybe copy&paste the output line from the svn merge command)

A more detailed and possibly more accurate description can be found at the documentation pages of subversion:

Basic Branch Merging

Resolve Merge Conflicts

Merging your changes back into the trunk

With the new version of SVN running on the server, merging your branch back into the trunk has become much easier:

Make sure all local changes have been committed to your branch and that you had merged the latest trunk version into your branch.

Then you check out the latest revision of the trunk and change into this directory. Again, the path to where you want to check out the trunk can be freely chosen, in the example it is the directory "trunk" in the current directory:

svn checkout https://svnsrv.desy.de/desy/marlintpc/trunk ./trunk
cd trunk

After this, you merge your branch into the local copy of the trunk using the reintegrate option:

svn merge --reintegrate https://svnsrv.desy.de/desy/marlintpc/branches/MYUSERNAME

MYUSERNAME should be replaced with the branch name you chose (see above).

After testing if everything works (i.e. compiles without errors or warnings), you can commit the local copy of the trunk into the repository:

svn commit -m "some meaningful message listing the changes"

Make sure to delete your private development branch afterwards since future changes and merges cannot be porperly reintegrated again after this action. For future developments you have to create a new branch from the trunk as described above (since the old one has been deleted it can of course have the same name).

More information about merge with reintegrate can be found in the SVN documentation:

Reintegrating a Branch

Changing the underlying repository

If you still have a working copy of the source code from the Bonn server, then the output of svn info should read something like:

[$] svn info
Path: .
URL: svn://pi.physik.uni-bonn.de/MarlinTPC/...
Repository Root: svn://pi.physik.uni-bonn.de/MarlinTPC

If you're revision is smaller than r2224 (which should be), then you can simply change the repository (or it's URL) to the new server. Simply issue the following command (if you're using kerberos based authentication):

svn switch --relocate svn://pi.physik.uni-bonn.de/MarlinTPC https://svnsrv.desy.de/desy/marlintpc

For other authentication methods, you have to change the url of the desy svn server accordingly: https://svnsrv.desy.de/svn/marlintpc (certificate based) or https://svnsrv.desy.de/basic/marlintpc (for basic authentication).

Afterwards the svn info command should print out:

[$] svn info
Path:
URL: https://svnsrv.desy.de/desy/marlintpc/...
Repository Root: https://svnsrv.desy.de/desy/marlintpc

Similarly, this should help if you have working copy that uses a different authentication method.

Revision Logging

All processors should register their revision number in the run header parameters of the datasets processed by them. This makes it easy to find out which data need to be re-processed in case a bug is found in a processor.

This only works for SVN, as it replaces the $Rev$ variable. For future versioning systems a different mechanism should be found.

To do that make sure to include something like

void MyProcessor::processRunHeader( LCRunHeader* run ) {

    run->parameters().setValue( _processorName + "_revision", "$Rev$" );

    // additional code might follow here

}

in the processRunHeader method of your processor.

In addition to that line of code you have to enable the "Rev" keyword substitution for the corresponding source file =myprocessor.cc= by executing the command

svn propset svn:keywords Rev myprocessor.cc

You can verify that the file properties are set correctly by using

svn proplist --verbose myprocessor.cc

Parameter Logging

In addition to the revision number, processors should also register the used parameters in the run header parameters of the datasets processed by them.

This is done by adding

void MyProcessor::processRunHeader( LCRunHeader* run ) {

    for ( ProcParamMap::iterator i = _map.begin(); i != _map.end(); i++ ) {
        if ( ! i->second->isOptional() || i->second->valueSet() ) {
            run->parameters().setValue( _processorName + "_" + i->second->name(), i->second->value() );
        }
    }

    // additional code might follow here
}

in the processRunHeader method of your processor.