iRODS in the Duke, NERSC, RENCI Collaboration

Research Data and Computation Scenario

Here are some basic characteristics of the problem space we’re addressing in our collaboration with Duke Physics and NERSC:

  • The Duke research team will run OSG grid jobs manipulating terabytes of data.
  • The team will use iRODS as the logical data grid infrastructure.
  • The collaboration will use an HPSS storage facility at NERSC.
  • NERSC will have full administrative control within its zone.
  • Duke Physics will have full administrative control within its zone.

Initially fewer than ten researchers will be involved but the model will be developed to scale to accommodate larger groups. Each researcher will have a grid certificate and Virtual Organization (VO) membership.

Files will generally be smaller than 3GB in size.

Initially, workflows will be created using shell scripts. As design progresses there will likely be applications for iRODS iRules also. Longer term, a visual workflow management system like Kepler will likely prove useful.

iRODS Basics

iRODS is a mature distributed data management system. The iRODS administrative database is called the iCAT. iRODS servers connected to each other in a network with a single iCAT enabled server are called a zone or data grid. The zone terminology is pervasive in the commands used to administer iRODS.

There are several available client interfaces to an iRODS data grid. The most fundamental is called iCommands. This is a set of command line utilities that resemble classic Unix file manipulation tools. These include ils, imkdir, irm, ichmod and so on. There are also commands with a distinctly iRODS flavor like iput, iget, ibun and the administrative interface controlled through iadmin.

System Context

Now that we’ve introduced some key terminology, here’s a system context diagram illustrating the connectivity of the pieces.

The Workflow

Here’s a preliminary design of the workflow envisioned for the collaboration.

iRODS Federation

iRODS provides the ability to view data from one zone within another zone. This is referred to as federation. The process involves using the iRODS mkzone command specifying an iRODS server as a remote zone. In this use case, separate administrative domains have a degree of access to each others’ data. Administrators in each zone are able to create and manage users and access control lists to maintain control of authentication and authorization.

Duke and NERSC will federate their respective iCAT enabled iRODS servers. This federated configuration will allow Duke researchers to put data objects into the NERSC iRODS zone while connected to the Duke iRODS server. All storage associated with paths relative to the NERSC zone will be physically located at NERSC and reside in the HPSS system.

Instructions on federating zones are available here. At a high level, the process involves creating a zone which points to a remote iRODS server. The contents of that server then become available within the context of the local iRODS server.

[scox@scox:~]$ iadmin mkzone zoneA remote
[scox@scox:~]$ iadmin mkuser rods#zoneA rodsuser
iRODS users can be created at the local server by the administrator. Their access to data can be configured at a granular level using the ichmod command. In particular, until the iRODS administrator at a federated site (eg, NERSC) grants write access to a folder in the federated zone, remote users will not be able to write files.

Grid Security Infrastructure (GSI) Authentication

As with other components of the grid infrastructure, authentication becomes a concern in this environment. We’d like to reuse the well-understood authentication schemes grid infrastructure is invested in – i.e. the Grid Security Infrastructure (GSI). Happily, iRODS appears to provide native support for GSI authentication.

The build environment for iRODS provides a script called irodssetup which provides a menu driven dialog (with intelligent defaults) for configuring a server. One of the options is GSI authentication. I selected this for my build. Here are the details of the portion of that interaction specific to GSI:


Setup, Configuration / Build

scox@engage-ce:/opt/iRODS/trunk/iRODS$ ./irodssetup
iRODS configuration setup
---------------------------------------------------------------- [snip]...

iRODS can make use of the Grid Security Infrastructure (GSI)
authentication system in addition to the iRODS secure
password system (challenge/response, no plain-text).
In most cases, the iRODS password system is sufficient but
if you are using GSI for other applications, you might want
to include GSI in iRODS.  Both the clients and servers need
to be built with GSI and then users can select it by setting
irodsAuthScheme=GSI in their .irodsEnv files (or still use
the iRODS password system if they want).

Include GSI [yes]? 

The GLOBUS_LOCATION and the 'install type' is needed to find include
and library files.  GLOBUS_LOCATION specifies the directory where
Globus is installed (see Globus documentation).  The 'install type' is
which 'flavor' of installation you want to use.  For this, use the,
exact name of one of the subdirectories under GLOBUS_LOCATION/include.

You also need to set up your Globus GSI environment before running

GLOBUS_LOCATION [/osg/2/osg-1.2.11/globus]? 

Available types appear to be: gcc32  gcc32dbg  gcc32dbgpthr  gcc32pthr

GSI Install Type to use [gcc32]?


Once that’s done, I created ~/.irods/.irodsEnvScox with the following contents:

irodsHost ''
irodsPort 1247
irodsDefResource 'demoResc'
irodsHome '/tempZone/home/scox'
irodsCwd '/tempZone/home/scox'
irodsUserName 'scox'
irodsZone 'tempZone'
irodsAuthScheme 'GSI'

Established a grid proxy using grid-proxy-init and created a new iRODS user as follows:

$ iadmin mkuser scox#tempZone rodsuser "/DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595"

Set required iRODS environment variables:

$ export irodsEnvFile=~/.irods/.irodsEnvScox
$ export irodsAuthScheme=GSI

And ran a couple of iRODS commands.

scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils
scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a
scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils
scox@engage-ce:/opt/iRODS/trunk/iRODS$ irm a
scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils

The next morning I tried to run an iput command knowing my grid proxy had expired:

scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a
DEBUG: Client side:GSS-API error acquiring credentials: GSS Major Status: General failure
DEBUG: Client side:GSS-API error acquiring credentials: GSS Minor Status Error Chain:
Error with GSI credential
Error with gss credential handle
Error with credential: The proxy credential: /tmp/x509up_u1143
      with subject: /DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595/CN=1263643209
      expired 290 minutes ago.

igsiSetupCreds failed with error -935000 GSI_ERROR_ACQUIRING_CREDS
scox@engage-ce:/opt/iRODS/trunk/iRODS$ grid-proxy-init
Your identity: /DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595
Enter GRID pass phrase for this identity:
Creating proxy .............................................................................. Done
Your proxy is valid until: Thu Sep  9 21:01:22 2010
scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a

Granular Access Control Lists (ACL)

Here’s a brief set of transitions that demonstrate the use of access control lists in iRODS. In this scenario, the administrator at zoneB is manipulating the ACL of a document called nersc-test.

The sequence shows that

  • Initially, only the local admin has access to the document.
  • There is a user, rods#zoneA belonging to a federated zone.
  • The admin grants read on nersc-test to rods#zoneA using ichmod.
  • The admin grants write on nersc-test to rods#zoneA.
  • The admin revokes rods#zoneA’s access to nersc-test entirely.
scox@engage-ce:~$ ils -A nersc-test
        ACL - rods#zoneB:own
scox@engage-ce:~$ iadmin lu
scox@engage-ce:~$ ichmod read rods#zoneA nersc-test
scox@engage-ce:~$ ils -A nersc-test
        ACL - rods#zoneA:read object   rods#zoneB:own
scox@engage-ce:~$ ichmod write rods#zoneA nersc-test
scox@engage-ce:~$ ils -A nersc-test
        ACL - rods#zoneA:modify object   rods#zoneB:own
scox@engage-ce:~$ ichmod null rods#zoneA nersc-test
scox@engage-ce:~$ ils -A nersc-test
        ACL - rods#zoneB:own


Once the two iRODS data grids are up and federated with one another, it is interesting to Duke researchers to be able to initiate programmatic workflows based on events occurring within iRODS.  While it’s not an immediate term target, the iRODS rule system is tailor made for this kind of requirement.

Here’s documentation of iRODS’ rule system.

Another concept to be aware of is iRODS micro-services. A micro-service is a small, well defined algorithm within iRODS, exposed to the the rule interface. That is, micro-services are basically functions which can be called and combined into programs.

Without further adieu, here’s an iRODS rule:


myrule||delayExec(“<PLUSET>3s</PLUSET><EF>3s REPEAT 5 TIMES</EF>”,msiExecCmd(“mycmd”,”hello”,null,null,null,*CmdOut)##msiGetStdoutInExecCmdOut(*CmdOut,*StdoutStr)##writeLine(stdout,*StdoutStr),nop)|nop





The main interesting aspects of this rule are:

delayExec: This causes a micro-service to be called on a repeated, scheduled basis. The first parameter defines the frequency of the repeat loop. Documentation on the syntax of this parameter is at the linked document.

msiExecCmd: This invokes an external program. The program must be executable by the user running iRODS and be in the path of the shell that launched the iRODS server.

This will cause the program mycmd to be executed periodically, according to the first parameter to delayExec. The invoked program can be a shell script which, in turn contains iCommands. Of course, this could be done with a cron job invoking the same shell script. To some extent, we’ll have to learn more about the workflow and overall environment to help make that decision.

Next Steps

We now have clearer requirements and a preliminary workflow.

Next steps are:

  • NERSC to install and configure iRODS with GSI and HPSS support.
  • Duke installing and configuring iRODS with GSI support.
  • RENCI to install and configure iRODS with GSI support.
  • Federation tests to ensure the data grid is configured to support requirements.
  • Continue analysis to determine scripts etc, required to implement workflow.




This entry was posted in Compute Grids, Data Grids, Globus, grid, iRODS, OSG. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s