Research Data and Computation Scenario
Here are some basic characteristics of the problem space we’re addressing in our collaboration with Duke Physics and NERSC:
- The Duke research team will run OSG grid jobs manipulating terabytes of data.
- The team will use iRODS as the logical data grid infrastructure.
- The collaboration will use an HPSS storage facility at NERSC.
- NERSC will have full administrative control within its zone.
- Duke Physics will have full administrative control within its zone.
Initially fewer than ten researchers will be involved but the model will be developed to scale to accommodate larger groups. Each researcher will have a grid certificate and Virtual Organization (VO) membership.
Files will generally be smaller than 3GB in size.
Initially, workflows will be created using shell scripts. As design progresses there will likely be applications for iRODS iRules also. Longer term, a visual workflow management system like Kepler will likely prove useful.
iRODS is a mature distributed data management system. The iRODS administrative database is called the iCAT. iRODS servers connected to each other in a network with a single iCAT enabled server are called a zone or data grid. The zone terminology is pervasive in the commands used to administer iRODS.
There are several available client interfaces to an iRODS data grid. The most fundamental is called iCommands. This is a set of command line utilities that resemble classic Unix file manipulation tools. These include ils, imkdir, irm, ichmod and so on. There are also commands with a distinctly iRODS flavor like iput, iget, ibun and the administrative interface controlled through iadmin.
Now that we’ve introduced some key terminology, here’s a system context diagram illustrating the connectivity of the pieces.
Here’s a preliminary design of the workflow envisioned for the collaboration.
iRODS provides the ability to view data from one zone within another zone. This is referred to as federation. The process involves using the iRODS mkzone command specifying an iRODS server as a remote zone. In this use case, separate administrative domains have a degree of access to each others’ data. Administrators in each zone are able to create and manage users and access control lists to maintain control of authentication and authorization.
Duke and NERSC will federate their respective iCAT enabled iRODS servers. This federated configuration will allow Duke researchers to put data objects into the NERSC iRODS zone while connected to the Duke iRODS server. All storage associated with paths relative to the NERSC zone will be physically located at NERSC and reside in the HPSS system.
Instructions on federating zones are available here. At a high level, the process involves creating a zone which points to a remote iRODS server. The contents of that server then become available within the context of the local iRODS server.
[scox@scox:~]$ iadmin mkzone zoneA remote scox.europa.renci.org:1247 [scox@scox:~]$ iadmin mkuser rods#zoneA rodsuser
Grid Security Infrastructure (GSI) Authentication
As with other components of the grid infrastructure, authentication becomes a concern in this environment. We’d like to reuse the well-understood authentication schemes grid infrastructure is invested in – i.e. the Grid Security Infrastructure (GSI). Happily, iRODS appears to provide native support for GSI authentication.
The build environment for iRODS provides a script called irodssetup which provides a menu driven dialog (with intelligent defaults) for configuring a server. One of the options is GSI authentication. I selected this for my build. Here are the details of the portion of that interaction specific to GSI:
=========================================== Setup, Configuration / Build scox@engage-ce:/opt/iRODS/trunk/iRODS$ ./irodssetup iRODS configuration setup ---------------------------------------------------------------- [snip]... iRODS can make use of the Grid Security Infrastructure (GSI) authentication system in addition to the iRODS secure password system (challenge/response, no plain-text). In most cases, the iRODS password system is sufficient but if you are using GSI for other applications, you might want to include GSI in iRODS. Both the clients and servers need to be built with GSI and then users can select it by setting irodsAuthScheme=GSI in their .irodsEnv files (or still use the iRODS password system if they want). Include GSI [yes]? The GLOBUS_LOCATION and the 'install type' is needed to find include and library files. GLOBUS_LOCATION specifies the directory where Globus is installed (see Globus documentation). The 'install type' is which 'flavor' of installation you want to use. For this, use the, exact name of one of the subdirectories under GLOBUS_LOCATION/include. You also need to set up your Globus GSI environment before running this. GLOBUS_LOCATION [/osg/2/osg-1.2.11/globus]? Available types appear to be: gcc32 gcc32dbg gcc32dbgpthr gcc32pthr GSI Install Type to use [gcc32]? ===========================================
Once that’s done, I created ~/.irods/.irodsEnvScox with the following contents:
irodsHost 'engage-ce.renci.org'irodsPort 1247irodsDefResource 'demoResc'irodsHome '/tempZone/home/scox'irodsCwd '/tempZone/home/scox'irodsUserName 'scox'irodsZone 'tempZone'irodsAuthScheme 'GSI'
Established a grid proxy using grid-proxy-init and created a new iRODS user as follows:
$ iadmin mkuser scox#tempZone rodsuser "/DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595"
Set required iRODS environment variables:
$ export irodsEnvFile=~/.irods/.irodsEnvScox $ export irodsAuthScheme=GSI
And ran a couple of iRODS commands.
scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils /tempZone/home/scox: scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils /tempZone/home/scox: a scox@engage-ce:/opt/iRODS/trunk/iRODS$ irm a scox@engage-ce:/opt/iRODS/trunk/iRODS$ ils /tempZone/home/scox: scox@engage-ce:/opt/iRODS/trunk/iRODS$
The next morning I tried to run an iput command knowing my grid proxy had expired:
scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a DEBUG: Client side:GSS-API error acquiring credentials: GSS Major Status: General failure DEBUG: Client side:GSS-API error acquiring credentials: GSS Minor Status Error Chain: acquire_cred.c:gss_acquire_cred:143: Error with GSI credential globus_i_gsi_gss_utils.c:globus_i_gsi_gss_cred_read:1342: Error with gss credential handle globus_gsi_credential.c:globus_gsi_cred_read:317: Error with credential: The proxy credential: /tmp/x509up_u1143 with subject: /DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595/CN=1263643209 expired 290 minutes ago. igsiSetupCreds failed with error -935000 GSI_ERROR_ACQUIRING_CREDS scox@engage-ce:/opt/iRODS/trunk/iRODS$ grid-proxy-init Your identity: /DC=org/DC=doegrids/OU=People/CN=Steven Cox 318595 Enter GRID pass phrase for this identity: Creating proxy .............................................................................. Done Your proxy is valid until: Thu Sep 9 21:01:22 2010 scox@engage-ce:/opt/iRODS/trunk/iRODS$ iput ~/a scox@engage-ce:/opt/iRODS/trunk/iRODS$
Granular Access Control Lists (ACL)
Here’s a brief set of transitions that demonstrate the use of access control lists in iRODS. In this scenario, the administrator at zoneB is manipulating the ACL of a document called nersc-test.
The sequence shows that
- Initially, only the local admin has access to the document.
- There is a user, rods#zoneA belonging to a federated zone.
- The admin grants read on nersc-test to rods#zoneA using ichmod.
- The admin grants write on nersc-test to rods#zoneA.
- The admin revokes rods#zoneA’s access to nersc-test entirely.
scox@engage-ce:~$ ils -A nersc-test /zoneB/home/rods/test/nersc-test ACL - rods#zoneB:own scox@engage-ce:~$ iadmin lu rods#zoneA rodsBoot#zoneB rods#zoneB scox#zoneB scox@engage-ce:~$ ichmod read rods#zoneA nersc-test scox@engage-ce:~$ ils -A nersc-test /zoneB/home/rods/test/nersc-test ACL - rods#zoneA:read object rods#zoneB:own scox@engage-ce:~$ ichmod write rods#zoneA nersc-test scox@engage-ce:~$ ils -A nersc-test /zoneB/home/rods/test/nersc-test ACL - rods#zoneA:modify object rods#zoneB:own scox@engage-ce:~$ ichmod null rods#zoneA nersc-test scox@engage-ce:~$ ils -A nersc-test /zoneB/home/rods/test/nersc-test ACL - rods#zoneB:own scox@engage-ce:~$
Once the two iRODS data grids are up and federated with one another, it is interesting to Duke researchers to be able to initiate programmatic workflows based on events occurring within iRODS. While it’s not an immediate term target, the iRODS rule system is tailor made for this kind of requirement.
Here’s documentation of iRODS’ rule system.
Another concept to be aware of is iRODS micro-services. A micro-service is a small, well defined algorithm within iRODS, exposed to the the rule interface. That is, micro-services are basically functions which can be called and combined into programs.
Without further adieu, here’s an iRODS rule:
myrule||delayExec(“<PLUSET>3s</PLUSET><EF>3s REPEAT 5 TIMES</EF>”,msiExecCmd(“mycmd”,”hello”,null,null,null,*CmdOut)##msiGetStdoutInExecCmdOut(*CmdOut,*StdoutStr)##writeLine(stdout,*StdoutStr),nop)|nop
The main interesting aspects of this rule are:
delayExec: This causes a micro-service to be called on a repeated, scheduled basis. The first parameter defines the frequency of the repeat loop. Documentation on the syntax of this parameter is at the linked document.
msiExecCmd: This invokes an external program. The program must be executable by the user running iRODS and be in the path of the shell that launched the iRODS server.
This will cause the program mycmd to be executed periodically, according to the first parameter to delayExec. The invoked program can be a shell script which, in turn contains iCommands. Of course, this could be done with a cron job invoking the same shell script. To some extent, we’ll have to learn more about the workflow and overall environment to help make that decision.
We now have clearer requirements and a preliminary workflow.
Next steps are:
- NERSC to install and configure iRODS with GSI and HPSS support.
- Duke installing and configuring iRODS with GSI support.
- RENCI to install and configure iRODS with GSI support.
- Federation tests to ensure the data grid is configured to support requirements.
- Continue analysis to determine scripts etc, required to implement workflow.