Pegasus WMS and NERSC Dirac

The Engage VO’s recently begun working with Duke Chemistry on a protein folding simulation  that’s taking us to interesting places.

In particular, deep inside the National Energy Research Scientific Center – NERSC and specifically, to their 50 node General Purpose Graphics Processor (GPGPU) cluster, Dirac. You know you’re having a good time when the cluster you’re using is at a cutting edge DOE research lab and they consider it experimental.

As noted in previous posts, performance for many molecular dynamics applications is greatly improved on GPUs. We’re using NAMD which provides pre-built binaries for GPUs. The binaries ship with dynamic libraries for CUDA, its only external dependency. As such, it’s practical for use on the Open Science Grid since it can be relocated to new clusters trivially.

NAMD, like Amber and many scientific codes, especially ones that simulate natural phenomena, supports restart. That is, output from one run can be used as input to another to continue the calculation. This is important since ultimately we’re modeling a biological system that doesn’t have a particular endpoint – it just keeps going. It also has implications for the shape of our workflow. In this case we have one computation activity – execute NAMD – which we’d like to repeat, feeding the input from one run into the next.

We used Pegasus WMS as the wrokflow execution engine. There are lots of reasons for this but here are some of the basics:

  • It’s state of the art for creating a useful layer of workflow abstraction over the OSG’s runtime infrastructure.
  • The support list is incredibly helpful.
  • It allows me to use my personal OSG/DOE X.509 user credential which is required by NERSC policy.

Each compute job stages in

  • The GPU accelerated version of NAMD
  • The input data files
  • A statically linked version of MPICH2
  • The ouptut file from the previous run if one exists

Output is written from this job, archived and transferred back to the submit host.

One of the interesting aspects of this is how to generate workflows with these characteristics. Each compute job is somewhat logically self contained. We’d like to be able to repeat them in chains and otherwise organize them hierarchically – for example – to do parallel simulations, each of which consists of a chain of compute jobs. More on this in a later post.

Another challenge, from a workflow maintenance and configuration perspective is how to get very hardware specific in our job specifications while keeping the workflow abstract. Here’s the job specification for running NAMD on Dirac’s Tesla GPUs.

<!– part 3: Definition of all jobs/dags/daxes (at least one) –>
<job id=”1n3″ namespace=”namd-flow.0″ name=”namd”>
<argument>–model=ternarycomplex119819 –config=ternarycomplex_popcwimineq-05 –slice=0 –namdType=CUDA –runLength=100</argument>
<profile namespace=”globus” key=”queue”>dirac_reg</profile>
<profile namespace=”globus” key=”xcount”>8:tesla</profile>
<profile namespace=”globus” key=”maxWallTime”>240</profile>
<profile namespace=”globus” key=”jobType”>mpi</profile>
<profile namespace=”globus” key=”host_xcount”>1</profile>
<profile namespace=”condor” key=”x509userproxy”>/home/scox/dev/grayson/var/proxy/x509_proxy_scox</profile>
<uses name=”mpich2-static-1.1.1p1.tar.gz” link=”input”/>
<uses name=”cpuinfo” link=”input”/>
<uses name=”beratan-0.tar.gz” link=”input”/>
<uses name=”namd.tar.gz” link=”input”/>
<uses name=”out-0.tar.gz” link=”output”/>

Again, more on how we’re managing this in a later post.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s