Engage Submit Host Architecture

Engage users log in to the submit host to run jobs on the OSG. It needs to move to a platform with better administrative support. It’s time to do some archaeology to find out what’s there then build a new submit host. Here goes:

The submit host has quite a bit of software installed and running on it. The rest of this post is dedicated to detailing what it is and how it’s configured. For now, let’s take a look at why we bother having it to start with.

Users log in via the secure shell (SSH) protocol. There, they create scripts which will be run as programs on the Open Science Grid (OSG).

They use Condor tools like condor_submit to submit jobs. These jobs target grid resources using the Condor ClassAd construct. ClassAds are attributes of grid resources which allow jobs to select execution sites which will suit them best.

Condor on the submit node is enhanced with the OSG MatchMaker. MatchMaker runs as a daemon on the submit node which provides improved targeting. The targeting enhancements are allowed by interaction with Engage Central, a node running MatchMaker verification tasks. These tasks interrogate and characterize OSG nodes. It then injects ClassAds into Condor which characterize the resources to enable better targeting.

Virtual Data Toolkit (VDT) services running on the submit provide the OSG PKI security infrastructure. This involves keeping certificate authority (CA) files up to date, getting Certificate Revocation Lists (CRL) as well as providing the ability to create a short lived proxy certificate for interaction with the grid.

The VDT stack is also used to provide a GridFTP (gsiftp://) server. This allows jobs running on OSG to use globus-url-copy to fetch data from the submit node and also to copy result data back from execution sites to the node from which the job was launched.

Configuration Approach

Here’s an overview of steps to configure the new engage submit host:

  • Install OSG Virtual Data Toolkit (VDT) 1.2.15
    • Condor (batch scheduler)
    • GridFTP
    • Certificates
    • Enable fetch-crl, update-certs and log rotation
  • Install and Configure OSGMM
    • Create an osgmm Unix account
    • Import configuration from engage-submit
    • Add OSGMM daemon to condor configuration
  • User Administration
    • Create user accounts
    • Install quotatool
    • Configure user storage quotas
  • Set hight ulimits for open files, user processes, etc.
  • Configure periodic tasks (via cron) to
    • Create home directories automatically
    • Maintain the Condor Queue
    • Kill old gridftp processes
    • Backup the system’s configuration
    • Warn of disk quota overruns
    • Update disk quota settings
  • Configure system services for the init daemon

Install the Virtual Data Toolkit (VDT)

VDT components are installed using the same osg library used in the installation of the Blueridge CE. The RCI Toolkit provides a number of functions which use the VDT package manager (pacman) to download and install components. The toolkit is not likely to be of interest to experienced OSG systems administrators but may be informative for admins newer to the OSG.

For this installation two new flags were introduced to osg_install_all for the following final form of the command:

osg_install_all --condor --isce=false

The condor flag causes the Condor batch system to be installed. Setting isce to false skips configuration steps unique to an OSG compute element gateway node.

The VDT’s services are configured as follows:


[root@engage-submit2:/opt/osg/current]$ vdt-control --list
Service                 | Type   | Desired State
------------------------+--------+--------------
fetch-crl              | cron    | enable        
vdt-rotate-logs        | cron    | enable        
vdt-update-certs       | cron    | enable        
globus-gatekeeper      | inetd   | do not enable 
gsiftp                 | inetd   | do not enable 
mysql5                 | init    | do not enable 
globus-ws              | init    | do not enable 
gums-host-cron         | cron    | do not enable 
MLD                    | init    | do not enable 
condor-cron            | init    | do not enable 
apache                 | init    | do not enable 
tomcat-55              | init    | do not enable 
condor                 | init    | enable        
gratia-condor          | cron    | do not enable 
edg-mkgridmap          | cron    | do not enable 

Install OSGMM

OSGMM (/home/osgmm) was copied from the old machine to the new to preserve all settings.

The condor configuration for OSGMM was also transferred:

# osgmm config

NEGOTIATOR_INTERVAL= 30
NEGOTIATOR_CYCLE_DELAY = 30
SCHEDD_INTERVAL = 30

#NEGOTIATOR_INTERVAL = 25
#SCHEDD_INTERVAL = 60
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=4000
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=30
NUM_VIRTUAL_MACHINES=0

## DAGMAN
DAGMAN_MAX_SUBMIT_ATTEMPTS = 12
DAGMAN_MAX_JOBS_IDLE = 500
DAGMAN_MAX_JOBS_SUBMITTED = 10000
#DAGMAN_SUBMIT_DELAY = 1

# Add the OSG Match Maker to the daemons managed by Condor.
DAEMON_LIST = $(DAEMON_LIST), OSGMM
OSGMM = /home/osgmm/installs/current/sbin/osgmm-wrapper

# To run the OSGMM as a different user account, uncomment this line and modify
# it to replace osgmm with the user in question.  The account must have a
# usable home directory.
OSGMM_USERID = osgmm

# If you are running a more complex security configuration with the OSGMM
# running as a different user, the following settings should allow the OSG Match
# Maker the access it requires.
ALLOW_WRITE = $(ALLOW_WRITE),$(OSGMM_USERID)@*/$(FULL_HOSTNAME)
SEC_DAEMON_AUTHENTICATION_METHODS = FS, $(SEC_DAEMON_AUTHENTICATION_METHODS)

Configure Ulimits

After these commands:

ulimit -n 20000
ulimit -u 65600

The machine’s ulimits are as follows:

[root@engage-submit2:/srv/backups]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 270336
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 20000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65600
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

These values are equal or higher than the original host’s settings.

Configure Disk Quotas

Below, we

  • Turn on quotas for the /home partition
  • Run the update quota script to set up the actual quotas
  • Have a look at the configured quotas

[root@engage-submit2:/root]$ emacs /etc/fstab
...
/dev/VGdata/LVhome      /home      ext3    defaults,usrquota        1 2
...
[root@engage-submit2:/root]$ touch /home/aquota.user
[root@engage-submit2:/root]$ chmod 600 /home/aquota.user 
[root@engage-submit2:/root]$ ls -lisa /home/aquota.user 
49153 0 -rw------- 1 root root 0 Dec  6 11:17 /home/aquota.user
[root@engage-submit2:/root]$ mount -o remount /home
[root@engage-submit2:/root]$ mount
...
/dev/mapper/VGdata-LVhome on /home type ext3 (rw,usrquota)
...
[root@engage-submit2:/root]$ quotacheck -vgum /home
...
done
quotacheck: Checked 382 directories and 577 files
[root@engage-submit2:/root]$ quotaon -av
/dev/mapper/VGdata-LVhome [/home]: user quotas turned on
[root@engage-submit2:/root]$ quota -u scox
Disk quotas for user scox (uid 1143): none
[root@engage-submit2:/root]$ /opt/quota/update-quotas 
[root@engage-submit2:/root]$ quota -u scox
Disk quotas for user scox (uid 1143): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/VGdata-LVhome
                  43956  15360000 20480000             283       0       0    


The quotatool package, used by the update-quota script was not installed by default. After “yum install quotatool”:

[root@engage-submit2:/opt/osg/1.2.13]$ yum info quotatool
Loaded plugins: dellsysid, fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.linux.duke.edu
 * base: mirror.linux.duke.edu
 * epel: archive.linux.duke.edu
 * extras: styx.biochem.wfubmc.edu
 * updates: www.gtlib.gatech.edu
Installed Packages
Name       : quotatool
Arch       : x86_64
Version    : 1.4.11
Release    : 8.el5
Size       : 46 k
Repo       : installed
Summary    : A utility to set filesystem quotas
URL        : http://quotatool.ekenberg.se
License    : GPLv2
Description: Quotatool is a utility to set filesystem quotas from the commandline.
           : It is suitable for use in scripts and other non-interactive situations.

Configure Cron

The following periodic tasks are configured:

  • kill-leftover-gridftp: As the name suggests, stop old gridftp processes.
  • nightly-backup: backup /opt and other system directories
  • quota: issue quota warnings if appropriate
  • update-quotas: calculate and apply per-user quota sizes
  • create-home-directories: Create home directories for users that don’t have one.
  • maintain-condor-q: Remove stale jobs.
[root@engage-submit2:/opt/osg/current]$ ls -lisa /etc/cron.daily/ | grep Dec
 262331  4 drwxr-xr-x   2 root root  4096 Dec  6 14:56 .
 262145 12 drwxr-xr-x 108 root root 12288 Dec  6 11:18 ..
 264409  4 -rwxr-xr-x   1 root root   187 Dec  6 13:51 kill-leftover-gridftp
 264478  4 -rwxr-x--x   1 root root  1719 Dec  6 14:56 nightly-backups
 264410  4 -rwxr-xr-x   1 root root   345 Dec  6 13:36 quota
 264412  4 -rwxr-xr-x   1 root root   820 Dec  6 13:38 update-quotas
 [root@engage-submit2:/opt/osg/current]$ ls -lisa /etc/cron.hourly/ | grep Dec
 262327  4 drwxr-xr-x   2 root root  4096 Dec  6 13:47 .
 262145 12 drwxr-xr-x 108 root root 12288 Dec  6 11:18 ..
 264476  4 -rwxr-xr-x   1 root root   656 Dec  6 13:39 create-home-directories
 264317  4 -rwxr-xr-x   1 root root   733 Dec  6 13:47 maintain-condor-q

Configure System Services

The following services are configured in /etc/init.d:

  • condor: Condor batch scheduler services
  • quota: The disk quota system
  • quotarpc: The disk quota system
  • osgmm: The OSG Match Maker meta-scheduler

Request and Install Certificates

[root@engage-submit2:/opt/osg/1.2.13]$ osg_host_cert_request 
checking CertLib version, V2-7,  This is the latest version, released 18 May 2009.
Processing OU=Services request.
Generating a 2048 bit RSA private key
......................................+++
...............................................................+++
writing new private key to './host-engage-submit2.renci.orgkey.pem'
-----
osg
OSG
OSG:ENGAGE

Your Certificate Request has been successfully submitted
Your Certificate Request id: 67177 

	You will receive a notification email from the CA when your certificate
	has been issued. Please disregard the instructions to download your
	certificate though a web browser and use the cert-retrieve script instead.

[root@engage-submit2:/opt/osg/osg-cert-requests]$ osg_http_cert_request 
checking CertLib version, V2-7,  This is the latest version, released 18 May 2009.
Processing OU=Services request.
Generating a 2048 bit RSA private key
............................+++
.........................................................+++
writing new private key to './host-engage-submit2.renci.orgkey.pem'
-----
osg
OSG
OSG:ENGAGE

Your Certificate Request has been successfully submitted
Your Certificate Request id: 67178 

	You will receive a notification email from the CA when your certificate
	has been issued. Please disregard the instructions to download your
	certificate though a web browser and use the cert-retrieve script instead.
[root@engage-submit2:/opt/osg/osg-cert-requests]$ 

Certificates were installed in /etc/grid-security following standard VDT conventions regarding naming, location, ownership and permissions.

Verify Installation

We’ll be taking the following steps to verify that the new submit host works as expected:

  • Verify job submission
    • Submit jobs exercising
      • Condor
      • OSGMM
      • GRAM
      • GridFTP
    • Monitor job execution with condor tools
    • Monitor job execution with OSGMM’s condor_grid_overview
  • Verify that each of the following cron services runs successfully
    • kill-leftover-gridftp
    • nightly-backup
    • quota
    • update-quotas
    • create-home-directories
    • maintain-condor-q
  • Verify services restarted on reboot
    • Condor
    • OSGMM
    • GridFTP
    • Quota

Here’s a job running on Blueridge submitted from the new submit node.


[scox@engage-submit2:~/dev/cpmemd]$ condor_grid_overview | grep scox
1180       (DAGMan)         scox                               Running     condor_dagman     0:02:23
1181         |-job_1        scox         RENCI-Blueridge       Running     environment.sh    0:01:29


To get running, this job transfers files from the submit node to the OSG worker node. Once it’s done, it transfers output files back to the submit node. Here’s a fragment of it’s standard output:

--(inf): =========================================================================
  /usr/bin/time /osg/osg-data/job.MWOqX16253/app/run/sbin/mpich2/mpiexec.gforker
  -np 16 /osg/osg-data/job.MWOqX16253/app/run/sbin/pmemd/pmemd.mpich2
  -O
  -i run/i14vnphdhf+_hipW1.in
  -c in/i14vnphdhf+_hipW1.crd
  -p in/i14vnphdhf+_hipW1.top
  -x run/out/i14vnphdhf+_hipW1.tra
  -r run/out/i14vnphdhf+_hipW1.rst
  -e run/out/i14vnphdhf+_hipW1.ene
  -o run/out/i14vnphdhf+_hipW1.out
 10611.50user 277.92system 11:32.33elapsed 1572%CPU (0avgtext+0avgdata 0maxresident)k
 0inputs+0outputs (0major+13517668minor)pagefaults 0swaps
 --(inf): --(pmemd-exec): execution time: 0:11:32
 --(inf): ========================================================================
 --(inf): --(end): /osg/osg-data/job.MWOqX16253/app/bin/cpmemd-osg.sh-@-compute-1-4.local-@-Fri Dec 10 11:19:57 EST 2010
 --(inf): total test execution time: 0:11:42
 --(inf): ========================================================================
 --(inf): --[copy] /osg/osg-data/job.MWOqX16253/app/run to /osg/osg-data/job.MWOqX16253/out
 --(inf): staging out /osg/osg-data/job.MWOqX16253/app.stdouterr
 --(inf): executing job cleanup...
 === RUN SUCCESSFUL ===

So this gives us a pretty good idea that the GridFTP server on the submit host is working.

Another important tool on the submit host, this one provided by OSGMM, is called condor_grid_overview. It provides a dashboard like view of all active jobs being managed by the submit host:

[scox@engage-submit2:~/dev/cpmemd]$ condor_grid_overview 

ID         DAG              Owner        Resource              Status      Command       TimeInState
========== ================ ============ ===================== =========== ===
864        (DAGMan)         rynge                              Running     condor_dagman     5:11:28
1184         |-job_318      rynge        RENCI-Engagement      Running     remote-job-wra    0:09:01
1594         |-job_728      rynge        BNL-ATLAS             Pending     remote-job-wra    0:53:22
1599         |-job_733      rynge        BNL-ATLAS             Pending     remote-job-wra    0:53:22
1603         |-job_737      rynge        Firefly               Running     remote-job-wra    0:45:02
1607         |-job_741      rynge        RENCI-Engagement      Running     remote-job-wra    0:49:37
1611         |-job_745      rynge        BNL-ATLAS             Pending     remote-job-wra    0:52:21
1615         |-job_749      rynge        BNL-ATLAS             Pending     remote-job-wra    0:48:19
1619         |-job_753      rynge        RENCI-Engagement      Running     remote-job-wra    0:45:02
1624         |-job_758      rynge        GLOW                  Running     remote-job-wra    0:40:32
1630         |-job_764      rynge        RENCI-Engagement      Running     remote-job-wra    0:36:02
1872         |-job_87       rynge        GLOW                  Pending     remote-job-wra    0:13:07

Site                      Total  Subm Stage  Pend  Run  Other Succes  Rank Comment
========================= ===== ===== ===== ===== ===== ===== ====== ===== =====
BNL-ATLAS                     4     0     0     4     0     0    91%   915 
CIT_CMS_T2                    0     0     0     0     0     0   100%     1 Non-verified site
CIT_CMS_T2                    0     0     0     0     0     0   100%     1 Non-verified site
Clemson-Palmetto              0     0     0     0     0     0   100%     1 Non-verified site
FNAL_FERMIGRID                0     0     0     0     0     0   100%     1 Non-verified site
Firefly                       1     0     0     0     1     0   100%   950 
Firefly                       0     0     0     0     0     0   100%   803 
GLOW                          0     0     0     0     0     0   100%     1 Non-verified site
GLOW                          2     0     0     1     1     0   100%   200 Many jobs pending (>25.0% of running jobs)
GridUNESP_CENTRAL             0     0     0     0     0     0   100%   841 
MIT_CMS                       0     0     0     0     0     0    82%   656 
MIT_CMS                       0     0     0     0     0     0    80%   642 
NWICG_NotreDame               0     0     0     0     0     0   100%     1 Non-verified site
NYSGRID_CORNELL_NYS1          0     0     0     0     0     0   100%     1 Non-verified site
Nebraska                      0     0     0     0     0     0   100%   807 
Nebraska                      0     0     0     0     0     0   100%   814 
Prairiefire                   0     0     0     0     0     0     9%    72 High job failure rate
Purdue-RCAC                   0     0     0     0     0     0    87%   698 
RENCI-Blueridge               0     0     0     0     0     0   100%   801 
RENCI-Engagement              4     0     0     0     4     0   100%   950 
SBGrid-Harvard-East           0     0     0     0     0     0   100%   800 
SPRACE                        0     0     0     0     0     0    93%   748 
SWT2_CPB                      0     0     0     0     0     0   100%     1 Non-verified site
UCR-HEP                       0     0     0     0     0     0   100%   802 
UCSDT2                        0     0     0     0     0     0   100%     1 Non-verified site
UCSDT2                        0     0     0     0     0     0   100%     1 Non-verified site
UConn-OSG                     0     0     0     0     0     0   100%   800 
UFlorida-HPC                  0     0     0     0     0     0   100%   853 
UFlorida-PG                   0     0     0     0     0     0   100%     1 Non-verified site
UJ-OSG                        0     0     0     0     0     0   100%     1 Non-verified site
UMissHEP                      0     0     0     0     0     0    96%   770 
USCMS-FNAL-WC1                0     0     0     0     0     0   100%   802 
UTA_SWT2                      0     0     0     0     0     0   100%     1 Non-verified site
WQCG-Harvard-OSG              0     0     0     0     0     0   100%     1 Non-verified site

12 jobs; 0 matching, 5 pending remotely, 7 running, 0 held, 0 other
[scox@engage-submit2:~/dev/cpmemd]$ 

Next

  • Develop a plan to migrate user data
  • Develop a plan to migrate users
  • Final phase:
    • Update hostname settings in Condor (and any other relevant) configs.
    • Switch DNS and certificates to the new machine for a weekend test only.
    • Validate.
    • Switch DNS to new machine.
    • Keep old machine online at alternate DNS record.
This entry was posted in Compute Grids, condor, Engage VO, Globus, grid, High Throughput Computing (HTC), High Throughput Parallel Computing (HTPC), OSG, RENCI. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s