An OSG Map

Finding OSG Stuff

The Open Science Grid is a large global collaboration. Thousands of computing resources are shared through its protocols and conventions. While this is good news, it’s not always a simple matter to understand how the resources are connected and what’s available.

In the process of setting up the RENCI Blueridge OSG Compute Element (CE) it has been useful to compare its settings to established CEs. It’s especially helpful to compare it to somewhat similar CEs. For example, there’s not much point comparing the configuration of my Tier-3 CE to anything running Atlas. So making the network easily navigable makes it easier for newcomers.

In a separate activity, I’ve been running parallel (MPI) jobs on OSG. This is a process that’s still in the early stages. As a result it’s often important to know the physical characteristics of the target execution host in a greater level of detail than we’d ordinarily want to. When I’ve targeted a half dozen clusters a searchable, dynamic hypertree browsing OSG is a somewhat friendlier interface than ldapsearch. Don’t get me wrong, ldapsearch is great. It was a big help in finding resources. But so is the hypertree.

These and other needs suggested it would be useful to have a visual map of the OSG.

Surely, This Exists

Of course, there are already many ways to interact with this data.

Within the OSG, the Berkeley Database Information Index (BDII) is used to coalesce information gathered about Compute Elements and Storage Elements (SE) via the CEMon and GIP components of the Virtual Data Toolkit (VDT). This is great data but it’s not visual, doesn’t aim to explain questions of topology and isn’t filterable.

There are also fairly ambitious monitoring and management tools including the Availability Computation Enginge and Grid Infrastructure Monitoring System Based on Nagios which seek to provide something akin to a network monitoring platform for grid services.

By contrast, the OSG Grid Map is best described as a simple searchable visualization of the OSG BDII. It seeks to add no new information to that data store. But it provides a visual representation of it that may be helpful in conceptualizing the system’s topology.


So the first time I loaded the topology I had that unfortunate feeling that tends to happen when you look at a visualization tool that’s no match for the thing we want to visualize. It looked like a rapidly emptied box of vermicelli.

But a few good things came of that. First, I realized, that’s because the structure of the OSG’s compute elements and sub-clusters is a little like a rapidly emptied box of vermicelli. YMMV. The spots in the visualization where multiple red nodes (CEs) connect to the same item on the outer perimeter (sub-clusters) indicate that multiple CEs may submit jobs to the same sub-cluster. I then added search, a clickable menu dynamically built from adjacent tree nodes and started clicking on things. So here are a few views of OSG territories.

One thing to notice is that the detailed attribute information for sub-clusters is available. It’s hidden by default but clicking on the sub-cluster name reveals the details.


You never know who will read these things. So even though the foregoing discussion is pretty geeky, I’ll take it down yet another level here for those who care about such things. The current approach is as follows:

  • The map is interactive –
    • Searches narrow the scope of the displayed toplogy
    • Clicking a nodes’ label centers that node
    • Subcluster details are expandable in the navigation pane
  • A python script
    • Reads the XML representation of the BDII index.
    • Reads detailed CEMon/GIP reports for each cluster
    • Writes a JSON structure representing the topology.
  • An (X)HTML page houses
    • The Infoviz JavaScript visualization toolkit.
    • JQuery
    • The OSG topology map written by the python script
    • A small JavaScript application using the above tools
  • Limitations:
    • Same Origin Policy: Web browsers require JavaScript to establish connections to the host from which it was served. Since the datasource is BDII, browser side JavaScript can’t build the topology. If this were served from the OSG BDII site, it should be possible to do the whole thing without Python or changes to server side code by reading the existing XML index and associated CEMon data.
    • It uses the HTML 5 Canvas element. Seems to work in all modern browsers.
    • Of course, loading the entire topology wont scale indefinitely. Breaking it up in a way that does scale should be doable without too much headache.
    • It’s not real-time and I don’t have plans to update it frequently.


There’s no particular conclusion outside of the thing itself. Give it a try and feel free to leave your opinions as comments. If you know of something that already does this I’d like to hear about that too.

This entry was posted in Compute Grids, Engage VO, grid, High Throughput Computing (HTC), OSG, RENCI. Bookmark the permalink.

2 Responses to An OSG Map

  1. Soichi Hayashi says:

    Hi. Great article!

    I am a software engineer at Grid Operations Center for OSG. I develop tools such as OIM, MyOSG, etc.. Currently, I am R&Ding a data discovery tool for BDII and making it available on MyOSG. I currently have something like following >

    My goal is to either greatly expand this page, or provide entirely different set of pages that allow users to easily navigate and discover real time data available in BDII, then let them download / subscribe to the data so that they can do subsequent analysis of the data.

    I’d like to get in touch with you and discuss how I can provide such tool.. Are you interested?


  2. soichih says:

    I forgot to click “Notify me of follow-up comments via email.” on my previous comment… FYI.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s