Finding OSG Stuff
The Open Science Grid is a large global collaboration. Thousands of computing resources are shared through its protocols and conventions. While this is good news, it’s not always a simple matter to understand how the resources are connected and what’s available.
In the process of setting up the RENCI Blueridge OSG Compute Element (CE) it has been useful to compare its settings to established CEs. It’s especially helpful to compare it to somewhat similar CEs. For example, there’s not much point comparing the configuration of my Tier-3 CE to anything running Atlas. So making the network easily navigable makes it easier for newcomers.
In a separate activity, I’ve been running parallel (MPI) jobs on OSG. This is a process that’s still in the early stages. As a result it’s often important to know the physical characteristics of the target execution host in a greater level of detail than we’d ordinarily want to. When I’ve targeted a half dozen clusters a searchable, dynamic hypertree browsing OSG is a somewhat friendlier interface than ldapsearch. Don’t get me wrong, ldapsearch is great. It was a big help in finding resources. But so is the hypertree.
These and other needs suggested it would be useful to have a visual map of the OSG.
Surely, This Exists
Of course, there are already many ways to interact with this data.
Within the OSG, the Berkeley Database Information Index (BDII) is used to coalesce information gathered about Compute Elements and Storage Elements (SE) via the CEMon and GIP components of the Virtual Data Toolkit (VDT). This is great data but it’s not visual, doesn’t aim to explain questions of topology and isn’t filterable.
There are also fairly ambitious monitoring and management tools including the Availability Computation Enginge and Grid Infrastructure Monitoring System Based on Nagios which seek to provide something akin to a network monitoring platform for grid services.
By contrast, the OSG Grid Map is best described as a simple searchable visualization of the OSG BDII. It seeks to add no new information to that data store. But it provides a visual representation of it that may be helpful in conceptualizing the system’s topology.
So the first time I loaded the topology I had that unfortunate feeling that tends to happen when you look at a visualization tool that’s no match for the thing we want to visualize. It looked like a rapidly emptied box of vermicelli.
But a few good things came of that. First, I realized, that’s because the structure of the OSG’s compute elements and sub-clusters is a little like a rapidly emptied box of vermicelli. YMMV. The spots in the visualization where multiple red nodes (CEs) connect to the same item on the outer perimeter (sub-clusters) indicate that multiple CEs may submit jobs to the same sub-cluster. I then added search, a clickable menu dynamically built from adjacent tree nodes and started clicking on things. So here are a few views of OSG territories.
One thing to notice is that the detailed attribute information for sub-clusters is available. It’s hidden by default but clicking on the sub-cluster name reveals the details.
You never know who will read these things. So even though the foregoing discussion is pretty geeky, I’ll take it down yet another level here for those who care about such things. The current approach is as follows:
- The map is interactive –
- Searches narrow the scope of the displayed toplogy
- Clicking a nodes’ label centers that node
- Subcluster details are expandable in the navigation pane
- A python script
- Reads the XML representation of the BDII index.
- Reads detailed CEMon/GIP reports for each cluster
- Writes a JSON structure representing the topology.
- An (X)HTML page houses
- The OSG topology map written by the python script
- It uses the HTML 5 Canvas element. Seems to work in all modern browsers.
- Of course, loading the entire topology wont scale indefinitely. Breaking it up in a way that does scale should be doable without too much headache.
- It’s not real-time and I don’t have plans to update it frequently.
There’s no particular conclusion outside of the thing itself. Give it a try and feel free to leave your opinions as comments. If you know of something that already does this I’d like to hear about that too.