spilth.org

NYU Geospatial Work

Thu, 28 May 2026 00:00:00 GMT

I just wrapped up a 2-year stint with New York University’s Digital Library department, mostly helping them with their geospatial data but also working on some ancillary projects as well.

GeoBlacklight

I was initially brought on to help with their GeoBlacklight instance, the NYU Spatial Data Repository. GBL is a Ruby on Rails project that sits in front of Solr to make finding Geospatial data fast and discoverable via a map interface. The combination of Ruby/Rails and geospatial data was right up my alley.

I upgraded their instance of GBL to version 4, made functional improvements, fixed bugs, addressed CVEs, integrated New Relic, updated the Capistrano deployment, did some exploratory work with Anubis and did some additional work to deter bots and crawlers. I also worked on an Ansible playbook to automate setting up GeoServer on Rocky Linux.

OpenGeoMetadata, GeoServer & PostGIS

The source of their data for the Spatial Data Repository is a collection of JSON files using the OGM Aardvark Schema, found in the edu.nyu OGM Repository.

To help manage this data I created a command-line tool called sdr-data-loader using Python to automate loading Shapefiles into PostGIS and publishing them to GeoServer.

I took this opportunity to generate an “audit report” to discover and diagnose any records with issues such as missing database records, unpublished layers or invalid IDs. In the end that report helped me fix about 1600 records with issues.

PMTiles & COG

I did some exploratory work to determine the feasibility of converting some of their geospatial data to PMTiles or Cloud Optimized GeoTIFFs to eventually remove the need for GeoServer. Due to the volume and heterogeneous nature of their data, as well as somewhat immature support at the time for those technologies, it was decided not to pursue those formats for now.

Invenio RDM

The latter half of my time with NYU was spent working on their Invenio RDM instance UltraViolet.

GeoSpatial Support

My main goal was to add geospatial functionality to Invenio RDM - the ability to add records with geospatial data that could be previewed using Geoserver both while editing and viewing records. I created a React component that talks to a Python backend to validate record data against GeoServer. This lets depositors get immediate feedback on whether the data they have entered is correct.

I created a data loading CLI (also in Python) to help automate ingesting their geospatial data into Ultraviolet. This included converting data from OGM’s Aardvark format to a format compatible with Invenio RDM

Upgrades and Entra ID Support

During the course of working with Invenio RDM I updated their instance from version 12 to 13 and added OAuth support for Microsoft Entra ID.

GeoJSON Previewer

As an extracurricular project I created a file previewer for GeoJSON after working with Invenio RDM’s previewers for so long. That should be available in the next release of the project.

Fun Work

I had a great time working with the folks in NYU’s Digital Libraries department and hope to engage with them again soon. It was also nice to be able to contribute to open source projects like Invenio RDM and GeoBlacklight.

I’m excited to takes some of the GeoSpatial learnings from this engagement and bring them over to the Lost Mapper, particularly how to set up an instance of GeoBlacklight and ingest your organizations data.

My gis-dockerized project came in handy many times while testing out scripts and working with data. A miniature version of it even made its way into the sdr-data-loader!

Stunt List for The Expanse RPG

Tue, 26 May 2026 00:00:00 GMT

This past weekend I ran my first ever session of The Expanse RPG. I’m running the adventure Cupbearer from the Quickstart but also picked up the Game Master’s Kit which includes hand outs with all the Stunts in the game - there’s like 130 of them.

My players found the lists to be a bit overwhelming and hard to share. I mentioned, “Wouldn’t it be neat if you each had the Stunts on your phone and could filter them by Category and how many Stunt Points you had to spend?”, so I went ahead and made that over the course of the day yesterday: The Expanse RPG Stunt List

After sharing it with various Expanse RPG communities I’ve come to realize that the Stunts list in the GM’s Kit and the ones in the core rulebook are a little different, so I’ll be fixing those over the next few days. But there seems to be only a few of those and their mostly off by 1 point, so it’s still totally usable!

arc2ogm - Initial Thoughts

Wed, 09 Jul 2025 00:00:00 GMT

The Struggle

One of the things I’ve struggled with as a hobbyist mapper is finding and keeping track of GIS data sources on the internet. And because ESRI is such a giant, a large number of state GIS departments end up using ArcGIS. And while there are a large number of ArcGIS REST Services out there, they are not well advertised, and the interface for browsing them isn’t very… mappy.

The Inspiration

I recently did some work for NYU on their Spatial Data Repository which is built on GeoBlacklight. GeoBlacklight is backed by Solr and ingests documents using the OpenGeoMetadata Aardvark Schema. The documents describe mapping resources of various kinds.

A minimal Aardvark file looks something like this:

{
  "id": "foo-trails",
  "dct_title_s": "Trails",
  "gbl_resourceClass_sm": [
    "Web services"
  ],
  "dct_accessRights_s": "Public",
  "gbl_mdModified_dt": "2025-07-09T00:57:51Z",
  "gbl_mdVersion_s": "Aardvark",
  "schema_provider_s": "Testing",
  "dct_references_s": "{\"urn:x-esri:serviceType:ArcGIS#FeatureLayer\":\"https://services6.arcgis.com/9QlSLDqa0P1cHLhu/ArcGIS/rest/services/WRD_WMA_Public/FeatureServer/13\"}",
  "locn_geometry": "ENVELOPE(-85.41306473369147, -81.12480564035837, 34.975019430744, 30.786567131737005)",
  "gbl_resourceType_sm": "Line data"
}

Ingesting that file results in a GeoBlacklight search results that look like this:

And a record page that looks like this:

It’s a nice visual way to browse and filter map data in order to find what you’re looking for. Imagine if there was a GeoBlacklight server that cataloged all the public ArcGIS REST Servers out there?

The Idea

My goal is to create a tool (tentatively called arc2ogm) that you can point at a collection of ArcGIS REST Service URLs, and it will crawl them and turn each compatible resource into an Aardvark document. Then I’d stand up a GeoBlacklight instance and ingest all those documents for others to browse and search.

It’s likely that the cost of the server is not something I can warrant right now, so at least the tool and the resulting data could be useful to others.

The Minimal Viable Project

The Aardvark schema only has a few required fields, though a few more make the record much more usable in GeoBlacklight:

Required Fields

Title

This is the name of the resource which should be pretty straightforward. A challenge here is that they can often be named somewhat generically because they are only thought about in the context of a county or state.

Resource Class

This field will likely be hard-coded because we’re always talking to a “web service.” There might be some subtlety here moving forward though.

Access Rights

This will likely always be Public since we’re crawling publicly available resources.

ID

This is meant to be a unique identifier which I will need some kind of strategy for generating. I suspect this value should be based on the URL (assuming they never change), but this requires further thought.

Modified

This will just be a timestamp for when the URL was crawled.

Metadata Version

This will be hardcoded to Aardvark.

Helpful Fields

Creator, Publisher, Provider

One of these fields will likely be necessary to identify (and make filterable) the source of the data. It could also identify the title or owner of the ArcGIS REST Server being crawled.

Resource Type

There’s a canned collection of values for this field, and for the most part, it will just map to whether it contains point, line or polygon data. As I move beyond Feature Services, I assume additional mappings will be necessary.

References

This field is a collection of URLs related to the data — where you might download or access it from. There are 4 ESRI-specific values for DynamicMapLayer, FeatureLayer, ImageMapLayer and TiledMapLayer.

The value here makes it possible to display the “Open in ArcGIS Online” link and “Web services” button. You could plug this value into QGIS to load that data on the fly.

Geometry

This field represents the extent of the data and powers the ability to search by the pannable/zoomable map in GeoBlacklight. This feels like a high priority field to me.

From my initial development, there will be a challenge here as the Spatial Reference Identifiers (SRIDs) used by some layers aren’t available in pyproj. From a cursory investigation they do show up in QGIS, but I suspect it’s less than trivial to bring QGIS’ data sources into a standalone CLI.