Child pages
  • DCS Globalization Project - Neatline Workflow
Skip to end of metadata
Go to start of metadata

 

Overview

It can be challenging to discern a workflow from the Neatline website and documentation. As I worked on the project, I found bits and pieces of information in many different places: within blog posts, within the documentation, or on other websites (such as the Scholars Lab blog and DH Q&A).

I've listed the basic workflow I followed to display a georectified map and group of historical events within a Neatline exhibit. I've noted problems I ran into and added additional information if the tutorial did not work for me.

 


Neatline workflow

1. Install Omeka/Neatline/Geoserver
2. Georectify a historical map
3. Upload GeoTIFF to Geoserver
4. Connect Geoserver and Neatline
5. Geocode your data
6. Ingest data into Neatline
7. Design your exhibit

 


Install Omeka/Neatline/Geoserver

You’ll need a server with LAMP stack

 

 


Georectify a historical map

Originally, we functioned under the assumption that we needed ArcMap to georeference our historical images. In retrospect, this is only recommended if the person doing the georectification is experienced using this software. Other options for georectification include QGIS or MapWarper (recommended). However, given that we accumulated knowledge from our trials and tribulations with ArcGIS I have included that documentation here as well.

 

MapWarper

 Due to our troubles with ArcGIS, the GIS Associate recommended that we use MapWarper, a browser-based tool, to georectify our images. It is easy, free, and quick.

 

  1. Go to mapwarper.net
  2. Create a free account.
  3. Click the "upload map" tab.
  4. Upload the image you want to georectify.
  5. Add as many control points as you'd like.
  6. Click the "export" tab and download the GeoTIFF.

ArcGIS

Take your historical map image (high res TIFF, 3000-5000 pixels recommended) and run it through ArcMap using Neatline's georeferencing tutorial. After I georeferenced my map, this is what it looked like (still within the ArcMap interface):

I didn't have any problems with the georeferencing process, but I did have a problem getting my georeferenced TIFF to display correctly. You can read my forum post documenting this issue.

To troubleshoot, I tried using ArcMap 2.0 and Photoshop CS6 (rather than ArcMap 1.0 and Photoshop CS5, which I had been using) but neither fixed my problem.

Ultimately, it was determined that the problem I experienced was due to the file being automatically saved as an incorrect file type. The file needs to be an uncompressed 8-bit TIFF.

Anna Radue of UITS GIS shared an updated tutorial that can be accessed here: Georeferencing Historic Map Workflow. This tutorial recommends exporting the aforementioned file type "as a personal database." Despite sitting down with the GIS Associate at the IU Libraries, we were unable to get Anna's tutorial to work.

 


Upload GeoTIFF to Geoserver

Follow the instructions listed under Option 2: http://neatline.org/2012/08/29/using-neatline-with-historical-maps-part-3-geoserver/

This tutorial I linked to gives the option of doing this via the Neatline Maps plugin or GeoServer itself, depending on file size, but Neatline Maps was discontinued in Neatline 2.0 so it is no longer an option.

There are a few things you may need to troubleshoot.

Initially we faced an issue of Geoserver not being configured correctly. When I tried to browse under "Connection Parameters," options didn't display. After checking to make sure pop-ups were enabled through my browser I checked with our sys admin and sure enough, there was a configuration problem with Geoserver.

You also may get the following message when you try to navigate to the file on the server through the Geoserver interface: "Could not list layers for this store, an error occurred retrieving them: Argument "value" should not be null." Using Geoloader should solve this problem. 

Geoloader


Geoloader GitHub page (contains zipped package, instructions for use): https://github.com/scholarslab/Geoloader

## Installing Geoloader (instructions from David McClure of UVA ScholarsLab)

1. Clone the repo w/ 'git clone https://github.com/scholarslab/Geoloader' or just download the zipfile from that GitHub URL if you don't have git.

2. From inside the Geoloader directory (wherever you put it), run 'bundle' then 'rake install'.

 

(One problem DCS ran into here is that Sycamore doesn't have libxslt2, so it can't install the 'nokogiri' gem that Geoloader needs to run. Bryan Brown installed it at his workstation but Brian Wheeler would need to do this with any servers we want to run Geoloader on.)


3. If it installed correctly, you should be able to type 'geoloader' and get a list of help instructions that looks like this:


Commands:

  geoloader clear [WORKSPACE]  # Clear a workspace
  geoloader help [COMMAND]     # Describe available commands or one specific command
  geoloader list               # List workspaces and asset counts
  geoloader load [FILES]       # Load a YAML batch manifest
  geoloader work               # Start a Resque worker

 

4. Create a Geoloader configuration file by typing 'cd ~', 'touch .geoloader.yaml' (pay attention to the dots!), and then paste this into the file w/ actual username/password:

geoserver:
  username: gs_username
  password: gs_password


5. Load geotiffs to Geoserver by typing something like this:

geoloader load /path/to/geotiff.tif --services geoserver

 

If you want to load a file to a specific workspace (the default is “geoloader"), use the —workspace flag:

geoloader load /path/to/geotiff.tif —-services geoserver --workspace different_workspace

 

To change the default workspace (the one used when no —workspace flag is specified), add this to the .geoloader.yaml file:

workspaces:
  default: different_workspace

 


Connect Geoserver and Neatline

You will need to add the WMS address and WMS layer to Neatline. 

The WMS address is the URL for your Geoserver account. For DCS, it was: http://poplar.dlib.indiana.edu:8235/geoserver/wms

To identify the WMS layer, go to the "Layers" section listed under "Data" on the left hand side of your Geoserver account. Find the workspace and layer name for the image you want to connect to Neatline. Separate these values with a colon and you have your WMS layer. For DCS, it was Globalization:newest_rebuilt

 

 

Having problems? Make sure you double check that your image is in the correct directory!

 


Geocode Your Data

There are multiple ways to geocode your data to get lat/lon coordinates. Unfortunately, Neatline cannot ingest lat/lon data so you will need to use an Excel formula to translate this into WKT data. So really, geocoding for Neatline is a two-step process.

Note: Wayne Graham provides a tutorial on geocoding for Neatline. He provides a lot of information specific to the programming language Ruby but you can still glean useful information from it to apply to other means of geocoding.

Getting lat/lon coordinates

Tools

Getting WKT data

You can use Wayne's Ruby tutorial to get WKT data.

Alternatively, if you aren't comfortable using Ruby, use a formula in Excel to convert your lat/long coordinates into WKT data. Leslie Barnes from UNC supplied the following formula.

If you geocode your data using Google, you'll need to convert the coordinates to meters, which is what Neatline uses (I'm not sure if they use the same projection as Google, and I cribbed this formula from the Scholars' Lab blog. This formula only works with this projection system.)

  1. Export your spreadsheet into Excel. (It may also be possible in Google docs.)
  2. Split the coordinates column into two columns, deleting the comma. (This just helps to keep things clear.) And though we always say "lat/lon", Google gives us lon/lat.
  3. Run this formula in the next column. (In this case, C2 is the lon and D2 is the lat): =(C2*20037508.34/180)&", "&(LN(TAN((90+D2)*PI()/360))/(PI()/180))*(20037508.34/180)
  4. Paste values in the next column and use find and replace or concatenate to add kml around it. All that is absolutely required for it to show up is the KML namespace, <Placemark>, <Point>, and <coordinates>.

Using this formula worked for me! Here's what my CSV file looked like after I ran the formula and added KML tags.


Ingest Data Into Neatline

There are multiple ways to ingest records into Neatline: by using Ruby, by using the CSV Import plugin, or by manually plotting points.

Using Ruby

Wayne Graham provides tutorials for using Ruby to ingest records into Neatline.

I asked about alternatives to importing using Ruby since the data needs to be in WKT format, not just lat/long coordinates. Using QGIS or ArcMap was suggested, but ultimately we decided to use the Ruby code already provided in their tutorial, thinking it would be simpler.

Here's the code (slightly adjusted from what Wayne provided) that I was able to make work: Batch Ingest into Omeka Using Ruby. However, one downfall of this code is that it only ingests your WKT data–not any other fields. Someone with a better understanding of Ruby would need to write additional code for it to ingest the additional data needed to automatically fill out the Omeka form.

Using Ruby to ingest the records is probably only a good solution if the person working on the project is adept at programming and knows Ruby well.

Using the CSV Import plugin

After I had my WKT data, I added metadata to populate the other fields.

Here is a screenshot of what my final CSV file looked like, just before uploading it to Omeka using CSV Import:

 

And here's what the CSV Import plugin looks like in Omeka. You'll be able to map columns on your spreadsheet to Dublin Core elements:

We hoped that by mapping our WKT data to the coverage field it would automatically populate the exhibit; however, found that our points all clustered in one place:

To see what was happening, I contacted David McClure, a Neatline developer. Here's what he said:

"You’re hitting up against an annoying complexity in web mapping projects, one that I’ve been trying to find ways to iron out. Basically, what’s going on here is that the two versions of the exhibit (when the WMS layer is set as the base layer, vs. when the WMS layer is added as on overlay) are using different coordinate projection systems. The WMS layer itself is projected in EPSG:4326, and when it’s the only layer in the exhibit, OpenLayers (the mapping library Neatline uses) will use that projection as the default. That means that when you plotted the points, they were stored in Neatline’s database in EPSG:4326. But then, when you add the WMS layer as an overlay on the OpenStreetMap base layer, OpenLayers is forced to switch to the projection used by OpenStreetMap (EPSG:900913). Since the points are in EPSG:4326, though, they all clump up around the “null island” down under the horn of Africa.

 

To fix this, you need to convert the points to EPSG:900913, assuming you want to use the OpenStreetMap base layer. This can be done in bulk with a little scripting, but in this case, since it looks like you’ve only got a handful of records, I’d just do this: With the OpenStreetMap base layer active, open up each record that has a point, clear out the geometry, and reposition the point where it belongs. They’ll stay in the right place as long as you use a regular web mapping base layer like Google/OSM/Stamen."

 

By the time we got to this part of the project, we were close to moving to a different platform for our mapping needs (Leaflet). For a quick fix we manually plotted the points.

 


Design Your Exhibit

We didn't get to this point.

 


Project-specific


Other resources

 

  • No labels