home

News

Backend Migration of Neotoma DB, new API release

For the last several years, the Neotoma IT team (Eric Grimm, Simon Goring, Mike Stryker, and Steve Crawford; special thanks to Anna George, Jack Williams, and Jessica Blois) has been migrating the Neotoma relational database from SQL Server (Windows-based) to PostgreSQL (Linux). Most of the effort associated with this migration has gone into transferring and testing all Tilia Data Steward services. The goal of this migration is to make Neotoma more sustainable and easily supported by a network of open-source developers.


All migration work so far has been done on a development version of the database, leaving the mainline production version (i.e. the one used by most people) unchanged.  As the last stage of the migration process, we are now switching over the production version of the database.  This process began Aug 10 and we hope to have the first wave of migration done by Friday Aug 21.  Once this migration is complete, there will be two versions of the database:  a primary living version in PostgreSQL, and a secondary static version, maintained for backwards compatibility, on SQL Server.  The migration is anticipated to affect services as follows:


Explorer:  Explorer services have been migrated to the new server and tested.  There may be a few temporary disruptions, with a known bug to be fixed for Stratigraphic Diagrammer.

Tilia:  Tilia has been migrated and tested against the PostgreSQL server and the updated API (currently at http://tilia-dev.neotomadb.org).  Some Tilia-server interactions may be slower than in the past as we optimize the Postgres database, but should return to normal shortly.

API:  There are new generations of APIs (v1.5, v2.0) that point to and work with the PostgreSQL version of the database, available temporarily here: http://api-dev.neotomadb.org/api-docs/ and a soon-to-be-established long-term URL here: http://api.neotomadb.org/api-docs/. The original APIs (v1.0) still point to the static SQLServer/Windows database.  These V1.0 APIs are being renamed with a ‘wn’ prefix added, e.g. https://wnapi.neotomadb.org/v1/data/sites?sitename=Marion%  

neotoma R package: The original neotoma R package is built on top of v1.0 APIs, and so points to the now-static SQLServer version of the database. The source code of the R package has been pointed to the new Windows server.  These changes have been pushed to GitHub (http://github.com/ropensci/neotoma) and we are awaiting a full release on CRAN. A new Neotoma R package (neotoma2, breaking compatibility with the old package) will point to the living PostgreSQL version.  V2 is expected to be released in late October, with pre-release expected soon.  Development for this package can be found here: http://github.com/NeotomaDB/neotoma2

DOIs:  As a newer feature, DOIs were designed to work with the PostgreSQL version. There may be a few bugs or temporary disruptions to DOI minting and accessibility through landing pages, but we anticipate that these will be brief.


If you run into issues, please post queries to Slack (neotomadb.slack.com) or send an email to neotoma-contact@googlegroups.com

Posted by Jack Williams on 08/13