Migration update Oct 13, 2020

Dear Colleagues,

The migration of the Neotoma database and software is nearing completion.  It’s been a complex process and we apologize for any delays or disruptions caused by it.  Here’s an update on where things stand.  See also this summary table of individual components, their current status, and locations.

Overview:  This has been a complete and two-stage migration of the Neotoma database and all primary software services (APIs, DOIs, Explorer, neotoma R package, Tilia).  In the first stage, we moved everything from an older Windows server to a newer Windows server at Penn State.  This process began in mid July and completed in late August, and was triggered by a reorganization of IT services at Penn State.  In the second stage, we moved everything from Windows (Windows server, SQL-Server implementation of database) to Linux (Linux server, PostgreSQL implementation of database.  This second-stage migration, in the works for several years, is motivated by sustainability goals.  By shifting to an open-source software framework (Windows software is proprietary; Linux software is open-source), we lower software licensing costs and broaden the pool of developers and scientists able to support and enhance the code.  

Current Status:  Two versions of the Neotoma ‘stack’ (i.e. the database and accompanying software) are currently running (see this summary table for details about individual components, their current status, and their locations).  

The original SQL-Server version of NeotomaDB is still running on a Windows server and all components are active.  However, all data uploads ceased to this version in late July 2020, so this is essentially a frozen version, maintained to support backwards compatibility.  Individual software components are stable, but the new server seems to be slow, so there are some speed issues for larger data retrieval queries.  This version is suitable for educational purposes and for comparative testing of research scripts against the old and new versions.  We will maintain this version for as long as feasible and at least the next several months; we will definitely keep it up while we wrap up last fixes with the new version.  

The new PostgreSQL version of NeotomaDB is now running on a Linux server and the backend database is fully migrated and operational.  Eric Grimm has begun test uploads of real data to the new database, so the new PostgreSQL version has advanced slightly beyond the Windows version.  APIs: migrated and released, with V1.5 and 2.0 the current version.  Current efforts are focusing on last testing and updating documentation.  Tilia:  Now migrated and in testing with a few stewards.  We anticipate a full release soon, i.e. in the next week or two.  Explorer:  Running and in last stages of bug fixing (e.g. making sure all links and pointers to APIs are updated correctly; removing case sensitivity).  Two current versions: Windows Version (Stable, FrozenDB) and Postgres Version (Beta Testing, LivingDB).  Once we get past this migration, we will start releasing updates to Explorer that enhance its functionality; several are in development.  DOIs:  The code for minting DOIs was entirely written for the PostgreSQL version of the database, and primarily draw upon the API 2.0 services.  So, simple DOI-minting capabilities should be ready soon after the API documentation is updated, with new functionality to be added over the next several months.  Neotoma R:  A V2.0 package is at the design and development stage; this new version will both point to the PostgreSQL version of the database and include new features in response to user requests.  The 1.0 package is functional and pointing to the Windows version of the database.  

For Further Updates:  You can track updates on the Neotoma Slack channel.  We will also post quick updates to this summary table on an approximate weekly basis  The status of individual components is changing quickly, as we port over individual components, find bugs (or have them reported to us), and fix them.  The written summary above is dated to Oct 10, 2020. See also this technical documentation.  Going forward we will post quick updates to the summary table and longer News updates as needed.

To Report Issues and Requests:  Please send an email to neotoma-contact@google.com or via Slack.  On Slack we have separate channels for each software service and we check it regularly.

-Jack Williams, Jessica Blois, Simon Goring, Eric Grimm, Doug Miller, Jonathan Nelson, & Mike Stryker

Posted by Jack Williams on 10/13