News

Eric24 Event, Good overview talks about Neotoma

On April 28, 2020, Neotoma leaders organized and convened an all-day (17hour) online zoom webinar in remembrance of Eric’s life and work, across four sessions organized by continent and timezone: Asia and IndoPacific, Europe, Africa, and the Americas. All talks were recorded and posted to YouTube.

• Website for Eric24H event: https://sites.google.com/view/ericgrimm/home

• YouTube Channel: https://www.youtube.com/channel/UCRED5fsSanfbhGBs4gt6iBA

The event includes a nice series of short (15 min) introductory overviews about Neotoma, that are excellent starting points for students or other newcomers to Neotoma.  They include talks about the history of Neotoma (Russ Graham), its current directions (Jack Williams), what makes Neotoma unique as a long-lasting data resource (Andrea Thomer), and lessons learned by the next generation of Neotoma stewards and leaders (Jessica Blois and Simon Goring).

Strat Diagrammer 2.0 released

Migration update Nov 20, 2020

Dear Colleagues,

I'm pleased to report several additional milestones in the porting of Neotoma services over from Windows/SQLServer to Linux/Postgres:

1) An updated version of Tilia (v3.0.1) was released today to Data Stewards, and is ready and enabled for Stewards to resume uploading data to Neotoma.  Eric had been working on this new version for several months, with roughly 70 new datasets uploaded, to ensure it was fully connected with the new postgres version of Neotoma.  This version of Tilia can download sites, upload new sites, and add new datasets to existing sites.  There are a few minor known issues that we are working on fixing.

2) Neotoma Explorer has now been fully repointed to the new version of Neotoma, so as new sites are added, you will be able to find them in Explorer.  Jonathan Nelson at Penn State soon will start to work on next-stage improvements to Explorer, in response to Neotoma user and steward requests.  [11/25/20 Update: Several bugs have been reported, in some search windows and in Stratigraphic Diagrammer and its display of chronologies.  We're working on them now and discussing in Slack: https://neotomadb.slack.com.]

3) Landing pages for individual Neotoma datasets are now re-enabled.  For example, https://data.neotomadb.org/14194   Simon Goring is also about to turn on the minting of DOIs for all Neotoma records.  This capability has been available for a while, but has been put on hold while bugs were ironed out of the uploading process; we wanted to avoid minting DOIs for datasets that might need to be removed if errors were found.

See the updated summary table for details about individual components, their current status, and their locations. 

Cheers,

Jack

In memoriam: Eric Grimm

Hi all,

It is with great sorrow that we pass along the news that Eric Grimm, one of the co-founders of the Neotoma Paleoecology Database, passed away unexpectedly over the weekend. Eric was a giant in the fields of palynology and paleoecology, whose work was marked both by great breadth of vision and deep meticulousness.  Eric’s vision and hard work was instrumental in creating a culture of open data and transparent science so that as individuals, we could all contribute to a greater good of answering globally relevant questions about the influence of climate on Earth’s vegetation and the interactions among species. His contributions to the fields of paleoecology and open biological and geological databases cannot be overemphasized.  He will be sorely missed by all of us, as a leader, and as our friend.

Eric’s vision of an open and shared data architecture for paleoecology, first realized as the continental pollen databases, and now as the multi-proxy Neotoma Paleoecology Database, lives on.  We have known that for an institution to be sustainable, it cannot be solely dependent on any single person.  Hence, several years ago, we embarked on a process of expanding and solidifying the governance and day-to-day work of maintaining the Neotoma database and its software ecosystem, following Eric’s example of open, collaborative work. So the Neotoma Executive Team and Leadership Council is in the process of planning our next steps forward.  We have many ideas here, but they will take some days to solidify into plans and can also can wait until we process the news and our grief.

In the meantime, if you have any questions about this news, or anything related to Tilia or Neotoma, you can contact the Neotoma Executive Team (neotoma-contact@googlegroups.com), or Jack Williams (Exec Chair, jwwilliams1@wisc.edu) or Jessica Blois (Assoc Exec Chair, jblois@ucmerced.edu) directly, and we can figure out how best to address them.

Best,

Jack and Jessica, on behalf of the Neotoma Executive Committee

R v1.7.6 available on GitHub

There is a new version of the R package available on GitHub.  This update has been submitted to but not accepted yet by CRAN. 

devtools::install_github(repo='ropensci/neotoma')

Migration update Oct 13, 2020

Dear Colleagues,

The migration of the Neotoma database and software is nearing completion.  It’s been a complex process and we apologize for any delays or disruptions caused by it.  Here’s an update on where things stand.  See also this summary table of individual components, their current status, and locations.

Overview:  This has been a complete and two-stage migration of the Neotoma database and all primary software services (APIs, DOIs, Explorer, neotoma R package, Tilia).  In the first stage, we moved everything from an older Windows server to a newer Windows server at Penn State.  This process began in mid July and completed in late August, and was triggered by a reorganization of IT services at Penn State.  In the second stage, we moved everything from Windows (Windows server, SQL-Server implementation of database) to Linux (Linux server, PostgreSQL implementation of database.  This second-stage migration, in the works for several years, is motivated by sustainability goals.  By shifting to an open-source software framework (Windows software is proprietary; Linux software is open-source), we lower software licensing costs and broaden the pool of developers and scientists able to support and enhance the code.  

Current Status:  Two versions of the Neotoma ‘stack’ (i.e. the database and accompanying software) are currently running (see this summary table for details about individual components, their current status, and their locations).  

The original SQL-Server version of NeotomaDB is still running on a Windows server and all components are active.  However, all data uploads ceased to this version in late July 2020, so this is essentially a frozen version, maintained to support backwards compatibility.  Individual software components are stable, but the new server seems to be slow, so there are some speed issues for larger data retrieval queries.  This version is suitable for educational purposes and for comparative testing of research scripts against the old and new versions.  We will maintain this version for as long as feasible and at least the next several months; we will definitely keep it up while we wrap up last fixes with the new version.  

The new PostgreSQL version of NeotomaDB is now running on a Linux server and the backend database is fully migrated and operational.  Eric Grimm has begun test uploads of real data to the new database, so the new PostgreSQL version has advanced slightly beyond the Windows version.  APIs: migrated and released, with V1.5 and 2.0 the current version.  Current efforts are focusing on last testing and updating documentation.  Tilia:  Now migrated and in testing with a few stewards.  We anticipate a full release soon, i.e. in the next week or two.  Explorer:  Running and in last stages of bug fixing (e.g. making sure all links and pointers to APIs are updated correctly; removing case sensitivity).  Two current versions: Windows Version (Stable, FrozenDB) and Postgres Version (Beta Testing, LivingDB).  Once we get past this migration, we will start releasing updates to Explorer that enhance its functionality; several are in development.  DOIs:  The code for minting DOIs was entirely written for the PostgreSQL version of the database, and primarily draw upon the API 2.0 services.  So, simple DOI-minting capabilities should be ready soon after the API documentation is updated, with new functionality to be added over the next several months.  Neotoma R:  A V2.0 package is at the design and development stage; this new version will both point to the PostgreSQL version of the database and include new features in response to user requests.  The 1.0 package is functional and pointing to the Windows version of the database.  

For Further Updates:  You can track updates on the Neotoma Slack channel.  We will also post quick updates to this summary table on an approximate weekly basis  The status of individual components is changing quickly, as we port over individual components, find bugs (or have them reported to us), and fix them.  The written summary above is dated to Oct 10, 2020. See also this technical documentation.  Going forward we will post quick updates to the summary table and longer News updates as needed.

To Report Issues and Requests:  Please send an email to neotoma-contact@google.com or via Slack.  On Slack we have separate channels for each software service and we check it regularly.

-Jack Williams, Jessica Blois, Simon Goring, Eric Grimm, Doug Miller, Jonathan Nelson, & Mike Stryker

Backend Migration of Neotoma DB, new API release

For the last several years, the Neotoma IT team (Eric Grimm, Simon Goring, Mike Stryker, and Steve Crawford; special thanks to Anna George, Jack Williams, and Jessica Blois) has been migrating the Neotoma relational database from SQL Server (Windows-based) to PostgreSQL (Linux). Most of the effort associated with this migration has gone into transferring and testing all Tilia Data Steward services. The goal of this migration is to make Neotoma more sustainable and easily supported by a network of open-source developers.

All migration work so far has been done on a development version of the database, leaving the mainline production version (i.e. the one used by most people) unchanged.  As the last stage of the migration process, we are now switching over the production version of the database.  This process began Aug 10 and we hope to have the first wave of migration done by Friday Aug 21.  Once this migration is complete, there will be two versions of the database:  a primary living version in PostgreSQL, and a secondary static version, maintained for backwards compatibility, on SQL Server.  The migration is anticipated to affect services as follows:

Explorer:  Explorer services have been migrated to the new server and tested.  There may be a few temporary disruptions, with a known bug to be fixed for Stratigraphic Diagrammer.

Tilia:  Tilia has been migrated and tested against the PostgreSQL server and the updated API (currently at http://tilia-dev.neotomadb.org).  Some Tilia-server interactions may be slower than in the past as we optimize the Postgres database, but should return to normal shortly.

API:  There are new generations of APIs (v1.5, v2.0) that point to and work with the PostgreSQL version of the database, available temporarily here: http://api-dev.neotomadb.org/api-docs/ and a soon-to-be-established long-term URL here: http://api.neotomadb.org/api-docs/. The original APIs (v1.0) still point to the static SQLServer/Windows database.  These V1.0 APIs are being renamed with a ‘wn’ prefix added, e.g. https://wnapi.neotomadb.org/v1/data/sites?sitename=Marion%  

neotoma R package: The original neotoma R package is built on top of v1.0 APIs, and so points to the now-static SQLServer version of the database. The source code of the R package has been pointed to the new Windows server.  These changes have been pushed to GitHub (http://github.com/ropensci/neotoma) and we are awaiting a full release on CRAN. A new Neotoma R package (neotoma2, breaking compatibility with the old package) will point to the living PostgreSQL version.  V2 is expected to be released in late October, with pre-release expected soon.  Development for this package can be found here: http://github.com/NeotomaDB/neotoma2

DOIs:  As a newer feature, DOIs were designed to work with the PostgreSQL version. There may be a few bugs or temporary disruptions to DOI minting and accessibility through landing pages, but we anticipate that these will be brief.

Science article: Human effects on ecosystems

Hi all,
Nice reporting by Science magazine on a talk presented at the 2020 Ecological Society of America meeting, by Allison Stegner and Trisha Spanbauer, showing evidence from Neotoma indicating that the impacts by human land use on ecosystems was as large or larger than the effects associated with the end of the last deglaciation https://www.sciencemag.org/news/2020/08/humans-have-altered-north-america-s-ecosystems-more-melting-glaciers

Posted by Jack Williams on 08/13