Beta version of neotoma2 R package released!

We are excited to announce, at long last, the beta release of the Neotoma R package, neotoma2. Many, many thanks to Simon Goring and Socorro Dominguez for this huge effort, and all of the folks who have contributed suggestions, code, vignette examples, etc. along the way.

The new R package works with the latest Neotoma web services, and is designed to work more closely with the R tidyverse, making data access and analysis more streamlined.

Some features:

  • Nested sites, datasets and samples to more closely mirror the data model of the database.

  • Filter functions to allow you remove sites and datasets easily

  • New set_*() functions, allowing you to add new records directly into your analysis seamlessly

The neotoma2 package is currently hosted on GitHub (https://github.com/NeotyomaDB/neotoma2) while we iron out the kinks and see how people use the package.

If you’d like to use the package we’ve provided some RMarkdown vignettes to get you started.  Installation is as easy as:



We’ll be running a live coding day on Wednesday Feb 16 from 9:00am - 12:00pm (Pacific) on Zoom (https://uwmadison.zoom.us/j/97083802266?pwd=OEd3a2hyUlJwLzJyc3Q1dUFtMlVjUT09).  Folks are invited to join and talk with us about their needs, and see the package in action as we work to improve our online resources.

Bug reports can be made to Simon Goring (goring@wisc.edu), online at https://github.com/NeotomaDB/neotoma2/issues or in the #it_r channel of our Slack workspace (https://join.slack.com/t/neotomadb/shared_invite/zt-cvsv53ep-wjGeCTkq7IhP6eUNA9NxYQ).

Folks are more than welcome to contribute to the project.  Please let us know if you’d like to help out with documentation, examples, code, or helping us craft useful case studies for using the R package!

Read the full story...
Posted by Jessica Blois on 02/11

NSF Facilities/Resources/Equipment standard text available

For PIs writing proposals to NSF, standard text about Neotoma resources is available here: https://www.neotomadb.org/documents
Read the full story...
Posted by Jack Williams on 09/20

Eric24 Event, Good overview talks about Neotoma

On April 28, 2020, Neotoma leaders organized and convened an all-day (17hour) online zoom webinar in remembrance of Eric’s life and work, across four sessions organized by continent and timezone: Asia and IndoPacific, Europe, Africa, and the Americas. All talks were recorded and posted to YouTube.

The event includes a nice series of short (15 min) introductory overviews about Neotoma, that are excellent starting points for students or other newcomers to Neotoma.  They include talks about the history of Neotoma (Russ Graham), its current directions (Jack Williams), what makes Neotoma unique as a long-lasting data resource (Andrea Thomer), and lessons learned by the next generation of Neotoma stewards and leaders (Jessica Blois and Simon Goring).

Read the full story...
Posted by Jack Williams on 05/14

Strat Diagrammer 2.0 released

I'm delighted to report that a new version of Stratigraphic Diagrammer is now released for Neotoma!  Stratigraphic Diagrammer is available in Neotoma Explorer and provides a stratigraphic view of a site-level dataset.  It can be accessed by first finding a site, then clicking on it to get a small popup box, then clicking on the 'view dataset' icon (looks like an eyeball) next to a dataset. 

The original version only supported pollen data, whereas this new version supports pollen, diatom, ostracodes, testate amoebae, and vertebrates.  This new version can be extended to other variables as well, with some case-by-case decisions needed about how to handle expected range values, missing values, units, etc. 
And the new version is quite beautiful as well and allows popups of values for individual points. Thanks to all on the Neotoma IT team and particularly Mike Stryker and Jonathan Nelson for their great work on designing and launching this.  

Read the full story...
Posted by Jack Williams on 02/02

Migration update Nov 20, 2020

Dear Colleagues,

I'm pleased to report several additional milestones in the porting of Neotoma services over from Windows/SQLServer to Linux/Postgres:

1) An updated version of Tilia (v3.0.1) was released today to Data Stewards, and is ready and enabled for Stewards to resume uploading data to Neotoma.  Eric had been working on this new version for several months, with roughly 70 new datasets uploaded, to ensure it was fully connected with the new postgres version of Neotoma.  This version of Tilia can download sites, upload new sites, and add new datasets to existing sites.  There are a few minor known issues that we are working on fixing.

2) Neotoma Explorer has now been fully repointed to the new version of Neotoma, so as new sites are added, you will be able to find them in Explorer.  Jonathan Nelson at Penn State soon will start to work on next-stage improvements to Explorer, in response to Neotoma user and steward requests.  [11/25/20 Update: Several bugs have been reported, in some search windows and in Stratigraphic Diagrammer and its display of chronologies.  We're working on them now and discussing in Slack: https://neotomadb.slack.com.]

3) Landing pages for individual Neotoma datasets are now re-enabled.  For example, https://data.neotomadb.org/14194   Simon Goring is also about to turn on the minting of DOIs for all Neotoma records.  This capability has been available for a while, but has been put on hold while bugs were ironed out of the uploading process; we wanted to avoid minting DOIs for datasets that might need to be removed if errors were found.

See the updated summary table for details about individual components, their current status, and their locations. 



Read the full story...
Posted by Jack Williams on 11/20

In memoriam: Eric Grimm

Hi all,

It is with great sorrow that we pass along the news that Eric Grimm, one of the co-founders of the Neotoma Paleoecology Database, passed away unexpectedly over the weekend. Eric was a giant in the fields of palynology and paleoecology, whose work was marked both by great breadth of vision and deep meticulousness.  Eric’s vision and hard work was instrumental in creating a culture of open data and transparent science so that as individuals, we could all contribute to a greater good of answering globally relevant questions about the influence of climate on Earth’s vegetation and the interactions among species. His contributions to the fields of paleoecology and open biological and geological databases cannot be overemphasized.  He will be sorely missed by all of us, as a leader, and as our friend.

Eric’s vision of an open and shared data architecture for paleoecology, first realized as the continental pollen databases, and now as the multi-proxy Neotoma Paleoecology Database, lives on.  We have known that for an institution to be sustainable, it cannot be solely dependent on any single person.  Hence, several years ago, we embarked on a process of expanding and solidifying the governance and day-to-day work of maintaining the Neotoma database and its software ecosystem, following Eric’s example of open, collaborative work. So the Neotoma Executive Team and Leadership Council is in the process of planning our next steps forward.  We have many ideas here, but they will take some days to solidify into plans and can also can wait until we process the news and our grief.

In the meantime, if you have any questions about this news, or anything related to Tilia or Neotoma, you can contact the Neotoma Executive Team (neotoma-contact@googlegroups.com), or Jack Williams (Exec Chair, jwwilliams1@wisc.edu) or Jessica Blois (Assoc Exec Chair, jblois@ucmerced.edu) directly, and we can figure out how best to address them.


Jack and Jessica, on behalf of the Neotoma Executive Committee

Read the full story...
Posted by Jack Williams on 11/16

R v1.7.6 available on GitHub

There is a new version of the R package available on GitHub.  This update has been submitted to but not accepted yet by CRAN. 

If you install the `devtools` package, you can update to the most recent version (v1.7.6) by simply typing:


Read the full story...
Posted by Jack Williams on 10/16

Migration update Oct 13, 2020

Dear Colleagues,

The migration of the Neotoma database and software is nearing completion.  It’s been a complex process and we apologize for any delays or disruptions caused by it.  Here’s an update on where things stand.  See also this summary table of individual components, their current status, and locations.

Overview:  This has been a complete and two-stage migration of the Neotoma database and all primary software services (APIs, DOIs, Explorer, neotoma R package, Tilia).  In the first stage, we moved everything from an older Windows server to a newer Windows server at Penn State.  This process began in mid July and completed in late August, and was triggered by a reorganization of IT services at Penn State.  In the second stage, we moved everything from Windows (Windows server, SQL-Server implementation of database) to Linux (Linux server, PostgreSQL implementation of database.  This second-stage migration, in the works for several years, is motivated by sustainability goals.  By shifting to an open-source software framework (Windows software is proprietary; Linux software is open-source), we lower software licensing costs and broaden the pool of developers and scientists able to support and enhance the code.  

Current Status:  Two versions of the Neotoma ‘stack’ (i.e. the database and accompanying software) are currently running (see this summary table for details about individual components, their current status, and their locations).  

The original SQL-Server version of NeotomaDB is still running on a Windows server and all components are active.  However, all data uploads ceased to this version in late July 2020, so this is essentially a frozen version, maintained to support backwards compatibility.  Individual software components are stable, but the new server seems to be slow, so there are some speed issues for larger data retrieval queries.  This version is suitable for educational purposes and for comparative testing of research scripts against the old and new versions.  We will maintain this version for as long as feasible and at least the next several months; we will definitely keep it up while we wrap up last fixes with the new version.  

The new PostgreSQL version of NeotomaDB is now running on a Linux server and the backend database is fully migrated and operational.  Eric Grimm has begun test uploads of real data to the new database, so the new PostgreSQL version has advanced slightly beyond the Windows version.  APIs: migrated and released, with V1.5 and 2.0 the current version.  Current efforts are focusing on last testing and updating documentation.  Tilia:  Now migrated and in testing with a few stewards.  We anticipate a full release soon, i.e. in the next week or two.  Explorer:  Running and in last stages of bug fixing (e.g. making sure all links and pointers to APIs are updated correctly; removing case sensitivity).  Two current versions: Windows Version (Stable, FrozenDB) and Postgres Version (Beta Testing, LivingDB).  Once we get past this migration, we will start releasing updates to Explorer that enhance its functionality; several are in development.  DOIs:  The code for minting DOIs was entirely written for the PostgreSQL version of the database, and primarily draw upon the API 2.0 services.  So, simple DOI-minting capabilities should be ready soon after the API documentation is updated, with new functionality to be added over the next several months.  Neotoma R:  A V2.0 package is at the design and development stage; this new version will both point to the PostgreSQL version of the database and include new features in response to user requests.  The 1.0 package is functional and pointing to the Windows version of the database.  

For Further Updates:  You can track updates on the Neotoma Slack channel.  We will also post quick updates to this summary table on an approximate weekly basis  The status of individual components is changing quickly, as we port over individual components, find bugs (or have them reported to us), and fix them.  The written summary above is dated to Oct 10, 2020. See also this technical documentation.  Going forward we will post quick updates to the summary table and longer News updates as needed.

To Report Issues and Requests:  Please send an email to neotoma-contact@google.com or via Slack.  On Slack we have separate channels for each software service and we check it regularly.

-Jack Williams, Jessica Blois, Simon Goring, Eric Grimm, Doug Miller, Jonathan Nelson, & Mike Stryker

Read the full story...
Posted by Jack Williams on 10/13

Page 1 of 8 pages  1 2 3 >  Last ›