Lots Of Copies Keeps Stuff Safe: Harvard / IPLC partnership key as team implements secure preservation solution for shared web collections

Written by: Julianna Barrera-Gomez (Digital Accessions Specialist), Digital Preservation Services

Many of us enjoy or even rely on the use of historical web collections – maybe we’re reminiscing about the early web design era (getting inspired by those pattered backgrounds, flashing GIFs, and bright blue bulleted hyperlinks of yore) or helping a library patron navigate decade-old web page captures as they work toward their research goals.  The most popular web collection platform – the free-to-access Wayback Machine – is provided for internet users from the forward-thinking folks at the Internet Archive.  We appreciate that we can easily find and access those websites that cultural stewards and everyday people before us thought to capture and curate over time – adding in descriptions and cataloging information to help us understand the context even more, so that we can enjoy easier access to our historical world wide web.  In addition to general collections built by Internet Archive, there is also a fee-based service called Archive-It that allows cultural heritage institutions to collect and curate websites into publicly discoverable collections – these are often thematic collections based on local collection development policies.  The Internet Archive’s intent as a non-profit library is to provide access to these collections and websites, but preservation experts have been calling for more robust action to ensure the long-term digital preservation of these web captures since web collecting began.

The problem: How do we start preserving what’s been collected? 

We team up with others, of course!  To that end, Harvard Library Digital Preservation Services is excited to share news about our involvement in an Ivy Plus Libraries Confederation project: implementation of a preservation workflow to ensure that the Confederation’s collective web collections are effectively preserved beyond the discovery service provided by Archive-It.  This problem had been reviewed beginning in 2019-2020, when the IPLC Web Resources Collection Program created the Preservation Task Force, a group of experts who compiled minimum requirements to guide this community preservation effort.  They evaluated and scored several options, ultimately authoring a report that recommended the formation of a private LOCKSS network (PLN) based on the success of a proof-of-concept pilot project using that software (link to the report).

Finding the solution with LOCKSS: Lots of Copies Keeps Stuff Safe 

What is LOCKSS? It’s an open-source software system developed by Stanford University (an Ivy Plus Libraries Confederation member) that utilizes peer-to-peer networks to replicate digital material across a secure private network.  You can learn more about the amazingness of LOCKSS from this Wikipedia article, the LOCKSS website, or for those who prefer to learn from videos featuring cats, this YouTube video [https://www.youtube.com/watch?v=TOE_Jw23cVg]. In short: LOCKSS allows for geographically distributed groups of people who share a common preservation goal – like IPLC and their shared web collections – to come together to distribute both the load of the preservation work as well as the preservation risk. Looking back at the Preservation Task Force’s report, LOCKSS really does stand out as an ideal solution that joins perfectly with the collaborative purpose of the IPLC. 

The process: How we got LOCKSS up and running 

Fast forward to Fall 2022 - the LOCKSS Implementation Project Team was appointed to start the work that the Preservation Task Force had laid out.  This is where Stephen Abrams (Head of Digital Preservation, DPS) and I (then Digital Preservation Analyst, DPS) began work as Implementation Project Managers representing Harvard.  We convened the group of IPLC partners who had taken part in the previous pilot investigation, worked with the IPLC Web Collections Librarian to estimate yearly growth projections for the next 5 years, and confirmed which partners could commit those resources to establish a node in the LOCKSS network.  We were delighted to confirm that five IPLC partners – Dartmouth, Harvard, Princeton, Stanford, and Yale – were able to provide nodes for the implementation. After agreeing upon a process to add web collections to the network, we were ready to begin testing the workflow. 

This is where our LOCKSS partners from Stanford began their complex technical work to test and set up the collections for ingest into the PLN, so that all 5 nodes are ready to begin ingesting and eventually checking over the copies on other nodes.  Having in-house LOCKSS experts from our Stanford partnership was certainly helpful for the project – but of course the LOCKSS program staff are always friendly and very interested in helping anyone achieve their PLN dreams with this easy-to-use (and open-source!) technology. 

The result: LOCKSS is a go! 

When Stephen and I wrapped-up our Implementation Project Management roles we were happy to report to IPLC that the PLN is operational, with over 17 TB of web collections data replicated per node.  Implementation was just the beginning – the IPLC Web Collecting Advisory Committee still has work to do to ensure the operational and financial security of this partnership into the future. But for me – especially as I changed roles in DPS just before this wrapped - I take pride in knowing that we were able to successfully implement a shared solution for this shared preservation problem. That’s worth all the animated GIFs on the web, ever.* 

*For a beautifully nostalgic sample of this, check out Cameron's World, “a web-collage of text and images excavated from the buried neighbourhoods of archived GeoCities pages (1994–2009)” https://www.cameronsworld.net/   

Special Thanks: 

People make partnerships work – Stephen and I would like to thank these key partners who made our Implementation Project work so much easier. 

• Galadriel Chilton as IPLC Director of Collections Initiatives
• Miranda Siler as IPLC Web Collection Librarian
• Clay Miller, Thib Guicherd-Callin, and Snowden Becker as LOCKSS technical experts
• The IPLC Web Collecting Advisory Committee for guiding this work