RRes
e-RA: the electronic Rothamsted Archive
Festival of Ideas - 23-24 June 2018
Rothamsted

The Data Extraction Tool

Background

The e-RA programs:

For more details, refer to the Key References listed below.

Phases of Development

Phase 1 1990-1993: Plans for e-RA were laid out in 1990 (Verrier & Perry, 1990). e-RA was then started in 1991 (Barnett, 1991), funded by the Lawes Agricultural Trust (LAT) and the Leverhulme Trust. The projected design for e-RA was proposed in 1991 (Caiger and Verrier 1991) and programs were developed in 1991 and 1992 (Caiger 1991, Caiger 1992) to join and load the data. At this stage it was based on an ORACLE database management system under UNIX and was a complex organisation of co-operating software systems. Programs specific to e-RA were written in C, PRO*C, SQL*Plus and Tcl-TK to perform tasks like data entry, description and extraction. Perl scripts were used to generate HTML compliant web pages (Potts et al, 1996). Initially only Broadbalk wheat yields, Park Grass hay yields and meteorological data were added. A dedicated website for data extraction and background information was developed (Potts et al, 1997). This was one of the first active scientific websites in the UK. The initial test version of the database Version 1 (V1) was available with a limited subset of data in 1993.

Phase 2 1994-1998: The LAT continued to fund the development of e-RA, mainly for data collection. This phase involved continual amendment and refinement and further testing as data began to be collated and input. This involved determination of the most useful forms of output from the database. The provision of a suitable interface between the database and the data extraction tool was addressed.

Phase 3 1998-2001: Various short-term grants enabled maintenance, updating existing datasets and adding new datasets.

Phase 4 2002-2004: No direct funding, basic maintenance. In these two years, lack of funds meant that the development stopped, and only basic maintenance was possible.

Phase 5 2005-2011: A major renovation began in 2005. This involved the allocation of a full time programmer, a full time curator and a web interface developer. During this period Oracle ceased to be supported at Rothamsted causing the database to be migrated to Microsoft SQL Server. V1 code had to be refactored to support the new SQL Server DBMS and the web user interface overhauled to provide a more user friendly experience.

Following migration to SQL Server, version 2 (V2) of the e-RA database was released to Rothamsted users in February 2009. It included substantial re-writing of the back end database, data query tools, data entry and data verification; extending existing datasets; adding new datasets; the development of a comprehensive searchable bibliography. V1 hardware and software was retired.

A new e-RA website was released (March 2011). This was an extensive overhaul of the previous site with major additions to the content including comprehensive background information about the field experiments and meteorological data.

Phase 6 2012-2017: The electronic Rothamsted Archive, e-RA, became part of the Long-term Experiments National Capability, funded by the BBSRC. Refinements were made to the database to provide public access to updated e-RA V2 and the launch of e-RA database externally to public and wider scientific community in May 2013 . This is accessed via the e-RA website using the e-RA data extraction tool (DET). Web pages were finalised and the website moved to a publicly accessible web server for external access.

Currently two versions of e-RA are maintained, the development version used by the curators on the internal website and a public database on the external website. The database itself has kept its complex structure but the new interface simplifies greatly the experience for the user. The end product is user friendly and available to anyone after a registration step. It is accompanied by extensive supporting documentation and the e-RA curators provide tailored assistance and support in selecting and understanding data to suit users' needs.

Data user statistics

 The e-RA curators monitor requests for data from e-RA. Data gathered includes:

What data is requested:-
Summary of what data is requested
inlcudes:
Which field experiment
Which field experiment variable
Which weather variable

Who is requesting the data:-
Summary of who is requesting the data
inlcudes:
Level of interest
Role of requestor
Requesting institute
Country

Project category:-
Summary of projects
Complete list of projects

Last update: December 2016

Key References

1997

1996

1992

1991

1990

RRes Logo - Back to RRes