NBN Gateway 5 update

Many of you are aware that the NBN Gateway version 5 has launched and some of you have been in touch regarding issues that you have encountered in relation to the new site.  Thank you for your feedback which has helped us to respond to and resolve these teething problems. 

We hope to begin loading data onto NBN Gateway 5 from the 18th November. You can submit datasets prior to this date to data@nbn.org.uk.

Below you can read about all of the new features of the NBN Gateway and, for information, we have also detailed the issues that cropped up.

NBN Gateway 5

The NBN Gateway was created to enable organisations who gather and manage wildlife data to share their data with others via the internet.
It was officially launched in 2004, following release of a prototype version in 2001. Over the years, the partnership has developed and the number of species records available through the Gateway has increased steadily.  Geographic datasets (habitats and sites) have been added to provide context or to act as filters for querying the species data. 
Over 150 data providers now use the NBN Gateway to make their data available for wider use and for public interest.  The NBN Gateway receives on average 15,000 visits* per month, and the data are also accessed via web services on around 40 other websites or apps.  The data are being used in many innovative ways within the fields of policy making, research, education, land management and public engagement, all contributing to the overall aim of biodiversity conservation – some examples can be found on the Use Data section of our website.
There have been a number of updates to the functionality of the NBN Gateway over the years.  Most notably, we saw the introduction of the New Interactive Map in 2011 and since then we have continued to gather user feedback and comments in preparation for the most significant update since the NBN Gateway's official launch almost 10 years ago.

In advance of the launch of NBN Gateway 5, we carried out a lengthy and thorough Access Controls consultation followed by user testing of the new functionality on a test site, which has taken place over the last few months. Once again we would like to thank all those users who have participated in this testing and who have helped to define the shape of the new NBN Gateway.

Not only have we responded to user requests and feedback, but we have invested in servers that are able to cope with the increasing pressure put upon them as data volume and usage continues to increase.  We are confident that the user experience will be much better and that the performance of the NBN Gateway 5 will be noticeably improved.  As ever, we would like to hear from you if you have any feedback on the new NBN Gateway or if you would like to make any suggestions for the future.  Please do get in touch with us through access@nbn.org.uk

* (a ‘visit’ is defined as use of the website that involves accessing at least one reporting page e.g. a species map, a site report, a dataset metadata page or organisation profile page etc)

So, what's new?

If you are a regular user of the NBN Gateway, then the following changes will be particularly noticeable to you:

  • Better performance, greater reliability and stability.  The new Gateway will be able to cope with increased volumes of data and new functionality. 
  • More flexible – users can request access and download precisely the records they need, filtering data by taxonomy, geography, date, datasets and designations.  
  • What you see is what you get – all publicly available records can be downloaded and their details are available to view on screen.
  • Improved download functionality – e.g. data downloads will be supplied in a single table rather than a separate table per dataset.  It will also be possible to download whole datasets.
  • Totally new interactive map – it is now possible to select and query multiple records and create maps of two or more species in different colours.

For data providers

  • It will be much easier to administer datasets and organisations, deal with access requests and proactively grant access.
  • You will receive better feedback on who is using your data and for what purpose.  
  • We will publish monthly reports summarising access requests and data downloads across the whole NBN to give you an insight into data access and use via the new Gateway.
  • Data security and Data Exchange Principles remain paramount.  Data providers can use access controls to restrict access where necessary, blurring resolution, hiding recorder names and attributes, and flagging records as sensitive.

Important for all users

  • Everyone will now need to log in if they want to view record details on screen or download data

Web Services

  • The NBN Gateway 5 runs off REST web services
  • Information for developers about the REST API will be available shortly from the NBN website.  An announcement will be made in the NBN Forum when this is available.
  • In time, coded examples of implementations of the REST services will also be available.
  • The SOAP web services will continue to function off the old Gateway 4 database until current users are able to migrate to the REST services.  However, the data accessible through the SOAP services will gradually go out of date as the old database (http://old-data.nbn.org.uk) will not be updated with new datasets.

New to the NBN Gateway?

  • ‘How do I…’ documentation will be available in due course, along with training videos to show how quickly and easily you can use the NBN Gateway to best effect.

Issues raised when the new version launched

We would like to stress that the issues did not caused any risk to data security.  They did, however, raise concerns and caused inconvenience for a number of dataset administrators and data users, for which we sincerely apologise.   We have followed up directly with those specifically affected.

The following issues were raised when the new Gateway first went “live”, but have now been resolved:
Interactive Mapping Tool (IMT)

There was a bug in the IMT that resulted in record squares being shown at enhanced resolution rather than at public resolution. The information was only available as an image on the map and therefore it was not possible to view the actual detail of the records on screen by selecting and querying records or by downloading the data.  This did not affect sensitive records, which were not shown, or affect any other areas of the NBN Gateway.  We restricted the IMT to show only 10km squares as soon as we became aware of this bug and the issue has now been fully resolved.

Download Notifier Bug
When anyone requested a data download, the file of downloaded data was created correctly in accordance with the user’s level of access. No data was leaked.  However, there was a bug in the notification e-mail system which meant that dataset administrators who had opted to receive an email when their records were downloaded  received a notification of every data download, even if it didn’t contain records from their dataset.  We disabled the download functionality as soon as we became aware of this bug on Monday 21st October.

Access Requests GridRef bug 
The Grid Reference filter was not being applied to access requests. This meant that when a request for a species within a particular 10km square was made, the system treated it as if it was a request for all of the records for that particular species throughout the UK, and made the request to all datasets containing those records. If such an erroneous request was granted, no actual record access would be granted as that part of the system was always functioning correctly.  We disabled the access request functionality as soon as we became aware of this bug, and will also take steps to remove these erroneous requests from the dataset administrators’ access request log.

Access Requests for Fully Public Datasets Bug 
A number of datasets on the Gateway are fully publicly available, and it should not be possible to request access to these datasets.  This part of the system wasn’t functioning correctly, which meant that dataset administrators received requests for better access to their data, when there was no better access they could give. 

Transfer of datasets

A small number of datasets were not transferred from Gateway 4 to Gateway 5 in time for the release of the new site.  We had aimed to load these by now, but had to give priority to fixing the following bugs.  These five datasets will be loaded at the earliest opportunity.  

  • Butterfly Conservation – Butterfly distributions for Great Britain for the period 1690-2004 from Butterfly Conservation and the Biological Records Centre
  • Butterfly Conservation – Macro-moth provisional distribution for the British Isles (excluding the Republic of Ireland) from the National Moth Recording Scheme
  • Marine Biological Association – DASSH Data Archive Centre Statutory Agency and commercial marine surveys
  • Marine Conservation Society –Seasearch Marine Surveys
  • RSPB – Big Garden Birdwatch winter sightings in the UK in 2009

The providers of these datasets have already been notified, with apologies for the delay.

Thank you

We would like to thank you for your patience whilst we worked through the issues.  If you do experience any other problems please do let us know as soon as possible by contacting access@nbn.org.uk

User guides for administering datasets and organisations will be available very soon.

Web design by Red Paint