Getting data from a WFS server

rowmaps

these web pages
are being produced by
Barry Cornelius

home

maps
preferences

original data
missing data
KMLs
GPXs
CSVs
GeoJSONs

using Organic Maps
app on a phone
and getting it to
display rights of way

tweets
blog
contact
privacy

Getting data from a WFS server

Introduction

The data.gov.uk website has lots of datasets some of which are delivered by WFS servers - WFS means Web Feature Service. I've written these notes because I think the information provided at data.gov.uk doesn't make it easy to download such datasets. Although my aim is to get data about public rights of way (PROWs), what is described below is appropriate no matter what your dataset is about (provided it is delivered by a WFS server). And most of the information is also appropriate if you are getting the data from somewhere other than data.gov.uk.

These notes are derived from knowledge I've acquired during the last few years. I should point out I'm not an expert in this area as the information presented here was found by trial and error. So, if you want to suggest changes to this document, please e-mail me.

Finding a dataset info web page

If you go to:
http://data.gov.uk
and type:

   "rights of way"

in the search box, you'll get a web page showing a lot of hits, each one associated with a dataset. When I recently did this, I got 76 hits. If you click on one of these links, your browser will go to a web page that describes that dataset. I call this new web page the dataset info web page.

My first hit was for Staffordshire's dataset. You might get something different as your first hit. When I clicked on the link for the first hit, the browser was taken to the web page at:
https://data.gov.uk/dataset/public-rights-of-way11
So this web page is the dataset info web page for Staffordshire's PROWs.

There is some introductory stuff on a dataset info web page. And then under the heading DATA RESOURCES, there are details of what you can access. Often it provides details about how you access the data using a WMS server or a WFS server. WMS (Web Map Service) is no good for me as it just provides images whereas WFS is good because it provides access to files containing data. So you're usually in luck if an icon or link says something like:

   ... WFS Service

or:

   INSPIRE Download Service

and/or:

   Link to WFS

FIX check "icon or link" and "LINK"

Now move your mouse so that it hovers over the LINK or Link to WFS. The status bar of your browser should indicate the url of the web page you would go to if you were to click the link. Look at this url. What you do next depends on whether the url include GetFeature, getfeature, GetCapabilities or getcapabilities. If it contains none of these, these notes do not help you!

Method 1: Using a GetFeature url

If the url is a GetFeature url, the url should say something like:
https://gis.beacons-npa.gov.uk/geoserver/inspire/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=inspire:public_rights_of_ways

It's crucial that there is a service parameter and it is either wfs or WFS. If this is not the case, the rest of these notes are probably no good to you.

Click on the link and some data will be downloaded. Because the data provider has chosen to give you a GetFeature url, you will not have a choice as to the format of the data. In the above example, the Brecon Beacons NPA deliver the data in GML.

Having, hopefully, got the data, have a look at the section on Licensing at the bottom of this web page.

Method 2: Using a GetCapabilties url

If the url is a GetCapabilities url, the url should say something like:
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?service=wfs&version=2.0.0&request=GetCapabilities

It's crucial that there is a service parameter and it is either wfs or WFS. If this is not the case, the rest of these notes are probably no good to you.

Click on LINK or Link to WFS. Hopefully, this takes you to a new web page that contains a lot of XML. I call this XML the GetCapabilities XML.

Aside: it doesn't matter as to the order in which the parameters of this url appear. So, for example, all the following are equivalent:
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?service=wfs&version=2.0.0&request=GetCapabilities
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?service=wfs&request=GetCapabilities&version=2.0.0
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?request=GetCapabilities&service=wfs&version=2.0.0

Method 2: Getting FFFF and TTTT from the GetCapabilities XML

Now use the ability of your browser to search the GetCapabilities XML for a line containing some string of characters. It's probably Ctrl-F (or use Edit|Find). So press Ctrl-F and the browser should provide a box in order to type the string. In the box, type:

   resultType

This should take you to a part of the GetCapabilities XML that says something like:

   <ows:Parameter name="resultType">
      <ows:AllowedValues>
         <ows:Value>results</ows:Value>
         <ows:Value>hits</ows:Value>
      </ows:AllowedValues>
   </ows:Parameter>
   <ows:Parameter name="outputFormat">
      <ows:AllowedValues>
         <ows:Value>text/xml; subtype=gml/3.2</ows:Value>
         <ows:Value>GML2</ows:Value>
         <ows:Value>KML</ows:Value>
         <ows:Value>SHAPE-ZIP</ows:Value>
         <ows:Value>application/gml+xml; version=3.2</ows:Value>
         <ows:Value>application/json</ows:Value>
         <ows:Value>application/vnd.google-earth.kml xml</ows:Value>
         <ows:Value>application/vnd.google-earth.kml+xml</ows:Value>
         <ows:Value>csv</ows:Value>
         <ows:Value>gml3</ows:Value>
         <ows:Value>gml32</ows:Value>
         <ows:Value>json</ows:Value>
         <ows:Value>text/xml; subtype=gml/2.1.2</ows:Value>
         <ows:Value>text/xml; subtype=gml/3.1.1</ows:Value>
      </ows:AllowedValues>
   </ows:Parameter>

You're interested in the AllowedValues for outputFormat. I'm particularly interested to see whether the allowed values include SHAPE-ZIP or KML or json as these are my preferred formats. Choose one of the allowed formats. I'll choose:

SHAPE-ZIP

I'll refer to what you've chosen as FFFF.

FIX dw ; and space in params

Now use Ctrl-F (or use Edit|Find) to find:

   FeatureTypeList

Hopefully, you will get a section that starts like this:

      <FeatureTypeList>
         <FeatureType>
            <Name>staffordshire_county_council_prowstaffordshire150316:staffordshire_county_council_prowstaffordshire150316</Name>
            <Title>prowstaffordshire150316</Title>
            ...
         </FeatureType>

There may be more than one FeatureType. I'm interested in one where the Name of the FeatureType says something like public right of way, right of way, prow or row. So, for the above, I would choose:

   staffordshire_county_council_prowstaffordshire150316:staffordshire_county_council_prowstaffordshire150316

If you're not dealing with PROWs, choose something that is appropriate. I'll refer to what you've chosen as TTTT.

Method 2: Using FFFF and TTTT to get the data

Now look at the GetCapabilities url, e.g.:
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?service=wfs&version=2.0.0&request=GetCapabilities
and produce a new url by changing this url in two places:

Change the parameter request=GetCapabilities to:
```
      request=GetFeature
```
Add the following two parameters to the end of the URL:
```
      &typeName=TTTT&outputFormat=FFFF
```
where TTTT and FFFF are what you found out from looking at the GetCapabilities XML.

That will produce something like:
http://inspire.misoportal.com/geoserver/staffordshire_county_council_prowstaffordshire150316/wfs?service=wfs&version=2.0.0&request=GetFeature&typeName=staffordshire_county_council_prowstaffordshire150316:staffordshire_county_council_prowstaffordshire150316&outputFormat=SHAPE-ZIP

Hopefully, when you go to this new url, it will download a file. If you have chosen SHAPE-ZIP, the browser will offer to download a zip file that you can later unzip in order to get the ESRI shape files.

Method 2: The easy way

After I first released these notes, @ReadDavid drew my attention to a @cgutteridge tweet (on September 5th 2016) saying:

Open data people: Do you get scared and confused by WFS XML on http://data.gov.uk? 
This can help: http://geo-explore.ecs.soton.ac.uk/

So I went to that url and I easily got the data for the two examples I tried. In particular, I tried it on a GetCapabilities XML that had many FeatureTypes and I got the data for the FeatureType I wanted. So give it a go: http://geo-explore.ecs.soton.ac.uk

Licensing

If you're are wanting to use the data, you'll need to be aware of the licence under which it is released. This is a minefield, and, unfortunately, many data providers aren't too aware of what it's all about. There seem to be two places to look for licensing information on the dataset info web page:

Under the title at the top of the dataset info web page it often says something like:
```
    Published by XXX.   Licensed under YYY.
    
```
Sometimes licensing information appears in an Access constraints subsection of the Additional information section that is towards the bottom of the dataset info web page. It'll say:
```
    Access constraints: ZZZ
    
```

If YYY or ZZZ say OGL or Open Government Licence, that's good. If instead YYY or ZZZ say OSODL or Ordnance Survey OpenData Licence, that's effectively just as good. This is because in May 2016 I e-mailed the Ordnance Survey about the local authorities that had, during the last few years, successfully obtained an exemption from the Public Sector Mapping Agreement and released their data under terms equivalent to the Ordnance Survey OpenData Licence and the OS replied All data exempted by Ordnance Survey is now covered by the Open Government Licence (OGL), which superseded its own OS OpenData licence in April 2015.

The OGL is described at:
http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/

Using the two methods

I've successfully used Method 1 with the following dataset info web pages:
Brecon Beacons NPA
Redcar and Cleveland

I've successfully used Method 2 with the following dataset info web pages:
inner Bradford
outer Bradford
East Sussex
Gateshead
Manchester
North Lincolnshire
Oldham
Portsmouth
Salford
Staffordshire
Wirral

Metadata

First released: 11th September 2016
Last updated: 14th September 2016
URL: http://www.rowmaps.com/blog/2016-09-11-getting-data-from-a-wfs-server/