OpenSource GIS applied

Monday, July 01, 2013

Open data Hackaton Amsterdam

Last saturday I visited the Open Data Hackathon in Amsterdam organised by "HackdeOverheid" (http://www.hackdeoverheid.nl). It was my first visit to such an event. I really liked the vibrant atmosphere, but in the days after some thoughts on it returned to me frequently which I would like to share with you here.

Identifying OpenData
These days it's quite popular at governments to publish data as OpenData. For sure any data published is interesting. However quite some datasets published are aggregated in some way, which makes it less usefull for usecases the publisher hadn't thought about it could also be used for. Sometimes the aggregation is done to facilitate developers, but most of the times it's done for other reasons like to protect the privacy of the persons mentioned in the data.
For example at this hackaton a dataset was presented by SVB (http://www.hackdeoverheid.nl/voornamen-data-beschikbaar-voor-apps), they summerized the given firstname over the whole country, which limits the dataset for a single purpose: firstname-popularity. If they had aggregated to street/postode/area (or not at all) people might have used the dataset to relate name-giving to region or even economical status.
Which leads me to a suggestion to publishers, provide us with the raw data please. Offer aggregations as a separate download.

Open standards
At the event there were frequent requests for open formats like CSV/JSON/TXT, also people requested API's. But there was not much awareness on open standards. At such a point I always realise that as a geo community we're quite far in development and implementation of open standards. The risk of every organisation implementing it's own formats and API's is that a data miner should develop specific format conversions for each organisation he wants to extract data from. Think of the dutch communities, we have some 250 communities, if they would all develop a specific api on their data, it would be very hard to extract similar data from all those api's. Quite some people are aware of this riks, that's why government develloped "basisregistraties": indications on how to store and comunicate data on certain thematic area's, to be implemented by for example all communities. And quite important for the Open Data movement, since most of the data available via the "basisregistraties" will be open data. A first example of this is "Basisregistratie gebouwen", a dataset (+ soap and WFS api) which contains all buildings in the Netherlands. Ok, this is not a simple json-rest API, but hey, we're developers, we are not afraid of a little XML. My colleague pointed me on http://5stardata.info/ where indeed complying to unified data models in not mentioned as a star, they point at linked data as a way to go. Which indeed might be a better pattern to interact with data from different origin. But afaik is quite experimental at most organisations.

GeoJSON in GitHub
Recently the Github team added GeoJSON support in Github. Uploaded GeoJSON files are displayed as maps on a nice backdrop using LeafletJS. Since then people started uploading masses of GeoJSON files, also in preparation of this Hackathon. For sure there is the risk this will be a single action, and the data will soon be outdated, but if done correctly it could mean a real change in how we're used to publishing data. Imagine:
- An automated proces updates the GeoJSON data in Github every... In the history of git you can then see very nicely the historic development of the dataset.
- You can fork the dataset, reformat it and publish it again, or even open a pull request for data optimisations
- To make the data accessible in traditional GIS you could add a WMS/WFS server which uses the Github GeoJSON as input-storage (using OGR)
- In the end people will love Git as storage and will introduce Git servers in their own organisation as master storage and just clone to Github.
Related to this there is a proposal by Max Ogden & OKFN https://github.com/maxogden/dat and another proposal by OpenGeo https://github.com/opengeo/GeoGit. Today I noticed a blog by Rufus Pollock on the matter, http://blog.okfn.org/2013/07/02/git-and-github-for-data, amazing to see the movement on this theme these days

OGC vs best practices
The last thought is on OGC vs best practices standards. These days we see projects like MapBox, CartoDB, LeafletJS, GeoJSON being very popular, but dissociating from OGC standards.
For sure they use conventions between the products (epsg:900913, TMS, GeoJSON), but those conventions are a result of best practices in the community and not designed in a highly technological and political process at OGC. These best practices standards are light weight, focus on performance, are much easier to implement, widespread and offer a more fluent user experience than any application using OGC standards. OGC should really focus on these leighter standards. We are at a point that data distributors and proprietary spatial software implementors get much pressure from users to also support the best practices standards, resulting in the fact that the best practices standards are widely implemented in both open source and proprietary systems without having been adopted by OGC.

Friday, June 21, 2013

which OGC standard to use to recieve info from mobile phones being used as sensors

We're cooperating in an european FP7 project called COBWEB (http://cobwebproject.eu) on enabling citizens to contribute nature observations in biospheres. From an interoperability aspect it's quite usefull to use an OGC standard to publish observations from the registered phones. But which one to use, there's a couple of them out there. I did some research, but I'm still not sure. Maybe this research helps others, or comment if you have additional ideas on the matter

Most prominent is the openLS standard, which is actually a set of "core" services around location. One of them being the location service, where a client registers with a server and the server sends out requests for postion at regular intervals, the client then responses with it's location. I'm not sure though, if such a model would work on mobile phones. I think a phone can't have a listener for incoming requests like this (what if it's stand-by or offline at that moment)? You can probably run some webserver like i-jetty or nanohttpd, but i guess it will drain the battery if you keep it alive constantly.

GeoSMS offers location info over SMS, which is a rather costly method, but could work out on phones without internet access.

NetCDF is a set of formats for creation, access, and sharing of array-oriented data. NetCDF is quite optimal in size and can be accessed with the thredds data server using OGC:CSW or OpeNDap. Seems not very optimal for single sensor transactions. Quite strong in aggregations. NetCDF is mainly used in oceanography and meteorology sensing

SensorWeb

SOS/O&M: standards for querying and inserting sensor observations and registring sensors (might be usefull for mobile use, although seems to follow same path as OpenLS in that the server takes initiative to read the sensor, and i'm not sure if this is possible on a mobile phone)

SES (event service): register for certain events, and get notified if event occurs (might be needed in some usecases)

SPS (planing service): can be used to query if a sensor is capable of performing a task and assign the actual task (usecase: send a message to all registered phones: "who is in neighboorhood of xxx,yyy", if yes: "please go to xxx,yyy")

Geopackage is mostly a database for use on a mobile phone in case of lack of network connectivity. Users can store their measurements in the local database and when back online use the data from the database to trigger posting of observations. Also users might cache image- or vector-tiles for an area so they still have a map while being offline. Geopackage offers some nice options over using plain file or sqlite storage, but comes at a price (extra dependencies for the app).

WFS-transactional can be used to upload measurements in GML format

WPS can be used to do any other request/response type even asynchronous, can be used to upload measurements

Conclusion:

Most suitable would be SOS or OpenLS, but i wonder if they'd fit in a usecase where a phone takes initiative to send observations as soon as it's back online.
If above doesn't work out WFS-t or WPS seem most appropriate

enable citizens living within Biosphere Reserves to collect environmental data using mobile devices - See more at: http://cobwebproject.eu/#sthash.tBOqUwad.dpuf

Wednesday, June 19, 2013

ArcGIS server and additional WMS styles

Today I had the opportunity to do some experimenting with publishing WMS services on an ArcGIS server instance. My goal was to publish some alternative styles for a WMS layer (most wms clients offer possibility to change a style on a layer, if multiple styles are published in capabilities). Unfortunately it appeared to be quite a challenge to get this done...

As mentioned here, http://resources.arcgis.com/en/help/main/10.1/index.html#/Using_Styled_Layer_Descriptors_with_WMS_services/015400000468000000/, Esri wants me to create an SLD file by hand (based on their snippets) and attach that to the service definition. Ok, quite experimental, but let's try

I used GeoCat bridge (http://geocat.net/bridge) to create the SLD from ArcMap. When I pointed to the SLD file created by bridge (on my local drive), no warning whatsoever, but no additional styles appeared in the WMS. It appears the file should either be located in the ArcGIS Server machine or accessible via the web (was it really strange for me to expect the file would be uploaded during publication?). So after having uploaded the SLD to a web account and inserting the url, the alternative style appeared, yeah!

The styles created by bridge need minor changes to get them working as extra styles in ArcGIS Server. For example the naming of layers is quite bizar in AGS. Layernames are actually increasing numbers (it must be quite a challenge to keep those numbers synchronised over time). So the named-layer section in the SLD should have something like

<sld:NamedLayer><sld:Name>1</sld:Name></sld:NamedLayer>

Also all styles for all layers in the service should be in a single SLD file, so the separate bridge SLD files should be concatenated to a single file.

Note that ArcGIS does not support the full set of SLD tags available in for example geoserver.
Test your style first before publishing to ArcGIS Server by adding the sld parameter to a single wms request.

http://AGS/Service/MapServer/WmsServer?LAYERS=0&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&STYLES=Velocity&FORMAT=image%2Fpng&SRS=EPSG%3A28992&BBOX=177900,314900,178000,315000&WIDTH=256&HEIGHT=256&sld=http://www/test.xml

During my small research I noticed following SLD tags/conventions not supported by AGS:
- <sld:rotation> on sld:mark is skipped
- having a ttf symbol as a wellknownname (convention within geoserver community), in stead use the well known names: circle, square, triangle, star, cross or x

Sunday, June 02, 2013

Amazing list of geonetworks around the globe

Wednesday, April 03, 2013

Auto ordering point layers on top of line/polygon layers in GeoExplorer

Once in a while we get this question for GeoExplorer (GXP):

Force the webviewer to draw points over lines over polygons regardless of the layer menu structure
Off course i get the idea, one will not hide the point layer by overlaying it with polygon features. But still implementing this feature is far les obvious as expected.

This feature is not common in Geo, one will not find it in common libraries. Impact will be quite high to change it code-wise in GXP. One would need a) a procedure to determine the featuretype of a layer (point/line/polygon/mixed/grid) and b) a procedure to reorder the layers triggered by a layer-add/move event, without altering the tree-structure. I'm not even sure a. is obvious, since featuretype is not advertised by default in WMS capabilities.

This might be one of those points where you'll have to decide if the GXP library is suitable for you, it presumes some default GIS workflow, and it's quite impactfull to change it. In such a case using the plain components OpenLayers/Geoext might solve your challenge more easily. But leaving you with a viewer with far less functionality by default.

If a question like this pops up, check the use case! Maybe there are alternatives to get similar behaviour, which do fit the common GIS workflow. For example similar behaviour could be managed by configuration, where one should configure all point layers on top.

By the way, in most usecases I'm not in favour of having tens of layers in a TOC. In most usecases I prefer to have just a couple of layers in a map and present the user another map with an alternative set of layers if they are working in another context (WMC/OWSContext). But give them the possibility of adding additional layers to the context via a CSW-interface.