5. Administration

This chapter describes the operation and maintenance aspects for the SOSPilot platform. For example:

  • how to start stop servers
  • managing the ETL
  • where to find logfiles
  • setting Postgres credentials

5.1. Data Management

The data is harvested and transformed in three ETL steps. See chapter on Data Management.

The cron job runs under account sadmin with this cronfile: https://github.com/Geonovum/sospilot/blob/master/src/rivm-lml/cronfile.txt . LOgfiles are under /var/log/sospilot.

Every 30 minutes all three steps are run (for now) via the script https://github.com/Geonovum/sospilot/blob/master/src/rivm-lml/etl-all.sh. It is best to stop the cronjob whenever doing any of the resets below. Also be aware that the ETL steps build up history.

Postgres credentials: set these once in the file https://github.com/Geonovum/sospilot/blob/master/src/rivm-lml/pgcreds.sh for your local Postgres. All scripts will use these values.

5.1.1. Reset ETL Step 1 - RIVM File Harvester

  • Empty (not drop) the table rivm_lml.lml_files

The Harvester tracks its progress via the unique file id in rivm_lml.lml_files.

5.1.2. Reset ETL Step 1 - Stations

Whenever there is a new set of stations CSV. This needs to be done. Note: also the SOS Sensor data (see below) needs to be updated. This may be problematic/refined. See https://github.com/Geonovum/sospilot/tree/master/data/rivm-lml/stations

  • Drop the table rivm_lml.stations
  • check with * stations2gml.sh
  • stations2postgis.sh

5.1.3. Reset ETL Step 2 - Files to Core AQ

  • Reset counter last_id to -1 in table rivm_lml.etl_progress for row where worker is files2measurements
  • Also you will need to empty (not drop) the table rivm_lml.measurements

5.1.4. Reset ETL Step 1 + Step 2 - Shortcut

Step 1 + Step 2 can be reset via a single script:


5.1.5. Reset ETL Step 3 - SOS-T Sensor Publishing

This re-publishes the Stations as Sensors.

5.1.6. Reset ETL Step 3 - SOS-T Observations Publishing

  • Reset counter last_id to -1 in table rivm_lml.etl_progress for row where worker is measurements2sos
  • Also you will need to clear the SOS database

5.1.7. Clean SOS data

See data and scripts at https://github.com/Geonovum/sospilot/tree/master/data/sosdb. Using this procedure, no reinstall of the .war file is requried or any other Admin reset (somehow an Admin reset did not work).

As root do

5.1.8. Reset ETL Step 3 - Shortcut

Step 3 can be reset via a single script:


This renders a clean/empty SOS. The Sensors (see above) need to be republished.

5.2. Web Services

The Tomcat server runs both GeoServer and the 52N SOS server. Logfiles in /var/log/tomcat/catalina.out. Stop/start with shorthand: /opt/bin/tcstop and /opt/bin/tcstart.

Admin GeoServer: http://sensors.geonovum.nl/gs/web

Admin SOS: http://sensors.geonovum.nl/sos

5.2.1. SOS Server

The original version received was 52N-SOS-INSPIRE-with-RestAPI_20140519.zip. This version has been patched since to solve some issues. See server-inrichting chapter.

Patching basically means: Stop Tomcat, copy a replacement .jar to /var/www/sensors.geonovum.nl/webapps/sos/WEB-INF/lib and start Tomcat.