10. Raspberry Pi Installation

Below the setup and installation of the Raspberry Pi (RPi) for weewx weather software with the Davis Vantage Pro2 weather station is described.

10.1. Conventions

General conventions.

10.1.1. Directories

We keep the following directory conventions:

  • /opt additional, manually installed software
  • /opt/<product>/<product-version> subdires e.g. /opt/weewx/weewx-2.7.0
  • /opt/bin own shell scripts
  • /opt/geonovum/sospilot/git all project software synced with GitHub https://github.com/Geonovum/sospilot
  • /home/sadmin home dir beheer account

Under /opt/<product>/<product-version> we often make a dynamic link to the latest version of a product, for example /opt/weewx/weewx is a dynlink to /opt/weewx/weewx-2.7.0.

Add to /etc/profile

export PATH=${PATH}:/opt/bin

10.1.2. System

Ordered Oct 16, 2014. Delivered Oct 17, 2014. From http://www.kiwi-electronics.nl.

_images/rasp-pi-all-s.jpg

Figure 1 - Raspberry Pi Package through Install

Specs.

Raspberry Pi Model B+ bundle                    € 68,95
  - with 2A power adapter
  - microSD 32GB Transcend class 10
  - Color case: Transparant Multicomp enclosure

Wi-Pi Draadloze USB Adapter voor Raspberry Pi   € 18,95

Belkin 4-Port Powered Mobile USB 2.0 Hub        € 16,95

Specs:
Chip Broadcom BCM2835 SoC
Core architecture ARM11
CPU 700 MHz Low Power ARM1176JZFS Applications Processor
GPU Dual Core VideoCore IV® Multimedia Co-Processor
Provides Open GL ES 2.0, hardware-accelerated OpenVG, and
1080p30 H.264 high-profile decode
Capable of 1Gpixel/s, 1.5Gtexel/s or 24GFLOPs with texture filtering
and DMA infrastructure
Memory 512MB SDRAM
Operating System Boots from Micro SD card, running Linux operating system
Dimensions 85 x 56 x 17mm
Power Micro USB socket 5V, 2A

Connectors:
Ethernet 10/100 BaseT Ethernet socket
Video Output HDMI (rev 1.3 & 1.4)
Composite RCA (PAL and NTSC)
Audio Output 3.5mm jack, HDMI
USB 4 x USB 2.0 Connector
GPIO Connector 40-pin 2.54 mm (100 mil) expansion header: 2x20 strip
Providing 27 GPIO pins as well as +3.3 V, +5 V and GND supply lines
Camera Connector 15-pin MIPI Camera Serial Interface (CSI-2)
JTAG Not populated
Display Connector Display Serial Interface (DSI) 15 way flat flex cable connector
with two data lanes and a clock lane
Memory Card Slot SDIO

See also https://www.adafruit.com/datasheets/pi-specs.pdf. The Belkin 4-Port Powered Mobile USB is recommended when attaching WiPi and Davis USB devices, as the RPi may have loss USB-power issues.

10.1.3. Hardware Setup

Multicomp enclosure install: https://www.youtube.com/watch?v=1uFMFZMGO_A

10.1.4. OS Setup

SD card appeared to be non-readable. Reformatted and put NOOBS (probably could have put Raspbian directly) on card. Described here http://www.raspberrypi.org/help/noobs-setup . We will install the Raspbian OS, a Debian Linux variant for the Pi:

format SD card using  https://www.sdcard.org/downloads/formatter_4
select complete format
unpack NOOBS and put files on card
put card in Pi and Reboot
select Raspian
Standard install steps
NB the password to be entered is for the user 'pi' not root!
user pi is sudoer
root password may be set using sudo passwd root
chosen non-GUI at startup, can always get GUI via startx command

10.1.5. Wifi Setup

Wifi setup with WiPi. See doc http://www.element14.com/community/servlet/JiveServlet/downloadBody/49107-102-1-257014/Wi_Pi.User_Manual.pdf Steps.

$ root@raspberrypi:~# sudo nano /etc/network/interfaces

The file will already have a network entry for the localhost, or loopback network interface, and the Ethernet socket, known as eth0. We’re going to add a new interface called wlan0, There are two slightly different ways of editing this file, depending on which of type broad types of encryption is in use on the WiFi network you wish to connect to. In the case of WPA/WPA2, add the following lines to the end of the interfaces document:

auto wlan0
iface wlan0 inet dhcp
wpa-ssid <name of your WiFi network>
wpa-psk <password of your WiFi network>

In the case of WEP, add the following instead

auto wlan0 iface wlan0 inet dhcp wireless-essid <name of your WiFi network> wireless-key <password of your WiFi network>

Result in /etc/network/interfaces

root@georasp:~# cat /etc/network/interfaces
auto lo

iface lo inet loopback
iface eth0 inet dhcp

# allow-hotplug wlan0
# iface wlan0 inet manual
# wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

auto wlan0
iface wlan0 inet dhcp
wpa-ssid <name of your WiFi network>
wpa-psk <password of your WiFi network>

But to have mutiple WLANs and not have passwords in files, this approach is more flexible and more secure. http://www.geeked.info/raspberry-pi-add-multiple-wifi-access-points/

Our /etc/network/interfaces is now

auto lo

iface lo inet loopback
# allow-hotplug eth0
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0
iface wlan0 inet dhcp

pre-up wpa_supplicant -Dwext -i wlan0 -c /etc/wpa_supplicant.conf -B

And in the file /etc/wpa_supplicant.conf configure multiple WIFI stations. For each station generate a PSK as follows wpa_passphrase <ssid> <passphrase>. /etc/wpa_supplicant.conf will become something like:

ctrl_interface=/var/run/wpa_supplicant
#ap_scan=2

network={
       ssid="<station #1 name>"
       scan_ssid=1
       proto=WPA RSN
       key_mgmt=WPA-PSK
       pairwise=CCMP TKIP
       group=CCMP TKIP
       psk=<generated PSK #1>
}

network={
       ssid="<station #2 name>"
       scan_ssid=1
       proto=WPA RSN
       key_mgmt=WPA-PSK
       pairwise=CCMP TKIP
       group=CCMP TKIP
       psk=<generated PSK #2>
}

The latter approach with wpa_supplicant did somehow not work so we remained in the first simple approach without wpa_supplicant, only a simple /etc/network/interfaces config.

Bogger: Wifi seems to go down from time to time with wlan0: CTRL-EVENT-DISCONNECTED reason=4 in syslog. Will use a script in cron to always keep Wifi up. For topic see http://www.raspberrypi.org/forums/viewtopic.php?t=54001&p=413095. See script at https://github.com/Geonovum/sospilot/blob/master/src/raspberry/wificheck.sh and Monitoring section below.

10.1.6. Hostname

In /etc/hostname set to georasp..

10.1.7. Accounts

Two standard accounts: root (“root admin”) en sadmin (“sensors admin”). NB account root is never a login account on Ubuntu/Debian!

Het beheer-account root heeft root-rechten.

Het account sadmin heeft ook wat rechten maar minder. Het account sadmin heeft lees/schrijfrechten op directories voor custom installaties (zie onder).

10.1.8. Software Installation

Via Ubuntu/Debian Advanced Packaging Tool (APT) . Hiermee is op zeer eenvoudige wijze niet alleen alle software, inclusief de meeste GIS tools gemakkelijk te installeren, maar ook up-to-date te houden. Bijvoorbeeld een complete Java installatie gaat met : apt-get install sun-java6-jdk. APT wordt altijd door het root account (met root via sudo of sudo -i) uitgevoerd.

Alleen in een uiterst geval waarbij een software product niet in het APT systeem zit of niet in een gewenste versie is een handmatige (“custom”) installatie gedaan. Hierbij is de volgende conventie aangehouden: custom installaties worden door het account root.

10.2. Software - General

Install of standard packages.

10.2.1. nginx Web Server

As Apache2 seems to have a relative large footprint, many prefer nginx as webserver on RPi. (Though for now, no webserver is used nor required). Setup.

apt-get install nginx

# start/stop server
/etc/init.d/nginx start
/etc/init.d/nginx stop

Config under /etc/nginx especially, default website at /etc/nginx/sites-available/default

server {
        #listen   80; ## listen for ipv4; this line is default and implied
        #listen   [::]:80 default_server ipv6only=on; ## listen for ipv6

        root /usr/share/nginx/www;
        index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name localhost;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ /index.html;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
        }

        location /doc/ {
                alias /usr/share/doc/;
                autoindex on;
                allow 127.0.0.1;
                allow ::1;
                deny all;
        }
}

10.3. Installation - Project Software

Software and documentation for the project, e.g. weewx config, are in the project GitHub: https://github.com/Geonovum/sospilot

Installed under /opt/geonovum/sospilot

cd /opt/geonovum/sospilot
git clone https://github.com/Geonovum/sospilot.git git

NB all documentation (Sphinx) is automagically published after each Git commit/push to ReadTheDocs.org: http://sospilot.readthedocs.org via a standard GitHub Post-commit hook.

The following refresh script is handy to undo local changes and sync with master.

# Refresh from original Repo
# WARNING will remove all local changes!!!
# except for files not in Git

git fetch --all
git reset --hard origin/master

See https://github.com/Geonovum/sospilot/blob/master/refresh-git.sh

10.4. Installation - Weather Software

10.4.1. weewx - Weather Station server

Home: weewx.

Install under /opt/weewx. Custom install as user sadmin in order to facilitate custimization.

See http://www.weewx.com/docs/setup.htm

Steps.

# Install Dependencies
# required packages:
apt-get install python-configobj
apt-get install python-cheetah
apt-get install python-imaging
apt-get install fonts-freefont-ttf  # Fonts in reporting

# optional for extended almanac information:
apt-get install python-dev
apt-get install python-setuptools
easy_install pip
pip install pyephem

# Weewx install after download
cd /opt/weewx
tar xzvf archive/weewx-2.7.0.tar.gz
ln -s weewx-2.7.0 weewx

cd weewx

# Change install dir in setup.cfg as follows
# Configuration file for weewx installer. The syntax is from module
# ConfigParser. See http://docs.python.org/library/configparser.html

[install]

# Set the following to the root directory where weewx should be installed
home = /opt/weewx/weewxinst

# Given the value of 'home' above, the following are reasonable values
prefix =
exec-prefix =
install_lib = %(home)s/bin
install_scripts = %(home)s/bin

# build en install in /opt/weewx/weewxinst
./setup.py build
./setup.py install

# test install
# change
cd /opt/weewx/weewxinst
change station_type = Simulator in weewx.conf

# link met aangepaste configs uit Geonovum GitHub (na backup oude versies)
ln -s /opt/geonovum/sospilot/git/src/weewx/test/weewx.conf /opt/weewx/weewxinst
ln -s /opt/geonovum/sospilot/git/src/weewx/test/skin.conf /opt/weewx/weewxinst/skins/Standard
ln -s /opt/geonovum/sospilot/git/src/weewx/test/weatherapidriver.py /opt/weewx/weewxinst/bin/user

# test OK
sadmin@georasp /opt/weewx/weewxinst $ ./bin/weewxd weewx.conf
LOOP:   2014-10-19 16:18:50 CEST (1413728330) {'heatindex': 32.67858297022247, 'barometer': 31.099999998967093, 'windchill': 32.67858297022247,
'dewpoint': 27.203560993945757, 'outTemp': 32.67858297022247, 'outHumidity': 79.99999996901272, 'UV': 2.5568864075841278,
'radiation': 182.63474339886625, 'rain': 0, 'dateTime': 1413728330, 'windDir': 359.9999998140763, 'pressure': 31.099999998967093,
'windSpeed': 5.164547900449179e-09, 'inTemp': 63.00000002065819, 'windGust': 6.197456769996279e-09, 'usUnits': 1, 'windGustDir': 359.9999998140763}
LOOP:   2014-10-19 16:18:52 CEST (1413728332) {'heatindex': 32.67676549144743, 'barometer': 31.099999990703814, 'windchill': 32.67676549144743,
'dewpoint': 27.20178958368346, 'outTemp': 32.67676549144743, 'outHumidity': 79.99999972111442, 'UV': 2.555313141990661,
'radiation': 182.52236728504724, 'rain': 0, 'dateTime': 1413728332, 'windDir': 359.9999983266865, 'pressure': 31.099999990703814,
'windSpeed': 4.648092932768577e-08, 'inTemp': 63.00000018592372, 'windGust': 5.577711537085861e-08, 'usUnits': 1, 'windGustDir': 359.9999983266865}

# install weewx daemon in /etc/init.d (als root)
# aanpassen settings in daemon in GitHub  /opt/geonovum/sospilot/git/src/weewx/test/weewx-daemon.sh

# PATH should only include /usr/* if it runs after the mountnfs.sh script
WEEWX_HOME=/opt/weewx/weewxinst
PATH=/sbin:/usr/sbin:/bin:/usr/bin
WEEWX_BIN=$WEEWX_HOME/bin/weewxd
WEEWX_CFG=$WEEWX_HOME/weewx.conf
DESC="weewx weather system"
NAME=weewx
WEEWX_USER=sadmin:sadmin
PIDFILE=$WEEWX_HOME/$NAME.pid
DAEMON=$WEEWX_BIN
DAEMON_ARGS="--daemon --pidfile=$PIDFILE $WEEWX_CFG"
SCRIPTNAME=/etc/init.d/$NAME

cp /opt/geonovum/sospilot/git/src/weewx/davis/weewx-deamon.sh /etc/init.d/weewx
update-rc.d weewx defaults
/etc/init.d/weewx start
/etc/init.d/weewx status
* Status of weewx weather system: running

# weewx log bekijken
tail -f /var/log/syslog

# memory in gaten houden
  PID USER      PR  NI    VIRT    RES    SHR  S  %CPU %MEM     TIME+ COMMAND
 4688 sadmin    20   0    170936  36776  4608 S   0.0  0.5   3:15.23 weewxd  (16.10.14 16:22)

# nginx ontsluiting
location /weewx {
    alias /opt/weewx/weewxinst/public_html;
    autoindex on;
    allow 127.0.0.1;
    allow ::1;
    allow all;
}

10.4.2. Extra voor TFA Nexus Pro

TE923 driver. Nodig pyusb

pip install pyusb
# geeft: DistributionNotFound: No distributions matching the version for pyusb

# tweede try:
apt-get install python-usb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-usb
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 17.7 kB of archives.
After this operation, 132 kB of additional disk space will be used.
Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main python-usb armhf 0.4.3-1 [17.7 kB]
Fetched 17.7 kB in 0s (37.9 kB/s)
Selecting previously unselected package python-usb.
(Reading database ... 83706 files and directories currently installed.)
Unpacking python-usb (from .../python-usb_0.4.3-1_armhf.deb) ...
Setting up python-usb (0.4.3-1) ...

# root@otterpi:/opt/weewx/weewxinst# lsusb
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 005: ID 1130:6801 Tenx Technology, Inc.
Bus 001 Device 006: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter

# nu andere error van weewx
Dec 10 15:02:51 otterpi weewx[2645]: te923: Found device on USB bus=001 device=005
Dec 10 15:02:51 otterpi weewx[2645]: te923: Unable to claim USB interface 0: could not claim interface 0: Operation not permitted
Dec 10 15:02:51 otterpi weewx[2645]: wxengine: Unable to open WX station hardware: could not claim interface 0: Operation not permitted
Dec 10 15:02:51 otterpi weewx[2645]: wxengine: Caught WeeWxIOError: could not claim interface 0: Operation not permitted
Dec 10 15:02:51 otterpi weewx[2645]:     ****  Exiting...

# may have to do with udev usb rules
http://www.tdressler.net/ipsymcon/te923.html

vi /etc/udev/rules.d/99-te923.rules
Inhalt:
ATTRS{idVendor}=="1130", ATTRS{idProduct}=="6801", MODE="0660", GROUP="plugdev", RUN="/bin/sh -c 'echo -n $id:1.0 > /sys/bus/usb/drivers/usbhid/unbind'"

udevadm control --reload-rules

adduser sadmin plugdev

10.5. Installation - Weather Hardware

The Davis weather station is mounted on the Geonovum roof. The Vantage Pro2 console is connected via RF to the outside station. The console includes Weatherlink for archiving and USB connectivity. Via the console the station can be configured. Via weewx the status can be obtained

Several sensors seem to be non-active.

./wee_config_vantage --info --config /opt/weewx/weewxinst/weewx.conf
Using configuration file /opt/weewx/weewxinst/weewx.conf.
Querying...
Davis Vantage EEPROM settings:

    CONSOLE TYPE:                   VantagePro2

    CONSOLE FIRMWARE:
      Date:                         Jul 14 2008
      Version:                      1.80

    CONSOLE SETTINGS:
      Archive interval:             1800 (seconds)
      Altitude:                     98 (meter)
      Wind cup type:                large
      Rain bucket type:             0.2 MM
      Rain year start:              10
      Onboard time:                 2014-11-03 14:48:51

    CONSOLE DISPLAY UNITS:
      Barometer:                    hPa
      Temperature:                  degree_10F
      Rain:                         mm
      Wind:                         km_per_hour

    CONSOLE STATION INFO:
      Latitude (onboard):           +52.2
      Longitude (onboard):          +5.4
      Use manual or auto DST?       AUTO
      DST setting:                  N/A
      Use GMT offset or zone code?  ZONE_CODE
      Time zone code:               21
      GMT offset:                   N/A

    TRANSMITTERS:
      Channel 1:                    iss
      Channel 2:                    (N/A)
      Channel 3:                    (N/A)
      Channel 4:                    (N/A)
      Channel 5:                    (N/A)
      Channel 6:                    (N/A)
      Channel 7:                    (N/A)
      Channel 8:                    (N/A)

    RECEPTION STATS:
      Total packets received:       109
      Total packets missed:         3
      Number of resynchronizations: 0
      Longest good stretch:         41
      Number of CRC errors:         0

    BAROMETER CALIBRATION DATA:
      Current barometer reading:    29.405 inHg
      Altitude:                     98 feet
      Dew point:                    255 F
      Virtual temperature:          -89 F
      Humidity correction factor:   23
      Correction ratio:             1.005
      Correction constant:          +0.000 inHg
      Gain:                         0.000
      Offset:                       9.000

    OFFSETS:
      Wind direction:               +0 deg
      Inside Temperature:           +0.0 F
      Inside Humidity:              +0%
      Outside Temperature:          -1.0 F
      Outside Humidity:             +0%

Also http://www.weewx.com/docs/usersguide.htm#wee_config_vantage to clear archive and set archive interval to 5 mins.

Problem: temperature and humidity sensors not working! Working after 3 hours on 20:00, then failing at 0800. Also low station battery message.

May also be the Davis Supercap Problem: http://vp-kb.wikispaces.com/Supercap+fault

10.6. Installation - ETL Tools

10.6.1. XSLT Processor

Zie http://en.wikipedia.org/wiki/XSLT. XSLT (XSL Transformations) is a declarative, XML-based language used for the transformation of XML documents into other XML documents.

Installatie van XSLT processor voor commandline. o.a. gebruikt voor INSPIRE GML transformaties.

apt-get install xsltproc

10.6.2. SQLite

weewx uses SQLite to store weather records. Command line tools.

apt-get install sqlite3

10.6.3. Postgres Client

Just need psql for now plus libs (psycopg2) for Stetl.

apt-get  install postgresql-client

10.6.4. GDAL/OGR

Volgens de website www.gdal.org.

GDAL is a translator library for raster geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. The related OGR library (which lives within the GDAL source tree) provides a similar capability for simple features vector data.

Installatie is simpel via APT.

$ apt-get install gdal-bin python-gdal

# Error...! 2e keer gaat goed na  apt-get update --fix-missing
Fetched 15.6 MB in 18s (838 kB/s)
Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/m/mysql-5.5/mysql-common_5.5.38-0+wheezy1_all.deb  404  Not Found
Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/m/mysql-5.5/libmysqlclient18_5.5.38-0+wheezy1_armhf.deb  404  Not Found

Setting up libgeos-3.3.3 (3.3.3-1.1) ...
Setting up proj-bin (4.7.0-2) ...
Setting up gdal-bin (1.9.0-3.1) ...
python-gdal_1.9.0-3.1_armhf.deb

10.6.5. Stetl - Streaming ETL

Zie http://stetl.org

First all dependencies!

apt-get install python-pip python-lxml libgdal-dev python-psycopg2

Normaal doen we pip install stetl maar nu even install uit Git vanwege te verwachten updates. Install vanuit GitHub versie onder /opt/stetl/git (als user sadmin).

$ mkdir /opt/stetl
$ cd /opt/stetl
$ git clone https://github.com/geopython/stetl.git git
$ cd git
$ python setup.py install  (als root)

$ stetl -h
# 2014-10-21 18:40:37,819 util INFO Found cStringIO, good!
# 2014-10-21 18:40:38,585 util INFO Found lxml.etree, native XML parsing, fabulous!
# 2014-10-21 18:40:41,636 util INFO Found GDAL/OGR Python bindings, super!!
# 2014-10-21 18:40:41,830 main INFO Stetl version = 1.0.7rc13

Installatie Testen.

$ which stetl
# /usr/local/bin/stetl

cd /opt/stetl/git/examples/basics
./runall.sh
# OK!

10.6.6. Python Jinja2

Needed for Stetl Jinja2 templating Filter.

pip install jinja2
Downloading/unpacking jinja2
  Downloading Jinja2-2.7.3.tar.gz (378kB): 378kB downloaded
  Running setup.py (path:/tmp/pip_build_root/jinja2/setup.py) egg_info for package jinja2

    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking markupsafe (from jinja2)
  Downloading MarkupSafe-0.23.tar.gz
  Running setup.py (path:/tmp/pip_build_root/markupsafe/setup.py) egg_info for package markupsafe

Installing collected packages: jinja2, markupsafe
  Running setup.py install for jinja2

    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
  Running setup.py install for markupsafe

    building 'markupsafe._speedups' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o
    markupsafe/_speedups.c:12:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    ==========================================================================
    WARNING: The C extension could not be compiled, speedups are not enabled.
    Failure information, if any, is above.
    Retrying the build without the C extension now.


    ==========================================================================
    WARNING: The C extension could not be compiled, speedups are not enabled.
    Plain-Python installation succeeded.
    ==========================================================================
Successfully installed jinja2 markupsafe
Cleaning up...

10.7. Installation - Maintenance

10.7.1. Remote Access

The RPi is not accessible from outside the LAN. For small maintenance purposes we may setup a reverse SSH tunnel such that we can access the RPi from a known system, ‘remote’, to which the RPi can connect via SSH. This way the RPi is only accessible from ‘remote’ and the communication is encrypted.

Setting up and maintaining a tunnel is best done with autossh. See more info at http://linuxaria.com/howto/permanent-ssh-tunnels-with-autossh

Steps as follows.

# install autossh
$ apt-get install autossh

# add user without shell on RPi and remote
useradd -m -s /bin/false autossh

# Generate keys op RPi
ssh-keygen -t rsa

# store on remote in /home/autossh/.ssh/authorized_keys

# add to /etc/rc.local on RPi/opt/bin/start-tunnels.sh with content
sleep 120
export AUTOSSH_LOGFILE=/var/log/autossh/autossh.log
export AUTOSSH_PIDFILE=/var/run/autossh/autossh.pid
# export AUTOSSH_POLL=60
# export AUTOSSH_FIRST_POLL=30
# export AUTOSSH_GATETIME=30
export AUTOSSH_DEBUG=1
rm -rf /var/run/autossh
mkdir  /var/run/autossh
chown autossh:autossh /var/run/autossh

su -s /bin/sh autossh -c
  'autossh -M 0 -q -f -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R <localport>:localhost:22 autossh@<remote>'

Now we can login to the RPi, but only from ‘remote’ with ssh <user>@localhost -p <localport>.

10.7.2. Monitoring

As the RPi will be running headless and unattended within a LAN, it is of utmost importance that ‘everything remains running’. To this end cronjobs are run with the following crontab file.

# Cronfile for keeping stuff alive on unattended Raspberry Pi
# Some bit crude like reboot, but effective mostly
# Author: Just van den Broecke <justb4@gmail.com>
#
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
SRC=/opt/geonovum/sospilot/git/src

# Do checks on weewx and network every N mins
*/6  * * * * $SRC/weewx/weewxcheck.sh
*/10 * * * * $SRC/raspberry/wificheck.sh
*/15 * * * * $SRC/raspberry/rpistatus.sh
0   4  * * *   shutdown -r +5
0   3  * * *   $SRC/weewx/backup-weewx.sh

The weewx daemon appears to be stopping randomly. Not clear why, but looks like this happens when there are network problems. To check and restart if needed the following script is run.

#! /bin/sh
# Author: Just van den Broecke <justb4@gmail.com>
# Restart weewx if not running.
#

WEEWX_HOME=/opt/weewx/weewxinst
WEEWX_BIN=$WEEWX_HOME/bin/weewxd

NPROC=`ps ax | grep $WEEWX_BIN | grep $NAME.pid | wc -l`
if [ $NPROC -gt 1 ]; then
    echo "weewx running multiple times on `date`! Attempting restart." >> /var/log/weewxcheck.log
    /etc/init.d/weewx restart
elif [ $NPROC = 1 ]; then
    echo "Weewx is ok: $status"
else
    echo "weewx not running on `date`! Attempting restart." >> /var/log/weewxcheck.log
    /etc/init.d/weewx restart
fi

Restarts are also logged so we can see how often this happens.

The WiPi seems to have stability problems. This is a whole area of investigation on WIFI-stations/drivers/parameters etc, that could take days if not weeks... For now a script is run, that checks if the WIfi (wlan0 device) is up or else restarts the interface/Wifi. For topic see http://www.raspberrypi.org/forums/viewtopic.php?t=54001&p=413095. See script at https://github.com/Geonovum/sospilot/blob/master/src/raspberry/wificheck.sh

#!/bin/bash
##################################################################
# NOTE! THIS IS A MODIFIED VERSION OF THE ORIGINAL PROGRAM
# WRITTEN BY KEVIN REED.  TO GET THE ORIGINAL PROGRAM SEE
# THE URL BELOW:
#
# A Project of TNET Services, Inc
#
# Title:     WiFi_Check
# Author:    Kevin Reed (Dweeber)
#            dweeber.dweebs@gmail.com
#            Small adaptions by Just van den Broecke <justb4@gmail.com>
# Project:   Raspberry Pi Stuff
#
# Copyright: Copyright (c) 2012 Kevin Reed <kreed@tnet.com>
#            https://github.com/dweeber/WiFi_Check
#
# Purpose:
#
# Script checks to see if WiFi has a network IP and if not
# restart WiFi
#
# Uses a lock file which prevents the script from running more
# than one at a time.  If lockfile is old, it removes it
#
# Instructions:
#
# o Install where you want to run it from like /usr/local/bin
# o chmod 0755 /usr/local/bin/WiFi_Check
# o Add to crontab
#
# Run Every 5 mins - Seems like ever min is over kill unless
# this is a very common problem.  If once a min change */5 to *
# once every 2 mins */5 to */2 ...
#
# */5 * * * * /usr/local/bin/WiFi_Check
#
##################################################################
# Settings
# Where and what you want to call the Lockfile
lockfile='/var/run/WiFi_Check.pid'

# Which Interface do you want to check/fix
wlan='wlan0'

# Which address do you want to ping to see if you can connect
pingip='194.109.6.93'

##################################################################
echo
echo "Starting WiFi check for $wlan"
date
echo

# Check to see if there is a lock file
if [ -e $lockfile ]; then
    # A lockfile exists... Lets check to see if it is still valid
    pid=`cat $lockfile`
    if kill -0 &>1 > /dev/null $pid; then
        # Still Valid... lets let it be...
        #echo "Process still running, Lockfile valid"
        exit 1
    else
        # Old Lockfile, Remove it
        #echo "Old lockfile, Removing Lockfile"
        rm $lockfile
    fi
fi
# If we get here, set a lock file using our current PID#
#echo "Setting Lockfile"
echo $$ > $lockfile

# We can perform check
echo "Performing Network check for $wlan"
/bin/ping -c 2 -I $wlan $pingip > /dev/null 2> /dev/null
if [ $? -ge 1 ] ; then
    echo "Network connection down on `date`! Attempting reconnection." >> /var/log/wificheck.log
    /sbin/ifdown $wlan
    sleep 10
    /sbin/ifup --force $wlan
else
    echo "Network is Okay"
fi


# Check is complete, Remove Lock file and exit
#echo "process is complete, removing lockfile"
rm $lockfile
exit 0

##################################################################
# End of Script

The overall RPi status is checked every 15 mins and the results posted to the VPS. In particular the network usage is monitored via vnstat. The script can be found at https://github.com/Geonovum/sospilot/blob/master/src/raspberry/rpistatus.sh and is as follows.

#! /bin/sh
# Author: Just van den Broecke <justb4@gmail.com>
# Status of RPi main resources. Post to VPS if possible.
#

log=/var/log/rpistatus.txt
remote=sadmin@sensors:/var/www/sensors.geonovum.nl/site/pi

echo "Status of `hostname` on date: `date`" > $log
uptime  >> $log 2>&1

echo "\n=== weewx ===" >> $log
/etc/init.d/weewx status >> $log
echo "archive stat: `ls -l /opt/weewx/weewxinst/archive`" >> $log 2>&1
echo "archive recs: `sqlite3 /opt/weewx/weewxinst/archive/weewx.sdb 'select count(*) from archive'`" >> $log 2>&1

echo "\n=== restarts ===" >> $log
echo "weewx:" >> $log
wc -l /var/log/weewxcheck.log | cut -d'/' -f1 >> $log 2>&1
echo "\nWifi:" >> $log
wc -l /var/log/wificheck.log  | cut -d'/' -f1 >> $log 2>&1

echo "\n=== bandwidth (vnstat)" >> $log
vnstat >> $log 2>&1

echo "\n=== network (ifconfig)" >> $log
ifconfig >> $log 2>&1

echo "\n=== disk usage (df -h) ===" >> $log
df -h >> $log 2>&1

echo "\n=== memory (free -m)===" >> $log
free -m >> $log 2>&1

scp $log $remote

A typical result is as follows. See http://sensors.geonovum.nl/pi/rpistatus.txt.

Status of georasp on date: Thu Oct 23 13:11:31 CEST 2014
 13:11:31 up 16:39,  3 users,  load average: 0.18, 0.17, 0.16

=== weewx ===
Status of weewx weather system:: running.
archive stat: total 196
-rw-r--r-- 1 sadmin sadmin    189 Oct 20 13:02 one_archive_rec.txt
-rw-r--r-- 1 sadmin sadmin  43008 Oct 23 13:08 stats.sdb
-rw-r--r-- 1 sadmin sadmin 145408 Oct 23 13:08 weewx.sdb
archive recs: 850

=== restarts ===
weewx:
0

Wifi:
0

=== bandwidth (vnstat)

                      rx      /      tx      /     total    /   estimated
 eth0: Not enough data available yet.
 wlan0:
       Oct '14      1.32 MiB  /    2.34 MiB  /    3.66 MiB  /    3.00 MiB
         today      1.32 MiB  /    2.34 MiB  /    3.66 MiB  /       4 MiB


=== network (ifconfig)
eth0      Link encap:Ethernet  HWaddr b8:27:eb:12:6a:ef
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:16829 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16829 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2825670 (2.6 MiB)  TX bytes:2825670 (2.6 MiB)

wlan0     Link encap:Ethernet  HWaddr 00:c1:41:06:0f:42
          inet addr:10.0.0.241  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:52305 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31157 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9209831 (8.7 MiB)  TX bytes:11348504 (10.8 MiB)


=== disk usage (df -h) ===
Filesystem      Size  Used Avail Use% Mounted on
rootfs           28G  3.0G   24G  12% /
/dev/root        28G  3.0G   24G  12% /
devtmpfs        215M     0  215M   0% /dev
tmpfs            44M  276K   44M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            88M     0   88M   0% /run/shm
/dev/mmcblk0p5   60M  9.6M   50M  17% /boot

=== memory (free -m)===
             total       used       free     shared    buffers     cached
Mem:           437        224        212          0         33        136
-/+ buffers/cache:         55        382
Swap:           99          0         99

10.7.3. Backup

weewx db backup

Only weewx.sdb the SQLite DB has to be backed up. The stats file will always be regenerated.

See script: https://github.com/Geonovum/sospilot/blob/master/src/weewx/backup-weewx.sh added to root cronfile

Local dir: /opt/weewx/weewxinst/backup

Backed up to sensors VPS: /home/sadmin/weewx-backup dir.

Pi SD Card Disk Backup

Follow instructions on http://sysmatt.blogspot.nl/2014/08/backup-restore-customize-and-clone-your.html to make a restorable .tar.gz (i.s.o. dd diskclone).

$ apt-get install dosfstools
# was already installed
# attach USB SDcardreader with 16GB SD Card
$ dmesg
[39798.700351] sd 0:0:0:1: [sdb] 31586304 512-byte logical blocks: (16.1 GB/15.0 GiB)
[39798.700855] sd 0:0:0:1: [sdb] Write Protect is off
[39798.700888] sd 0:0:0:1: [sdb] Mode Sense: 03 00 00 00
[39798.701388] sd 0:0:0:1: [sdb] No Caching mode page found
[39798.701451] sd 0:0:0:1: [sdb] Assuming drive cache: write through
[39798.706669] sd 0:0:0:2: [sdc] Attached SCSI removable disk
[39798.707165] sd 0:0:0:2: Attached scsi generic sg2 type 0
[39798.709292] sd 0:0:0:1: [sdb] No Caching mode page found
[39798.709355] sd 0:0:0:1: [sdb] Assuming drive cache: write through
[39798.710838]  sdb: sdb1
[39798.714637] sd 0:0:0:1: [sdb] No Caching mode page found
[39798.714677] sd 0:0:0:1: [sdb] Assuming drive cache: write through
[39798.714701] sd 0:0:0:1: [sdb] Attached SCSI removable disk
[39798.715493] scsi 0:0:0:3: Direct-Access     Generic  SM/xD-Picture    0.00 PQ: 0 ANSI: 2
[39798.718181] sd 0:0:0:3: [sdd] Attached SCSI removable disk
[39798.724978] sd 0:0:0:3: Attached scsi generic sg3 type 0

root@georasp:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs          29077488 3081180  24496200  12% /
/dev/root       29077488 3081180  24496200  12% /
devtmpfs          219764       0    219764   0% /dev
tmpfs              44788     280     44508   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs              89560       0     89560   0% /run/shm
/dev/mmcblk0p5     60479    9779     50700  17% /boot

root@georasp:~# parted -l
Model: Generic SD/MMC (scsi)
Disk /dev/sdb: 16.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  16.2GB  16.2GB  primary  fat32        lba


Model: SD USD (sd/mmc)
Disk /dev/mmcblk0: 31.3GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system  Flags
 1      4194kB  828MB   824MB   primary   fat32        lba
 2      830MB   31.3GB  30.5GB  extended
 5      835MB   898MB   62.9MB  logical   fat32        lba
 6      902MB   31.3GB  30.4GB  logical   ext4
 3      31.3GB  31.3GB  33.6MB  primary   ext4


root@georasp:~# parted /dev/sdb
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
(parted) mkpart primary fat16 1MiB 64MB
(parted) mkpart primary ext4 64MB -1s
(parted) print
Model: Generic SD/MMC (scsi)
Disk /dev/sdb: 16.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  64.0MB  62.9MB  primary               lba
 2      64.0MB  16.2GB  16.1GB  primary

(parted) quit
Information: You may need to update /etc/fstab.

root@georasp:~# parted -l
Model: Generic SD/MMC (scsi)
Disk /dev/sdb: 16.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  64.0MB  62.9MB  primary               lba
 2      64.0MB  16.2GB  16.1GB  primary


Model: SD USD (sd/mmc)
Disk /dev/mmcblk0: 31.3GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system  Flags
 1      4194kB  828MB   824MB   primary   fat32        lba
 2      830MB   31.3GB  30.5GB  extended
 5      835MB   898MB   62.9MB  logical   fat32        lba
 6      902MB   31.3GB  30.4GB  logical   ext4
 3      31.3GB  31.3GB  33.6MB  primary   ext4


root@georasp:~# mkfs.vfat /dev/sdb1
mkfs.vfat 3.0.13 (30 Jun 2012)

root@georasp:~# mkfs.ext4 -j  /dev/sdb2
mke2fs 1.42.5 (29-Jul-2012)
warning: 512 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
984960 inodes, 3932160 blocks
196633 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4026531840
120 block groups
32768 blocks per group, 32768 fragments per group
8208 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

root@georasp:~# mkdir /tmp/newpi
root@georasp:~# mount /dev/sdb2 /tmp/newpi
root@georasp:~# mkdir /tmp/newpi/boot
root@georasp:~# mount /dev/sdb1 /tmp/newpi/boot

root@georasp:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           28G  3.0G   24G  12% /
/dev/root        28G  3.0G   24G  12% /
devtmpfs        215M     0  215M   0% /dev
tmpfs            44M  284K   44M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            88M     0   88M   0% /run/shm
/dev/mmcblk0p5   60M  9.6M   50M  17% /boot
/dev/sdb2        15G   38M   14G   1% /tmp/newpi
/dev/sdb1        60M     0   60M   0% /tmp/newpi/boot
root@georasp:~# crontab -l
# Cronfile for keeping stuff alive on unattended Raspberry Pi
# Some bit crude like reboot, but effective mostly
# Author: Just van den Broecke <justb4@gmail.com>
#
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
SRC=/opt/geonovum/sospilot/git/src

# Do checks on weewx and network every N mins
*/6  * * * * $SRC/weewx/weewxcheck.sh
*/10 * * * * $SRC/raspberry/wificheck.sh
*/15 * * * * $SRC/raspberry/rpistatus.sh
0   4  * * *   shutdown -r +5
root@georasp:~# crontab -r
root@georasp:~# ls
Desktop
root@georasp:~# /etc/init.d/weewx stop
[ ok ] Stopping weewx weather system: weewx.
root@georasp:~#

# get the backup tools
wget -O sysmatt-rpi-tools.zip  https://github.com/sysmatt-industries/sysmatt-rpi-tools/archive/master.zip

# do rsync
$ rsync -av --one-file-system / /boot /tmp/newpi/

# ...wait long time, many files....

root@georasp:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           28G  3.0G   24G  12% /
/dev/root        28G  3.0G   24G  12% /
devtmpfs        215M     0  215M   0% /dev
tmpfs            44M  284K   44M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            88M     0   88M   0% /run/shm
/dev/mmcblk0p5   60M  9.6M   50M  17% /boot
/dev/sdb2        15G  3.0G   11G  22% /tmp/newpi
/dev/sdb1        60M  9.6M   51M  16% /tmp/newpi/boot

# NOOBS stuff repair
$ edit /tmp/newpi/boot/cmdline.txt
# root=/dev/mmcblk0p6 must become root=/dev/mmcblk0p2
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

# fstab
# in /tmp/newpi/etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p5  /boot           vfat    defaults          0       2
/dev/mmcblk0p6  /               ext4    defaults,noatime  0       1
# should become
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p   /               ext4    defaults,noatime  0       1

Overdracht. /etc/network/interfaces aanpassen.

# SD-card in USBReader mounten:
mkdir /tmp/oldpi
mount /dev/sdb6 /tmp/oldpi
/tmp/oldpi/etc/network/interfaces

10.7.4. Kiosk Mode

The RPi will be conencted to a TV-screen in a public room at Geonovum (kitchen). As to have something interesting to see, the Rpi will be put in a “kiosk” mode. This can be achieved quite simply using a webbrowser and a website. As there is no user interaction possible via mouse/keyboard this simple setup suffices.

The website will be a continuous change of pages/URLs. For this a simple JavaScript app is made that changes pages into an iframe. See https://github.com/Geonovum/sospilot/blob/master/www/kiosk/index.html. This app will be published to http://sensors.geonovum.nl/kiosk thus can be updated at all times without needing access to the RPi. If bandwidth becomes an issue we may move the app to the RPi later.

In order to always start a browser, X-Windows needs to be started at boottime. This is described here: http://www.opentechguides.com/how-to/article/raspberry-pi/5/raspberry-pi-auto-start.html and here a complete tutorial https://www.danpurdy.co.uk/web-development/raspberry-pi-kiosk-screen-tutorial

In order to start the browser (Chromium) the following is useful: https://lokir.wordpress.com/2012/09/16/raspberry-pi-kiosk-mode-with-chromium and http://askubuntu.com/questions/487488/how-to-open-chromium-in-full-screen-kiosk-mode-in-minimal-windows-manager-enviro

The following was executed.

# install chromium and tools
apt-get install chromium x11-xserver-utils unclutter

# enable X desktop at boot
raspi-config

# choose option3

# edit /etc/xdg/lxsession/LXDE/autostart as follows
@lxpanel --profile LXDE
@pcmanfm --desktop --profile LXDE
# @xscreensaver -no-splash
@xset s off
@xset -dpms
@xset s noblank
@chromium --noerrdialogs --kiosk http://sensors.geonovum.nl/kiosk

apt-get install tightvncserver
tightvncserver
# start als user pi http://www.penguintutor.com/linux/tightvnc
su pi -c '/usr/bin/tightvncserver :1'