Tag Archives: NodeRed

Setup and Configuration of a Commercial LoRaWAN Temperature Sensor

Introducton

This document is part of the PAE-IoT Parks and Gardens Project.

The aim of this post is to document the configuration details needed to setup a new temperature sensor which was deployed as part of the Port Adelaide-Enfield Library/Parks and Gardens collaboration, for monitoring the environment around a newly built sports ground and playing surface.

It is a fairly long document and aims to explicitly describe the code that is needed to set up the new data path. It is built on top of the existing data handling system (The Things Network, NodeRED, InfluxDB and Grafana) which has been described in previous posts, so it isn’t a complete explaination of this system.

Grafana graph from installed temperature sensors

More details about the project, including the live sensor data, can be found on the project pages:

The Hardware – Temperature Sensor

General information (from The Things Network)

Device Id:       eui-a840416eb183975f
Description:     LSN50V2-D23 LORAWAN WATERPROOF TEMPERATURE SENSOR (3 PROBES)
Frequency plan:  Australia 915-928 MHz, FSB 2 (used by TTN)
LoRaWAN version: LoRaWAN Specification 1.0.3
Regional Parameters version: RP001 Regional Parameters 1.0.3 revision A

Hardware

Brand:            dragino
Model:            lsn50v2-d20-d22-d23
Hardware version: unknown_hw_version
Firmware version: lsn50v2-d20-d22-d23 1.7.5

Note: The LoRaWAN Regional Parameters is set to “Version 1.0.3 Revision A”. This is different to what was found to work with custom developed sensors. It is unclear at this stage what the actual difference is.

The Things Network (TTN) Configuraton

The sensor was easily registered with The Things Network, as the hardware has been pre-configured with a hardware profile from The Things Network Device Repository. This configures the network parameters (as described in the previous section) as well as data formatting functions.

The current Formatter Code returns four data points (Temp_Black, Temp_Red, Temp White, and BatV) and two status values (Work_Mode and ALARM_status).

The code for this uplink data formatting function is as follows:

function decodeUplink(input) {
	var port = input.fPort;
	var bytes = input.bytes;
	var mode=(bytes[6] & 0x7C)>>2;
	var data = {};
	 switch (input.fPort) {
		 case 2:
if(mode=='3')
{
  data.Work_mode="DS18B20";
  data.BatV=(bytes[0]<<8 | bytes[1])/1000;
  data.ALARM_status=(bytes[6] & 0x01)? "TRUE":"FALSE";
  
  if((bytes[2]==0xff)&& (bytes[3]==0xff))
  {
    data.Temp_Red="NULL";
  }
  else
  {
    data.Temp_Red= parseFloat(((bytes[2]<<24>>16 | bytes[3])/10).toFixed(1));
  }

  if((bytes[7]==0xff)&& (bytes[8]==0xff))
  {
    data.Temp_White="NULL";
  }
  else
  {
  	data.Temp_White=parseFloat(((bytes[7]<<24>>16 | bytes[8])/10).toFixed(1));
  }
  
  if((bytes[9]==0xff)&& (bytes[10]==0xff))
  {
    data.Temp_Black="NULL";
  }
  else
  {
  	data.Temp_Black=parseFloat(((bytes[9]<<8 | bytes[10])/10) .toFixed(1)); 
  }
}
else if(mode=='31')
{
  data.Work_mode="ALARM";
  data.Temp_Red_MIN= bytes[4]<<24>>24;
  data.Temp_Red_MAX= bytes[5]<<24>>24; 
  data.Temp_White_MIN= bytes[7]<<24>>24;
  data.Temp_White_MAX= bytes[8]<<24>>24; 
  data.Temp_Black_MIN= bytes[9]<<24>>24;
  data.Temp_Black_MAX= bytes[10]<<24>>24;  
}

  if(bytes.length==11)
  return {
      data: data,
    }
	break;
default:
    return {
      errors: ["unknown FPort"]
    }
  }
}

The data that is then sent through to NodeRED via the MQTT service is a text formatted JSON data structure.

Comments

In previous PAEIoT projects, the decoding of the LoRaWAN uplink data has been done at a later step, in NodeRed. It was done like this for two reasons: 1) The encoded LoRaWAN packets were smaller for transmitting over the network; and 2) Understanding how to decode and process the data could be done separately to anything that TTN does.

Both of these conditions have changed.

In Version 3 of the TTN service, a lot more meta-data about the network is sent with the sensor data, so any advantages of savings made with minimising the size of the internet packet is lost.

As the uplink data formatting code is automatically included by the manucaturer, is available and open-source, provided thatit works, there is no advantage to not using it.

NodeRed and InfluxDB – Data Procesing and Storage

The following flow diagram describes the new path and changes added to support the new temperature sensor.

Specifically, the following changes and details were added.

TTN Decode (minor change)

The “TTN Decode” node now also passes the entire “uplink_message”, rather than just “payload_raw”. This means that the sensor location (longitude, latitude, and elevation) can be set via the registraion of the device in TTN.

payload.dev_id – Switch (New Output Added)

Added switch to separate flow for the Parks and Garden device (dev_id=eui-a840416eb183975f).

Note: When additional sensors are added, this switch node will need to be changed to also select the new sensor.

Parks Decode – Function (New)

This is the main function which pulls out and renames the data. The ‘msg.payload’ contains this data, and ‘msg.meta’ contains tags for the data.

// Parks Decode
// Decode temperature sensing device
// LSN50V2-D23 LORAWAN WATERPROOF TEMPERATURE SENSOR (3 PROBES)

var device_id = msg.payload.dev_id;

var batv       = msg.payload.uplink_message.decoded_payload.BatV;
var temp_black = msg.payload.uplink_message.decoded_payload.Temp_Black;
var temp_red   = msg.payload.uplink_message.decoded_payload.Temp_Red;
var temp_white = msg.payload.uplink_message.decoded_payload.Temp_White;

var latitude  = msg.payload.uplink_message.locations.user.latitude;
var longitude = msg.payload.uplink_message.locations.user.latitude;
var altitude  = msg.payload.uplink_message.locations.user.altitude;

var data = {};
data.batv       = batv;

data.temp_black = temp_black;
data.temp_red   = temp_red;
data.temp_white = temp_white;

data.temp_air     = temp_white;
data.temp_surface = temp_red;
data.temp_ground  = temp_black;

data.latitude   = latitude;
data.longitude  = longitude;
data.altitude   = altitude;

msg.payload = data;
msg.meta = {
    "device_id": device_id,
};

return msg;

InfluxDB Encode – Function (Copied)

This function is the same as prevously used to format the payload prior to sending it to the InfluxDB node (and into the database).

// InfluxDB Encode
var bucket    = "paeiot-bucket"

var device_id = msg.meta.device_id;
var fields    = msg.payload;

msg.payload = {   
  bucket: bucket,
  data: [{
    measurement: 'sensor',
    tags: {
      "dev_id":  device_id
    }, 
    fields: fields
  }]};

return msg;

InfluxDB – Database Node (Copied)

This uses the existing configured node without any changes.

Grafana Dashboard

A new folder and dashboardwas created to display the temperature data.

Parks and Gardens / Playing Ground Temperatures

The query used to return the air temperature (temp_air) is the following:

from(bucket: "paeiot-bucket")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "sensor")
 |> filter(fn: (r) => r["dev_id"] == "eui-a840416eb183975f") 
 |> filter(fn: (r) => r["_field"] == "temp_air")
 |> drop(columns: ["dev_id", "latitude", "longitude", "altitude"])
 |> aggregateWindow(every: v.windowPeriod, fn: mean)
 |> yield(name: "mean")

An addtional two queries pull out the data for the surface temperature (temp_surface) and ground temperature (temp_ground) with the filter for the ‘_field’ value being changed respecfully. eg.

 filter(fn: (r) => r["_field"] == "temp_surface")
 filter(fn: (r) => r["_field"] == "temp_ground")

The resulting graph (as shown at the top of this post) also has some additional formatting changes (overrides), as pseudo-code:

Where field has name "temp_air", change Display Name to  "Air";
Where field has name "temp_surface", change Display Name to  "Surface";
Where field has name "temp_ground", change Display Name to  "Ground";

Conclusion

The existing PAE-IoT data processing system was modified to allow data from a newly added, commercially available temperature sensor, to be stored, displayed and dynamically updated.

A moderate amount of changes were required due to choices made by the hardware and system vendors. The system design choices that were made were informed by the desire to minimise additional changes if more sensors are added or if the system is extended in other straight forward ways.

This documentation is part of this process.

IoT Workshop (Part 4)

In the last part of this blog post series, we look at pulling everything together with InfluxDB and Grafana to store and display outr IoT sensor data.

Tools to Collect and Display Data

In the workshop we are using the following software tools for processing our collected data.

  • Node-RED – Receive, Process and Push IoT sensor data
  • InfluxDB – Store the time series of sensor data
  • Grafana – Display and manipulate the data visualisation

These packages can easily be installed on a Ubuntu system (desktop or laptop) as well as Raspberry Pi Raspian system[1]. In the following I have included the instructions for installing on Ubuntu.

This combination of software will allow data to be displayed is graphs which can be interactively arranged and manipulated, and shared with multiple users on your network.

Node-RED

The installation and configuration of Node-RED was discussed in the previous post. An additional module needs to be install to allow Node-RED to send data to InfluxDB.

node-red-contrib-influxdb

InfluxDB

InfluxBD is a time series data store, which is like a database, but different. It is designed for storing and retrieving sequential data which contains timestamps in a more efficient way. 

To install 

$ sudo apt install influxdb influxdb-client

It uses the default port of 8086.

Before we are able to start storing data, a database needs to be created in InfluxDB, which we can then push our time series data into. This is done from the Linux commad line ($) as follows: First open a influx prompt (>), then create the ‘iot’ database.

$ influx -precision rfc3339
> create database iot

From the influx prompt (>) you can find out more about the available databases

> show databases

Node-RED can then be configured to push data to this database with the InfluxDB output node, configured as follows.

Configuring InfluxDB node to write to local InfluxDB instance.

A flow can then be configured to pull the data from a Node-RED message and send it to the InfluxDB.

The function used to extract moisture data (analog to digital data in the range 0-1023) is as follows. An additional format check (message starts with “Sensor”) has been added in case some other format is received. The device (device_id) that is reporting the data is specified in the ‘msg.measurement’ field.

var payload = msg.payload.payload_fields.receivedString
var dev_id = msg.payload.dev_id
var re = /^Sensor/;
var moisture = 0.0;
if(re.test(payload)){
payload = payload.substring(10)
var integer = parseInt(payload,16)
moisture = integer/1024.0 * 100.0
}
msg.measurement = dev_id
msg.payload = moisture
return msg;

Grafana

Grafana is a graphing and visualisation package. It can display the information from InfluxDB[3] and display it.

To install[4]

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt update
sudo apt install grafana
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

Connect to the Grafana server with the URL

http://localhost:3000

When logging in for the first time, you can use any username and password and you will be then prompted to change it.

Grafana is a very powerful piece of software but does a good job of helping the first time user through the initial setup. It is recommended that you have a read through the Getting Started documentation. The following is a very brief summary of the confugration process, and I have glossed over a lot of the details.

Once you login in, you will then be prompted for a data source. The details to connect to the local InfluxDB is as follows. Click and set the URL to http://localhost:8086 (it may just be displayed in grey which is a trap). There is no username or password needed if InfluxDB is installed as mentioned above.[5]

Grafana data query builder.

Grafana plots are configured by specifying a data query. In this case we are looking for data from the ‘iot-workshop-1’ device.

The following resulting plot came from defining two data queries for two IoT devices. The second one was configured to generate random data at 10 second intervals for testing purposes.

Notes:
[1] See: http://blog.centurio.net/2018/10/28/howto-install-influxdb-and-grafana-on-a-raspberry-pi-3/
[2] NodeRED is a general purpose automation framework and can do a lot more than mentioned here.
[3] As well as several other databases, including MySQL and PostgreSQL.
[4] See: https://grafana.com/docs/installation/debian/
[5] You realise that this this needs to be changed if your system is being run on a Internet accessible server, right?

Summary

This series of posts covered the process of building a private IoT platform for collecting and displaying sensor information that uses the Internet to move data about but does not rely on cloud or other 3rd party hosted services other than The Things Network.

There are no ongoing charges or fees with this system, and it is a good introduction to software which can then be adapated for other purposes. Have a look for YouTube videos on how to connect Node-RED to home management systems (Google Assistant or Amazon’s Echo) or create different dashboards with Grafana plugins

IoT Workshop

On Thursday (24 Oct 2019) we started running the first session of a new IoT workshop, being held at the local libray (Enfield Branch of the Port Adelaide Enfield Council). This workshop aims to be an introduction to creating sensors and nodes for the LoRaWAN IoT network, using The Things Network website.

This first sesssion (of four) was a whirlwind dump of details which we will unpack and work on through the remainder of the sessions, with hands on electroncs and programming.

We went though the architecture of the system and got everyone to create an account on The Things Network, and set them up so that they could see the IoT data that was being transmitted over the network.

Architecture of LoRAWAN Network (from The Things Network)

Technolgy discussed..

A quick summary of the various bits-n-pieces that are being used follows.

LoRa

The radio technology that is used by sensors for ‘Long Range’, low bandwidth communication. These radios may be programmed to be used in either point-to-point or point-to-gateway (LoRaWAN) mode, configurable with software. They use the unlicensed ISM radio bands, 911MHz in Australia.

https://en.wikipedia.org/wiki/LoRa

LoRaWAN

The radio network and infrastructure used to collect sensor data, via publically accessible gateways.

Arduino

A microprocessor board and programming tools, which use the Atmel (and other) microprocessor. The original aim was to create a programming system which was easy to use for hobbyists and members of the maker community. Arduino microprocessor boards can be used with a LoRa radio to create sensors which can transmit data via LoRAWAN and the internet to be collected and used by other applications. https://www.arduino.cc/

Arduino Shield

This is additional hardware that has been designed to plug directly into an Arduino board and allow it to be easy to use.

Arduino IDE

The Interactive Development Environment (IDE) used to program an Arduino microprocessor board. It is available as a downloadable Java program, and as a web based application. The software is available from the Arduino website.

Arduino Libraries

Additional software packages which can be installed to extend Arduino programs. For example, the LMIC library is used to allow our Arduino programs to use the LoRaWAN radio shield.  

Libraries can be downloaded from within the Arduino IDE, or downloaded and installed manually.

LoRaWAN Sensor Node

A device which collects data and transmits to the LoRaWAN network. It uses a LoRa radio. They need to be registered and authenticated with the LoRaWAN network to transmit correctly.

LoRaWAN Gateway

A device which listens for LoRaWAN radio transmissions from nodes in its area and forwards them to the servers on the LoRaWAN network. They need to be connected to the internet and be registered to operate.

The Things Network

A free website (registration required) which allows access to data collected by LoRaWAN sensors. It is used to register LoRaWAN gateways on the network, and create LoRaWAN applications.

https://www.thethingsnetwork.org/

MQTT

MQ Telemetry Transport. This is the communication software and protocol  used to allow sensor data to be distributed and used once it has been collected by The Things Network.

NodeRed

A web server program, which can be installed on a home computer or server (or Raspberry Pi) which can collect and use sensor data collected. It uses MQTT to access sensor data.

NodeRed is an automation environment and is useful for a lot more than collecting and displaying our IoT data. See the website for more details.

https://nodered.org/

TTNmapper

Mobile application which can be used to map LoRaWAN coverage.  https://ttnmapper.org/

Sensor/Node Example

The following are some example of the sensors that will be built in the workshop.

Moisture Sensor

A soil moisture sensor which sends measurements via LoRaWAN to a monitoring application created NodeRed

The data is displayed in a web browser as the following.

Next Post: IoT Workshop (Part 2)