Posts - Page 5 of 5 - Arba Minch Air Quality

LoRa gateway installed

Mekuanint Hailemichael, a colleague of Arba Minch University (senior network manager) has installed a LoRa gateway at a high building of the Nech Sar Campus of Arba Minch University. Installation of the gateway is part of our research project titled “Determining the feasibility and reliability of a do-it-yourself air quality sensor network in Arba Minch, Ethiopia.”.

The gateway is installed in a metal cage.

The building with the gateway is in the middle of Arba Minch, at Nech Sar Campus.

We are planning to install and test multiple Arduino-based particulate matter (PM) sensor systems in Arba Minch. Apart from testing their operationality and data quality, we want to test the possibility of real-time online data through LoRa functionality. The sensor systems transmit small packets of data over a frequency of 868 MHz. A well-positioned gateway can receive this data and put it online. For this purpose, Mekuannint (co-investigator in the research project) has installed a Lorank-8 gateway together with antenna and lighting protection. It is connected to the Arba Minch University network.

Arduino PM sensor system 3/3: sensor data online

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This third post: getting the PM sensor data available online.

Preferably, the measurement data is available real-time online. With the setup of the previous post, we can only access the data by getting it from the SD card. To an Arduino system, a WiFi module can be added to provide online functionality. However, in and around Arba Minch the availability of WiFi is unreliable and limited. I therefore chose to add LoRa (Long Range) functionality to the sensor system, to get the PM sensor data available online.

With LoRa, small packets of data are sent over frequencies of around 433 or 868 MHz. On this frequency, data can be sent over large distances (10+ km). A receiver with internet connection (‘gateway’) can pick up these transmissions and put them online. For small-scale and personal projects, TheThingsNetwork provides network servers for these transmissions for free. In this way, only at one position (the gateway) internet is needed, while within range of that gateway sensor systems can transmit their data.

1 Materials

Individual sensor systems get LoRa functionality through a Dragino LoRa shield. Apart from that, a gateway is needed to receive the data transfers of individual sensor systems. I bought this gateway.

2 Software

To run the LoRa transmission on an arduino board, I used the LMIC arduino library. Apart from that, I created an account on TheThingsNetwork. On that account, I registered an application, and under that application, I registered individual sensor systems as devices.

3 Connections

I mounted the LoRa shield on an Arduino Mega, and connected the SPS30 Sensirion, SD module and DS3231 to the Arduino Mega according to below connections.

SPS030......Mega
1 VCC.......5V
2 SDA.......SDA
3 SCL.......SCL
4 Select....GND
5 GND.......GND

SD module...Mega
GND.........GND
VCC.........5V
MISO........12
MOSI........11
SCK.........13
CS..........53

DS3231......Mega
GND.........GND
VCC.........5V
SDA.........SDA
SCL.........SCL

I had difficulty in getting both the LoRa shield and the SD module to work. Somehow, the communication over MISO/MOSI disturbed each other. While for the default SD library, on the Arduino Mega pins 51-53 are used, I managed to get it working by using SoftwareSPI of the SDfat library and using pins 11-13.

4 Sketch

I used the following sketches for the components:

Example1_sps30_BasicReadings of the sps30 library
ds3231 of the RTClib library
SoftwareSPI of the SDfat library
ttn-abp of the LMIC library

I combined these sketches to create a full sketch that operates all parts together. In that sketch, for every individual sensor system I had to provide the TheThingsNetwork device’s NwkSKey, AppSKey and DevAddr.

Show code of the whole sketch

String Version = "0-5"; //Sketch version
String NodeID = "SPSAXX"; //Instrument ID
String TTNtype = "ABP"; //Type of connection to TTN

///////////////////////include libraries//////////////////////////////////////
#include <SPI.h>
#include <Wire.h>
#include <EEPROM.h>

#include <lmic.h>
#include <hal/hal.h>

#include <DS3231.h>
#include "SparkFunBME280.h"
#include "SdFat.h"
#include "sps30.h"

//***************************************************************************************************
// TTN settings. Change the keys according to the configuration in TTN.                             *
//***************************************************************************************************
// LoRaWAN NwkSKey, network session key
static const PROGMEM u1_t NWKSKEY[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
// LoRaWAN AppSKey, application session key
static const u1_t PROGMEM APPSKEY[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
// LoRaWAN end-device address (DevAddr)
static const u4_t DEVADDR = 0x00000000 ; // <-- Change this address for every node!
// Empty callbacks - only filled when using OTAA
void os_getArtEui (u1_t* buf) { }
void os_getDevEui (u1_t* buf) { }
void os_getDevKey (u1_t* buf) { }

//***************************************************************************************************
// Global data.                                                                                     *
//***************************************************************************************************
// Constructors
SPS30 sps30;
BME280 bme280;
DS3231 Clock;

//SD pin definitions
const uint8_t SOFT_MOSI_PIN = 11;
const uint8_t SOFT_MISO_PIN = 12;
const uint8_t SOFT_SCK_PIN  = 13;
const uint8_t SD_CHIP_SELECT_PIN = 53;
SdFatSoftSpi<SOFT_MISO_PIN, SOFT_MOSI_PIN, SOFT_SCK_PIN> sd;

// LMIC Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = LMIC_UNUSED_PIN,
    .dio = {2, 6, 7},
};

//SPS030 commands and function prototypes
#define SP30_COMMS I2C_COMMS //SPS030
#define TX_PIN 0 //SPS030
#define RX_PIN 0 //SPS030
#define DEBUG 0 //SPS030
#define PERFORMCLEANNOW 1 //SPS030 cleaning
void ErrtoMess(char *mess, uint8_t r);
void Errorloop(char *mess, uint8_t r);
struct sps_values readSPSdata();
struct sps_values avgSPSdata(int counter, sps_values valIn);

//Empty file and strings
SdFile file;
String dataString;
String timeString;
String spsMassS;
String spsNumS;
String bmeString;
String fileHead = "YYYY-MM-DD;UU:MM:SS;RH;T;PM1mass;PM2.5mass;PM4mass;PM10mass;PM0num;PM1num;PM2.5num;PM4num;PM10num;Partsize";

//Variables
bool Century=false;                               //DS3231
bool h12, PM, ADy, A12h, Apm;                     //DS3231
byte ADay, AHour, AMinute, ASecond, ABits;        //DS3231
byte year, month, date, DoW, hour, minute, second;//DS3231
int pm25, rh, temp, counter;                      //Data storage and counter for average

int restartEvent = 1; //to indicate a restart / reset has happened
int CleanNeed = 1; //Always 1, then at the chosen hour SPS will do 1 cleaning. For the remainder of that hour it will be 0.

int seqnoUp;                                    // To store the TTN upload framecounter
int eeAddress = 0;                              // Writing address in EEPROM
const unsigned TX_INTERVAL = 300;               // #seconds between each send job
static uint8_t mydata[] = {0,0,0,0,0,0};        // Structure for the data to send over LoRa
static osjob_t sendjob;

//Structures
struct bmeData_t                               // Stucture for BME data
{
  int temperature ;                           
  int rhumidity ;                                           
  String bmeString ;
} ;
struct timeData_t                               // Structure for time data
{
  String timeString ;
  long timeInYear ;
  String month ;
  int hour ;
} ;

sps_values spsValues;
bmeData_t bmeValues;
timeData_t timeValues;

//***************************************************************************************************
// Setup and loop.                                                                                  *
//***************************************************************************************************
void setup() {
  Serial.begin(115200);
  Wire.begin();
  Serial.println("Node: " + NodeID + "; Version: " + Version + "; Connection via: " + TTNtype);
  pinMode(3, OUTPUT);

  EEPROM.get(eeAddress,seqnoUp);
  if (seqnoUp<1) {
      Serial.println("EEPROM empty; setting seqnoUp as 20 and writing to EEPROM. Do not forget to reset frame counters in TTN.");
      seqnoUp = 20;
      EEPROM.put(eeAddress,seqnoUp);
  }
  else {
      seqnoUp += 20;
      EEPROM.put(eeAddress,seqnoUp);
      Serial.println("Writing updated seqnoUP "+String(seqnoUp)+" to EEPROM");
  }
  
  setupLMIC();
  setupBME();
  setupSPS();
}  

void loop() {
  //After restart, let the loop start at the start of a minute
  while (restartEvent==1) {                     
    if (Clock.getSecond() == 0) {
      break;
    }
  }

  //For 50 seconds, take average of SPS values
  counter = 0;
  spsValues = readSPSdata();
  float startMeas = millis()/1000;
  float endMeas = millis()/1000-startMeas;
  while (endMeas < 50) {                          
      if (restartEvent == 0) {
        os_runloop_once();
      }
      spsValues = avgSPSdata(counter,spsValues);
      counter = counter + 1;
      endMeas = millis()/1000-startMeas;
  }   

  //Take all other values, and prepare them for LMIC and SD
  bmeValues = loopBME();
  timeValues = loopDS3231();
  pm25 = int(spsValues.MassPM2+0.5);
  rh = bmeValues.rhumidity;
  temp = bmeValues.temperature;
  mydata[0] = {highByte(pm25)};
  mydata[1] = {lowByte(pm25)};
  mydata[2] = {rh};
  mydata[3] = {temp};
  mydata[4] = {highByte(timeValues.timeInYear)};
  mydata[5] = {lowByte(timeValues.timeInYear)};

  spsMassS = String(spsValues.MassPM1)+';'+String(spsValues.MassPM2)+';'+String(spsValues.MassPM4)+';'+String(spsValues.MassPM10);
  spsNumS = String(spsValues.NumPM0)+';'+String(spsValues.NumPM1)+';'+String(spsValues.NumPM2)+';'+String(spsValues.NumPM4)+';'+String(spsValues.NumPM10)+';'+String(spsValues.PartSize);

  if (restartEvent == 1){
    restartEvent = 0;
    loopSD(fileHead,timeValues.month);    //After every restart, a header line is printed
  }

  //Wait untill full minute, and after that print to SD
  while (true) {
    os_runloop_once();
    if (Clock.getSecond() == 0) {
      break;
    }
  }
  timeValues = loopDS3231();
  dataString = timeValues.timeString + ";" + bmeValues.bmeString + ";" + spsMassS + ";" + spsNumS;
  Serial.print('\n');
  Serial.print(dataString);
  Serial.print('\n');
  loopSD(dataString,timeValues.month); //this loop includes the LED

  CleanNeed = cleanNeed(timeValues.hour,CleanNeed); //SPS cleaning check.
}

//***************************************************************************************************
// Setup functions.                                                                                 *
//***************************************************************************************************
void setupLMIC() {
    #ifdef VCC_ENABLE
    // For Pinoccio Scout boards
    pinMode(VCC_ENABLE, OUTPUT);
    digitalWrite(VCC_ENABLE, HIGH);
    delay(1000);
    #endif

    // LMIC init
    os_init();
    // Reset the MAC state. Session and pending data transfers will be discarded.
    LMIC_reset();

    // Set static session parameters. Instead of dynamically establishing a session
    // by joining the network, precomputed session parameters are be provided.
    #ifdef PROGMEM
    // On AVR, these values are stored in flash and only copied to RAM
    // once. Copy them to a temporary buffer here, LMIC_setSession will
    // copy them into a buffer of its own again.
    uint8_t appskey[sizeof(APPSKEY)];
    uint8_t nwkskey[sizeof(NWKSKEY)];
    memcpy_P(appskey, APPSKEY, sizeof(APPSKEY));
    memcpy_P(nwkskey, NWKSKEY, sizeof(NWKSKEY));
    LMIC_setSession (0x1, DEVADDR, nwkskey, appskey);
    #else
    // If not running an AVR with PROGMEM, just use the arrays directly
    LMIC_setSession (0x1, DEVADDR, NWKSKEY, APPSKEY);
    #endif

    #if defined(CFG_eu868)
    // Set up the channels used by the Things Network, which corresponds
    // to the defaults of most gateways. Without this, only three base
    // channels from the LoRaWAN specification are used, which certainly
    // works, so it is good for debugging, but can overload those
    // frequencies, so be sure to configure the full frequency range of
    // your network here (unless your network autoconfigures them).
    // Setting up channels should happen after LMIC_setSession, as that
    // configures the minimal channel set.
    // NA-US channels 0-71 are configured automatically
    LMIC_setupChannel(0, 868100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(1, 868300000, DR_RANGE_MAP(DR_SF12, DR_SF7B), BAND_CENTI);      // g-band
    LMIC_setupChannel(2, 868500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(3, 867100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(4, 867300000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(5, 867500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(6, 867700000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(7, 867900000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(8, 868800000, DR_RANGE_MAP(DR_FSK,  DR_FSK),  BAND_MILLI);      // g2-band
    // TTN defines an additional channel at 869.525Mhz using SF9 for class B
    // devices' ping slots. LMIC does not have an easy way to define set this
    // frequency and support for class B is spotty and untested, so this
    // frequency is not configured here.
    #elif defined(CFG_us915)
    // NA-US channels 0-71 are configured automatically
    // but only one group of 8 should (a subband) should be active
    // TTN recommends the second sub band, 1 in a zero based count.
    // https://github.com/TheThingsNetwork/gateway-conf/blob/master/US-global_conf.json
    LMIC_selectSubBand(1);
    #endif

    // Disable link check validation
    LMIC_setLinkCheckMode(0);

    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow(DR_SF10,14);

    LMIC.seqnoUp = seqnoUp;

    // Start job
    do_send(&sendjob);  
}

void setupBME() {
  bme280.setI2CAddress(0x76); //check adress with i2c check

  if (bme280.beginI2C() == false) //Begin communication over I2C
  {
    Serial.println("BME280 did not respond. Please check wiring.");
    while(1); //Freeze
  }
}

void setupSPS() {
  // set driver debug level
  sps30.EnableDebugging(DEBUG);

  // Begin communication channel;
  if (sps30.begin(SP30_COMMS) == false) {
    Errorloop("could not initialize communication channel.", 0);
  }

  // check for SPS30 connection
  if (sps30.probe() == false) {
    Errorloop("could not probe / connect with SPS30.", 0);
  }
  else
    Serial.println(F("Detected SPS30."));

  // reset SPS30 connection
  if (sps30.reset() == false) {
    Errorloop("could not reset.", 0);
  }

  // start measurement
  if (sps30.start() == true)
    Serial.println(F("Measurement started"));
  else
    Errorloop("Could NOT start measurement", 0);

  // clean now requested -- at every reset 
  if (PERFORMCLEANNOW) {
    // clean now
    if (sps30.clean() == true)
      Serial.println(F("fan-cleaning started"));
    else
      Serial.println(F("Could NOT start fan-cleaning"));
    delay(15000);
  }
}


//***************************************************************************************************
// Loop functions.                                                                                  *
//***************************************************************************************************
struct bmeData_t loopBME() {
  float rhum,temper;
  rhum = bme280.readFloatHumidity();
  temper = bme280.readTempC();
  bmeData_t bmeData;
  bmeData.temperature = int(temper+0.5);
  bmeData.rhumidity = int(rhum+0.5);
  bmeData.bmeString = String(rhum) + ';' + String(temper);
  return(bmeData);
}

struct timeData_t loopDS3231() {
  int second,minute,hour,date,month,year,temperature; 
  String monthS;
  second=Clock.getSecond();
  minute=Clock.getMinute();
  hour=Clock.getHour(h12, PM);
  date=Clock.getDate();
  month=Clock.getMonth(Century);
  year=Clock.getYear();

  timeData_t timeData;
  timeData.timeString = "20" + String(year) + '-' + String(month) + '-' + String(date) + ';' + String(hour) + ':' + String(minute) + ':' + String(second);
  timeData.timeInYear = int(((month-1)*(31*24*6) + date*24*6 + hour*6 + minute/10 + second/(60*10))-32768)+32768; //#10 minutes in the year, minus 32768 to make full use of arduino int capacity.
  monthS = String(month);
  if (monthS.length()==1) {
    monthS="0"+monthS;
  };
  timeData.month = "20" + String(year) + monthS;
  timeData.hour = hour;

  return(timeData);
}

void loopSD(String dataS, String month) {
  //create a fileName per month, and as such, a new file per month
  String extension = ".txt";
  String fileNameStr = NodeID + "_V" + Version + "_" + month + extension;
  char fileName[fileNameStr.length()+1];
  fileNameStr.toCharArray(fileName, sizeof(fileName));
  
  if (!sd.begin(SD_CHIP_SELECT_PIN)) {
    sd.initErrorHalt();
  }

  if (!file.open(fileName, FILE_WRITE)) {
    sd.errorHalt(F("open failed"));
  }
  file.println(dataS);
  Serial.println("printed to SD");
  digitalWrite(3, HIGH); //when successfully written to SD: switch LED on
  file.close();
  delay(500);
  digitalWrite(3, LOW); //switch LED off
}

int cleanNeed(int hourCheck,int Need) {
  if (hourCheck == 3) {
    if (Need == 1) {
      // clean now requested
      if (PERFORMCLEANNOW) {
        // clean now
        if (sps30.clean() == true)
          Serial.println(F("fan-cleaning started"));
        else
          Serial.println(F("Could NOT start fan-cleaning"));
      }
      Need = 0;    
      delay(15000);
    }
  }
  else {
    Need = 1;
  }
  return(Need);
}

//***************************************************************************************************
// Other functions.                                                                                  *
//***************************************************************************************************
// LMIC functions
void onEvent (ev_t ev) {
    Serial.print(os_getTime());
    Serial.print(": ");
    switch(ev) {
        case EV_SCAN_TIMEOUT:
            Serial.println(F("EV_SCAN_TIMEOUT"));
            break;
        case EV_BEACON_FOUND:
            Serial.println(F("EV_BEACON_FOUND"));
            break;
        case EV_BEACON_MISSED:
            Serial.println(F("EV_BEACON_MISSED"));
            break;
        case EV_BEACON_TRACKED:
            Serial.println(F("EV_BEACON_TRACKED"));
            break;
        case EV_JOINING:
            Serial.println(F("EV_JOINING"));
            break;
        case EV_JOINED:
            Serial.println(F("EV_JOINED"));
            break;
        case EV_RFU1:
            Serial.println(F("EV_RFU1"));
            break;
        case EV_JOIN_FAILED:
            Serial.println(F("EV_JOIN_FAILED"));
            break;
        case EV_REJOIN_FAILED:
            Serial.println(F("EV_REJOIN_FAILED"));
            break;
        case EV_TXCOMPLETE:
            Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
            if (LMIC.txrxFlags & TXRX_ACK)
              Serial.println(F("Received ack"));
            if (LMIC.dataLen) {
              Serial.println(F("Received "));
              Serial.println(LMIC.dataLen);
              Serial.println(F(" bytes of payload"));
            }
            // Schedule next transmission
            os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
            break;
        case EV_LOST_TSYNC:
            Serial.println(F("EV_LOST_TSYNC"));
            break;
        case EV_RESET:
            Serial.println(F("EV_RESET"));
            break;
        case EV_RXCOMPLETE:
            // data received in ping slot
            Serial.println(F("EV_RXCOMPLETE"));
            break;
        case EV_LINK_DEAD:
            Serial.println(F("EV_LINK_DEAD"));
            break;
        case EV_LINK_ALIVE:
            Serial.println(F("EV_LINK_ALIVE"));
            break;
         default:
            Serial.println(F("Unknown event"));
            break;
    }
}

void do_send(osjob_t* j){
    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        // Prepare upstream data transmission at the next possible time.
        LMIC_setTxData2(1, mydata, sizeof(mydata), 0);
        seqnoUp = LMIC.seqnoUp ;
        EEPROM.put(eeAddress,seqnoUp);
        Serial.println("Writing updated seqnoUP "+String(seqnoUp)+" to EEPROM");
        serial_printf(Serial,"Packet [%2x %2x %2x %2x %2x %2x] queued \n",
        mydata[0],mydata[1],mydata[2],mydata[3],mydata[4],mydata[5]);
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

// SPS030 functions
struct sps_values readSPSdata() {
  uint8_t ret, error_cnt = 0;
  struct sps_values val;

  // loop to get data
  do {
    ret = sps30.GetValues(&val);
    // data might not have been ready
    if (ret == ERR_DATALENGTH){
        if (error_cnt++ > 3) {
          ErrtoMess("Error during reading values: ",ret);
        }
        delay(1000);
    }
    // if other error
    else if(ret != ERR_OK) {
      ErrtoMess("Error during reading values: ",ret);
    }

  } while (ret != ERR_OK);
  return(val);
}

struct sps_values avgSPSdata(int counter, sps_values valIn) {
  struct sps_values valAvg; //Structure in which to store average values
  struct sps_values valTemp; //Structure in which to store temporal values
    if (counter == 0) {
    valAvg = valIn;
  }
  else {
    valTemp = readSPSdata();
    valAvg.MassPM1 = (valIn.MassPM1 * (counter - 1) + valTemp.MassPM1) / counter;
    valAvg.MassPM2 = (valIn.MassPM2 * (counter - 1) + valTemp.MassPM2) / counter;
    valAvg.MassPM4 = (valIn.MassPM4 * (counter - 1) + valTemp.MassPM4) / counter;
    valAvg.MassPM10 = (valIn.MassPM10 * (counter - 1) + valTemp.MassPM10) / counter;
    valAvg.NumPM0 = (valIn.NumPM0 * (counter - 1) + valTemp.NumPM0) / counter;
    valAvg.NumPM1 = (valIn.NumPM1 * (counter - 1) + valTemp.NumPM1) / counter;
    valAvg.NumPM2 = (valIn.NumPM2 * (counter - 1) + valTemp.NumPM2) / counter;
    valAvg.NumPM4 = (valIn.NumPM4 * (counter - 1) + valTemp.NumPM4) / counter;
    valAvg.NumPM10 = (valIn.NumPM10 * (counter - 1) + valTemp.NumPM10) / counter;
    valAvg.PartSize = (valIn.PartSize * (counter - 1) + valTemp.PartSize) / counter;
  }
  return(valAvg);    
}

void Errorloop(char *mess, uint8_t r)
{
  if (r) ErrtoMess(mess, r);
  else Serial.println(mess);
  Serial.println(F("Program on hold"));
  for(;;) delay(100000);
}

void ErrtoMess(char *mess, uint8_t r)
{
  char buf[80];

  Serial.print(mess);

  sps30.GetErrDescription(r, buf, 80);
  Serial.println(buf);
}

// Print formatting function
void serial_printf(HardwareSerial& serial, const char* fmt, ...) { 
    va_list argv;
    va_start(argv, fmt);

    for (int i = 0; fmt[i] != '\0'; i++) {
        if (fmt[i] == '%') {
            // Look for specification of number of decimal places
            int places = 2;
            if (fmt[i+1] >= '0' && fmt[i+1] <= '9') {
                places = fmt[i+1] - '0';
                i++;
            }
            
            switch (fmt[++i]) {
                case 'B':
                    serial.print("0b"); // Fall through intended
                case 'b':
                    serial.print(va_arg(argv, int), BIN);
                    break;
                case 'c': 
                    serial.print((char) va_arg(argv, int));
                    break;
                case 'd': 
                case 'i':
                    serial.print(va_arg(argv, int), DEC);
                    break;
                case 'f': 
                    serial.print(va_arg(argv, double), places);
                    break;
                case 'l': 
                    serial.print(va_arg(argv, long), DEC);
                    break;
                case 'o':
                    serial.print(va_arg(argv, int) == 0 ? "off" : "on");
                    break;
                case 's': 
                    serial.print(va_arg(argv, const char*));
                    break;
                case 'X':
                    serial.print("0x"); // Fall through intended
                case 'x':
                    serial.print(va_arg(argv, int), HEX);
                    break;
                case '%': 
                    serial.print(fmt[i]);
                    break;
                default:
                    serial.print("?");
                    break;
            }
        } else {
            serial.print(fmt[i]);
        }
    }
    va_end(argv);
}

5 Operation

In The Things Network, I activated the Storage Integration. In Python I wrote a script that, when ran, downloads the data from TheThingsNetwork, saves it locally as a csv file, creates a graph, and uploads this graph to a website. See the code below.

Downloading the data

import requests
import pandas as pd
import json
import numpy as np

applicationid = 'set_application_id'
deviceid = 'set_device_id'
apikey = 'set_api_key'
url1 = f'https://eu1.cloud.thethings.network/api/v3/as/applications/{applicationid}/devices/{deviceid}/packages/storage/uplink_message'

headers = {'Accept':'text/event-stream','Authorization':f'Bearer {apikey}'}
r = requests.get(url1,headers=headers)

dataList = r.text.strip('\n').split(sep='\n\n')

jsonList = []
timeList = []
PM25List = []
for i,data in enumerate(dataList):
    jsonList.append(json.loads(data)['result']) #data will be a dictionary with one key: result. So, grab directly that one.
    timeList.append(pd.to_datetime(jsonList[i]['received_at']))
    PM25List.append(jsonList[i]['uplink_message']['decoded_payload']['pm25'])
timeArr = np.array(timeList)
pm25Arr = np.array(PM25List)

timeDifference = 3 # For setting the time to local time.
dataDf = pd.DataFrame({'pm25':pm25Arr},index=timeArr+pd.to_timedelta(timeDifference,'H'))

Creating a graph

import matplotlib.pyplot as plt
import mpld3 as mpd
import pandas as pd
import numpy as np

dataDf = pd.read_csv('ttnData.csv',index_col=0)
dataDf.loc[:,'time'] = pd.to_datetime(dataDf.time)
device = 'id_arpm-1'
data = dataDf.loc[dataDf.device_id==device]
fig,ax=plt.subplots(figsize=(16,12))
ax.plot(data.PM25,label=device,lw=4)
ax.set_ylabel('Concentration (\u00B5g/m\u00B2)',size=20)
ax.set_title('PM2.5 concentration for device '+device,size=20)

html_str = mpd.fig_to_html(fig)
html_file = open('figure.html','w')
html_file.write(html_str)
html_file.close()

Uploading the graph

# -*- coding: utf-8 -*-
#sFTP upload (following https://stackoverflow.com/questions/432385/sftp-in-python-platform-independent)
import paramiko
host = "set_website_hostname"                    #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "set_website-hosting_password"                #hard-coded
username = "set_website-hosting_username"                #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

sftp.put('output/figure.html','/www/figure.html')

sftp.close()
transport.close()
print('Upload done.')

This all resulted in a graph on my website, provided that I had a sensor system running, a gateway connected to the internet, and the Python code running.

Arduino PM sensor system 2/3: sensor data on an SD card

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This second post: storing the data on an SD card.

In the previous post I described how I got the sensor data on my computer. Preferably, measurements are done without continuously having a computer around. I wanted to add local data storage to the sensor system with a memory (micro SD) card.

1 Materials

Next to the setup part 1 materials, I bought the following materials:

DS3231 Real Time Clock
SD card module
Micro SD card (in bulk from Aliexpress much cheaper)
10 jumper cables

I found out that for a memory card, it was better to have a relatively small card (< 1GB). Furthermore, after running into battery failure of the real-time clock multiple times, I learned that the DS3231 charging system could destroy non-rechargeable batteries. To resolve this, the charging circuit had to be disabled (for example by removing resistor 201 from the DS3231 shield).

2 Software

Both the real-time clock and the SD card work with libraries that by default are included in the Arduino IDE installation. Additional software was not needed.

3 Connections

I made the following connections between the Aduino Mega and the SD and real-time clock modules:

SD module...Mega
GND.........GND
VCC.........5V
MISO........50
MOSI........51
SCK.........52
CS..........53

DS3231......Mega
GND.........GND
VCC.........5V
SDA.........SDA
SCL.........SCL

I also put the micro SD card in the SD module.

4 Sketch

I used the following sketches for the components:

Example1_sps30_BasicReadings of the sps30 library
ds3231 of the RTClib library
SoftwareSPI of the SDfat library

In the below sketch, these are combined.

See sketch

#include <SPI.h>
#include "SdFat.h"
#include <Wire.h>
#include "RTClib.h"
#include "sps30.h"

SPS30 sps30;
SdFat SD;
RTC_DS3231 rtc;

#define SD_CS_PIN SS
#define SP30_COMMS I2C_COMMS
#define DEBUG 0

File dataFile;
String data_filename= "measurement_data";
String month_cov = "";
String dataheader = "counter;datetime;PM1;PM2_5;PM4;PM10;NumPM1;NumPM2_5;NumPM4;NumPM10;PartSize";
String data ="";
String serialdata = "";
struct sps_values val;
String pmString = "";

long counter = 0; // For every restart, a counter is included

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.print("Initializing RTC module...");
  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }

  Serial.print("Initializing SD card...");

  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
// set driver debug level
  sps30.EnableDebugging(DEBUG);
  // Begin communication channel TO sp30;
  if (!sps30.begin(SP30_COMMS)) 
    {
   while(1);
    }
  
  // check for SPS30 connection
 if (!sps30.probe()) 
   {
    while(1);
   }
  // reset SPS30 connection
  if (!sps30.reset()){
    
    while(1);
  }

  // start measurement
  sps30.start();    
  delay(10);
}

void loop() 
{
  DateTime now = rtc.now();
  sps30.GetValues(&val);
  long pm1 = static_cast<long>(floor(val.MassPM1+0.5));
  long pm2 = static_cast<long>(floor(val.MassPM2+0.5));
  long pm4 = static_cast<long>(floor(val.MassPM4+0.5));
  long pm10 = static_cast<long>(floor(val.MassPM10+0.5));
  long num1 = static_cast<long>(floor(val.NumPM1+0.5));
  long num2 = static_cast<long>(floor(val.NumPM2+0.5));
  long num4 = static_cast<long>(floor(val.NumPM4+0.5));
  long num10 = static_cast<long>(floor(val.NumPM10+0.5));

  pmString = String(pm1)+";"+String(pm2)+";"+String(pm4)+";"+String(pm10)+';'+String(num1)+';'+String(num2)+';'+String(num4)+';'+String(num10)+';'+String(val.PartSize);
  data = String(String(counter)+';'+now.timestamp(DateTime::TIMESTAMP_FULL))+";"+pmString;
  serialdata = String(String(counter)+'\t'+now.timestamp(DateTime::TIMESTAMP_FULL))+'\t'+pmString;
  Serial.println(serialdata);

  // Decide whether headers need to be included
  bool datafile_exists = SD.exists(data_filename);

  dataFile = SD.open(data_filename, FILE_WRITE);
  // if the file opened okay, write to it:
  if (dataFile) {
    if(!datafile_exists) {
      dataFile.println(dataheader);
    }

    dataFile.println(data);
    // close the file:
    dataFile.close();
          
  } else {
    // if the file didn't open, print an errors:
    Serial.println("error opening datafile");      
  }
  ++counter;
  delay(8500);
}

5 Operation and next

After uploading the sketch to the Arduino board, it started to collect data and store it on the SD card. I could get the data from the SD card. This is already sufficient for many measurement projects, but online real-time data would be even nicer. Post three: adding LoRa functionality to the sensor system.

Arduino PM sensor system 1/3: sensor data on a computer

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This first post: getting the sensor data on a computer.

Nowadays, more and more Do-It-Yourself air quality sensors come available. A huge advantage of this is the price: while plug and play particulate matter (PM) instruments are (at least) €300, PM sensors can already be bought for under €20. The two main disadvantages are 1) you have to build it yourself, and 2) the reliability of the instrument might be unknown.

Because of the cost-advantage, I chose to discover these options, despite of the two disadvantages. Across three posts, I describe the development of a PM sensor system, based on the Sensirion SPS30 PM sensor, with help of Arduino. For me, the starting point was absolutely zero knowledge on this topic, hence the posts will cover the full learning curve. If you have more knowledge on this topic, it might well be that you can give me relevant advice on how to improve the eventual sensor measurement set.

1 Materials

Below a list of materials, with links to randomly selected suppliers.

2 Software

I installed the Arduino IDE, downloaded a SPS30 library, and placed it in my Arduino folder (Documents/Arduino/libraries).

3 Connections

The SPS30 has five pins: GND, SEL, SCL, SDA and VCC. I connected these pins according to the below schematic (where the two 10K resistors are connected to the SDA and SCL wiring and 5V as ‘pull-up resistors’).

4 Starting the sensor

I connected the Arduino board to my laptop, and in the Arduino IDE, I selected the right COM port (Tools -> Port).

In the example sketch sps30/Example1_sps30_BasicReadings, I edited the following in the sketch:

Line 161: #define SP30_COMMS I2C_COMMS

I uploaded the sketch, clicked on the serial monitor and put the baud rate on 115200. After hitting enter, I get PM readings on my screen.

5 What’s next?

Now I have PM data on the screen. The data is out there: but how to save it? Post two: storing the data on an SD card.

Personal exposure exceeding guideline values

For the 10 participants of a pilot study, personal exposures to Carbon Monoxide (CO) and PM2.5 concentrations exceed guideline values. You can read more about these measurements on the project page.

Guideline values

The World Health Organization (WHO) advises a PM2.5 guideline of 25 μg/m³ as 24-hour average (source), while Ethiopia uses a guideline value of 65 μg/m³ as 24-hour average (Guideline Ambient Environment Standards for Ethiopia, 2003).

For CO, guideline values are given at shorter time average periods, as health effects from CO can be more direct. Both the WHO and Ethiopia use guideline values at averaging periods of 15 minutes, 30 minutes, 1 hour and 8 hours of respectively 90, 50, 25 and 10 PPM.

Measured CO exposure

During the 24-hour measurements, peaks up to 400 PPM were measured. The figures show the measured CO concentration as boxplots for each of the participants, without and with values outside the whiskers.

CO concentrations, without data outside whiskers. Whiskers extend to 1.5 * (Q3-Q1).

CO concentrations, with data outside whiskers. Whiskers extend to 1.5 * (Q3-Q1).

24-hour average concentrations are below 6 PPM, and for all participants 75% of all data is lower than 5 PPM. There are however peak concentrations of sometimes up to 400 PPM.

To compare the measured concentrations to guideline values, we have to use 15 minute, 30 minute, 1 hour and 8 hour average concentrations. Each of these time averaging period have their own guideline value. To compare them in one graph, the below graph shows the highest average value of each of these time periods, relative to the guideline value for that averaging period. Averaging concentrations higher than 100% imply that the concentration exceeds the guideline value.

CO maximum average concentrations, relative to the respective guideline values.

Measured PM2.5 exposure

Comparing the measured PM2.5 exposure to the guideline value is more straightforward. Below figure shows the 24-hour average concentrations combined with the guideline values.

At all households, the WHO guideline value (25 μg/m³) is exceeded. Only for the participant from household 6, the Ethiopian guideline value (65 μg/m³) is not exceeded.

However, during the 24-hour periods there is a large variability. Periods of relatively low concentrations are mixed with moments of high concentrations. Below figures show the CO and PM2.5 concentration during the whole period, for each of the participants.

TimePlot-HH01

TimePlot-HH02

TimePlot-HH03

TimePlot-HH04

TimePlot-HH06

TimePlot-HH07

TimePlot-HH05

TimePlot-HH08

TimePlot-HH09

TimePlot-HH10

Air pollution courses 2019

From October 2019 – January 2020 I offered two air pollution courses to four sections of students. I offered the course Air, Noise and Soil pollution to year 4 Water Supply and Environmental Engineering students (three sections; 150 students). I offered the course Atmospheric Chemistry and Air Pollution to year 2 Meteorology and Hydrology students (29 students). For both courses, I used 50% for student air pollution research projects. 33 groups studied a self-selected topic with respect to PM2.5, CO and/or CO2 concentrations on or around campus. They collected a combined 17,431 data points across various scenarios.

Course outline

I spread out the courses over approximately 12 weeks. I used fifty percent of the course time for ‘normal’ course contents: lectures on seven chapters, tests, and a written exam. Those activities were spread out over the whole semester. Simultaneously, the students used the other fifty percent for various parts of research projects. The first four weeks, students (with knowledge from the first lectures) selected topics and worked on a measurement plan. The students chose topics like ‘CO2 concentration in and outside a dormitory’, ‘CO concentration in a kitchen during cooking’, ‘PM concentration close to a generator’, etcetera.

Part of the measurement plan was to select measurement instruments and measurement times. In total I had 6 Lascar El-USB-CO, 6 UCB-PATS+, and 6 IQ Air Airvisual Pro instruments available. Students could select the instruments and use them across a period of four weeks. Between weeks 8-11, students analyzed their data and wrote a report. In the final week, all groups gave a small presentation on their research project.

Research and experience

This project was new for the students. Most courses are theory-based and lecture oriented. While uncertain at the start, students clearly liked the challenge and the opportunity to study air pollution themselves rather than only learn about it from lectures. On beforehand, colleagues warned me about giving instruments to students. However, the students rewarded me with their motivation for the trust I put in them. They returned all instruments in good working order. I am glad the instruments were put to good use, rather than only being on display. The students’ work gave me access to air pollution measurements across various scenarios – with some very high concentrations. I am planning to analyze the data of the students combined, and most likely will apply this teaching method in my future courses as well.

[UPDATE]: data collected by the students during these courses have been the basis for two publications:

For the second publication, three students from the courses were co-authors.

Students measure high concentrations

Outdoor concentrations of PM2.5, CO and CO2 are generally about 10 μg/m3, 0.3 PPM and 450 PPM, respectively. However, close to air pollution sources and/or with circumstances of little ventilation, concentrations can be much higher. This is also witnessed by students of Arba Minch University. So far, they have set records of measured concentrations at 1,800 μg/m3 PM2.5 ( coffee ceremony) 1,040 PPM CO (student’s cafeteria kitchen) and 4.200 PPM CO2 (a closed dormitory with students present).

A student measuring PM2.5 during coffee ceremony

The measurements

33 groups of students from Arba Minch University (Bachelor studies Meteorology & Hydrology, year 2, and Water Supply & Environmental Engineering, year 4) are conducting a small air pollution measurement research.

New Year’s Eve concentrations

Happy new year! On September 12, 2019, according to the Ethiopian calendar, a new year (2012 according to the Ethiopian calendar) has started. While in Ethiopia a change of years is not (yet) celebrated with much fireworks, it is thoroughly celebrated with delicious meals on New Year’s day. Of course, these meals will be prepared on the day before. Many of these meals are cooked on biomass fuel. This was visible on our Home Measurements location (see this project page for more information).

PM2.5 measurements
At the Home Measurements location PM2.5 is measured since August 23. The below graph shows half-hour averages between 17:00 and 2:00 (the next day) for August 23 – September 10, versus September 11 (New Year’s Eve).

On New Year's Eve the average concentration between 17:00 - 2:00 was 64 µg/m3, while on the other evenings this ranged from 10 - 39 µg/m3. Half-hour average PM2.5 concentrations were significantly higher on New Year's Eve than on any other evening (t(df:27)>=2.356, p<=0.013).

Doro Wot
The higher concentrations counted for something: on New Year's day we could enjoy delicious Doro Wot (chicken stew) at some of Arba Minch's most welcoming family homes.

First measurement location

The first measurement location has been installed. An IQAir AirVisual sensor is placed at the front of the house of Geertje and Jan Dirk Dingemanse in Arba Minch, Ethiopia. With its user-friendly interface, the installation consists merely out of supplying power and Wi-Fi. If electricity is available, continuous measurements of PM2.5, CO2, Temperature and Humidity are conducted and logged. The time series will give an insight in the air quality, weather conditions and availability of electricity.

A description of the measurement location and realtime measurement data can be viewed on this page. On the same page, the time series data will (irregularly) be made available for download.