jddingemanse, Author at Arba Minch Air Quality

Another Python training

Many colleagues of Arba Minch Water Technology Institute (AWTI) showed interest in attending the Python training which I offered last March. I therefore offered another round of the same training. You can find the training materials through this link.

Based on the previous round, I made a couple of changes:

I introduced practice sessions. Experience from the previous round showed that participants mainly followed the training sessions, but did not practice much by themselves. By offering assignments and sessions for working on those assignments, I hoped to stimulate personal practice.
Training assistants joined the sessions. From the earlier training I invited colleagues to help during the training. In this way, there were multiple persons available for personal support of the participants. I was assisted by Daniel Asele, Getachew Enssa, Manyazwal Getachew and Samrawit Dereje.
I included the possibility of earning a certificate. Earlier, I was unaware that a big reason for joining trainings is to get a certificate. I do however not believe in handing out a certificate based on only training attendance. Therefore, participants need to make a final assignment to be eligible for a certificate, by using their own data.

Interest versus application

The interest in the training was high (66 registrations). However, similar to the previous round, there was quite a difference between interest and actively participating in the full training. 54 persons attended at least one session, but 32 of these attended most sessions for the four core modules. At the moment, only the fifth (bonus) session on multidimensional (satellite) data remains to be offered. Apart from participating in the training, there is also the gap between following a training and using it yourself. Many participants do not realize that skills in Python depend on personal application much more than on following a training. Initially I thought that I could help 60+ persons towards better data processing. I have to adjust my expectations, and I must be happy with the five to ten colleagues that truly get started on using Python.

Air pollution course 2021

The first semester of academic year 2019/2020 I offered two air pollution courses, in which 33 groups of students collected air pollution data. The corona outbreak upended the academic calendar, so that this year’s air pollution course for year 4 Water Supply and Environmental Engineering students became scheduled from April 2021 instead of October 2020. This year’s batch was three sections, with a total of 101 students. One of the lessons of previous round was a lack of time. While a full research project requires due attention for all parts of the research cycle (from formulating research questions all the way to presenting findings), it is challenging to squeeze that into a single course. Therefore, for this year I decided to cooperate with my colleague (Bewketu Aweke), who would offer the course Research Methods to the same batch. From his course, he could cover research methodology, and from my course, I could cover air pollution. The students would conduct one research project that would be assessed for both courses.

Condensed course outline

An oncoming election upended a well-constructed schedule. In the first week of the course, we were informed that there would be six instead of the usual 12-15 weeks available. Students would be sent home after six weeks, and only come back a month later for conducting a final exam. We let the students execute a condensed form of the project, with the following outline:

Week 1: topic and instrument introduction, start on proposals;
Week 2: deadline of proposals.
Weeks 3-4: data collection.
Week 5: Data analysis.
Week 6: Draft report. Based on feedback on the draft report, students could submit a final report some weeks later.

Simultaneous to the project activities across these weeks, there were the course lectures of the two courses. These lectures covered the theoretical course contents, but also provided input for the project activities. In total 21 groups of students conducted measurements of PM2.5, CO and/or CO2 at self-selected locations on or around campus.

Individual assignment

Apart from time limitation, a lesson of the earlier project was that with group work, it was difficult to let all students participate equally. This year, all students were sent to their hometown after six weeks. This provided a unique opportunity to let students individually apply what their group found during the measurements. Every student got the individual assignment to interpret their groups’ measurement for their hometown situation. This resulted in 101 students thinking about their family members or neighbors that might encounter situations in which they had conducted measurements. For example, a group member that measured during coffee preparation:

We measured an average concentration of 288 microgram/m3 PM2.5 during preparation of coffee ceremony at the campus coffee place. In my hometown we have coffee ceremony. My mom spends three hours of the day in coffee preparation. If my mom is exposed for three hours to a concentration of 288 μg/m3, her 24-hour average will be at least 3*288/24 = 36 μg/m3. Likely, it will be higher, because during the other 21 hours of the day she also cooks food for us. So, her average exposure is most likely higher than the standard of 25 μg/m3. I recommend to my mom to use electric stove.

Or, a group member that conducted measurements close to waste burning:

We measured for 51 minutes while waste was burning. We measured an average concentration of 874 μg/m3. My sister works as a waste collector. She works probably for about two hours per day close to waste burning. If she is exposed for two hours to a concentration of 874 μg/m3, then her 24-hour average exposure will be at least 2*874/24 = 72.83 μg/m3. I recommend that the community reduces waste, and that instead of burning it, they use other waste disposal methods. My sister needs to be careful, wear a mask and stay far away from burning waste.

Advanced Excel training with sixty participants

The past week I have offered a training on Advanced Excel for data processing. Experts in several fields world-wide use Microsoft Excel. In scientific circles, programming languages like Python, Matlab or R are often mentioned. However, Microsoft Excel is encountered a lot in the work field. Advanced use of Microsoft Excel (wildcards, array formulas, nested formulas, assigning variable names, etcetera) enables data processing on a professional level. I gave a training in Python recently. An increased knowledge in Excel helps colleagues as well.

Due to high interest in the training (68 registrations), I offered the two-day training in two shifts. One of the shift was even in two simultaneous sessions. Daniel Asele, Samrawit Dereje and Geertje Dingemanse assisted me in offering the sessions. A small part of the training consisted out of one-way communication with slide presentations. The main part of the training required participants to work themselves through worksheets with step-by-step instructions, under supervision of the instructors.

I divided the training into five modules: assigning names, conditions and searching, grouping data, array formulas and miscellaneous topics. The training materials are available on an OSF repository.

A sample of the Advanced Excel training instructions.

Python training on environmental data processing

Over a period of five weeks, I have offered a five-day Python training for environmental data processing. You can find the training materials on my OSF repository.

During the year 2020, I found myself with a lot of time (students were sent home due to corona) and data (during air pollution courses I offered, students collected data across 30 situations). I decided to spend my time on learning to use Python for data processing. [UPDATE: this resulted in data visualizations for an article about the students’ data collection.] To foster my newfound Python addiction, I decided to turn what I learned into a training for colleagues of the Arba Minch Water Technology Institute. This resulted in a ‘Basic Python for Environmental Data Processing’ training. The training ran from the absolute basics (variables, functions) up to data processing and visualization (Pandas, Matplotlib) and working with multidimensional data (NetCDF).

PowerPoint slide with the outline of the Basic Python for Environmental Data Processing training

The training leaned heavily on lecture slides and participants simultaneously practicing on their own laptop. I offered four modules and one bonus-day, distributed over five weeks (one day per week). To keep groups small, I offered a singe module four times to four groups of participants. While the interest in Python appears to be high (48 participants at the start), the motivation to actively participate in all sessions lacks behind (17 participants followed all sessions). I will have to find out in future trainings whether that is something I can influence.

GMB to donate 30 laptops

GMB, a Dutch civil engineering company, has promised to donate 30 used but well maintained laptops to the air quality laboratory of Arba Minch University through their “Sociaal Fonds GMB“.

My students (undergraduate Water Supply and Environmental Engineering) have hardly any exposure to computers. The few computer classes they take are plagued by electrical outages and absent computer lab assistants. I saw this in their work in the previous two air quality courses. For typing a few sentences or making one graph in Excel, students spend hours, often frustrated about the inability to get the result they want. The few students who can afford their own laptop have considerably better computer skills, simply because they are exposed more.

When the laptops will arrive, I will include Excel lab classes in my course, and lend laptops to (groups of) students in need of a laptop. This gift of GMB leads to more practically educated water engineers, which will contribute in a more efficient way to Ethiopia’s environment and infrastructure.

LoRa gateway installed

Mekuanint Hailemichael, a colleague of Arba Minch University (senior network manager) has installed a LoRa gateway at a high building of the Nech Sar Campus of Arba Minch University. Installation of the gateway is part of our research project titled “Determining the feasibility and reliability of a do-it-yourself air quality sensor network in Arba Minch, Ethiopia.”.

The gateway is installed in a metal cage.

The building with the gateway is in the middle of Arba Minch, at Nech Sar Campus.

We are planning to install and test multiple Arduino-based particulate matter (PM) sensor systems in Arba Minch. Apart from testing their operationality and data quality, we want to test the possibility of real-time online data through LoRa functionality. The sensor systems transmit small packets of data over a frequency of 868 MHz. A well-positioned gateway can receive this data and put it online. For this purpose, Mekuannint (co-investigator in the research project) has installed a Lorank-8 gateway together with antenna and lighting protection. It is connected to the Arba Minch University network.

Arduino PM sensor system 3/3: sensor data online

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This third post: getting the PM sensor data available online.

Preferably, the measurement data is available real-time online. With the setup of the previous post, we can only access the data by getting it from the SD card. To an Arduino system, a WiFi module can be added to provide online functionality. However, in and around Arba Minch the availability of WiFi is unreliable and limited. I therefore chose to add LoRa (Long Range) functionality to the sensor system, to get the PM sensor data available online.

With LoRa, small packets of data are sent over frequencies of around 433 or 868 MHz. On this frequency, data can be sent over large distances (10+ km). A receiver with internet connection (‘gateway’) can pick up these transmissions and put them online. For small-scale and personal projects, TheThingsNetwork provides network servers for these transmissions for free. In this way, only at one position (the gateway) internet is needed, while within range of that gateway sensor systems can transmit their data.

1 Materials

Individual sensor systems get LoRa functionality through a Dragino LoRa shield. Apart from that, a gateway is needed to receive the data transfers of individual sensor systems. I bought this gateway.

2 Software

To run the LoRa transmission on an arduino board, I used the LMIC arduino library. Apart from that, I created an account on TheThingsNetwork. On that account, I registered an application, and under that application, I registered individual sensor systems as devices.

3 Connections

I mounted the LoRa shield on an Arduino Mega, and connected the SPS30 Sensirion, SD module and DS3231 to the Arduino Mega according to below connections.

SPS030......Mega
1 VCC.......5V
2 SDA.......SDA
3 SCL.......SCL
4 Select....GND
5 GND.......GND

SD module...Mega
GND.........GND
VCC.........5V
MISO........12
MOSI........11
SCK.........13
CS..........53

DS3231......Mega
GND.........GND
VCC.........5V
SDA.........SDA
SCL.........SCL

I had difficulty in getting both the LoRa shield and the SD module to work. Somehow, the communication over MISO/MOSI disturbed each other. While for the default SD library, on the Arduino Mega pins 51-53 are used, I managed to get it working by using SoftwareSPI of the SDfat library and using pins 11-13.

4 Sketch

I used the following sketches for the components:

Example1_sps30_BasicReadings of the sps30 library
ds3231 of the RTClib library
SoftwareSPI of the SDfat library
ttn-abp of the LMIC library

I combined these sketches to create a full sketch that operates all parts together. In that sketch, for every individual sensor system I had to provide the TheThingsNetwork device’s NwkSKey, AppSKey and DevAddr.

Show code of the whole sketch

String Version = "0-5"; //Sketch version
String NodeID = "SPSAXX"; //Instrument ID
String TTNtype = "ABP"; //Type of connection to TTN

///////////////////////include libraries//////////////////////////////////////
#include <SPI.h>
#include <Wire.h>
#include <EEPROM.h>

#include <lmic.h>
#include <hal/hal.h>

#include <DS3231.h>
#include "SparkFunBME280.h"
#include "SdFat.h"
#include "sps30.h"

//***************************************************************************************************
// TTN settings. Change the keys according to the configuration in TTN.                             *
//***************************************************************************************************
// LoRaWAN NwkSKey, network session key
static const PROGMEM u1_t NWKSKEY[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
// LoRaWAN AppSKey, application session key
static const u1_t PROGMEM APPSKEY[16] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
// LoRaWAN end-device address (DevAddr)
static const u4_t DEVADDR = 0x00000000 ; // <-- Change this address for every node!
// Empty callbacks - only filled when using OTAA
void os_getArtEui (u1_t* buf) { }
void os_getDevEui (u1_t* buf) { }
void os_getDevKey (u1_t* buf) { }

//***************************************************************************************************
// Global data.                                                                                     *
//***************************************************************************************************
// Constructors
SPS30 sps30;
BME280 bme280;
DS3231 Clock;

//SD pin definitions
const uint8_t SOFT_MOSI_PIN = 11;
const uint8_t SOFT_MISO_PIN = 12;
const uint8_t SOFT_SCK_PIN  = 13;
const uint8_t SD_CHIP_SELECT_PIN = 53;
SdFatSoftSpi<SOFT_MISO_PIN, SOFT_MOSI_PIN, SOFT_SCK_PIN> sd;

// LMIC Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = LMIC_UNUSED_PIN,
    .dio = {2, 6, 7},
};

//SPS030 commands and function prototypes
#define SP30_COMMS I2C_COMMS //SPS030
#define TX_PIN 0 //SPS030
#define RX_PIN 0 //SPS030
#define DEBUG 0 //SPS030
#define PERFORMCLEANNOW 1 //SPS030 cleaning
void ErrtoMess(char *mess, uint8_t r);
void Errorloop(char *mess, uint8_t r);
struct sps_values readSPSdata();
struct sps_values avgSPSdata(int counter, sps_values valIn);

//Empty file and strings
SdFile file;
String dataString;
String timeString;
String spsMassS;
String spsNumS;
String bmeString;
String fileHead = "YYYY-MM-DD;UU:MM:SS;RH;T;PM1mass;PM2.5mass;PM4mass;PM10mass;PM0num;PM1num;PM2.5num;PM4num;PM10num;Partsize";

//Variables
bool Century=false;                               //DS3231
bool h12, PM, ADy, A12h, Apm;                     //DS3231
byte ADay, AHour, AMinute, ASecond, ABits;        //DS3231
byte year, month, date, DoW, hour, minute, second;//DS3231
int pm25, rh, temp, counter;                      //Data storage and counter for average

int restartEvent = 1; //to indicate a restart / reset has happened
int CleanNeed = 1; //Always 1, then at the chosen hour SPS will do 1 cleaning. For the remainder of that hour it will be 0.

int seqnoUp;                                    // To store the TTN upload framecounter
int eeAddress = 0;                              // Writing address in EEPROM
const unsigned TX_INTERVAL = 300;               // #seconds between each send job
static uint8_t mydata[] = {0,0,0,0,0,0};        // Structure for the data to send over LoRa
static osjob_t sendjob;

//Structures
struct bmeData_t                               // Stucture for BME data
{
  int temperature ;                           
  int rhumidity ;                                           
  String bmeString ;
} ;
struct timeData_t                               // Structure for time data
{
  String timeString ;
  long timeInYear ;
  String month ;
  int hour ;
} ;

sps_values spsValues;
bmeData_t bmeValues;
timeData_t timeValues;

//***************************************************************************************************
// Setup and loop.                                                                                  *
//***************************************************************************************************
void setup() {
  Serial.begin(115200);
  Wire.begin();
  Serial.println("Node: " + NodeID + "; Version: " + Version + "; Connection via: " + TTNtype);
  pinMode(3, OUTPUT);

  EEPROM.get(eeAddress,seqnoUp);
  if (seqnoUp<1) {
      Serial.println("EEPROM empty; setting seqnoUp as 20 and writing to EEPROM. Do not forget to reset frame counters in TTN.");
      seqnoUp = 20;
      EEPROM.put(eeAddress,seqnoUp);
  }
  else {
      seqnoUp += 20;
      EEPROM.put(eeAddress,seqnoUp);
      Serial.println("Writing updated seqnoUP "+String(seqnoUp)+" to EEPROM");
  }
  
  setupLMIC();
  setupBME();
  setupSPS();
}  

void loop() {
  //After restart, let the loop start at the start of a minute
  while (restartEvent==1) {                     
    if (Clock.getSecond() == 0) {
      break;
    }
  }

  //For 50 seconds, take average of SPS values
  counter = 0;
  spsValues = readSPSdata();
  float startMeas = millis()/1000;
  float endMeas = millis()/1000-startMeas;
  while (endMeas < 50) {                          
      if (restartEvent == 0) {
        os_runloop_once();
      }
      spsValues = avgSPSdata(counter,spsValues);
      counter = counter + 1;
      endMeas = millis()/1000-startMeas;
  }   

  //Take all other values, and prepare them for LMIC and SD
  bmeValues = loopBME();
  timeValues = loopDS3231();
  pm25 = int(spsValues.MassPM2+0.5);
  rh = bmeValues.rhumidity;
  temp = bmeValues.temperature;
  mydata[0] = {highByte(pm25)};
  mydata[1] = {lowByte(pm25)};
  mydata[2] = {rh};
  mydata[3] = {temp};
  mydata[4] = {highByte(timeValues.timeInYear)};
  mydata[5] = {lowByte(timeValues.timeInYear)};

  spsMassS = String(spsValues.MassPM1)+';'+String(spsValues.MassPM2)+';'+String(spsValues.MassPM4)+';'+String(spsValues.MassPM10);
  spsNumS = String(spsValues.NumPM0)+';'+String(spsValues.NumPM1)+';'+String(spsValues.NumPM2)+';'+String(spsValues.NumPM4)+';'+String(spsValues.NumPM10)+';'+String(spsValues.PartSize);

  if (restartEvent == 1){
    restartEvent = 0;
    loopSD(fileHead,timeValues.month);    //After every restart, a header line is printed
  }

  //Wait untill full minute, and after that print to SD
  while (true) {
    os_runloop_once();
    if (Clock.getSecond() == 0) {
      break;
    }
  }
  timeValues = loopDS3231();
  dataString = timeValues.timeString + ";" + bmeValues.bmeString + ";" + spsMassS + ";" + spsNumS;
  Serial.print('\n');
  Serial.print(dataString);
  Serial.print('\n');
  loopSD(dataString,timeValues.month); //this loop includes the LED

  CleanNeed = cleanNeed(timeValues.hour,CleanNeed); //SPS cleaning check.
}

//***************************************************************************************************
// Setup functions.                                                                                 *
//***************************************************************************************************
void setupLMIC() {
    #ifdef VCC_ENABLE
    // For Pinoccio Scout boards
    pinMode(VCC_ENABLE, OUTPUT);
    digitalWrite(VCC_ENABLE, HIGH);
    delay(1000);
    #endif

    // LMIC init
    os_init();
    // Reset the MAC state. Session and pending data transfers will be discarded.
    LMIC_reset();

    // Set static session parameters. Instead of dynamically establishing a session
    // by joining the network, precomputed session parameters are be provided.
    #ifdef PROGMEM
    // On AVR, these values are stored in flash and only copied to RAM
    // once. Copy them to a temporary buffer here, LMIC_setSession will
    // copy them into a buffer of its own again.
    uint8_t appskey[sizeof(APPSKEY)];
    uint8_t nwkskey[sizeof(NWKSKEY)];
    memcpy_P(appskey, APPSKEY, sizeof(APPSKEY));
    memcpy_P(nwkskey, NWKSKEY, sizeof(NWKSKEY));
    LMIC_setSession (0x1, DEVADDR, nwkskey, appskey);
    #else
    // If not running an AVR with PROGMEM, just use the arrays directly
    LMIC_setSession (0x1, DEVADDR, NWKSKEY, APPSKEY);
    #endif

    #if defined(CFG_eu868)
    // Set up the channels used by the Things Network, which corresponds
    // to the defaults of most gateways. Without this, only three base
    // channels from the LoRaWAN specification are used, which certainly
    // works, so it is good for debugging, but can overload those
    // frequencies, so be sure to configure the full frequency range of
    // your network here (unless your network autoconfigures them).
    // Setting up channels should happen after LMIC_setSession, as that
    // configures the minimal channel set.
    // NA-US channels 0-71 are configured automatically
    LMIC_setupChannel(0, 868100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(1, 868300000, DR_RANGE_MAP(DR_SF12, DR_SF7B), BAND_CENTI);      // g-band
    LMIC_setupChannel(2, 868500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(3, 867100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(4, 867300000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(5, 867500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(6, 867700000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(7, 867900000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(8, 868800000, DR_RANGE_MAP(DR_FSK,  DR_FSK),  BAND_MILLI);      // g2-band
    // TTN defines an additional channel at 869.525Mhz using SF9 for class B
    // devices' ping slots. LMIC does not have an easy way to define set this
    // frequency and support for class B is spotty and untested, so this
    // frequency is not configured here.
    #elif defined(CFG_us915)
    // NA-US channels 0-71 are configured automatically
    // but only one group of 8 should (a subband) should be active
    // TTN recommends the second sub band, 1 in a zero based count.
    // https://github.com/TheThingsNetwork/gateway-conf/blob/master/US-global_conf.json
    LMIC_selectSubBand(1);
    #endif

    // Disable link check validation
    LMIC_setLinkCheckMode(0);

    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow(DR_SF10,14);

    LMIC.seqnoUp = seqnoUp;

    // Start job
    do_send(&sendjob);  
}

void setupBME() {
  bme280.setI2CAddress(0x76); //check adress with i2c check

  if (bme280.beginI2C() == false) //Begin communication over I2C
  {
    Serial.println("BME280 did not respond. Please check wiring.");
    while(1); //Freeze
  }
}

void setupSPS() {
  // set driver debug level
  sps30.EnableDebugging(DEBUG);

  // Begin communication channel;
  if (sps30.begin(SP30_COMMS) == false) {
    Errorloop("could not initialize communication channel.", 0);
  }

  // check for SPS30 connection
  if (sps30.probe() == false) {
    Errorloop("could not probe / connect with SPS30.", 0);
  }
  else
    Serial.println(F("Detected SPS30."));

  // reset SPS30 connection
  if (sps30.reset() == false) {
    Errorloop("could not reset.", 0);
  }

  // start measurement
  if (sps30.start() == true)
    Serial.println(F("Measurement started"));
  else
    Errorloop("Could NOT start measurement", 0);

  // clean now requested -- at every reset 
  if (PERFORMCLEANNOW) {
    // clean now
    if (sps30.clean() == true)
      Serial.println(F("fan-cleaning started"));
    else
      Serial.println(F("Could NOT start fan-cleaning"));
    delay(15000);
  }
}


//***************************************************************************************************
// Loop functions.                                                                                  *
//***************************************************************************************************
struct bmeData_t loopBME() {
  float rhum,temper;
  rhum = bme280.readFloatHumidity();
  temper = bme280.readTempC();
  bmeData_t bmeData;
  bmeData.temperature = int(temper+0.5);
  bmeData.rhumidity = int(rhum+0.5);
  bmeData.bmeString = String(rhum) + ';' + String(temper);
  return(bmeData);
}

struct timeData_t loopDS3231() {
  int second,minute,hour,date,month,year,temperature; 
  String monthS;
  second=Clock.getSecond();
  minute=Clock.getMinute();
  hour=Clock.getHour(h12, PM);
  date=Clock.getDate();
  month=Clock.getMonth(Century);
  year=Clock.getYear();

  timeData_t timeData;
  timeData.timeString = "20" + String(year) + '-' + String(month) + '-' + String(date) + ';' + String(hour) + ':' + String(minute) + ':' + String(second);
  timeData.timeInYear = int(((month-1)*(31*24*6) + date*24*6 + hour*6 + minute/10 + second/(60*10))-32768)+32768; //#10 minutes in the year, minus 32768 to make full use of arduino int capacity.
  monthS = String(month);
  if (monthS.length()==1) {
    monthS="0"+monthS;
  };
  timeData.month = "20" + String(year) + monthS;
  timeData.hour = hour;

  return(timeData);
}

void loopSD(String dataS, String month) {
  //create a fileName per month, and as such, a new file per month
  String extension = ".txt";
  String fileNameStr = NodeID + "_V" + Version + "_" + month + extension;
  char fileName[fileNameStr.length()+1];
  fileNameStr.toCharArray(fileName, sizeof(fileName));
  
  if (!sd.begin(SD_CHIP_SELECT_PIN)) {
    sd.initErrorHalt();
  }

  if (!file.open(fileName, FILE_WRITE)) {
    sd.errorHalt(F("open failed"));
  }
  file.println(dataS);
  Serial.println("printed to SD");
  digitalWrite(3, HIGH); //when successfully written to SD: switch LED on
  file.close();
  delay(500);
  digitalWrite(3, LOW); //switch LED off
}

int cleanNeed(int hourCheck,int Need) {
  if (hourCheck == 3) {
    if (Need == 1) {
      // clean now requested
      if (PERFORMCLEANNOW) {
        // clean now
        if (sps30.clean() == true)
          Serial.println(F("fan-cleaning started"));
        else
          Serial.println(F("Could NOT start fan-cleaning"));
      }
      Need = 0;    
      delay(15000);
    }
  }
  else {
    Need = 1;
  }
  return(Need);
}

//***************************************************************************************************
// Other functions.                                                                                  *
//***************************************************************************************************
// LMIC functions
void onEvent (ev_t ev) {
    Serial.print(os_getTime());
    Serial.print(": ");
    switch(ev) {
        case EV_SCAN_TIMEOUT:
            Serial.println(F("EV_SCAN_TIMEOUT"));
            break;
        case EV_BEACON_FOUND:
            Serial.println(F("EV_BEACON_FOUND"));
            break;
        case EV_BEACON_MISSED:
            Serial.println(F("EV_BEACON_MISSED"));
            break;
        case EV_BEACON_TRACKED:
            Serial.println(F("EV_BEACON_TRACKED"));
            break;
        case EV_JOINING:
            Serial.println(F("EV_JOINING"));
            break;
        case EV_JOINED:
            Serial.println(F("EV_JOINED"));
            break;
        case EV_RFU1:
            Serial.println(F("EV_RFU1"));
            break;
        case EV_JOIN_FAILED:
            Serial.println(F("EV_JOIN_FAILED"));
            break;
        case EV_REJOIN_FAILED:
            Serial.println(F("EV_REJOIN_FAILED"));
            break;
        case EV_TXCOMPLETE:
            Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
            if (LMIC.txrxFlags & TXRX_ACK)
              Serial.println(F("Received ack"));
            if (LMIC.dataLen) {
              Serial.println(F("Received "));
              Serial.println(LMIC.dataLen);
              Serial.println(F(" bytes of payload"));
            }
            // Schedule next transmission
            os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
            break;
        case EV_LOST_TSYNC:
            Serial.println(F("EV_LOST_TSYNC"));
            break;
        case EV_RESET:
            Serial.println(F("EV_RESET"));
            break;
        case EV_RXCOMPLETE:
            // data received in ping slot
            Serial.println(F("EV_RXCOMPLETE"));
            break;
        case EV_LINK_DEAD:
            Serial.println(F("EV_LINK_DEAD"));
            break;
        case EV_LINK_ALIVE:
            Serial.println(F("EV_LINK_ALIVE"));
            break;
         default:
            Serial.println(F("Unknown event"));
            break;
    }
}

void do_send(osjob_t* j){
    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        // Prepare upstream data transmission at the next possible time.
        LMIC_setTxData2(1, mydata, sizeof(mydata), 0);
        seqnoUp = LMIC.seqnoUp ;
        EEPROM.put(eeAddress,seqnoUp);
        Serial.println("Writing updated seqnoUP "+String(seqnoUp)+" to EEPROM");
        serial_printf(Serial,"Packet [%2x %2x %2x %2x %2x %2x] queued \n",
        mydata[0],mydata[1],mydata[2],mydata[3],mydata[4],mydata[5]);
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

// SPS030 functions
struct sps_values readSPSdata() {
  uint8_t ret, error_cnt = 0;
  struct sps_values val;

  // loop to get data
  do {
    ret = sps30.GetValues(&val);
    // data might not have been ready
    if (ret == ERR_DATALENGTH){
        if (error_cnt++ > 3) {
          ErrtoMess("Error during reading values: ",ret);
        }
        delay(1000);
    }
    // if other error
    else if(ret != ERR_OK) {
      ErrtoMess("Error during reading values: ",ret);
    }

  } while (ret != ERR_OK);
  return(val);
}

struct sps_values avgSPSdata(int counter, sps_values valIn) {
  struct sps_values valAvg; //Structure in which to store average values
  struct sps_values valTemp; //Structure in which to store temporal values
    if (counter == 0) {
    valAvg = valIn;
  }
  else {
    valTemp = readSPSdata();
    valAvg.MassPM1 = (valIn.MassPM1 * (counter - 1) + valTemp.MassPM1) / counter;
    valAvg.MassPM2 = (valIn.MassPM2 * (counter - 1) + valTemp.MassPM2) / counter;
    valAvg.MassPM4 = (valIn.MassPM4 * (counter - 1) + valTemp.MassPM4) / counter;
    valAvg.MassPM10 = (valIn.MassPM10 * (counter - 1) + valTemp.MassPM10) / counter;
    valAvg.NumPM0 = (valIn.NumPM0 * (counter - 1) + valTemp.NumPM0) / counter;
    valAvg.NumPM1 = (valIn.NumPM1 * (counter - 1) + valTemp.NumPM1) / counter;
    valAvg.NumPM2 = (valIn.NumPM2 * (counter - 1) + valTemp.NumPM2) / counter;
    valAvg.NumPM4 = (valIn.NumPM4 * (counter - 1) + valTemp.NumPM4) / counter;
    valAvg.NumPM10 = (valIn.NumPM10 * (counter - 1) + valTemp.NumPM10) / counter;
    valAvg.PartSize = (valIn.PartSize * (counter - 1) + valTemp.PartSize) / counter;
  }
  return(valAvg);    
}

void Errorloop(char *mess, uint8_t r)
{
  if (r) ErrtoMess(mess, r);
  else Serial.println(mess);
  Serial.println(F("Program on hold"));
  for(;;) delay(100000);
}

void ErrtoMess(char *mess, uint8_t r)
{
  char buf[80];

  Serial.print(mess);

  sps30.GetErrDescription(r, buf, 80);
  Serial.println(buf);
}

// Print formatting function
void serial_printf(HardwareSerial& serial, const char* fmt, ...) { 
    va_list argv;
    va_start(argv, fmt);

    for (int i = 0; fmt[i] != '\0'; i++) {
        if (fmt[i] == '%') {
            // Look for specification of number of decimal places
            int places = 2;
            if (fmt[i+1] >= '0' && fmt[i+1] <= '9') {
                places = fmt[i+1] - '0';
                i++;
            }
            
            switch (fmt[++i]) {
                case 'B':
                    serial.print("0b"); // Fall through intended
                case 'b':
                    serial.print(va_arg(argv, int), BIN);
                    break;
                case 'c': 
                    serial.print((char) va_arg(argv, int));
                    break;
                case 'd': 
                case 'i':
                    serial.print(va_arg(argv, int), DEC);
                    break;
                case 'f': 
                    serial.print(va_arg(argv, double), places);
                    break;
                case 'l': 
                    serial.print(va_arg(argv, long), DEC);
                    break;
                case 'o':
                    serial.print(va_arg(argv, int) == 0 ? "off" : "on");
                    break;
                case 's': 
                    serial.print(va_arg(argv, const char*));
                    break;
                case 'X':
                    serial.print("0x"); // Fall through intended
                case 'x':
                    serial.print(va_arg(argv, int), HEX);
                    break;
                case '%': 
                    serial.print(fmt[i]);
                    break;
                default:
                    serial.print("?");
                    break;
            }
        } else {
            serial.print(fmt[i]);
        }
    }
    va_end(argv);
}

5 Operation

In The Things Network, I activated the Storage Integration. In Python I wrote a script that, when ran, downloads the data from TheThingsNetwork, saves it locally as a csv file, creates a graph, and uploads this graph to a website. See the code below.

Downloading the data

import requests
import pandas as pd
import json
import numpy as np

applicationid = 'set_application_id'
deviceid = 'set_device_id'
apikey = 'set_api_key'
url1 = f'https://eu1.cloud.thethings.network/api/v3/as/applications/{applicationid}/devices/{deviceid}/packages/storage/uplink_message'

headers = {'Accept':'text/event-stream','Authorization':f'Bearer {apikey}'}
r = requests.get(url1,headers=headers)

dataList = r.text.strip('\n').split(sep='\n\n')

jsonList = []
timeList = []
PM25List = []
for i,data in enumerate(dataList):
    jsonList.append(json.loads(data)['result']) #data will be a dictionary with one key: result. So, grab directly that one.
    timeList.append(pd.to_datetime(jsonList[i]['received_at']))
    PM25List.append(jsonList[i]['uplink_message']['decoded_payload']['pm25'])
timeArr = np.array(timeList)
pm25Arr = np.array(PM25List)

timeDifference = 3 # For setting the time to local time.
dataDf = pd.DataFrame({'pm25':pm25Arr},index=timeArr+pd.to_timedelta(timeDifference,'H'))

Creating a graph

import matplotlib.pyplot as plt
import mpld3 as mpd
import pandas as pd
import numpy as np

dataDf = pd.read_csv('ttnData.csv',index_col=0)
dataDf.loc[:,'time'] = pd.to_datetime(dataDf.time)
device = 'id_arpm-1'
data = dataDf.loc[dataDf.device_id==device]
fig,ax=plt.subplots(figsize=(16,12))
ax.plot(data.PM25,label=device,lw=4)
ax.set_ylabel('Concentration (\u00B5g/m\u00B2)',size=20)
ax.set_title('PM2.5 concentration for device '+device,size=20)

html_str = mpd.fig_to_html(fig)
html_file = open('figure.html','w')
html_file.write(html_str)
html_file.close()

Uploading the graph

# -*- coding: utf-8 -*-
#sFTP upload (following https://stackoverflow.com/questions/432385/sftp-in-python-platform-independent)
import paramiko
host = "set_website_hostname"                    #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "set_website-hosting_password"                #hard-coded
username = "set_website-hosting_username"                #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

sftp.put('output/figure.html','/www/figure.html')

sftp.close()
transport.close()
print('Upload done.')

This all resulted in a graph on my website, provided that I had a sensor system running, a gateway connected to the internet, and the Python code running.

Arduino PM sensor system 2/3: sensor data on an SD card

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This second post: storing the data on an SD card.

In the previous post I described how I got the sensor data on my computer. Preferably, measurements are done without continuously having a computer around. I wanted to add local data storage to the sensor system with a memory (micro SD) card.

1 Materials

Next to the setup part 1 materials, I bought the following materials:

DS3231 Real Time Clock
SD card module
Micro SD card (in bulk from Aliexpress much cheaper)
10 jumper cables

I found out that for a memory card, it was better to have a relatively small card (< 1GB). Furthermore, after running into battery failure of the real-time clock multiple times, I learned that the DS3231 charging system could destroy non-rechargeable batteries. To resolve this, the charging circuit had to be disabled (for example by removing resistor 201 from the DS3231 shield).

2 Software

Both the real-time clock and the SD card work with libraries that by default are included in the Arduino IDE installation. Additional software was not needed.

3 Connections

I made the following connections between the Aduino Mega and the SD and real-time clock modules:

SD module...Mega
GND.........GND
VCC.........5V
MISO........50
MOSI........51
SCK.........52
CS..........53

DS3231......Mega
GND.........GND
VCC.........5V
SDA.........SDA
SCL.........SCL

I also put the micro SD card in the SD module.

4 Sketch

I used the following sketches for the components:

Example1_sps30_BasicReadings of the sps30 library
ds3231 of the RTClib library
SoftwareSPI of the SDfat library

In the below sketch, these are combined.

See sketch

#include <SPI.h>
#include "SdFat.h"
#include <Wire.h>
#include "RTClib.h"
#include "sps30.h"

SPS30 sps30;
SdFat SD;
RTC_DS3231 rtc;

#define SD_CS_PIN SS
#define SP30_COMMS I2C_COMMS
#define DEBUG 0

File dataFile;
String data_filename= "measurement_data";
String month_cov = "";
String dataheader = "counter;datetime;PM1;PM2_5;PM4;PM10;NumPM1;NumPM2_5;NumPM4;NumPM10;PartSize";
String data ="";
String serialdata = "";
struct sps_values val;
String pmString = "";

long counter = 0; // For every restart, a counter is included

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.print("Initializing RTC module...");
  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }

  Serial.print("Initializing SD card...");

  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
// set driver debug level
  sps30.EnableDebugging(DEBUG);
  // Begin communication channel TO sp30;
  if (!sps30.begin(SP30_COMMS)) 
    {
   while(1);
    }
  
  // check for SPS30 connection
 if (!sps30.probe()) 
   {
    while(1);
   }
  // reset SPS30 connection
  if (!sps30.reset()){
    
    while(1);
  }

  // start measurement
  sps30.start();    
  delay(10);
}

void loop() 
{
  DateTime now = rtc.now();
  sps30.GetValues(&val);
  long pm1 = static_cast<long>(floor(val.MassPM1+0.5));
  long pm2 = static_cast<long>(floor(val.MassPM2+0.5));
  long pm4 = static_cast<long>(floor(val.MassPM4+0.5));
  long pm10 = static_cast<long>(floor(val.MassPM10+0.5));
  long num1 = static_cast<long>(floor(val.NumPM1+0.5));
  long num2 = static_cast<long>(floor(val.NumPM2+0.5));
  long num4 = static_cast<long>(floor(val.NumPM4+0.5));
  long num10 = static_cast<long>(floor(val.NumPM10+0.5));

  pmString = String(pm1)+";"+String(pm2)+";"+String(pm4)+";"+String(pm10)+';'+String(num1)+';'+String(num2)+';'+String(num4)+';'+String(num10)+';'+String(val.PartSize);
  data = String(String(counter)+';'+now.timestamp(DateTime::TIMESTAMP_FULL))+";"+pmString;
  serialdata = String(String(counter)+'\t'+now.timestamp(DateTime::TIMESTAMP_FULL))+'\t'+pmString;
  Serial.println(serialdata);

  // Decide whether headers need to be included
  bool datafile_exists = SD.exists(data_filename);

  dataFile = SD.open(data_filename, FILE_WRITE);
  // if the file opened okay, write to it:
  if (dataFile) {
    if(!datafile_exists) {
      dataFile.println(dataheader);
    }

    dataFile.println(data);
    // close the file:
    dataFile.close();
          
  } else {
    // if the file didn't open, print an errors:
    Serial.println("error opening datafile");      
  }
  ++counter;
  delay(8500);
}

5 Operation and next

After uploading the sketch to the Arduino board, it started to collect data and store it on the SD card. I could get the data from the SD card. This is already sufficient for many measurement projects, but online real-time data would be even nicer. Post three: adding LoRa functionality to the sensor system.

Arduino PM sensor system 1/3: sensor data on a computer

In a series of three posts, I share my road towards a PM sensor system with help of Arduino and the SPS30 Sensirion PM sensor. This first post: getting the sensor data on a computer.

Nowadays, more and more Do-It-Yourself air quality sensors come available. A huge advantage of this is the price: while plug and play particulate matter (PM) instruments are (at least) €300, PM sensors can already be bought for under €20. The two main disadvantages are 1) you have to build it yourself, and 2) the reliability of the instrument might be unknown.

Because of the cost-advantage, I chose to discover these options, despite of the two disadvantages. Across three posts, I describe the development of a PM sensor system, based on the Sensirion SPS30 PM sensor, with help of Arduino. For me, the starting point was absolutely zero knowledge on this topic, hence the posts will cover the full learning curve. If you have more knowledge on this topic, it might well be that you can give me relevant advice on how to improve the eventual sensor measurement set.

1 Materials

Below a list of materials, with links to randomly selected suppliers.

2 Software

I installed the Arduino IDE, downloaded a SPS30 library, and placed it in my Arduino folder (Documents/Arduino/libraries).

3 Connections

The SPS30 has five pins: GND, SEL, SCL, SDA and VCC. I connected these pins according to the below schematic (where the two 10K resistors are connected to the SDA and SCL wiring and 5V as ‘pull-up resistors’).

4 Starting the sensor

I connected the Arduino board to my laptop, and in the Arduino IDE, I selected the right COM port (Tools -> Port).

In the example sketch sps30/Example1_sps30_BasicReadings, I edited the following in the sketch:

Line 161: #define SP30_COMMS I2C_COMMS

I uploaded the sketch, clicked on the serial monitor and put the baud rate on 115200. After hitting enter, I get PM readings on my screen.

5 What’s next?

Now I have PM data on the screen. The data is out there: but how to save it? Post two: storing the data on an SD card.

Personal exposure exceeding guideline values

For the 10 participants of a pilot study, personal exposures to Carbon Monoxide (CO) and PM2.5 concentrations exceed guideline values. You can read more about these measurements on the project page.

Guideline values

The World Health Organization (WHO) advises a PM2.5 guideline of 25 μg/m³ as 24-hour average (source), while Ethiopia uses a guideline value of 65 μg/m³ as 24-hour average (Guideline Ambient Environment Standards for Ethiopia, 2003).

For CO, guideline values are given at shorter time average periods, as health effects from CO can be more direct. Both the WHO and Ethiopia use guideline values at averaging periods of 15 minutes, 30 minutes, 1 hour and 8 hours of respectively 90, 50, 25 and 10 PPM.

Measured CO exposure

During the 24-hour measurements, peaks up to 400 PPM were measured. The figures show the measured CO concentration as boxplots for each of the participants, without and with values outside the whiskers.

CO concentrations, without data outside whiskers. Whiskers extend to 1.5 * (Q3-Q1).

CO concentrations, with data outside whiskers. Whiskers extend to 1.5 * (Q3-Q1).

24-hour average concentrations are below 6 PPM, and for all participants 75% of all data is lower than 5 PPM. There are however peak concentrations of sometimes up to 400 PPM.

To compare the measured concentrations to guideline values, we have to use 15 minute, 30 minute, 1 hour and 8 hour average concentrations. Each of these time averaging period have their own guideline value. To compare them in one graph, the below graph shows the highest average value of each of these time periods, relative to the guideline value for that averaging period. Averaging concentrations higher than 100% imply that the concentration exceeds the guideline value.

CO maximum average concentrations, relative to the respective guideline values.

Measured PM2.5 exposure

Comparing the measured PM2.5 exposure to the guideline value is more straightforward. Below figure shows the 24-hour average concentrations combined with the guideline values.

At all households, the WHO guideline value (25 μg/m³) is exceeded. Only for the participant from household 6, the Ethiopian guideline value (65 μg/m³) is not exceeded.

However, during the 24-hour periods there is a large variability. Periods of relatively low concentrations are mixed with moments of high concentrations. Below figures show the CO and PM2.5 concentration during the whole period, for each of the participants.

TimePlot-HH01

TimePlot-HH02

TimePlot-HH03

TimePlot-HH04

TimePlot-HH06

TimePlot-HH07

TimePlot-HH05

TimePlot-HH08

TimePlot-HH09

TimePlot-HH10