Posts

Python package development for meteorological data

The Ethiopian Meteorology Institute (EMI) collects meteorological data from several weather stations all over Ethiopia. Data is processed manually in Microsoft Excel. To validate this data, or to produce tables or graphs, employees spend much time. Zerihun Bikila, employee of EMI and a master student in one of my Python classes, came up with the idea to create automated data processing tools with Python, under the name ‘PyCAMT’ (Python based Climate Data Processing and Visualization). Based on this, we worked on the development of a python package for meteorological data in Ethiopia.

Sample use of the preliminary package: visualizing meteorological data with one line of code.

For a month, I have enthusiastically coded together a preliminary package for importing, processing and visualizing EMI station data. Zerihun worked on a user interface in Jupyter Notebook. We tried to pitch the package to EMI management, but this so far proved difficult. Anyways, it was good practice for me to create a package in Python, and I am sure that I will use what I learned in the near future to help people automize and professionalize data processing and visualization.

I hosted the Python code as pycamtET on my GitHub repository. The Jupyter Notebook interface is hosted as pycamtETinterface. Below, you can see slides that introduce the package.

Measurements in Addis Ababa

With help of Tofikk Redi (Ethiopian Meteorology Institute, EMI), I have installed three low-cost self-assembled low-cost sensors and one IQAir Airvisual Pro at the monitoring station of EMI in Addis Ababa. Tofikk will irregularly download and share the data, but I also want to find out how the set-up is working for a longer period without regular checkup. As for the data itself, I am interested in the intercorrelation as well as the correlation with the IQAir instrument under concentration levels in Addis Ababa. Also, I hope to compare data of my sensors with the Beta Attenuation Monitor (BAM) at this site. The BAM is a reference-grade PM monitor. The BAM at this site is currently not operational, but EMI expects it to be back in business within the coming months.

Installation of low-cost sensor systems at the EMI monitoring station in Addis Ababa.

EMI monitoring station in Addis Ababa

Kitchen validation measurements

With help of my colleague Dagmawi Matewos, I have installed two measurement boxes filled with instruments in the university campus kitchen. In my courses and research I use low-cost measurement instruments, for which the lower costs usually comes at a cost of quality.  Validation is therefore important. We installed the instruments in this kitchen to validate the instruments under high and variable concentration circumstances. In this kitchen, food is prepared for students at various fire pits.

The following instruments are installed across two boxes:

  • Four self-developed PM2.5 sensor systems (ASPM);
  • Six UCB-PATS+ PM2.5 sampling instruments;
  • Ten IQAir Airvisual Pro instruments (measuring PM2.5 and CO2);
  • Ten Lascar EL-USB_CO carbon monoxide sensors;
  • Three UPAS gravimetric PM2.5 sampling instruments.
Measurement box
Validation preparation
Instruments
Kitchen measurements
previous arrow
next arrow

All PM2.5 measurements will be compared with each other. The ASPM, PATS and IQAV measure continuously (at a frequency of <20 seconds). The UPAS instrument is used to collect filters for gravimetric analysis, and can be considered as golden standard for PM2.5 measurements. Hence, instruments can be evaluated based on intra-correlation (how well do they compare to their own type), inter-correlation amongst low-cost sensor types (comparing ASPM with PATS and IQAV, and vice versa), and correlation to the reference (UPAS).

For CO2 and CO I only have one instrument, so for those there is only intra-comparison possible. We have installed an additional six IQAV instruments in a student dormitory, to specifically test the instrument under varying CO2 concentrations.

Article on student science

Aquademia published our article titled An Evaluation of Best Practices in an Air Quality Student Science Project in Ethiopia.

The publication is based on my students’ data from October 2019 – January 2020. We evaluated this practical course’s educational and scientific outcomes to see if combining research and teaching is feasible in Ethiopia’s public university context. Spoiler: it is.

Figure 1 of the publication: comparison of student measurements with other studies, across various scenarios. Measurements by my students are in most cases comparable to other studies.

We speak about ‘student science’ to keep a strong link with ‘citizen science’ (science done by non-scientists), but to stress that it is with students. The benefit of this approach is that, contrary to other citizens, university students are supposed to learn about doing research. Therefore, involving students in conducting research has both benefits for the research field and for the students’ education. I am planning to keep applying this method, and I hope that others at Ethiopian universities will start applying it as well. In a country with both theoretical education and limited research resources, student science is a win-win.

First use of donated laptops

The first 15 GMB laptops arrived in Ethiopia by mail. While already promised in March and transported in May, finally they are in the lab, and students can start to use them!

Donated laptops ready in the air quality lab

 

The laptops were sent by post, but once arrived in Ethiopia, it was quite a challenge to get them to Arba Minch. While the resale value per laptop was 150 euro, the laptops were taxed as new. Furthermore, initially a penalty of 300% was awarded because it was seen as importing laptops without license. After multiple visits to Addis Ababa and a lot of paperwork, the penalty was waived and the laptops were forwarded to Arba Minch. The remaining tax will be refunded by Arba Minch University. Thanks to them and thanks to GMB my students can now get practical computer classes which are not affected by power outages and absent lab assistants!

GMB promised to donate in total 30 laptops. Based on the experience with the first batch, the next batch of 15 laptops will be transported by people visiting us from The Netherlands.

Another Python training

Many colleagues of Arba Minch Water Technology Institute (AWTI) showed interest in attending the Python training which I offered last March. I therefore offered another round of the same training. You can find the training materials through this link.

Based on the previous round, I made a couple of changes:

  • I introduced practice sessions. Experience from the previous round showed that participants mainly followed the training sessions, but did not practice much by themselves. By offering assignments and sessions for working on those assignments, I hoped to stimulate personal practice.
  • Training assistants joined the sessions. From the earlier training I invited colleagues to help during the training. In this way, there were multiple persons available for personal support of the participants. I was assisted by Daniel Asele, Getachew Enssa, Manyazwal Getachew and Samrawit Dereje.
  • I included the possibility of earning a certificate. Earlier, I was unaware that a big reason for joining trainings is to get a certificate. I do however not believe in handing out a certificate based on only training attendance. Therefore, participants need to make a final assignment to be eligible for a certificate, by using their own data.

Interest versus application

The interest in the training was high (66 registrations). However, similar to the previous round, there was quite a difference between interest and actively participating in the full training. 54 persons attended at least one session, but 32 of these attended most sessions for the four core modules. At the moment, only the fifth (bonus) session on multidimensional (satellite) data remains to be offered. Apart from participating in the training, there is also the gap between following a training and using it yourself. Many participants do not realize that skills in Python depend on personal application much more than on following a training. Initially I thought that I could help 60+ persons towards better data processing. I have to adjust my expectations, and I must be happy with the five to ten colleagues that truly get started on using Python.

Air pollution course 2021

The first semester of academic year 2019/2020 I offered two air pollution courses, in which 33 groups of students collected air pollution data. The corona outbreak upended the academic calendar, so that this year’s air pollution course for year 4 Water Supply and Environmental Engineering students became scheduled from April 2021 instead of October 2020. This year’s batch was three sections, with a total of 101 students. One of the lessons of previous round was a lack of time. While a full research project requires due attention for all parts of the research cycle (from formulating research questions all the way to presenting findings), it is challenging to squeeze that into a single course. Therefore, for this year I decided to cooperate with my colleague (Bewketu Aweke), who would offer the course Research Methods to the same batch. From his course, he could cover research methodology, and from my course, I could cover air pollution. The students would conduct one research project that would be assessed for both courses.

Condensed course outline

An oncoming election upended a well-constructed schedule. In the first week of the course, we were informed that there would be six instead of the usual 12-15 weeks available. Students would be sent home after six weeks, and only come back a month later for conducting a final exam. We let the students execute a condensed form of the project, with the following outline:

  • Week 1: topic and instrument introduction, start on proposals;
  • Week 2: deadline of proposals.
  • Weeks 3-4: data collection.
  • Week 5: Data analysis.
  • Week 6: Draft report. Based on feedback on the draft report, students could submit a final report some weeks later.

Simultaneous to the project activities across these weeks, there were the course lectures of the two courses. These lectures covered the theoretical course contents, but also provided input for the project activities. In total 21 groups of students conducted measurements of PM2.5, CO and/or CO2 at self-selected locations on or around campus.

Individual assignment

Apart from time limitation, a lesson of the earlier project was that with group work, it was difficult to let all students participate equally. This year, all students were sent to their hometown after six weeks. This provided a unique opportunity to let students individually apply what their group found during the measurements. Every student got the individual assignment to interpret their groups’ measurement for their hometown situation. This resulted in 101 students thinking about their family members or neighbors that might encounter situations in which they had conducted measurements. For example, a group member that measured during coffee preparation:

We measured an average concentration of 288 microgram/m3 PM2.5 during preparation of coffee ceremony at the campus coffee place. In my hometown we have coffee ceremony. My mom spends three hours of the day in coffee preparation. If my mom is exposed for three hours to a concentration of 288 μg/m3, her 24-hour average will be at least 3*288/24 = 36 μg/m3. Likely, it will be higher, because during the other 21 hours of the day she also cooks food for us. So, her average exposure is most likely higher than the standard of 25 μg/m3. I recommend to my mom to use electric stove.

Or, a group member that conducted measurements close to waste burning:

We measured for 51 minutes while waste was burning. We measured an average concentration of 874 μg/m3. My sister works as a waste collector. She works probably for about two hours per day close to waste burning. If she is exposed for two hours to a concentration of 874 μg/m3, then her 24-hour average exposure will be at least 2*874/24 = 72.83 μg/m3. I recommend that the community reduces waste, and that instead of burning it, they use other waste disposal methods. My sister needs to be careful, wear a mask and stay far away from burning waste.

Advanced Excel training with sixty participants

The past week I have offered a training on Advanced Excel for data processing. Experts in several fields world-wide use Microsoft Excel. In scientific circles, programming languages like Python, Matlab or R are often mentioned. However, Microsoft Excel is encountered a lot in the work field. Advanced use of Microsoft Excel (wildcards, array formulas, nested formulas, assigning variable names, etcetera) enables data processing on a professional level. I gave a training in Python recently. An increased knowledge in Excel helps colleagues as well.

Due to high interest in the training (68 registrations), I offered the two-day training in two shifts. One of the shift was even in two simultaneous sessions. Daniel Asele, Samrawit Dereje and Geertje Dingemanse assisted me in offering the sessions. A small part of the training consisted out of one-way communication with slide presentations. The main part of the training required participants to work themselves through worksheets with step-by-step instructions, under supervision of the instructors.

I divided the training into five modules: assigning names, conditions and searching, grouping data, array formulas and miscellaneous topics. The training materials are available on an OSF repository.

A sample of the Advanced Excel training instructions.

Python training on environmental data processing

Over a period of five weeks, I have offered a five-day Python training for environmental data processing. You can find the training materials on my OSF repository.

During the year 2020, I found myself with a lot of time (students were sent home due to corona) and data (during air pollution courses I offered, students collected data across 30 situations). I decided to spend my time on learning to use Python for data processing. [UPDATE: this resulted in data visualizations for an article about the students’ data collection.] To foster my newfound Python addiction, I decided to turn what I learned into a training for colleagues of the Arba Minch Water Technology Institute. This resulted in a ‘Basic Python for Environmental Data Processing’ training. The training ran from the absolute basics (variables, functions) up to data processing and visualization (Pandas, Matplotlib) and working with multidimensional data (NetCDF).

PowerPoint slide with the outline of the Basic Python for Environmental Data Processing training

The training leaned heavily on lecture slides and participants simultaneously practicing on their own laptop. I offered four modules and one bonus-day, distributed over five weeks (one day per week). To keep groups small, I offered a singe module four times to four groups of participants. While the interest in Python appears to be high (48 participants at the start), the motivation to actively participate in all sessions lacks behind (17 participants followed all sessions). I will have to find out in future trainings whether that is something I can influence.

GMB to donate 30 laptops

GMB, a Dutch civil engineering company, has promised to donate 30 used but well maintained laptops to the air quality laboratory of Arba Minch University through their “Sociaal Fonds GMB“.

My students (undergraduate Water Supply and Environmental Engineering) have hardly any exposure to computers. The few computer classes they take are plagued by electrical outages and absent computer lab assistants. I saw this in their work in the previous two air quality courses. For typing a few sentences or making one graph in Excel, students spend hours, often frustrated about the inability to get the result they want. The few students who can afford their own laptop have considerably better computer skills, simply because they are exposed more.

When the laptops will arrive, I will include Excel lab classes in my course, and lend laptops to (groups of) students in need of a laptop. This gift of GMB leads to more practically educated water engineers, which will contribute in a more efficient way to Ethiopia’s environment and infrastructure.