Posted on 1 Comment

#41: Datalogging with MQTT, Node-RED, InfluxDB, and Grafana

Note: This page is still a draft! The instructions are still incomplete.

Resources

The plan

Whatever data source you want to record, there are a few common building blocks that you will need. Each of these building blocks has multiple alternatives. There are also totally different architectures that do things in a different way.

However, this particular combination of software is ideal for home automation, because these software building blocks can also be used for other purposes within your overall system.

You can install the various software elements on different computers, or in virtual machines, or using Docker containers, or on a NAS, or in many other ways.

To keep it simple and because many people already have one spare, we’re going to use a Raspberry Pi.

Prepare Raspberry Pi

Begin by installing Raspberry Pi OS (formerly known as “Raspbian”) on your Raspberry Pi. The easiest way to do it these days is to use the Raspberry Pi Imager, which runs on Windows, MacOS, or Linux and can be downloaded from www.raspberrypi.org/downloads/.

Use the Imager to download and install Raspberry Pi OS onto a micro SD card, insert it into your Pi, and start it up.

TODO: logging in first time.

Make sure your Pi has the latest packages by applying all updates:

sudo apt update
sudo apt upgrade

You now have a Raspberry Pi with a basic Raspbian OS installation, ready for customisation.

Step 0: Get the IP address

We’ll need to know the IP address of the Raspberry Pi later. If you’ve connected using the hostname or you’re using a screen and keyboard, you can get the

Step 1: Install the Mosquitto MQTT broker and clients

There are many different MQTT brokers available, but Mosquitto is the most popular and it has served me well for many years. Install both the “mosquitto” package and the “mosquitto-clients” package, so that you have both the broker and some handy command line clients that you can use for testing:

sudo apt install mosquitto mosquitto-clients

The Mosquitto broker will be set up with a default configuration and will work fine out of the box, but we’re going to change the configuration to make it more secure.

You can leave your MQTT broker unprotected, but it’s a good idea to set a username and password on it.

First, create a text file that contains the username and password with a colon to separate them. You can do this on the command line with a single command:

echo "mqtt_username:mqtt_password" > pwfile

This will create a text file called “pwfile” with the details in them.

Use the passwd utility provided as part of the Mosquitto package to encrypt the file:

mosquitto_passwd -U pwfile

Display the contents of the file to verify that it has now been encrypted:

cat pwfile

You will see your username in plaintext, followed by a string of gibberish which is the encrypted form of the password. If the password is still plaintext, it means the encryption command above didn’t work.

Move the encrypted file into the Mosquitto configuration directory:

sudo mv pwfile /etc/mosquitto/

Mosquitto needs to be told where the new file is located. Use an editor such as Nano to open the Mosquitto configuration file:

sudo nano /etc/mosquitto/mosquitto.conf

Near the bottom of the file, just above the “include_dir” line, add these lines:

allow_anonymous false
password_file /etc/mosquitto/pwfile

The first line tells Mosquitto to reject all anonymous connections, and require a password.

The second line tells Mosquitto where to find the list of allowed usernames and passwords.

To save your changes and exit, type CTRL+X, then Y, then ENTER.

Restart Mosquitto so that it uses the new configuration:

sudo /etc/init.d/mosquitto restart

From now on, all connections to your MQTT broker will need to supply the username and password that you configured.

Step 2: Install the InfluxDB time-series database

To install InfluxDB we can use its official repository, because the developers have provided a packages specifically for different operating systems on the Raspberry Pi.

Start by fetching the official repository key and adding it to the local keyring:

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -

Now you can add the repository. There are a few different versions available, so you need to copy and paste the command that matches your operating system.

To find out what version you’re running, you can type the following command:

lsb_release -a

This is will report the operating system type and version, usually with a codename like “Stretch” or “Buster”. Copy and paste the command that matches your system.

For Raspbian Stretch:

echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Raspbian Buster:

echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Ubuntu 20.04LTS:

echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Ubuntu 18.04LTS:

echo "deb https://repos.influxdata.com/ubuntu bionic stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Now that the repository has been added, we need to update the list of packages that are available:

sudo apt update

Install the database package:

sudo apt install influxdb

Tell the systemctl service manager to enable InfluxDB at startup:

sudo systemctl unmask influxdb
sudo systemctl enable influxdb

Start InfluxDB manually this time. In future, it will be started automatically whenever your Raspberry Pi boots up:

sudo systemctl start influxdb

Let’s set up access control for InfluxDB before we do anything else. The default installation of InfluxDB leaves the system wide open, so we’ll start by creating an admin user and setting a password.

Connect to InfluxDB by running the client. We don’t need to use a username or password this time, because nothing has been set yet:

influx

Create a user called “admin”, and put in the password you want to use for it:

CREATE USER admin WITH PASSWORD 'adminpassword' WITH ALL PRIVILEGES

Now you can exit out of InfluxDB. Simply type “exit” and press ENTER.

exit

The InfluxDB configuration needs to be edited so that it will use authentication. Otherwise the admin user that we just created will be ignored. Use a text editor to open the InfluxDB config file:

sudo nano /etc/influxdb/influxdb.conf

Press CTRL+W to search for the section called [HTTP].

auth-enabled = true
pprof-enabled = true
pprof-auth-enabled = true
ping-auth-enabled = true

To save your changes and exit, type CTRL+X, then Y, then ENTER.

The config change won’t be applied until InfluxDB has been restarted, so restart it manually:

sudo systemctl restart influxdb

From now on, any time you want to connect to the InfluxDB command line you will need to supply the username and password.

We need to do that now, so substitute the password you set:

influx -username admin -password <adminpassword>

If the previous changes worked, you should now be connected to InfluxDB again and authenticated as the admin user that you just created.

Next we need to tell InfluxDB to create a database where we can store sensor data. In this example I’ve simply called the database “sensors”:

CREATE DATABASE sensors

That was easy! Because of the way InfluxDB works, there’s no need to create a schema with tables and columns like you would be a relational database such as MariaDB, MySQL, Postgres, or SQL Server.

All you need to do is create an empty database, and it will be automatically populated when you start sending data to it.

Leave the InfluxDB client by typing exit, as before:

exit

Step 3: Install Node-RED

There are several different ways to install Node-RED, and it’s readily available in the Raspbery OS packaging system. However, the Node-RED team recommend that you do NOT use the packaged version.

Instead, they provide a handy script that installs the latest version of Node-RED from the official release, and helps you keep it updated. The installation script goes far beyond that, however. It also:

  • makes sure there’s no existing installation of Node-RED and removes any that it finds in order to prevent conflicts
  • installs the latest Node.js
  • gives you the option of installing a set of useful nodes specifically designed to run on a Pi
  • sets up Node-RED to run as a service and installs tools to help you manage it

Before running the Node-RED installation script, install the tools needed by NPM to build binary modules:

sudo apt install build-essential git

Now you can run the official Node-RED installation script:

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

The script will ask you a couple of questions about whether you are sure you want to proceed, and whether to install Pi-specific nodes. Say "y" (yes) to both questions.

Node-RED can be extended by installing modules to give it extra features. We need to do that now so that it can connect to your InfluxDB database. Use NPM to install the InfluxDB nodes:

npm install node-red-contrib-influxdb

At this point Node-RED is be installed, but just like with InfluxDB you need to configure it to be automatically started on boot:

sudo systemctl enable nodered.service

You can start the service manually this time, but in future it will happen automatically on when your Pi starts up:

sudo systemctl start nodered.service

Step 4: Install Grafana

Just like with InfluxDB, we can install Grafana by adding the official repository and installing the package. Start by fetching the public key for the repository and adding it to the local keyring:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Now add the repository itself:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update the package list (again!) and install the Grafana package:

sudo apt update
sudo apt install grafana

Just like the other packages we've installed, we need to enable the service so that it will start automatically:

sudo systemctl enable grafana-server

That takes care of starting Grafana at boot. For now, let's start it manually:

sudo systemctl start grafana-server

Step 5: Configure your sensors to send data to MQTT

For each sensor that you want to log and report, you will need to go through a series of steps to ensure that the data is received by your Raspberry Pi, processed by Node-RED, stored in InfluxDB, and then charted using Grafana. This is a process that you will repeat multiple times as your build up your home automation system.

Let's break it down into a series of simple steps.

Begin watching MQTT for messages. There won't be any messages yet, but this is a handy technique for discovering new devices when they begin reporting to MQTT. On your Raspberry Pi, open a terminal and run the Mosquitto client command that we ran earlier:

mosquitto_sub -u mqtt_username -P mqtt_password -v -t "#"

This launches the Mosquitto client in "verbose" mode (the "-v" flag) and subscribes to all topics using the wildcard character. What this means is the client will display every message that is published by any client to any topic, and report not just the message but also the topic that it appeared in.

On a busy MQTT broker this is like drinking from a fire-hose!

However, we don't have any sensors using our MQTT broker yet so you shouldn't see anything happen: the terminal will just sit there waiting to display messages. Leave the terminal open so that you can see any messages that are published.

Configure sensor to report using MQTT. The exact process for this will depend on your device. Typically you will need to configure three things in your sensor:

  1. The IP address (or hostname) of your MQTT broker. In this case, it's the IP address of the Raspberry Pi.
  2. The username and password for MQTT. We configured this way back at the start of the process. If you don't have a username and password on your broker, you can skip this.
  3. The MQTT topics for reporting. In many devices this should default to something sensible, and you may not need to change it. For our Air Quality Sensor project, the topic will be generated automatically based on the chip ID of the ESP chip.

If you are using the example Arduino sketch for the Air Quality Sensor, open it in the Arduino IDE and go to the tab called "config.h". Edit the broker IP address and the MQTT username / password to match your own settings:

Compile the sketch and flash it to the Air Quality Sensor, just like in the previous episodes.

After the sensor starts up with the new settings, you should see some action in the Mosquitto client! You'll see a startup message from the sensor, and then it will periodically begin publishing its readings.

Step 6: Receiving and storing single-value sensor readings

aoeu

Step 7: Receiving and storing JSON-formatted sensor readings

aoeu

Step 8: Create a dashboard and widgets in Grafana

You can access the Grafana user interface using a web browser, by connecting to port 3000 on your Raspberry Pi. The URL will look something like:

http://192.168.1.248:3000

Substitute the IP address or hostname for your Raspberry Pi.

To log in, use the username and password "admin" and "admin". Grafana will ask you to change the password on first login, so set it to something secure.

Posted on 3 Comments

Wemos D1 Mini random PCB thickness

If you order a “Wemos D1 Mini” you’d expect to get the same thing every time, right?

Wrong!

The Wemos brand has become a victim of counterfeiters, who make a near-exact copy of the board and then produce it as cheaply as they can. These two D1 Minis look superficially similar, and if they were lying on the bench you would probably have trouble telling them apart:

But look closer. The shape of the PCB is slightly different around the top edges, and the silkscreen is slightly different. It’s most obvious around the “5V” label near the bottom left. There’s also slightly different soldermask pullback around the pads. The board on the left has several parts out of alignment, although electrically they’re still connected and the board works fine.

Part of reducing the cost is to use whatever is the cheapest PCB substrate they can get their hands on at the time. These D1 Minis were made using totally different PCB material:

One PCB is 1mm thick, and the other is 1.6mm.

I’ve seen D1 Minis with 0.8mm, 1.0mm, 1.2mm, 1.4mm, and 1.6mm PCBs! You really can’t predict what you’re going to get.

Why does this matter?

Most of the time, it doesn’t. But recently I heard from someone trying to build my Air Quality Sensor project that their D1 Mini wouldn’t fit in the 3D printed case, because the case only allows PCBs up to 1.2mm thick! Oops.

So I’ve now generated STLs for both 1.2mm and 1.6mm slots, for both the “Basic” and “Display” versions of the case. You can grab the latest STLs from the product pages, linked from the respective episodes:

Posted on Leave a comment

ESP32 pin allocation spreadsheet

The ESP32 is a fantastic MCU, but when you are designing a project with it you need to choose the I/O pins very carefully.

Many of the I/O pins have special purposes, such as being bootstrapping pins that change the way the ESP32 boots up. Some have limitations, such as being OK to use as inputs but they don’t have pullup or pulldown resistors, so you have to bias them externally.

And some of the strange behaviour is really obscure: for example, you can’t use analog input on any pin associated with ADC2 while WiFi is in use!

To help me navigate these limitations I created a spreadsheet that lists the pins and their strange behaviour:

When I start a new project, I use this spreadsheet as a template and make a duplicate just for the project. Then I add notes to the “Purpose” column to say what I’m going to use each pin for.

I’ve made the template public, so you can make copies of it yourself for your projects. Click the preview above to open it in Google Sheets.

Many of the cells have notes attached to them, explaining specific limitations.

Because I mostly use Wroom32 modules, the spreadsheet has a column for Wroom32 pin numbers. You can change this to suit whatever module you use in your project, or ignore it and work from the direct ESP32 pin numbers which are also listed.

Posted on 3 Comments

Vlog #69: Easy laser-cut PCB tray

I needed PCB trays to keep boards organised during the production process, so I laser-cut a simple design.

The trays used by professional PCB assemblers are usually made from static-dissipative material that can also withstand immersion in an ultrasonic bath. My design isn’t intended as a replacement for those, but just as a handy way for hobbyists to keep small batches of boards organised on the workbench or shelves. This is much better than stacking populated PCBs on top of each other because it saves the parts and boards being scratched or damaged.

The source files for this project have been published at https://github.com/jonoxer/PCB-Tray

The Fusion360 file includes parameters so that you can tune the material thickness and the slot size to suit your own requirements. There is also a parameter for board width, but it’s broken at the moment.

DXFs have also been included, for 3mm MDF and 2mm PCB slots.

Posted on Leave a comment

International shipping is back! (Mostly)

Limited international shipping is back!

Some countries and services are not available, and the situation is changing by the day. As of today, there is a list of about 119 countries that I can’t ship to at all. For the rest, “Economy” class packages aren’t being processed, and for some classes of package there are delays of 2 to 5 weeks added to the normal shipping time.

If you add items to your cart and begin the checkout process, the online store will use your delivery address to determine if it’s possible to ship to you, and only offer you shipping methods that are currently available for your country.

So feel free to go ahead and place an order. I’m watching the situation very carefully and checking every order, so if there are any problems I’ll follow up with you.

For the latest status from Australia Post, you can check on their COVID-19 updates page:

auspost.com.au/about-us/news-media/important-updates/coronavirus/coronavirus-international-updates

Thanks for your patience!

Posted on 3 Comments

International shipping suspended

Because of the decrease in international travel over the last couple of months, the postal service doesn’t have enough flights available to carry international letters and parcels.

As of this week, Australia Post can’t process “Economy” class international packages at all. Some premium classes of package, such as trackable services and International Express, are still being accepted at post offices. However, even these more expensive shipping options are experiencing weeks of delays.

Sadly, this means that I simply can’t ship international packages at the moment. I can still ship orders to Australian customers without any problems: domestic packages are being processed and transported as normal. It’s the lack of international flights that is the big problem.

I’ve now turned off all shipping methods for international customers.

The moment that the postal service has resumed international service, I’ll turn international shipping back on. Until then, I can only process orders for Australian customers.

I’m really sorry about that 🙁

In the meantime, please join the SuperHouse Discord chat server and the SuperHouse forum, and join me every Sunday at 10am Melbourne time for a livestream to chat about whatever fun projects we’re working on during the lockdown! This period of isolation can be hard, but we can also use it as an opportunity to connect more online and learn from each other.

Jon

Posted on 17 Comments

#40: DIY air quality sensor, part 3 – software

You can build the Air Quality Sensor project without understanding how the software works, but if you want to know what’s really going on behind the scenes you can join me for a deep dive into the source code.

Part 1 showed how to make the simplest possible air quality sensor. Part 2 added a 128×32 pixel OLED display and a mode button.

Resources

Posted on 33 Comments

#39: DIY air quality sensor, part 2 – “Display” version

It’s surprisingly easy to make your own simple air quality sensor. All you need is a cheap laser-scattering particulate matter sensor, a Wemos D1 Mini, a soldering iron, and Tasmota.

Part 1 showed how to make the simplest possible air quality sensor. Make sure you’ve seen that first, because Part 2 continues from Part 1 to add a 128×32 pixel OLED display and a mode button. We’re also going to install custom firmware to make the sensor last longer.

Resources

3D print the case

Printing a case is totally optional, of course. You can use whatever enclosure you like. My case has been designed to be a press-fit over the PMS5003, which holds the two halves together by friction. See the link above to download the STLs if you want to print it yourself, or you can buy a case from me if you don’t have access to a printer. I’ll include a 6x6x9mm tact switch with the “Display” version of the case:

Put switch into case

The space for the tact switch is very tight.

To fit it into the slot, trim off the pins from the right side and trim the pins on the left side so they are a couple of millimeters long. Bend those leads out at a 45 degree angle, and solder a pair of jumper wires to them.

Push the switch into the cavity in the case as shown above, and guide the wires around the slot to the left and then down.

The button connects between GND and I/O pin D3:

Connect 128×32 OLED to D1 Mini

The 0.91″ 128×32 OLED module already has I2C pull-ups included, so there’s no need to add them. All we need to do is connect power and I2C:

The connections are:

  • OLED GND to D1 Mini GND
  • OLED VCC to D1 Mini 3.3V
  • OLED SCK to D1 Mini pin D1
  • OLED SDA to D1 Mini pin D2

Connect PMS5003 to D1 Mini

In the “Basic” version of this project, we only needed to connect to the 5V, GND, and Tx pins of the PMS5003. Now we also need to connect I/O pin D6 to the sensor’s Rx pin, to allow commands to be sent to the sensor:

The connections are:

  • PMS5003 pin 1 (5V) to D1 Mini 5V
  • PMS5003 pin 2 (GND) to D1 Mini GND
  • PMS5003 pin 4 (Rx) to D1 Mini pin D6
  • PMS5003 pin 5 (Tx) to D1 Mini pin D4

Close the case

This can be more tricky than it sounds! Be gentle so you don’t damage anything.

With the electronics sitting neatly in the “back” (or “right”) half of the case, make sure all the wiring is neatly tucked into the slots. It should look something like this:

Slide the other half of the case over the modules, checking that no wires are pinched between the halves. The case should close completely.

Set up Arduino IDE for ESP8266

If you don’t already have it, download and install the Arduino IDE:

https://www.arduino.cc/en/Main/Software

By default, the Arduino IDE does not support the ESP8266 processor on the D1 Mini. To add this support, go to the following URL and follow the instructions. The easiest method is to use the Boards Manager:

https://github.com/esp8266/Arduino

Install libraries

The sketch uses 4 libraries.

The “PMS” library is directly embedded within the program, so you don’t need to install it. This is done to force a specific version of the library, which was forked from the official version by GitHub user SwapBap to add extra features. There’s nothing you need to do because it’s part of the project.

In the Arduino IDE, go into “Sketch -> Include Library -> Manage Libraries…

Then search for each of these libraries, and click “Install”:

  • GFX library by Adafruit
  • SSD1306 library by Adafruit
  • PubSubClient library by Nick O’Leary

If you need more help, check out this tutorial:

https://www.arduino.cc/en/guide/libraries

Download “AirQualitySensorD1Mini” sketch

Download the AirQualitySensorD1Mini sketch from:

https://github.com/SuperHouse/AirQualitySensorD1Mini

The easiest way is to click the green “Clone or download” button near the top right of the page and select “Download ZIP”. Once you’ve downloaded it, extract the folder and put it in your sketchbook directory. You can find the correct location for this by looking in the Arduino IDE preferences.

Configure and compile sketch

Open the Arduino IDE, open the sketch, and have a look at the 4 tabs across the top. You can ignore the first three. Open the “config.h” tab, and modify the settings for WiFi and your MQTT broker:

Plug in the Air Quality Sensor using a USB cable.

Select “Tools -> Board:” and set it to “LOLIN(WEMOS) D1 R2 & mini”. Select the port, and upload the sketch:

After the sketch has uploaded the Air Quality Sensor will reboot, connect to your WiFi and MQTT broker, and show values on the display once it has successfully received data from the PMS5003.

Stay tuned for part 3, which will explain how the sketch works and how to read the data from it.