Posted on 2 Comments

#42: The ESPFlash programming header for ESP8266 / ESP32

There are far too many one-off programming headers for Espressif MCUs. I wanted a simple programming header that I could use consistently across my projects, and I was surprised that there hasn’t been any consensus around a single header.

This is my attempt to specify a header format that can be used in future ESP8266 / ESP32 projects, and hopefully could even be adopted by companies that make popular products based on Espressif chips.

Resources

Posted on 6 Comments

#41: Datalogging with MQTT, Node-RED, InfluxDB, and Grafana

Note: This page is still a draft! The instructions are still incomplete.

Resources

The plan

Whatever data source you want to record, there are a few common building blocks that you will need. Each of these building blocks has multiple alternatives. There are also totally different architectures that do things in a different way.

However, this particular combination of software is ideal for home automation, because these software building blocks can also be used for other purposes within your overall system.

You can install the various software elements on different computers, or in virtual machines, or using Docker containers, or on a NAS, or in many other ways.

To keep it simple and because many people already have one spare, we’re going to use a Raspberry Pi.

Prepare Raspberry Pi

Begin by installing Raspberry Pi OS (formerly known as “Raspbian”) on your Raspberry Pi. The easiest way to do it these days is to use the Raspberry Pi Imager, which runs on Windows, MacOS, or Linux and can be downloaded from www.raspberrypi.org/downloads/.

Use the Imager to download and install Raspberry Pi OS onto a micro SD card, insert it into your Pi, and start it up.

TODO: logging in first time.

Make sure your Pi has the latest packages by applying all updates:

sudo apt update
sudo apt upgrade

You now have a Raspberry Pi with a basic Raspbian OS installation, ready for customisation.

Step 0: Get the IP address

We’ll need to know the IP address of the Raspberry Pi later. If you’ve connected using the hostname or you’re using a screen and keyboard, you can get the

Step 1: Install the Mosquitto MQTT broker and clients

There are many different MQTT brokers available, but Mosquitto is the most popular and it has served me well for many years. Install both the “mosquitto” package and the “mosquitto-clients” package, so that you have both the broker and some handy command line clients that you can use for testing:

sudo apt install mosquitto mosquitto-clients

The Mosquitto broker will be set up with a default configuration and will work fine out of the box, but we’re going to change the configuration to make it more secure.

You can leave your MQTT broker unprotected, but it’s a good idea to set a username and password on it.

First, create a text file that contains the username and password with a colon to separate them. You can do this on the command line with a single command:

echo "mqtt_username:mqtt_password" > pwfile

This will create a text file called “pwfile” with the details in them.

Use the passwd utility provided as part of the Mosquitto package to encrypt the file:

mosquitto_passwd -U pwfile

Display the contents of the file to verify that it has now been encrypted:

cat pwfile

You will see your username in plaintext, followed by a string of gibberish which is the encrypted form of the password. If the password is still plaintext, it means the encryption command above didn’t work.

Move the encrypted file into the Mosquitto configuration directory:

sudo mv pwfile /etc/mosquitto/

Mosquitto needs to be told where the new file is located. Use an editor such as Nano to open the Mosquitto configuration file:

sudo nano /etc/mosquitto/mosquitto.conf

Near the bottom of the file, just above the “include_dir” line, add these lines:

allow_anonymous false
password_file /etc/mosquitto/pwfile

The first line tells Mosquitto to reject all anonymous connections, and require a password.

The second line tells Mosquitto where to find the list of allowed usernames and passwords.

To save your changes and exit, type CTRL+X, then Y, then ENTER.

Restart Mosquitto so that it uses the new configuration:

sudo /etc/init.d/mosquitto restart

From now on, all connections to your MQTT broker will need to supply the username and password that you configured.

Step 2: Install the InfluxDB time-series database

To install InfluxDB we can use its official repository, because the developers have provided a packages specifically for different operating systems on the Raspberry Pi.

Start by fetching the official repository key and adding it to the local keyring:

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -

Now you can add the repository. There are a few different versions available, so you need to copy and paste the command that matches your operating system.

To find out what version you’re running, you can type the following command:

lsb_release -a

This is will report the operating system type and version, usually with a codename like “Stretch” or “Buster”. Copy and paste the command that matches your system.

For Raspbian Stretch:

echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Raspbian Buster:

echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Ubuntu 20.04LTS:

echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

For Ubuntu 18.04LTS:

echo "deb https://repos.influxdata.com/ubuntu bionic stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Now that the repository has been added, we need to update the list of packages that are available:

sudo apt update

Install the database package:

sudo apt install influxdb

Tell the systemctl service manager to enable InfluxDB at startup:

sudo systemctl unmask influxdb
sudo systemctl enable influxdb

Start InfluxDB manually this time. In future, it will be started automatically whenever your Raspberry Pi boots up:

sudo systemctl start influxdb

Let’s set up access control for InfluxDB before we do anything else. The default installation of InfluxDB leaves the system wide open, so we’ll start by creating an admin user and setting a password.

Connect to InfluxDB by running the client. We don’t need to use a username or password this time, because nothing has been set yet:

influx

Create a user called “admin”, and put in the password you want to use for it:

CREATE USER admin WITH PASSWORD 'adminpassword' WITH ALL PRIVILEGES

Now you can exit out of InfluxDB. Simply type “exit” and press ENTER.

exit

The InfluxDB configuration needs to be edited so that it will use authentication. Otherwise the admin user that we just created will be ignored. Use a text editor to open the InfluxDB config file:

sudo nano /etc/influxdb/influxdb.conf

Press CTRL+W to search for the section called [HTTP].

auth-enabled = true
pprof-enabled = true
pprof-auth-enabled = true
ping-auth-enabled = true

To save your changes and exit, type CTRL+X, then Y, then ENTER.

The config change won’t be applied until InfluxDB has been restarted, so restart it manually:

sudo systemctl restart influxdb

From now on, any time you want to connect to the InfluxDB command line you will need to supply the username and password.

We need to do that now, so substitute the password you set:

influx -username admin -password <adminpassword>

If the previous changes worked, you should now be connected to InfluxDB again and authenticated as the admin user that you just created.

Next we need to tell InfluxDB to create a database where we can store sensor data. In this example I’ve simply called the database “sensors”:

CREATE DATABASE sensors

That was easy! Because of the way InfluxDB works, there’s no need to create a schema with tables and columns like you would be a relational database such as MariaDB, MySQL, Postgres, or SQL Server.

All you need to do is create an empty database, and it will be automatically populated when you start sending data to it.

Leave the InfluxDB client by typing exit, as before:

exit

Step 3: Install Node-RED

There are several different ways to install Node-RED, and it’s readily available in the Raspbery OS packaging system. However, the Node-RED team recommend that you do NOT use the packaged version.

Instead, they provide a handy script that installs the latest version of Node-RED from the official release, and helps you keep it updated. The installation script goes far beyond that, however. It also:

  • makes sure there’s no existing installation of Node-RED and removes any that it finds in order to prevent conflicts
  • installs the latest Node.js
  • gives you the option of installing a set of useful nodes specifically designed to run on a Pi
  • sets up Node-RED to run as a service and installs tools to help you manage it

Before running the Node-RED installation script, install the tools needed by NPM to build binary modules:

sudo apt install build-essential git

Now you can run the official Node-RED installation script:

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

The script will ask you a couple of questions about whether you are sure you want to proceed, and whether to install Pi-specific nodes. Say "y" (yes) to both questions.

Node-RED can be extended by installing modules to give it extra features. We need to do that now so that it can connect to your InfluxDB database. Use NPM to install the InfluxDB nodes:

npm install node-red-contrib-influxdb

At this point Node-RED is be installed, but just like with InfluxDB you need to configure it to be automatically started on boot:

sudo systemctl enable nodered.service

You can start the service manually this time, but in future it will happen automatically on when your Pi starts up:

sudo systemctl start nodered.service

Step 4: Install Grafana

Just like with InfluxDB, we can install Grafana by adding the official repository and installing the package. Start by fetching the public key for the repository and adding it to the local keyring:

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Now add the repository itself:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update the package list (again!) and install the Grafana package:

sudo apt update
sudo apt install grafana

Just like the other packages we've installed, we need to enable the service so that it will start automatically:

sudo systemctl enable grafana-server

That takes care of starting Grafana at boot. For now, let's start it manually:

sudo systemctl start grafana-server

Step 5: Configure your sensors to send data to MQTT

For each sensor that you want to log and report, you will need to go through a series of steps to ensure that the data is received by your Raspberry Pi, processed by Node-RED, stored in InfluxDB, and then charted using Grafana. This is a process that you will repeat multiple times as your build up your home automation system.

Let's break it down into a series of simple steps.

Begin watching MQTT for messages. There won't be any messages yet, but this is a handy technique for discovering new devices when they begin reporting to MQTT. On your Raspberry Pi, open a terminal and run the Mosquitto client command that we ran earlier:

mosquitto_sub -u mqtt_username -P mqtt_password -v -t "#"

This launches the Mosquitto client in "verbose" mode (the "-v" flag) and subscribes to all topics using the wildcard character. What this means is the client will display every message that is published by any client to any topic, and report not just the message but also the topic that it appeared in.

On a busy MQTT broker this is like drinking from a fire-hose!

However, we don't have any sensors using our MQTT broker yet so you shouldn't see anything happen: the terminal will just sit there waiting to display messages. Leave the terminal open so that you can see any messages that are published.

Configure sensor to report using MQTT. The exact process for this will depend on your device. Typically you will need to configure three things in your sensor:

  1. The IP address (or hostname) of your MQTT broker. In this case, it's the IP address of the Raspberry Pi.
  2. The username and password for MQTT. We configured this way back at the start of the process. If you don't have a username and password on your broker, you can skip this.
  3. The MQTT topics for reporting. In many devices this should default to something sensible, and you may not need to change it. For our Air Quality Sensor project, the topic will be generated automatically based on the chip ID of the ESP chip.

If you are using the example Arduino sketch for the Air Quality Sensor, open it in the Arduino IDE and go to the tab called "config.h". Edit the broker IP address and the MQTT username / password to match your own settings:

Compile the sketch and flash it to the Air Quality Sensor, just like in the previous episodes.

After the sensor starts up with the new settings, you should see some action in the Mosquitto client! You'll see a startup message from the sensor, and then it will periodically begin publishing its readings.

Step 6: Receiving and storing single-value sensor readings

aoeu

Step 7: Receiving and storing JSON-formatted sensor readings

aoeu

Step 8: Create a dashboard and widgets in Grafana

You can access the Grafana user interface using a web browser, by connecting to port 3000 on your Raspberry Pi. The URL will look something like:

http://192.168.1.248:3000

Substitute the IP address or hostname for your Raspberry Pi.

To log in, use the username and password "admin" and "admin". Grafana will ask you to change the password on first login, so set it to something secure.

Posted on 18 Comments

#40: DIY air quality sensor, part 3 – software

You can build the Air Quality Sensor project without understanding how the software works, but if you want to know what’s really going on behind the scenes you can join me for a deep dive into the source code.

Part 1 showed how to make the simplest possible air quality sensor. Part 2 added a 128×32 pixel OLED display and a mode button.

Resources

Posted on 35 Comments

#39: DIY air quality sensor, part 2 – “Display” version

It’s surprisingly easy to make your own simple air quality sensor. All you need is a cheap laser-scattering particulate matter sensor, a Wemos D1 Mini, a soldering iron, and Tasmota.

Part 1 showed how to make the simplest possible air quality sensor. Make sure you’ve seen that first, because Part 2 continues from Part 1 to add a 128×32 pixel OLED display and a mode button. We’re also going to install custom firmware to make the sensor last longer.

Resources

3D print the case

Printing a case is totally optional, of course. You can use whatever enclosure you like. My case has been designed to be a press-fit over the PMS5003, which holds the two halves together by friction. See the link above to download the STLs if you want to print it yourself, or you can buy a case from me if you don’t have access to a printer. I’ll include a 6x6x9mm tact switch with the “Display” version of the case:

Put switch into case

The space for the tact switch is very tight.

To fit it into the slot, trim off the pins from the right side and trim the pins on the left side so they are a couple of millimeters long. Bend those leads out at a 45 degree angle, and solder a pair of jumper wires to them.

Push the switch into the cavity in the case as shown above, and guide the wires around the slot to the left and then down.

The button connects between GND and I/O pin D3:

Connect 128×32 OLED to D1 Mini

The 0.91″ 128×32 OLED module already has I2C pull-ups included, so there’s no need to add them. All we need to do is connect power and I2C:

The connections are:

  • OLED GND to D1 Mini GND
  • OLED VCC to D1 Mini 3.3V
  • OLED SCK to D1 Mini pin D1
  • OLED SDA to D1 Mini pin D2

Connect PMS5003 to D1 Mini

In the “Basic” version of this project, we only needed to connect to the 5V, GND, and Tx pins of the PMS5003. Now we also need to connect I/O pin D6 to the sensor’s Rx pin, to allow commands to be sent to the sensor:

The connections are:

  • PMS5003 pin 1 (5V) to D1 Mini 5V
  • PMS5003 pin 2 (GND) to D1 Mini GND
  • PMS5003 pin 4 (Rx) to D1 Mini pin D6
  • PMS5003 pin 5 (Tx) to D1 Mini pin D4

Close the case

This can be more tricky than it sounds! Be gentle so you don’t damage anything.

With the electronics sitting neatly in the “back” (or “right”) half of the case, make sure all the wiring is neatly tucked into the slots. It should look something like this:

Slide the other half of the case over the modules, checking that no wires are pinched between the halves. The case should close completely.

Set up Arduino IDE for ESP8266

If you don’t already have it, download and install the Arduino IDE:

https://www.arduino.cc/en/Main/Software

By default, the Arduino IDE does not support the ESP8266 processor on the D1 Mini. To add this support, go to the following URL and follow the instructions. The easiest method is to use the Boards Manager:

https://github.com/esp8266/Arduino

Install libraries

The sketch uses 4 libraries.

The “PMS” library is directly embedded within the program, so you don’t need to install it. This is done to force a specific version of the library, which was forked from the official version by GitHub user SwapBap to add extra features. There’s nothing you need to do because it’s part of the project.

In the Arduino IDE, go into “Sketch -> Include Library -> Manage Libraries…

Then search for each of these libraries, and click “Install”:

  • GFX library by Adafruit
  • SSD1306 library by Adafruit
  • PubSubClient library by Nick O’Leary

If you need more help, check out this tutorial:

https://www.arduino.cc/en/guide/libraries

Download “AirQualitySensorD1Mini” sketch

Download the AirQualitySensorD1Mini sketch from:

https://github.com/SuperHouse/AirQualitySensorD1Mini

The easiest way is to click the green “Clone or download” button near the top right of the page and select “Download ZIP”. Once you’ve downloaded it, extract the folder and put it in your sketchbook directory. You can find the correct location for this by looking in the Arduino IDE preferences.

Configure and compile sketch

Open the Arduino IDE, open the sketch, and have a look at the 4 tabs across the top. You can ignore the first three. Open the “config.h” tab, and modify the settings for WiFi and your MQTT broker:

Plug in the Air Quality Sensor using a USB cable.

Select “Tools -> Board:” and set it to “LOLIN(WEMOS) D1 R2 & mini”. Select the port, and upload the sketch:

After the sketch has uploaded the Air Quality Sensor will reboot, connect to your WiFi and MQTT broker, and show values on the display once it has successfully received data from the PMS5003.

Stay tuned for part 3, which will explain how the sketch works and how to read the data from it.

Posted on 31 Comments

#38: DIY air quality sensor, part 1 – “Basic” version

It’s surprisingly easy to make your own simple air quality sensor. All you need is a cheap laser-scattering particulate matter sensor, a Wemos D1 Mini, a soldering iron, and Tasmota.

Part 1 shows how to make the simplest possible version of the SuperHouse Air Quality Sensor, the “Basic” version.

Part 2 shows how to improve it to make a more advanced “Display” version, Part 3 is a detailed walkthrough of how the firmware works, Part 4 will explain how various Air Quality Index calculations are done around the world, and Part 5 will step it right up with the “Pro” version with VOC sensor.

Resources

Background

Here in Australia we’ve recently had a terrible bushfire season, and there’s been a lot of concern about the effects of smoke on air quality. Smoke particles, pollen, and many other forms of air pollution are tiny. Really tiny. Many people have been wearing “PM2.5” face masks, which means they are designed to filter out particles as small as 2.5 microns in size, but pollen and smoke can be even smaller than that.

Detecting tiny particles down in the 0.1 to 10 micron range is hard. Normally you’d need an electron microscope to detect something this small. In fact these particles are so small that they are often smaller than the wavelength of light, which is in the region of 0.5 microns. This means light doesn’t interact with them in the usual way: instead of bouncing off them like a particle, it can bend around them like a wave, and it can also refract inside them like tiny lenses.

Laser-scattering particle detectors pass a laser through a test chamber, with a deflection sensor on the other side. Particles in the test chamber cause the beam to be scattered in different ways, and the deflection sensor can use this scattering to determine the number and size of particles in the chamber.

This process creates a series of concentric rings on the deflection sensor, which can use the spacing of the rings to determine the characteristics of the particles.

The particles that are detected are then categorised, or “binned”, into different size ranges for reporting.

All of this functionality is wrapped up in the tiny Plantower PMS5003 sensor, which provides a serial interface to report the statistics for the particles that it detects.

Reporting is normally provided in a couple of forms.

The simplest number is the PPD value, which stands for “particles per deciliter”. The higher the number, the more particles there are in the air sample.

The most commonly used number is the “ug/m^3” value, which is micrograms per cubic meter. This is the total mass of particles in that size range in a cubic meter of air, so smaller particles will require a higher raw count to add up to the same mass as larger particles.

Parts required

The “Basic” version of the SuperHouse Air Quality Sensor only requires a a Plantower PMS5003, the connection cable that comes with it, and a Wemos D1 Mini. See the links above for sources to buy them.

You can also print your own 3D-printed enclosure, or buy one from me if you don’t have easy access to a printer.

Connect PMS5003 to D1 Mini

The PMS5003 requires a 5V supply because it has an internal fan to keep air flowing through the test chamber, but all its I/O connections are 3.3V.

This is perfect for a D1 Mini, because we can connect the power pins to 5V on the D1 Mini and connect the sensor “TX” pin to an I/O header on the Mini.

Cut the supplied cable off at about 2/3rds of its length. Plug it into the PMS5003 to check the orientation, and then strip and tin the ends of the wires connected to pins 1, 2, and 5.

Connect VCC and GND from the PMS5003 to the 5V and GND headers on the D1 Mini.

Connect TX from the PMS5003 to the D4 header on the D1 Mini, which is GPIO2 for the ESP8266 MCU.

Connections between PMS5003 and D1 Mini

3D printed case

I’ve designed a case that slips over the PMS5003 and provides a space for the D1 Mini and wires. You can download the STLs and print it yourself, or if you don’t have access to a 3D printer you can buy it from me in a variety of colour options.

The case comes in two halves. Insert the PMS5003 into the case first, and then slip the D1 Mini into the space at the bottom. The case is oriented so that if the USB socket is on the side towards the PMS5003 it will be accessible through a slot.

Push the wires down into the case so everything is neat, and then slide the other half of the case over the top. It may take some jiggling to get the wires out of the way and the D1 Mini properly inserted into the slot.

Install Tasmota

There are many ways to install Tasmota on the D1 Mini, but my new favorite is to use Tasmotizer. I did a whole video about how to install and use Tasmotizer, so follow the instructions here if you need to install it:

SuperHouse #37: Installing Tasmota with Tasmotizer

Once you have Tasmotizer running, connect the D1 Mini using a USB cable and select the port.

Note: As of Tasmota v8.2, support for the PMS5003 has been removed from the normal “tasmota.bin” release binary. Instead, you must select “tasmota-sensors.bin”.

  1. Click the button next to “Release“.
  2. From the drop-down, select “tasmota-sensors.bin” (not the regular “tasmota.bin” as shown above)
  3. Click “Tasmotize!” and wait about a minute for the D1 Mini to be flashed and reboot.

Configure WiFi and apply the “SuperHouse AQS” template

After the D1 Mini has been flashed, click “Send config” in Tasmotizer to open the configuration screen.

  1. Click the checkbox to enable the WiFi section, and put in the details for your WiFi network.
  2. If you use MQTT, click the checkbox to enable the WiFi section and put in the address for your broker. My personal preference is to change the “Topic” entry to “tasmota-%06X” so that the MQTT topics will be dynamically generated using the unique ID of each D1 Mini.
  3. Click the button next to “Template”.
  4. Copy the template below, and paste it into the text box at the bottom of the config screen:
{"NAME":"SuperHouse AQS","GPIO":[255,255,69,255,255,255,255,255,255,255,255,255,255],"FLAG":15,"BASE":18}

Click “Save” to send the new configuration to the D1 Mini.

It will then restart, and will connect to your WiFi network using the values you just provided.

Connect to Tasmota web interface

You can use a serial console connected at 115200bps to get the IP address directly from the D1 Mini, or look in the management interface for your DHCP server to find the recently added device.

Once you’ve found the IP address, put it into a web browser to load up Tasmota’s web interface.

The module will report its latest readings, updating as fast as the sensor is sending them.

You can also configure the D1 Mini manually if you didn’t use the template above. Starting from a default Tasmota installation, log into the web interface and set up the module as shown below.

Access data via MQTT

Tasmota periodically publishes all the PMS5003 readings aggregated into a single JSON string, in the “../SENSOR” topic.

The topic will depend on the settings you applied for MQTT. In my case, with the “Topic” modified as explained above, the readings are published to a topic that looks similar to “tele/tasmota-5F596F/SENSOR” with the unique ID of this particular device substituted.

The published value will look something like this:

{"Time":"2020-03-17T13:08:11","PMS5003":{"CF1":2,"CF2.5":5,"CF10":5,"PM1":2,"PM2.5":5,"PM10":5,"PB0.3":516,"PB0.5":144,"PB1":38,"PB2.5":0,"PB5":0,"PB10":0}}

You can parse the JSON using your favorite method, such as with Node-RED. I’ll go over this in more detail in Part 2 when we look at more advanced software options.

The “PMx” values are micrograms per cubic meter. The “PBx” values are particles per deciliter.

Further improvements

This “Basic” version of the SuperHouse Air Quality Sensor is very simple to make, but it has a couple of limitations.

The laser and the deflection sensor in the PMS5003 can degrade over time. After about 6000 hours of continuous use it may begin to decrease in sensitivity. The life of the sensor can be dramatically extended by shutting it down for a couple of minutes, then waking it up, giving it time to stabilise, taking a reading, and shutting it down again. With this change the sensor can last for many years.

There is also no way to interact with the device directly. You have to access it over the network, either by loading the Tasmota web interface or by using the data published to MQTT. Adding a screen and a mode button can make it easy to read particulate levels directly on the device.

Both of these limitations are fixed in the “Display” version of the SuperHouse Air Quality Sensor in the next episode. Stay tuned!

Posted on 11 Comments

#36: 3D-printed PCB workstation using acupuncture needles

This was a surprisingly fun and useful project!

Connecting test probes to PCBs can be difficult when the contact points are very small, or when you need to keep the probes in place while using your hands to run tests or use a computer. Normal test probes for multimeters, oscilloscopes, and other equipment have to be held in place.

This amazing 3D-printed PCB workstation uses acupuncture needles as test probes. The test probes are attached to adjustable arms that can hold them in position on the device under test.

Print the parts and make one for your own lab:

Resources

Parts required

You can print the plastic parts yourself using the files provided on Thingiverse, or you can buy a kit from the designer. I printed the parts over the space of a few days while I was working on other things. The base takes a few hours to print and there are many other parts, so don’t try to rush through it. Collect everything you need and lay it out to make sure you have it all.

  • Probe arms: 3D-printed parts, M4 bolt with hex head, washer, and M4 wing nut
  • Pack of acupuncture needles (I used these ones in 0.35x40mm size)
  • 3D-printed base plate
  • PCB mounts: 3D-printed bracket, M5 bolt with hex head, washer, and M5 wing nut
  • light-weight, flexible hook-up wire
  • Pin headers
  • Heat-shrink tubing (I used 3mm on the pin headers, and 1.5mm on the needles)
  • Self-adhesive rubber feet
  • Ferrules and crimper: useful if acupuncture needs have stainless steel ends

Assemble base and pcb holders

The Thingiverse project includes both large and small PCB holders. I’ve only printed the small ones so far. Thread an M5 bolt up through the base and a bracket, and put an M5 washer and wing-nut on top. Make sure the bracket can slide along the slot.

Stick a rubber foot under each corner of the base, to help it sit securely on your bench and give the bolt heads enough clearance to slide without sticking.

Assemble probe brackets

Insert the vertical bracket into the mounting base. I used a drop of superglue to lock it in place.

Danger! If you put superglue into the mounting base and then squeeze in the vertical bracket, the superglue can squirt out under high pressure. Be very careful that you don’t squirt it into your eyes!

The handles of the acupuncture probes that I bought are about 1.3mm in diameter, and didn’t fit into the mounting clips. I drilled out the clips with a 1.5mm drill, and used super-glue to attach them in place with most of the handle sticking out the top.

The mounting clips are a press-fit into the horizontal arm. Use super-glue to fix them permanently.

Pass an M4 bolt through the vertical mount and horizontal arm, then put an M4 washer and wing-nut on it.

If the end of the acupuncture needle is plain steel, you can solder the wire directly onto it. My acupuncture needles are all stainless steel so I used a ferrule with the plastic cover removed, and crimped the wire onto the end of the needle.

I put 1.5mm heat-shrink tubing over the needle, with just the end exposed. This is optional but it may help prevent the probes from short-circuiting against each other.

Thread the wire along the horizontal arm. What you put on the other end of the wire is up to you: I soldered on a pin header and then put heat-shrink tubing over the joint. Alternatively, you could put on an alligator clip, a banana plug, a spring clip, or whatever suits you.

Usage

With the device under test mounted on the base, press-fit test probes into the base. Use the handles on the test probes to rotate them, and tighten the wing nut when the needle is in position.

The needles are quite springy, so it’s easy to adjust their position with a pair of tweezers after they are approximately right. The heat-shrink on the needle helps with this, because it’s easy to grip with the tweezers.

Posted on 8 Comments

#35: Sonoff BasicZBR3: Zigbee, not WiFi!

Every single Sonoff model has used the Espressif ESP8266 / ESP8285 processor – until now.

But now ITEAD has gone in a different direction, and released the Sonoff BasicZBR3 which drops WiFi entirely in favour of Zigbee. It doesn’t even use an Espressif processor: instead it uses a Texas Instruments CC2530, which is based on the 40-year-old Intel 8051 processor architecture.

What is Zigbee, and how does it compare to WiFi? Is the Sonoff BasicZBR3 any good?

Let’s find out!

Resources

Zigbee basics

Zigbee is a wireless communications system that’s similar to WiFi in some ways, but they are optimised for very different purposes. They both use the 2.4GHz unlicensed ISM band, and they’re both based on standards managed by the IEEE: 802.11 for WiFi, 802.15.4 for Zigbee.

WiFi is optimised for devices with plenty of power available, and that also require very high throughput. It provides bandwidth in the region of hundreds of Mbps. Perfect for laptops, smartphones, tablets, and security cameras.

Zigbee is optimised for devices that need to run at extremely low power for a long time, and that don’t need to transfer much data. It provides bandwidth of 250kbps, so only about 1/1000th as much as WiFi. It’s suited to tiny devices that need to run on a coin cell for several years, and only pass small amounts of data. Great for temperature sensors and similar IoT devices that only need to send tiny amounts of data every few minutes.

Star topology vs mesh topology

WiFi uses a star topology, with each client device connecting directly to an access point. All communication is arbitrated through the access points, and end devices don’t talk to each other directly. The access points are linked together using some sort of backhaul, usually wired Ethernet. With this topology, you need enough access points to provide coverage of the entire area where you will be deploying devices:

Zigbee is much more flexible in its topology. It can be operated as a simple star just like WiFi, but it can also operate as a mesh because some Zigbee devices can act as relays to extend the coverage of the network:

There are three types of Zigbee device.

Zigbee End Devices (ZED) are nodes that only connect to one other device. They can enter a deep sleep mode to go offline when they aren’t sending data, so they can typically operate for 2 years or more from a tiny coin cell. Typical ZEDs include temperature sensors, light switches, and motion detectors. There can be many ZEDs in a typical Zigbee network.

Zigbee Routers (ZR) are nodes that don’t go into sleep mode. They stay awake continually, ready to pass on messages between other Zigbee devices. They are also functional devices in their own right: they include smart power plugs and light globes, because these typically have mains power available and can remain operating indefinitely. They don’t need to go into sleep mode to conserve a battery. With a few ZRs spread across your coverage area, your network can provide connections for many ZEDs.

Zigbee Coordinators (ZC) are the most important type of node. There can only ever be one ZC in each network. It always takes the first network address, and then it assigns addresses to all the other devices in the network as they join. You can think of it as being like the DHCP server in a typical LAN. Usually the ZC will also act as a gateway to other types of network, such as connecting your Zigbee network onto your WiFi network or wired LAN. Many home automation hubs can act as a ZC.

Pairing with Alexa

The Sonoff BasicZBR3 is wired up the same way as other Sonoff models, with active and neutral coming in on the input side on the left, and then going out to the load (such as a lamp) on the right. Follow the instructions provided by ITEAD and the guides in my previous videos to make the connections.

Once the BasicZBR3 is installed, turn on the power. It will begin in pairing mode, ready to join an existing Zigbee network. Startup is very fast, and it will be ready within a second or so.

You will need a home automation hub that can operate as a Zigbee Coordinator, such as an Amazon Echo Plus.

With the BasicZBR3 turned on and in pairing mode, and within range of your Echo, say “Alexa, discover devices.”

Your Echo will tell you that it is looking for new devices, and the BasicZBR3 will flash its output when it is discovered. The Echo will wait for some time in case there are devices that are slow to respond, so you will have to wait a minute before it will report back that it has found your new device.

That’s it, you’re done! You can now control the BasicZBR3 using commands such as “Alexa, turn on the first plug.”

Customisation of the node names can be done through the Alexa app.

CC2530 processor connections

Programming the CC2530 requires power to the device, plus DD (Debug Data), DC (Debug Clock), and Reset.

In the BasicZBR3, the output relay is controlled by I/O pin P0_7, and it reads the tactile switch using pin P1_3:

The BasicZBR3 provides the debug connections for the CC2530 in a handy 5-pin header, so if you want to mess around with the firmware and load your own code into it, these are the connections on the PCB:

If you write any custom firmware for the Sonoff BasicZBR3, please join the forum or Discord server and share your results.

Posted on 14 Comments

#34: Science or sorcery? Sonoff T4EU1C no-neutral wifi touch switch

The new Sonoff T4EU1C no-neutral touch switch seems to defy the laws of physics. It works with just a live wire, and doesn’t need a neutral wire.

How can that possibly work? And can we install Tasmota on it?

Let’s find out.

Resources

No-neutral power connections

Traditional house wiring has an active and a neutral that meander around the ceiling to each light location, with the neutral connected directly to the light. The active comes down through the wall to the light switch, and then the output from the switch goes back up the wall to the active side of the light fitting.

This works well with passive switches, but most smart light switches need a neutral connection to power their own electronics. With only the active wiring going to the light switch, it can be very difficult to replace existing switches with smart switches.

Building codes in many countries now require that all light switches must have a neutral connection, but this is a recent change so most existing houses don’t have a neutral wire at the switches.

Smart light switches that are designed to be installed in place of existing switches and don’t require a neutral connection are generally called “no-neutral” switches.

Sonoff T4EU1C

The Sonoff T4EU1C is a no-neutral smart light switch with WiFi, so it only needs the active-in and active-out connections that are already found at most light switch locations. It operates by drawing a small amount of power through the light even when the light is turned off, so that it can run its own electronics.

The T4EU1C consists of two PCBs joined together by an 8-way header. The top PCB is the low-voltage logic board, which includes the processor, WiFi antenna, 3.3V voltage regulator, touch sensor pad and driver IC, and the programming header:

The bottom PCB is the high-voltage mains board, which includes a 12V-output mains power supply, and a relay and Triac for controlling the active output to the load:

Re-flashing the firmware with Tasmota

WARNING: There are problems running Tasmota on this particular model. It worked for me, but many other people have problems with the touch switch even when Tasmota is installed correctly. The rest of the Sonoff operates properly and it can be controlled by WiFi, but the touch sensor doesn’t work. There are a few people collaborating right now to figure out why this happens. If you’re feeling lucky, you can give it a try. However, it’s possible that you won’t be able to use the touch sensor until this problem is solved. There is ongoing discussion about it on the SuperHouse Discord server so please join in.

Just like all other Sonoff models so far, the firmware on the T4EU1C can be replaced with alternative open-source firmware to add new features such as local-network operation, MQTT, etc.

However, out of all the Sonoff models that have been released this is perhaps the most difficult one to re-flash. It can be done if you have the equipment and ability to make soldered connections to very tiny parts on the PCB, and this particular board also seems to have some strange requirements to force it into bootloader mode.

The common process for putting a Sonoff into bootloader mode is to hold the GPIO0 pin low (ie: to GND) while powering-up the board. In my testing, I found that this would result in the Sonoff appearing to accept the upload of the new firmware, but a strange checksum error would occur in the flash memory and it wouldn’t boot properly afterwards.

The process that I’ve found to work requires making a connection to the RESET pin as well as GPIO0, so that it can be put into bootloader mode after it has already been powered-up and connected to the programming adapter.

Programming connections

Re-flashing the firmware on the T4EU1C requires the usual connections to the programming header found on the logic board as shown in the picture above, plus connections to GPIO0 and RESET to force it into bootloader mode. Connection to the programming header is easy, because it’s exposed on the logic board in the same format as almost all other Sonoff models.

The difficulty is gaining access to GPIO0 and RESET, because neither of these lines are brought out onto pads for easy connections.

The most convenient place to make a connection to GPIO0 is the very thin track that runs diagonally on the top side of the logic board. Use a scalpel to carefully scrape away some of the green solder mask along the track. Be very careful that you don’t cut the track, and don’t scrape away the solder mask from the surrounding copper areas. If you do, it will be much harder to connect a wire to the track without causing a short circuit to the surrounding copper.

RESET is even harder. The only place it is exposed is on the end of a tiny resistor labelled “R3”, with a very small gap to an adjacent resistor. The connection must be made very carefully and quickly. If the resistor is allowed to heat up for more than about one second, the solder on the other end will also melt, and the resistor will move.

To control the GPIO0 and RESET pins, I soldered a pair of momentary buttons together and connected 3 lengths of very thin wire-wrap wire to them:

  • The common (blue) wire is soldered to GND on the logic board.
  • One switched wire (yellow, in my example) is soldered to the GPIO0 track after scraping off the soldermask.
  • One switched wire (red, in my example) is soldered to the RESET line by attaching it to the bottom end of the resistor marked R3.

Be extremely careful when making these connections to the PCB. It’s very easy to damage the tracks or the parts, and if you don’t have the necessary equipment and skill to rework the board you may destroy your Sonoff.

Programming header connections

Use a USB-to-Serial adapter and jumper wires to connect the Sonoff to your computer. Make sure the USB-to-Serial adapter supports 3.3V mode, because the ESP8285 can be damaged if you connect 5V to it. The connections are the same as explained in previous videos:

In this case the power switch doesn’t matter, because we will use GPIO0 and the RESET line to force the Sonoff into bootloader mode.

The connections can be made with loose jumper wire, or you can use my Sonoff Programming Adapter which does the same thing.

Tasmota upload

The T4EU1C has the same pin assignments as a Sonoff Basic, so download the generic pre-compiled Tasmota binary.

Install the “Esptool” program, and open a terminal in the same directory as the Tasmota binary.

With the Sonoff connected to your computer as described above, and the buttons ready to assert GPIO0 and RESET, run Esptool as below:

esptool.py -p /dev/tty.SLAB_USBtoUART write_flash -fm dout 0x0 sonoff.bin

You may need to adjust the serial port address to match your computer.

Immediately after you press “enter”, follow this sequence:

  1. Press and hold both buttons that you have attached to the Sonoff.
  2. Wait about 2 seconds.
  3. Release the RESET button.
  4. Wait about 2 seconds.
  5. Release the GPIO0 button.

You should see Esptool immediately identify the processor on the Sonoff and begin uploading Tasmota. This entire process will take about 30 seconds. You should see output similar to this:

 esptool.py v2.6
 Serial port /dev/tty.SLAB_USBtoUART
 Connecting…….._
 Detecting chip type… ESP8266
 Chip is ESP8285
 Features: WiFi, Embedded Flash
 MAC: 2c:f4:32:a8:18:8b
 Uploading stub…
 Running stub…
 Stub running…
 Configuring flash size…
 Auto-detected Flash size: 1MB
 Compressed 515872 bytes to 355877…
 Wrote 515872 bytes (355877 compressed) at 0x00000000 in 31.3 seconds (effective 131.7 kbit/s)…
 Hash of data verified.
 Leaving…
 Hard resetting via RTS pin…

At the end of the upload, Esptool will report 100% completion and say that it is resetting the board. However, it can’t actually reset the board because it doesn’t have any connection to the RESET line. You can press the reset button yourself, or disconnect the Sonoff from the programmer and re-connect it to cycle the power.

The Sonoff will then begin the normal Tasmota setup process, creating a WiFi network and opening a serial connection through the programming header. Follow the usual steps as shown in the Tasmota documentation to set it up, selecting “Sonoff Basic” as the module type.

The T4EU1C should then behave as expected, with control using the touch sensor on the front panel and also via WiFi.

ESP8266 / ESP8285 boot messages

If the ESP8285 didn’t enter bootloader mode properly, the Tasmota upload may have appeared to work in the previous step but the Sonoff still won’t boot. Esptool can show its normal upload progress, ending with the message saying it is 100% complete and resetting the board. But then when you power cycle or reset the board, it never comes online and you can’t tell why.

While booting, the ESP8285 outputs debug information to the programming header at the unusual baud rate of 74880bps. Most serial terminal programs can’t operate at that speed, because it’s not a standard baud rate.

The “CoolTerm” serial console is available for Mac, Linux, and Windows, and can be configured to support 74880bps.

Download and install CoolTerm following the instructions on its site.

After it has been installed, find where the binary is located on your computer. On a Mac, it’s typically inside the “/Applications” directory.

In that same directory, make a text file called “baudrates.ini” and put in the number “74880” without quotes or any other characters. Save the file, then start up CoolTerm.

You should now find 74880bps listed as a supported baud rate in the “Options” menu.

Select 74880 as the baud rate, select the serial port for your programming adapter, and click “OK”:

Click the “Connect” button to open the connection to your Sonoff, and then press the reset button on the Sonoff to force it to reboot.

If the ESP8285 has failed to boot, it will display an error similar to this:

ets Jan 8 2013,rst cause:1, boot mode:(3,0)
load 0x40100000, len 27144, room 16
tail 8
chksum 0xef
load 0x00000000, len 0, room 0
tail 0
chksum 0xef
load 0x00000000, len 0, room 8
tail 0
chksum 0xef
csum 0xef
csum err
ets_main.c

It will then hang, and won’t proceed any further.

If you see this message, there is an error with the program stored in flash memory and you should try following the steps above to flash the Sonoff again.

Posted on 33 Comments

#33: Sonoff Mini and S55: ALMOST perfect

The new Sonoff Mini DIY and Sonoff S55 are (so far) the best Sonoff devices that ITEAD has made. Flash them with Tasmota, and they get even better.

Resources

Background

The original Sonoff Basic didn’t have any provision for an external button or switch. One of the most common hacks is to add a switch, so ITEAD decided to make it easy and put screw terminals right there on the Mini.

The Mini is also the smallest Sonoff model so far, which makes it ideal for retrofitting behind an existing switch. Combined with the screw terminals to connect the switch, the Mini is probably the best general-purpose Sonoff for anyone wanting to retrofit home automation to an existing home.

The S55 is an exterior-rated GPO (General Purpose Outlet) or power point. It allows you to have both local control of the power point using a button, and also control from your home automation system.

Both are nice devices, but they become even better when the amazing Open Source Tasmota firmware is installed on them.

Over the years there have been many methods developed to do OTA (Over The Air) replacement of the existing Sonoff firmware with an alternative such as Tasmota. However, I’ve found that these methods are either frustratingly complex or just stop working because ITEAD change things with new releases. I don’t even bother trying these methods anymore: I just go straight to connecting a USB-to-Serial adapter to the programming pins on the Sonoff’s processor. It can be tricky to get the connections sometimes, but it’s guaranteed to work every time single time.

Warning: Do not attempt to connect a programming adapter to any Sonoff model while mains power is connected. ALWAYS disconnect the Sonoff and power it ONLY through the 3.3V connection.

Install Tasmota on Sonoff Mini

The programming connections for the Sonoff Mini are provided on tiny flat pads on the bottom of the PCB:

The button on the top of the PCB is connected to GPIO0, so if you connect to GND, 3.3V, TX, and RX, you can put the Sonoff Mini into programming mode by holding down the button while connecting power from the programmer.

Unfortunately the pads on the PCB are very small, so soldering jumper wires onto them is tricky. If you have a small soldering iron and good eyesight (or magnification) you can solder wires directly onto the pads.

Just be careful you don’t apply too much heat, because the pads can come away from the PCB!

To get around that problem I used my 3D printer to make a programming jig using a design published on Thingiverse. This programming jig uses “pogo pins”, which are spring-loaded pins that can make temporary connections to a PCB. Pogo pins come in a variety of sizes and shapes. I used P75-B1 pins, which means they have a 0.75mm diameter pin with a simple conical point:

This design has 3 parts: a base that holds 4 pogo pins, a small spacer that slips over the pins to help keep them aligned, and a top that goes over the pins and then provides alignment guides for the Sonoff Mini.

I used super-glue to attach the top to the base, once I’d checked everything was aligned:

The Sonoff Mini can be attached using an elastic band:

The pogo pins push against the pads on the PCB:

Hold down the button on the PCB while plugging in the programming adapter to force the Sonoff to go into programming mode, then you can use Esptool or whatever other method you prefer to load the Tasmota binary. In my case I used Esptool, and the command looked like this:

esptool.py -p /dev/tty.usbmodem141101 write_flash -fm dout 0x0 sonoff-6_6_0.bin

However, this command is specific to the USB port that I used on my computer. You may need to adjust the command to suit your needs. This is all documented well on the Tasmota site.

Because the Sonoff Mini has different pin arrangements to a normal Sonoff Basic, and it’s designed to use a normal switch instead of a button, you need to load a special configuration for it.

After you’ve been through the normal Tasmota setup process and connected it to your WiFi as described in the Tasmota docs, go to the Tasmota templates site at blakadder.github.io/templates/ and search for “Mini”. You’ll find a template that looks like this:

{"NAME":"Sonoff Mini","GPIO":[17,0,0,0,9,0,0,0,21,56,0,0,255],"FLAG":0,"BASE":1}

In the Tasmota interface, go to Configuration -> Configure Other and paste the template into the “Template” field.

Make sure the “Activate” check-box is ON, then click “Save”.

Your Sonoff Mini will now operate as expected, with the external switch operating in the normal way.

Install Tasmota on Sonoff S55

The Sonoff S55 doesn’t have a “DIY mode” jumper, so that’s not an option. Luckily, the usual programming header is provided on the PCB ready for you to solder on wires, solder on a header, or just press connections against it. The button for external control is linked to GPIO0, so it’s very easy to put it into programming mode.

The header is in the normal 0.1″ pitch so it’s large enough to solder on a header directly. There’s no need to use a complex pogo pin programming jig.

Solder some hookup wires directly onto the pads, or solder on a header:

I soldered a 4-way 0.1″ header socket onto the pads, with the pins bent a little sideways to make the head sit almost horizontal with the PCB. This makes it easy to plug in a Sonoff Programming Adapter, or use jumper wires to your own USB-to-serial converter.

Hold down the GPIO0 button while you connect the programming header to put it into programming mode, then follow the normal instructions for installing Tasmota. This is very well documented on the Tasmota site.

There’s no specific device profile for the S55 in Tasmota, but it uses the same pinouts as the Sonoff S26 and other mains-plug adapters. Simply select “Sonoff S2x” as the module type in the configuration interface, and it will work as expected.

Posted on 37 Comments

#32: Rock solid WiFi for home automation, part 1

WiFi is a critical part of most home automation systems, but many people simply install the WiFi router that came with their Internet plan and forget about it. However, setting up a good home WiFi network that will handle dozens of IoT devices and still keep your home secure takes a bit more work.

Resources

Planning your network

The typical home network uses an all-in-one router to provide many different network services, including:

  • Router
  • Firewall
  • WiFi access point
  • Ethernet switch
  • Media streaming
  • VPN server
  • DNS resolution
  • DHCP server
  • Printer sharing
  • File serving
  • … and more

This usually results in a network topology that looks like this:

All-in-one WiFi routers try to do too much, so they end up being terrible at everything.

Well structured networks generally separate critical services into different devices that are optimised to do that job, and do it well. Access points don’t try to be file servers, and DHCP servers aren’t Ethernet switches.

Learning from the way that small corporate networks are structured is a great way to improve your home network and improve its reliability, performance, and security.

A better way to structure a home network is like this:

To convert the typical network into the improved network, we’re going to make changes in a series of steps.

For this tutorial I’m using parts from the UniFi range from Ubiquiti Networks, which is a range of networking products designed to be installed across small to medium corporate networks. It uses a management system that provides a single interface to control all your devices, even if they’re installed in physically separate locations.

Shopping list

The specific devices that I installed included:

  • 1 x UniFi Security Gateway
  • 2 x UniFi UAP-AC-Pro access points (probably should have got Nano HD though)
  • 1 x Raspberry Pi 3 B+ (or you can use a UniFi Cloud Key)
  • 1 x UniFi US-8-60W 8-port managed Ethernet switches (not shown in this episode: will feature in Part 2)
  • 2 x UniFi US-24 24-port managed Ethernet switches (not shown in this episode: will feature in Part 2)

Step 1: Set up management system

Ubiquiti Networks provides management software called UniFi Network Controller. It has a web interface, so you run the UniFi Network Controller software on a computer somewhere on your network (or even outside your network) and then access it using a web browser.

The UniFi Network Controller can be run in three different ways:

  1. On your local computer. You can download the software and run it on your Mac, Windows, or Linux computer. This is a great way to test it out and see how it works, but if your computer is turned off the management software will also stop running. If you have a computer that you leave on all the time, such as a home media server, you can run the UniFi Network Controller on it. Download it for your computer from Ubiquiti Networks.
  2. On a UniFi Cloud Key. This is a tiny embedded computer a bit like a Raspberry Pi, which comes with the UniFi Network Controller software preinstalled. You just plug it into your network, open a web browser on your own computer, and log into it. The Cloud Key is a low-power device so you can leave it running continuously.
  3. On a Raspberry Pi or other low power computer. This is what I’m doing. A board such as a Raspberry Pi B+ has plenty of power to run the UniFi Network Controller and still do other things as well, such as run PiHole or other services. Effectively this is just like a DIY version of the Cloud Key, but it’s cheaper and you can do other things with it.

Step 1a: Install Raspbian on a Raspberry Pi

Start with a standard installation of Raspbian, and update all the packages in the usual way. I won’t go into detail of this because it’s very well documented elsewhere. You can either follow the official guide to installing OS images, or use NOOBS to install Raspbian.

Step 1b: Change network settings and password

After installation, make sure your Raspberry Pi is connected to wired Ethernet (not just to WiFi) and log in to the Raspberry Pi as the default “pi” user. The default password is “raspberry”.

Set a static IP address so that the Raspberry Pi will always boot up with the same address, to make it easier for you to log into it in future. Do this by editing the file /etc/dhcpcd.conf:

sudo nano /etc/dhcpcd.conf

Go to the bottom of the file, and add lines similar to this:

interface eth0
static ip_address=192.168.1.2/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

Those particular settings suit my network, which has the gateway running at 192.168.1.1 and also providing DNS resolution. You may need to change the values to suit your network.

Press Control-X to exit, then save the file.

Now run the “raspi-config” utility:

sudo raspi-config

Use the menu system to select “Change User Password”, and set a strong password.

Then go into the “Hostname” section, and change the hostname from “raspberrypi” to something more specific. I changed mine to “controller“.

Exit from raspi-config, and allow it to reboot.

A minute or so later your Raspberry Pi should come back up on the network. Now it will be at the IP address you specified above, it will have hostname you set, and the password will have changed.

Step 1c: Install UniFi Network Controller software

Install the haveged daemon to provide a good source of entropy. This isn’t essential, but it will help the UniFi Network Controller boot up faster, because it will be able to generate cryptographic keys more quickly:

sudo apt install haveged -y

Install a version of Java that works nicely with the Controller:

sudo apt install openjdk-8-jre-headless -y

Add the Ubiquiti Networks software repository and GPG key:

echo 'deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti' | sudo tee /etc/apt/sources.list.d/100-ubnt-unifi.list
sudo wget https://dl.ubnt.com/unifi/unifi-repo.gpg -O /etc/apt/trusted.gpg.d/unifi-repo.gpg

Those lines above may have wrapped in the browser. It should be 2 lines, with the second line starting with “sudo”.

Now update the list of available packages, and install the UniFi Network Controller package:

sudo apt update
sudo apt install unifi -y

In future you can update the Controller package just like you would with any other Raspbian package. If you regularly update your Raspberry Pi using the following commands, it will stay up to date along with all the other packages:

sudo apt update
sudo apt dist-upgrade

Step 1d: Log in to Controller software

After the installation finishes and the Controller software has finished booting, you can log in using a web browser. Use your browser to open the IP address you configured for your Controller, but make sure you prefix it with “https” and use port 8443. With the example configuration I gave above, the URL is:

https://192.168.1.2:8443

Your browser will complain because it doesn’t trust the SSL certificate being used by the Controller. You can click through and load it anyway.

Step through the setup wizard and answer each question. There aren’t many questions and they’re fairly simple to work out. You can skip the WiFi configuration and do it later, as I show in the video.

Click around and explore the Controller interface if you like. There won’t be anything much happening in it because there aren’t any devices configured.

Step 1e: Plan your access point placement

The UniFi Network Controller has an interesting feature that lets you create a plan of the physical layout of your house, and place network devices on the plan. It can then use obstructions such as walls to do estimates of the signal strength in different parts of your house.

Go to the “MAP” menu item on the left, and you’ll see an example map. Edit the example or create a new map and upload a plan of your house, then use the Map Designer tool to draw lines where all your walls are located. Place access points, turn on coverage display in the “Layers” menu, and experiment with different locations for your APs.

Step 2: Replace all-in-one router with dedicated router

If your Internet connection uses a technology such as ADSL, or Cable, or optical fibre, you need to have it terminated in a way that gives you a simple Ethernet connection. A device that does this is called a Network Termination Unit, or NTU. The NTU is specific to the type of connection that your ISP provides.

In some cases, your ISP will have provided an NTU so that you can simply plug your new router straight into it. If that’s the case, you can skip ahead to installing your dedicated router. I’m going to use a UniFi Security Gateway (USG) as my new dedicated router for the rest of this example.

I have a Cable internet connection that comes in on coaxial cable, so it needs a device to convert the coaxial connection into an Ethernet connection. Instead of using an NTU, I used the original Telstra Gateway Max that was provided by my provider, and changed its configuration to put it into “bridge” mode.

Bridge mode is a simplified mode that turns off almost all the features of your existing router and turns it into a dumb adapter that just passes data through without trying to manage it. Effectively this is like taking a router and giving it a lobotomy to turn it into an NTU. This will also disable any WiFi functionality, so it won’t be useful as an access point anymore.

The specifics of how to do this depend entirely on your ISP setup and your existing router, so I can’t provide instructions here.

Once you have converted your ISP’s router into a bridge or replaced it with an NTU, it won’t provide any of the normal services that you need such as routing and DHCP. That will now by done by your dedicated router.

Disconnect everything from your existing router except the ISP connection.

Use an Ethernet cable to connect the “WAN” port of your new router to one of the “LAN” ports of the old router. The connection will then go:

Your ISP -> Your NTU / bridge -> New router [REPLACE WITH IMAGE]

Connect from the “LAN” port of your new router to an Ethernet switch, so that you have somewhere to connect wired devices to your network. This will include the Raspberry Pi that is now running the UniFi Network Controller software!

ADD IMAGE HERE

Turn on the new router, wait for it to start up, and then go back into the management interface on the Controller. If you’re using a UniFi compatible router such as the UniFi Security Gateway (USG) that I show in the video, the Controller should now see that you have a new device on the network. You can “adopt” the new device, which sets up a secure communication link between the Controller and the USG.

The rest of the setup of the gateway is done through the Controller interface. There’s no need to log in directly to the USG itself, because it receives its configuration from the Controller.

Step 3: Install dedicated access points

Ubiquity specialise in WiFi, so they have a huge range of access points available. In my case I stuck to the UniFi range because I wanted APs that could be managed using the UniFi Network Controller. I chose UAP-AC-Pro models because they seemed the best for my needs, but I’ve been told that I probably should have got the nanoHD model instead.

The particular APs that I got use PoE (Power-over-Ethernet) instead of plugging in a power supply directly. This means you can run a single Ethernet cable from your Ethernet switch to your AP, and the AP will take its power from the cable.

If you have a PoE Ethernet switch, there’s nothing else you need to do. Just plug in the cable, and the AP will receive power.

If you have a non-PoE switch, you can use a POE injector mounted near the Ethernet switch as an intermediary, to provide the power source. My APs came with PoE injectors included in the box, but some models of UniFi AP can be bought both with and without the injector in case you already have a PoE switch.

If you want a bit more background on Power-over-Ethernet, I did an introduction to it way back in episode #3! Check it out here:

SuperHouse Episode #3: Power-over-Ethernet for Arduino home automation

I also wrote a tutorial about it on the Freetronics site:

Power-over-Ethernet for Arduino

Once your APs are plugged in and powered up, go back into the UniFi Network Controller interface to find them listed as new devices. You can then adopt them, just like you did with the Security Gateway.

However, don’t configure them directly unless you want to achieve something specific. The simplest thing is to leave them on default settings, which allows them to inherit the settings that you can apply to the entire site. In this context, “site” means your physical premises. The UniFi Network Controller is designed to scale up to installations with many devices across many physical sites, such as a group of remote branch offices, all managed through one central interface. For domestic use, you’ll probably only have one site.

To create a WiFi network that will be advertised through your fancy new access points, go to the “SETTINGS” menu item in the very bottom left of the Controller screen, then select “Wireless Networks“.

Click “CREATE NEW WIRELESS NETWORK“, and enter the details:

  1. Set “Name/SSID” for the WiFi name that you want.
  2. Leave the “Enabled” box ticked. Obviously!
  3. Change the “Security” setting to “WPA Personal“. This is the common authentication type that most domestic WiFi networks use, with a single shared password for all devices.
  4. Type the WiFi password that you want to use in the “Security Key” box.
  5. Leave “Guest Policy” un-ticked, unless you’re specifically creating a guest network. Your first WiFi network will almost certainly be a normal network, not a guest network.

You don’t need to worry about the ADVANCED OPTIONS section for now. We’ll be coming back here in Part 2, when we look at more advanced things that you can do to improve the security of your network and provide compartmentalisation for troublesome devices.

Click the green SAVE button, and your new WiFi network will be created. This will also be automatically applied to all the access points that you have linked to your Controller, so it will become visible and you’ll be able to connect devices to the new network.

That’s all for now, but come back for Part 2 where I’ll show you how to apply more advanced concepts to really make your home network the best it can be.