The Hub of Heliopolis - IoT

IoT with WebSockets and Python's AsyncIO

2018-03-03T22:49:00+01:00

Introduction¶

In this post, we will get back to the topic of IoT to introduce two new technologies by example: WebSockets and Asynchronous I/O in Python. The project that will allow us to explore them is a simple system of two LEDs that we will control with the gravity sensor of an Android device via a Raspberry Pi over a network.

The post is divided into two parts. The first one is theoretical in nature and will cover the essential technical details of the two main subjects, that is WebSockets and Python's asyncio. In the second part we will have a look at the circuit that we are going to control over the network and discuss the server and client code.

But before we dive into the study of the topics of this post, it is perhaps best to first have a high level view of how all the pieces fit together, thus motivating the choices of technologies mentioned above. The idea of the project stems from the following scenario: suppose you want to build a device that can be controlled over the network, e.g. an hand-held device like a phone. As we have seen in a previous post, one way of achieving this is by using a single-board computer like a Raspberry Pi and control it via a web server.

Now, what if you wanted more control over the connection method to the device, and perhaps a bi-directional channel to let data from, e.g. sensors on the device, to flow upstream to the controlling device? An elegant way of achieving this goal these days is with WebSockets, which allow (full) duplex communication between pairs of connected devices. So, for instance, we could have a WebSocket server running on a Raspberry Pi, and have a native application on an Android device to run a WebSocket client. Control commands can then flow from the Android device to the single-board computer, while the server can feed data from any sensors that the device is equipped with back to the client, with just a single connection.

The code samples that we will look at towards the end of the post implement a Python WebSocket server that will run on, e.g., a Raspberry Pi, and a native Android application that will act as a WebSocket client to control a pair of LEDs mounted on a breadboard. The data that we will transfer from the Android device to our circuit is coming from a gravity sensor, so that when we twist the hand-held device clockwise or anti-clockwise, a different LED will turn on. Furthermore, the brightness will depend on how much the device is tilted.

So, read on to find out more!

Connecting Devices over a Network: WebSockets¶

In A Gentle Introduction to IoT, we saw how to control a single LED over a network by running a web server hosting the controlling web application. This was a simple web page that displayed a button that not only showed the current state of the LED, but allowed to control it by turning it on or off. All we needed to do was to point a web browser to the web server running on the Raspberry Pi and play around with the only web page on it.

There might be situations where we are not happy with a web server, but we actually want more control over the way devices connect with each other. In a simple client-server relationship, surely TCP/IP sockets spring to mind, but what if we want to allow for data to flow in both directions?

Contrary to ordinary sockets, WebSockets offer a full-duplex communication channel over a single TCP connection. They are compatible with the HTTP protocol, but make use of their own protocol (the WebSocket protocol), which is switched to by including an HTTP Upgrade header in the HTTP handshake. By default, they are supposed to operate on the standard HTTP ports (80 for HTTP and 443 for secured HTTP), but a different one can be used for custom use, as we will see in our example. The advantage over any other solutions is that WebSockets have been designed to allow for fast bi-directional communication between client and server via an TCP connection that is kept open until manually closed (or dropped for other reasons), but without the overheads of HTTP headers.

By design, WebSockets are then the perfect tool for exchanging short messages between devices. As to what kind of messages we can send back and forth, we will see that we can either send text messages, or raw binary data, by either operating the WebSocket with text or binary frames.

Having justified the use of WebSockets for our project, we now have to motivate the use of the other mentioned technology: asyncio.

Like in A Gentle Introduction to IoT, the choice of Python is motivated by the use of the Raspberry Pi. In Python, WebSocket support is provided by the websockets module, which is built on top of asyncio.

Here is where we encounter the first constraint though, since asyncio was introduced in Python 3.4. We then have to ensure that we are using a version of Python greater than or at least equal to 3.4. But our coding can also vary based on whether we are using Python 3.5 and later. That is because this version of Python introduces the async and await syntax for natively defining coroutines (PEP 492).

In the last part of this post we will have a look at Python code samples written with the standard asyncio syntax, as well as the new one introduced by PEP 492. For now, we shall have a quick overview of the new features that are brought to Python 3 by the asyncio module to better familiarise with it and its usage.

Coroutines¶

I believe that the best way to understand asyncio is to first look at the basic concepts involved and recall the notion of coroutine. In Python we have the concept of Generators since version 2.2. Generators look and feel like normal functions, but rather than returning a value, they yield one, and are normally used in loops to provide iterators.

The typical scenario where you'd want to opt for a generator rather than a function is when you have to keep track of some state in between the different values returned. A simple example is a generator that generates the first $n$ Fibonacci numbers:

def fibonacci(n):
    if n == 0: return

    yield 0
    if n == 1: return

    yield 1
    if n == 2: return

    a0, a1 = 0, 1
    i = 2

    while i < n:
        a2 = a0 + a1
        yield a2
        a0, a1 = a1, a2
        i += 1

If $n=0$, the generator doesn't yield any number and we can then return. When $n=1$, the generator must yield the first Fibonacci number only, which is 0, and so on. The difference between return and yield is that the former terminates the iteration, while the latter allows it to continue. To better understand what is going on here, observe that a call to fibonacci, like f = fibonacci(10) doesn't return a Fibonacci number (you might at first expect this to return the first Fibonacci number), but a generator object instead, that is, something that we can use with a for loop, or any function that expects an iterable object, like sum. The first two Fibonacci numbers can then be generated with

f = fibonacci(2)
print(f)
print(next(f))
print(next(f))
# Expected output
# <generator object fibonacci at 0x7f2e0940baf0>
# 0
# 1

but if we now try to print yet another Fibonacci number, we get a StopIteration exception, which signals that the generator has returned and that there are no more values to be generated:

print(next(f))
# Expected output
# Traceback (most recent call last):
#   File "<stdin>", line 5, in <module>
#     print(next(f))
#
# StopIteration

The following code snippet illustrates the use of a generator with a for loop

test_cases = [0, 1, 2, 3, 10, 20]

for t in test_cases:
    print("Test case: n = {}".format(t))
    for f in fibonacci(t): print(f)

I am omitting the expected output to avoid cluttering the page, but it should be quite clear what the above code is supposed to do. To produce a more compact result we can do something like

def sfibonacci(n):
    for f in fibonacci(n): yield str(f)

for t in test_cases:
    print("Test case: n = {}".format(t))
    print(" ".join(sfibonacci(t)))

# Expected output:
#
# Test case: n = 0
#
# Test case: n = 1
# 0
# Test case: n = 2
# 0 1
# Test case: n = 3
# 0 1 1
# Test case: n = 10
# 0 1 1 2 3 5 8 13 21 34
# Test case: n = 20
# 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181

which also shows how to construct a generator out of another generator.

From the above example we deduce that the yield keyword is generating a value while also "pausing" the execution of the function, until another value is requested from it. In this case, the execution resumes from the instruction that comes soon after the yield, with all the values of local variables (the state) preserved.

Generators are also known as semicoroutine, and this leads us to talk about coroutines. This is a more general concept, because it encompasses those code elements that not only pass a value to the caller, but also receive and process a value passed to them by another code element (e.g. a generator, or another coroutine). This definition looks a bit circular, and this is due to the fact that, with coroutines, the relation is not of caller-callee, but symmetric. In more concrete terms, we are talking about functions that can retain a state in between invocations, and that can call to other functions, suspending and resuming execution from certain points of the code.

Starting with Python 2.5, coroutines have become an integral part of the language. They can be easily constructed with yield, which has been turned into an expression. The value yields evaluates to is passed to the coroutine with a call to send on the generator, like so

def coroutine():
    c = 0
    while True:
        text = yield
        c += 1
        print("#{} : {}".format(c, text))

coro = coroutine()
coro.next()
coro.send("Hello")
coro.send("World")
# Expected output
# #1 : Hello
# #2 : World

If you look to code like the above for the first time, you might be wondering why I have included a call to next. Remember from our discussion on generators that, contrary to normal functions, a call to coroutine returns a generator object and not the first generated value. After creation, a coroutine needs to be primed, that is it needs to be started so that it can execute its code until the first occurrence of the yield expression. The coroutine then halts and waits for values to be sent to it. In Python, there are two equivalent ways of priming a coroutine: either call next, as we have done in the code above, or send it a None with coro.send(None).

Since it is quite easy to forget to prime a coroutine, a good idea is to define a decorator, e.g. @coroutine that creates, primes and returns a coroutine.

If you want to know more about coroutines, I recommend that you have a look at David Beazley's A Curious Course on Coroutines and Concurrency. Here, I have just stated the essential details that we are going to need for our project. These should be enough to convince you that, with coroutines in Python we can implement single-threaded concurrency, which offer an ideal ground for asynchronous I/O operations, without the overhead of many context switches between different threads.

The `asyncio` module¶

If you have ever done any sort of I/O before, like reading from/writing to a file, or creating a socket and waiting for a connection etc... you will surely know that most of the I/O operations are blocking. For example, if you are trying to read from a file descriptor by making a SYS_READ system call, your code will hand control over to the OS until your request can be honoured. The normal execution flow then resumes.

The problem with this wait is that, in most cases, you don't know when there will be enough data available from the file descriptor to read. Your application then halts while it might be doing something useful instead.

The typical workaround is to poll the file descriptor periodically, and only read from it when data is actually available. As you can easily imagine, the solutions come in different patterns, and every time you have to deal with this there is some boilerplate code that you would have to write. This amounts to writing your event loop to cycle through your tasks, which include I/O polling. Wouldn't it be nice if we were provided with such boilerplate code encapsulated in a module that we can use whenever we need to perform asynchronous I/O operations?

This must be what the people behind asyncio must have thought, and that's why the essential element that this Python module offers is an event loop. This is designed to register and schedule Tasks which, with just a few words, can be described as objects decorating coroutines (hence, in practice, they are coroutines). Some of these tasks might involve I/O operations, and some of these operations might be blocking. By continuously polling for the I/O status of file descriptors, sockets etc..., asyncio allows you to write single-threaded concurrent code that performs I/O.

In this post, we are interested in working with WebSockets, as as an example of what we have just seen we can play around with some WebSocket servers and clients. In Python, we can make use of the websockets modules, which builds its functionalities on top of asyncio. The Getting Started page on PyPI shows how simple it is to create a a WebSocket client, and an echo server to test it. The following examples are based on them, but with just some slight modifications, and a twist: the client sends a string read from STDIN and the echo server reverses it.

So here is the server code

#!/usr/bin/env python3

# Echo server. Will reverse everything we throw at it.

import asyncio
import websockets

async def echo(websocket, path):
    async for message in websocket:
        await websocket.send(message[::-1])

loop = asyncio.get_event_loop()
loop.run_until_complete(websockets.serve(echo, 'localhost', 8765))
loop.run_forever()

and here is the client code

#!/usr/bin/env python3

# Client. Sends stuff from STDIN to the server.

import asyncio
import websockets

async def hello(uri):
    async with websockets.connect(uri) as websocket:
        await websocket.send("Hello world!")
        async for message in websocket:
            print(message)
            await websocket.send(input())

asyncio.get_event_loop().run_until_complete(hello('ws://localhost:8765'))

And that's where we make our first encounter with the new async/await keywords, introduced by the already cited PEP 492. The new syntax async def is used to declare a native coroutine in Python 3.5 and later versions. Perhaps more interesting are async with and async for. The former introduces native asynchronous context managers for classes that define the new magic methods __aenter__ and __aexit__, but apart from this, its usage is analogous to the synchronous counterpart with. The new syntax async for is used to consume asynchronous iterable, i.e. instances of classes that implement the __aiter__ magic method. When the await keyword is on its own, it defines an expression that execute the coroutine it appears in to execute the one passed as its argument until it completes. In the case of the code above, we just wait for the websocket object to complete the task of sending the data.

After the above discussion, the code for the client application should be quite clear. The first call to websocket.send is used to prime the socket, so that async for message in websocket won't hang indefinitely, waiting for something to show up on the socket's reading end.

Before we look at how to rewrite the above code snippets for Python 3.4, where we do not have native coroutines, I would like to briefly comment on the last two lines of code of the server application. The first time I came across that code, they gave me a bit of thinking as to why we need to call run_until_complete and run_forever. Wouldn't the former alone suffice? If you recall our discussion about blocking I/O operations and the necessity of constantly polling in order not to halt the execution of our application, you realise that the call to run_until_complete will register the socket with the I/O polling task. Hence, apart from this task, we have nothing else running, and if we do not start the event loop, the newly created socket won't be checked and the application simply terminates. The last line is there to ensure that we keep monitoring the socket for new incoming connections. When a client connects, a new task is scheduled to serve the connection with the passed handler, which must be a coroutine. This can be verified by peeking at the source code of both asyncio and websockets.

And now to the Python 3.4 version of the above code snippets. There are a few rule of thumb that we can use to convert from 3.5 and later to 3.4, where we don't have await and async. The first is the use of the decorator @asyncio.coroutine instead of async def. The situation is a bit more complicated for async with, which requires a replacement for await, which is yield from. The latter is substantially equivalent to

for a in foo(): yield a    # Same as: yield from foo()

With this in mind, async with can be coded with a more traditional try ... finally block as

# async with coro() as foo:
#     # <code>
#     pass
foo = yield from coro()
try:
    # <code>
    pass
finally:
    # Whatever coro().__aexit__() would have done.

The async for loop would translate to something like the following

# async for foo in bar():
#     # <code>
#     pass
try:
    while True:
        foo = yield from bar().__anext__()
        # <code>
except StopAsyncIteration:
    return

This coding is not strict, since you could, or you might have to replace the code of bar().__anext__() with the actual coding inside this coroutine. This is the case for the code that we are about to see, in which we perform the above translations. The server code for Python 3.4 is the following

#!/usr/bin/env python3

# Echo server. Will reverse everything we throw at it.
# For Python 3.4

import asyncio
import websockets

@asyncio.coroutine
def echo(websocket, path):
    while True:
        message = yield from websocket.recv()
        yield from websocket.send(message[::-1])

asyncio.get_event_loop().run_until_complete(websockets.serve(echo, 'localhost', 8765))
asyncio.get_event_loop().run_forever()

while the client code now looks like this

#!/usr/bin/env python3

# Client. Sends stuff from STDIN to the server.
# For Python 3.4

import asyncio
import websockets

@asyncio.coroutine
def hello(uri):
    websocket = yield from websockets.connect(uri)
    try:
        yield from websocket.send("Hello world!")
        while True:
            message = yield from websocket.recv()
            print(message)
            yield from  websocket.send(input())
    finally:
        yield from websocket.close()

asyncio.get_event_loop().run_until_complete(hello('ws://localhost:8765'))

Note how to execute a native coroutine, or one decorated with @asyncio.coroutine in Python 3.4, needs to be executed by an event loop from the asyncio module.

Setting Things Up¶

We shall now present the circuit that makes up the device that we want to control over the network, and both the server and client code that will allow us to use the gravity sensor of an Android device to operate it.

The Circuitry¶

This part of the post will be brief, because we are going to recycle part of the previous post on IoT A Gentle Introduction to IoT. In fact, we are going to double it up by making a circuit with two LEDs, each one following the same design in the just mentioned post. The idea is to turn either one or the other on, depending on the rotation angle of our Android device. For example, if we tilt our device to the right, the green LED will become brighter, while when we tilt it to the left, the red LED will become brighter.

So, based on the knowledge acquired with the previous post on IoT, the circuit that we want to build will look like this.

For this project I am using a Raspberry Pi 3 Model B. As you can see from the picture above, we need two LEDs, preferably of different colours, e.g. green and red, and two 220 Ω resistors. The green LED will be controlled via the BCM 18 (8) pin (green jumper wire), while the red one is controlled via the BCM 5 (29) pin (red jumper wire) of the Raspberry Pi. The black jumper wires indicate a connection on a ground pin.

The Server Code¶

The server code will run on the Raspberry Pi and will open a WebSocket server, listening for connection requests. At the moment of writing, Raspbian Jesse has Python 3.4.2 in its repository, so that we cannot benefit of the native Python coroutine offered by Python 3.5, unless we install this version manually.

All the server code can be found inside the gravity_led/server folder of the iot repository. Due to its length I will refrain from embedding it on this post, but I will comment on the essential aspects.

The code is based on an abstract WebSocket Server class, WSServer, contained in lib/wss.py. An actual WebSocket server has to inherit from this class and implement the handler method, which is the one that bootstraps the logic of the server application. This will be called as soon as the WebSocket server starts serving connection requests.

On creation, we can specify the address and port the server is to listen on, as well as a possible limit on the number of simultaneous connections that the server is allowed to serve. Why would such a limit be ever necessary? The reason is that, in this particular example, we only have one device, and this can only be controlled by one client at a time. It makes no sense to accept more than one connection. By setting the limit to 1, we then prevent other clients from connecting and finding out that the device is already being controlled by another client. For the way it is implemented though, the class is flexible enough to allow changing this limit at run-time with a call to the set_server_limit.

After instantiation, we can run the server with a call to the start method, and we can check whether it is running at any time with is_running. Most of the code in the former method provides logging information and a graceful shutdown when the user presses Ctrl+C to halt the server.

The code contained in main.py is used to bootstrap the class GyroListener contained in gyro_listener.py, which holds the actual logic of the server. As you can see, all that we have to do is inherit from WSServer and implement the handler coroutine, as shown here.

@asyncio.coroutine
def handler(self, websocket, path, conn_id):
    self.logger.info("Connection ID {} established [path {}]".format(conn_id, path))

    G.setmode(G.BOARD)
    G.setup(CHR, G.OUT)
    G.setup(CHL, G.OUT)

    pr = G.PWM(CHR, FREQ)
    pl = G.PWM(CHL, FREQ)
    pr.start(0)
    pl.start(0)

    while self.is_running:
        try:
            gyro_data = yield from websocket.recv()
            g         = gyro_data.split()
            val       = int(float(g[1]))

            self.logger.debug("Received datum {}".format(val))
            if val > 0:
                pr.ChangeDutyCycle(val)
                pl.ChangeDutyCycle(0)
            else:
                pr.ChangeDutyCycle(0)
                pl.ChangeDutyCycle(-val)

        except websockets.exceptions.ConnectionClosed:
            self.logger.info("Connection ID {} closed.".format(conn_id))
            pr.stop()
            break

    G.cleanup()

Here we initialise the Pi's GPIO and start listening to data over the socket until the server is running or the connection is closed by the client. The two pins that we have chosen to use to control the LEDs (CHR and CHL) are set on the PWM (Pulse Width Modulation) mode so that we can control the brightness: the more the Android device is tilted in one direction, the brighter the LED for that direction.

Here we also notice the "contract" between the server and the client: the latter will send the three coordinates of the gravity sensor as a space-separated list of three floating point values. The server will split this string and use the second component (the y-axis of the gravity sensor) to control the LEDs. When the value is positive the LED on the CHR pin will turn on, with a duty cycle proportional to the value passed by the client; when the value is negative the LED on the CHL pin will start to turn on.

To launch the server, run main.py and pass the IPv4 address and a port number, for example

python3 main.py 0.0.0.0 5678

to listen on any IP address on the Raspberry Pi and accept connections from everywhere.

The Client Code¶

For the client, we are going to develop a minimalist Android application, with the same project structure that we have encountered in the previous post on Android Development from the Command Line. Again, the code is quite extensive, but you can find it in the gravity_iot/client folder. Apart from the minimal Android project setup, with the AndroidManifest.xml and the resource files for the UI, all that we need is a single activity where we can specify the IP address and the port of the server to connect to, and a toggle button to start and close a connection to the server.

For dealing with WebSockets in Java, we make use of the Java WebSockets library, which offers the WebSocketClient abstract class. As you can see from the code in MainActivity.java, all that we have to do is extend the WebSocketClient class and implement a few methods, which we just use for logging purposes. The actual signalling is done via callbacks triggered by the on-board gravity sensor on the Android device:

@Override
public void onSensorChanged(final SensorEvent event) {
  float x = event.values[0];
  float y = event.values[1];
  float z = event.values[2];
  float g = (float) sqrt(x*x + y*y + z*z);
  try {
    webSocketClient.send(Float.toString(x/g*100) + " " + Float.toString(y/g*100) + " " + Float.toString(z/g*100));
  } catch (WebsocketNotConnectedException e) {
    Log.w("ctrl_ws_client", "Sensor updated but socket not connected");
  }
}

This method is part of the MainActivity class since we decided that this should implement the SensorEventListener interface. When a connection is successfully started, we register the instance of this class with the sensor, so that when the value is changed a white-space separated list of the three components of the gravity vector are sent in text form to the server via the WebSocket.

We can use Gradle to build and deploy the application for testing on a device with the command

./gradlew installDebug

assuming that you have an Android device in developer mode connected to your building machine via USB. This is what it looks like on my Nexus 5

Before moving to the conclusion, just a quick note on the contents of the AndroidManifest.xml file. If you have had a look at it, you might have noticed the attribute android:configChanges="orientation|screenSize" on the activity element. This allows the application to rotate with the device orientation, without the application being restarted. If we do not put this attribute, any previous connection would be closed and we would have to restart it by pressing on the toggle button. The use of this attribute is not mandatory, but bear in mind that if you decide not to use it, then you would have to make sure that the same WebSocket connection is persisted across every screen rotation.

Conclusions¶

In the introduction, I have mentioned that WebSockets can be operated in two modes: text and binary. In the code that we have seen in this post we have sent text frames between the server and the client. This is because we have used data of type String in the Java client code, which is then received as str on the Python server end.

In most practical cases, it is more convenient to send a stream of bytes instead, thus increasing throughput. On the Java side, this is achieved by passing an array of bytes (bytes[]) to the send method. On the Python end, data will then be received as bytes (that is, the return value of the recv coroutine is of type bytes) that one can iterate over, or perform any necessary operation to make sense of the received information.

The same holds on reverse: if we send an array of bytes from Python, we will receive an binary frame on Java. In this case, though, we will have to implement the ByteBuffer version of the onMessage callback of the WebSocketClient class, i.e. the one with signature

public void onMessage( ByteBuffer bytes )

A Gentle Introduction to IoT

2017-07-31T19:30:15+01:00

Preamble¶

No, I'm afraid this post is not on how to build your army of toy soldiers. If the disappointment hasn't stopped you from reading any further, what I will discuss here is something simpler and more peaceful, involving general electronics single board computers, LEDs, Python, web servers and web applications. All in just one place!

I will try to introduce the fundamental concepts of IoT by example, and I will do so by sharing my recent hands-on experience with my first Raspberry Pi, and how this has led me to get to know of fascinating technologies of the modern era. If your main objective is still to build an army of toy soldiers, this is definitely a first step (however you should consider using technology for better purposes, really!).

I will try not to give much for granted, but of course I will have to draw a line at some point, or this post would have never been finished! The subject is software, but it is also hardware, for the former would make little sense without the latter.

The approach will be very practical. We will start with a concrete problem, and we shall see how to solve it, both from a hardware and a software perspective.

What's IoT?¶

Can you eat it? Well, although your are free of munching on a breadboard if you really want to, IoT, as you probably know already, stands for Internet of Things. This term is used to indicate the inter-networking of physical devices (the things) that are equipped with electronics, sensors, motors etc.. and that communicate information with each other, sometimes taking actions based on the received inputs. In this sense, every-day life devices, like the fridges or the washing machines in our homes, or vending machines, or cars even, become smart.

Even though the first example of smart things appeared in the 1982 (and you can surely consider the toy army of Toy Story as an example of IoT in early animation movies), it is around 2016 that the IoT has evolved greatly, and the key aspect is the almost ubiquitous availability of wireless networks that allow an increasing pool of diverse devices to communicate with one another.

The IoT example of this post is somewhat a classic, but that will hopefully give you a rough idea of what the IoT is also about, in case this is the first time that you came across it. It is also quite simple and with a contained cost, but nonetheless will spawn many interesting connections with some fascinating modern technologies.

The Project¶

So what is exactly the project described in this post about? The idea is to turn some LEDs on and off by sending commands to a Raspberry Pi through a web page. This might seem quite a trivial project, but it has given me the opportunity to refresh some rusty knowledge of electronics that dated back to my undergraduate years, as well as learn a great deal of new things from the software side too.

The Hardware¶

In this first part I will describe all the hardware that is necessary for the project. The main one is, I would say, a single-board computer like a Raspberry Pi. In my case, I'm using a Raspberry Pi 3 Model B running the Raspbian OS. To turn an LED on and off we will use one of the GPIO pins. For commodity, I will also use a T-cobbler to connect all the GPIO pins on a breadboard, where the rest of the circuit will be assembled. All that we still need are a couple of jumper wires, a 220 Ω resistor and an LED of your favourite colour. To summarise, here is the minimum required:

1x Raspberry Pi
1x breadboard
1x T-cobbler and a bus cable (alternatively 2x male-to-female jumper wires)
2x male-to-male jumper wires
1x 220 Ω resistor
1x LED (about 20 mA max current)

The GPIO Pins¶

Before looking at the circuit, it is probably best to mention a few facts about the Raspberry Pi. In particular, the key part here is the set of GPIO pins that it provides.

The General Purpose Input-Output pins represent, as the name itself suggests, a way to connect to external devices in order to send commands to and from the Raspberry Pi. Some of these can be set to work as input or output, and can be set either high (3.3 V or a logic 1) or low (0 V or a logic 0). Some pins can be used together for other purposes, like communicating with another device through a Serial Peripheral Interface (SPI), attaching LCD screen through the Display Parallel Interface (DPI) etc.... Good references for the Raspberry Pi GPIO pins are this website and the Broadcom BCM2835 ARM Peripherals manual.

From the software side, the pins can be conveniently configured and controlled by means of the RPi Python module. The thing to be mindful of is that there are a bunch of different naming conventions for the pins on the Raspberry Pi. The main ones, to use a terminology proper of the RPi module, is BOARD and BCM. The former is a pin numbering that reflects the physical location of the pins on the PCB. Pin number one is on the top-left corner and gives a 3.3 V fixed output. Pin 2 is the one to its right, Pin 3 is the one below it, and so forth. The latter is the numbering convention used in the Broadcom manual.

The Circuit¶

Here is the schematic of the circuit that we want to build.

The schematic of the circuit, showing all the components used for this project.

As I have already mentioned, I prefer using a T-cobbler to connect all the GPIO pins to the breadboard. In case you are not using one, this is what your breadboard should look like this picture.

Another schematic representation of the circuit, showing how the components are physically connected with each other on the breadboard and the Raspberry Pi 3 Model B.

Where did the magic number 220 Ω come from? The explanation is very simple and essentially based on Ohm's law. Across a resistor $R$ to which a voltage difference of $V$ is applied, the current flowing through it is given by

$$I = \frac VR.$$

An LED is a diode, i.e. a p-n junction, that is capable of emitting light. In forward bias, an order-one approximation of a diode is given by a small resistor (order of 10 Ω) in series with a voltage generator (of about -0.67 V). The manufacturer of the LED usually provides the maximum current that the diode can withstand. In the case of common LEDs, this value is around 20 mA. Considering that, when a GPIO pin is on, it will provide 3.3 V to our circuit, in order not to burn our LED we need to use a resistor of resistance $R$ given by the inequality

$$\frac{3.3\text{ V} - 0.67\text{ V}} R \leq 20\text{ mA},$$

which yields

$$R\geq 130\ \Omega.$$

For a better estimate, we can look at the V-I chart provided by the manufacturer, which would probably give us a minimum value closer to 200 Ω. Anything below and you might risk frying your LED. Using a way bigger resistor, however, would starve it of current and it would not turn on at all. But with 220 Ω we should be perfectly fine (and safe!).

The Software¶

Now that we have assembled the required hardware, it is time to see how to control it. What we have done so far is to connect the Raspberry Pi with a very simple one-wire device, i.e. an LED. The IoT is the possibility of controlling a device from the internet, anywhere in the world, with the actual device that we want to control possibly miles and miles away from us.

What we now need is then a simple interface, accessible from the internet, that allows us to control the LED. Let's see how to create such interface, step-by-step.

But before we go any further, let's have a look, like we did with the hardware part, at all that we will need.

Python (2.7 or later; note that the project has been tested with Python2.7 and it might have to be adapted slightly to work with Python3)
a web server e.g. Apache2
a text editor of your choice.

Yep, that's all.

The RPi Python Module¶

On Raspbian there is a pretty simple way to control the GPIO pin that comes already bundled with the OS. I'm talking about the RPi Python module. For the simple task that we want to achieve here, RPi exposes all the features that we need. However, keep in mind that more advanced tasks, like real-time applications, cannot be solved by this module. This is not just because it is a Python module, but a "limitation" of any OS based on the Linux kernel, which is multitasking in nature. This means that the kernel can decide on its own how to allocate resources for all the running processes, potentially giving rise to jitter in your applications.

Another important limitation that should push you towards other approaches, like wiringPi, is the lack of hardware PWM. PWM stands for Pulse-Width Modulation and is a technique used to encode a message in a pulsing signal. Some of the common uses are that of dimming an LED (recall that an LED response is exponential, even though we treated it as linear in our first-order approximation discussed above), or controlling a motor, but this is a topic that would take us away from the main focus of this post, and it might be the subject of a future one.

Returning to our project, let's have a look at how to turn our LED on. Recall from the schematic above that we are using the GPIO Pin 16 (according to the Broadcom convention), which is the 36th physical pin on the GPIO.

import RPi.GPIO as G     # Import the GPIO component from the RPi module

LED = 36                 # Define a constant with the pin number where the LED
                         # is connected

G.setmode(G.BOARD)       # Set the pin numbering mode to the physical number

G.setup(LED, G.OUTPUT)   # Set the LED pin to output mode

G.output(LED, 1)         # Set the pin to high (3.3 V)

We can type the above lines of Python code directly into the Python interpreter. To turn the LED off we can either set the value on the pin 36 back to 0

G.output(LED, 0)

or clean up the GPIO configuration with

G.cleanup()

If we'll ever want to use more than just one LED on our breadboard, we can encapsulate most of the above code inside a Python class so that it can be reused instead of having to type it every time with setup a new LED. A minimalist Python class that would represent a physical LED would be something like

# File: led.py
import RPi.GPIO as G

G.setmode(G.BOARD)

class LED(object):
  def __init__(self, ch):
    self._ch = ch
    G.setup(ch, G.OUT)

  def on(self):
    G.output(self._ch, 1)

  def off(self):
    G.output(self._ch, 0)

  @property
  def state(self):
    return bool(G.input(self._ch))

  @state.setter
  def state(self, value):
    G.output(self._ch, bool(value))

  def toggle(self):
    G.output(self._ch, not G.input(self._ch))

Our code for turning an LED on and off would then reduce to the following few lines

from led  import LED
from time import sleep

led_red = LED(36)

led_red.on()

sleep(1)

led_red.off()

The extra method toggle can be used, as the name suggests, to toggle the LED state from on to off and vice-versa. In the above example we could then replace both led_red.on and led_red.off() by led_red.toggle(). The property state is used to get the state of the LED as a Boolean value (True for on and False for off), and it can also be used to set it. For instance, something like

led_red.state = [1]

would turn the LED on, since [1] evaluates to True when converted to a Boolean. Analogously, the following line of code

led_red.state = {}

would turn the LED off, since bool({}) = False in Python.

Sweet! We now know how to control our LED with code and all that's left to do is build a nice web interface that will execute this code on demand.

The WSGI Specification¶

Given that we already have some code in Python to control our LED, it would be good if the web interface could use that code directly. Is this possible? The (short) answer is yes!

A more articulated answer to the above question leads us into the realm of the Web Server Gateway Interface specification, or WSGI for short. It is a universal specification that was born out of the necessity of putting some order among all the Python frameworks for developing web applications. Before the specification, each of said frameworks would be compatible with just a few web servers. Choosing one of them would then restrict your choice of a web server to go with it, and vice-versa. To overcome this limitation, the WSGI specification was proposed in the PEP 333, which dates back ton 2003. The technical details can be found in the linked page. Here we just limit ourself to the essential details of the specification that will allow us to write a simple web application to control the LED over the internet.

In very simple terms, a Python web application consists of a bootstrap callable object that is called by the web server. The python code contained in the callable is executed, and the web server expects a response consisting of a code (200 for OK, 403 for Forbidden, 404 for Not Found etc...) and a stream of bytes (usually the HTML code to be rendered by the browser). A callable can be any Python object that exposes a __call__ function like, e.g., classes and functions.

My favourite web server is Apache2 and it will be the one that I will discuss in this post. Its functionalities can be extended with modules, and the mod_wsgi project provides a WSGI-compliant module for Apache. The documentation is very detailed and covers all the aspects, from the installation to the configuration and debugging techniques.

Regarding the installation process, this can be carried out in two modes. Either on the Apache-side, or on the Python-side. If you are into the IoT, chances are you will have your web server running on a Raspberry Pi. For this reason, I will discuss how to install the mod_wsgi module for the Apache web server. On Raspbian, this can be done with

sudo apt install libapache2-mod-wsgi

After the installation, the module should already be enabled. If this isn't the case, you can enable it with

sudo a2enmod wsgi

Writing the Web Application¶

The next steps are to actually write our web application and configure Apache to run it when a request comes in. In the configuration process we will specify our bootstrap Python script. By default, Apache expects to find a callable object inside it with the name application. The simplest thing that we could do is then to create a Python script and define a function with such a name. The code of our application would then be contained in this function, or called by it, in case we decide for a more modular approach.

In case things would go wrong while we develop our web application, we might want to be able to have a look at the Python stack trace to see where the problems are. Normally we would have to look in the Apache error log (usually in /var/log/apache2/error.log). However, we can make use of a middleware from the paste Python package, which is specifically designed for WSGI applications. Our bootstrap script will then look like this.

# File: bootstrap.wsgi
import sys, os
sys.path.append(os.path.dirname(__file__))

from index import Index

def main(env, start_response):
    status = '200 OK'

    output = Index.main(env)

    response_headers = [
        ('Content-type'   , 'text/html'     )
       ,('Content-Length' , str(len(output)))
    ]
    start_response(status, response_headers)

    return [output]

from paste.exceptions.errormiddleware import ErrorMiddleware
application = ErrorMiddleware(main, debug = True)

The first two lines are necessary if we want to be able to import modules and packages that are in the same folder as the bootstrap script.

We then import the class Index from the index module. This is just a design choice. The bootstrap script contains the essential code to get us to the main page (by means of the Index class), and returns us the full stack trace in case of errors (by means of the ErrorMiddleware class from the paste package). To make sense of the rest of the code, have a look at the already referenced documentation of the mod_wsgi module for Apache2.

The core code of our application is contained in the Index class from the index module.

# File index.py
from util import templ, qs
from led  import LED

led = LED(15)

class Index(object):

    @staticmethod
    def main(env):
        params = qs(env)
        if "action" in params and 'toggle' in params['action']:
            led.toggle()
        return templ('index', state = "on" if led.state else "off")

In fact, this module looks more like a view rather than a controller, as the actual code for controlling the LED is buried in the led module that we have analysed previously. The util.py module contains some helper functions to conveniently deal with HTML templates and query strings. We refrain from showing the code in this post, but you can find it in the dedicated GitHub repository.

The HTML template contained in the index.html file is very simple and looks like this.

<!DOCTYPE html>
<html lang="en-us">
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="theme-color" content="#666666" />
    <link rel="stylesheet" type="text/css" href="led_app/led.css">
  </head>
  <body>
    <a href="?action=toggle"><div class="button {state}"></div></a>
  </body>
</html>

The body is essentially just a link that executes a request with the parameter action set to toggle, containing a placeholder div element. The look-and-feel of a 3D button is then provided by the classes contained in the linked stylesheet, which you can find in the GitHub repository. Note how we use the {state} placeholder in the class attribute of the div element. This allows us setting the button appearance according to the LED current state. In the stylesheet we have two classes, .on and .off, the former giving a bright red colour to the button, while the latter giving a darker shade. The value is passed by the line

        return templ('index', state = "on" if led.state else "off")

in the static method main of the Index class of index.py.

Configuring Apache¶

We are almost ready to start playing with our LED over the internet. The last step is to put our application up and running on the Apache web server. To this end there is a tiny bit of configuration that we need to do. We can start by making a copy of the default web site Apache2 comes with. On Raspbian, this is located in /etc/apache2/sites-available and is contained in the 000-default.conf configuration file. Make a copy of this file in, say, led.conf and modify it to look like the following one.

# File: led.conf
<VirtualHost *:80>
  ServerAdmin webmaster@localhost

  # Required by static data storage access (e.g. css files)
  DocumentRoot /home/pi/Projects/www

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # WSGI Configuration
  WSGIScriptAlias /led /home/pi/Projects/www/led_app/bootstrap.wsgi

  <Directory /home/pi/Projects/www/led_app>
    Require all granted
  </Directory>

</VirtualHost>

Even though we are implementing a WSGI application, the DocumentRoot /home/pi/Projects/www is needed because we are importing a css file in index.html. The above configuration file assumes that the web application resides in the /home/pi/Projects/www/led_app folder. This way, static files can be accessed with a relative path referring to the parent folder /home/pi/Projects/www. This explains why we are importing the led.css file as

    <link rel="stylesheet" type="text/css" href="led_app/led.css">

The section below the # WSGI Configuration comment is the WSGI part of the configuration. The first line of this section tells Apache which script to use to bootstrap the web application. This is the script that is assumed to contain the callable object named application. The rest of the WSGI configuration section is required to set the rights to read the bootstrap and the python scripts contained in the web application folder.

That's it! All we have to do now is deactivate any other website on port 80 with a2dissite (this is only required if the web sites are not named), and enable led.conf with a2ensite led.conf. Restart the Apache2 web server with

sudo service apache2 restart

and point the browser on any device that is on the same local network as the Pi to its local IP address, append '/led' to it to start the web application (e.g. http://192.168.0.203/led) and you should now be in the web application, with a red button that will now allow you to control the LED state over the LAN.

The final look of the web application. The red button in the middle is used to toggle the LED state.

If you are behind a router, of course you will need to forward the port 80 to the Pi's local address before you could be able to access your web application from the internet, outside of your local network. In this case you will have to use the router's public IP instead of the local IP.

Conclusions¶

We have come to the end of this post on a simple IoT project. Its main purpose, like most of the other posts in this blog, is two-fold. On one hand, it is a way for me to take notes of new things that I have discovered and that I can later come back to if I need to. All the information gathered from the cited sources is gathered in a single place, which makes it more convenient than having to go through them separately. On the other hand, it is a way to share my experience with others, in the hope that it could be useful somehow. Even though the project in it self, as I have remarked many times now, is quite simple, the post includes references to many topics, e.g. electronics, programming, web servers and applications, and it shows you how all these different aspects can be organically combined together to create something.

The Hub of Heliopolis - IoT

IoT with WebSockets and Python's AsyncIO

Introduction¶

Connecting Devices over a Network: WebSockets¶

Coroutines¶

The asyncio module¶

Setting Things Up¶

The Circuitry¶

The Server Code¶

The Client Code¶

Conclusions¶

A Gentle Introduction to IoT

Preamble¶

What's IoT?¶

The Project¶

The Hardware¶

The GPIO Pins¶

The Circuit¶

The Software¶

The RPi Python Module¶

The WSGI Specification¶

Writing the Web Application¶

Configuring Apache¶

Conclusions¶

The `asyncio` module¶