The Hub of Heliopolis - Programming

The State of Python Profiling in 2026

2026-10-30T12:00:00+00:00

Python profiling has changed more in the last two years than in the decade before. CPython 3.14 gave external tools an official way to look inside a running interpreter, and made asyncio task graphs inspectable from outside the process. CPython 3.15 ships a sampling profiler in the standard library. Free-threaded builds have gone from experiment to something people actually deploy. And a growing share of the people reading profiles aren't people at all, but coding agents.

This post looks at where things stand: what the main sampling profilers can do today, where each one fits, and what changed in the latest release of Austin.

As many of you know, I've been working on Austin for the best part of eight years now, so I have a view from the inside on how this landscape came to be, and, naturally, a soft spot for one of the tools in it. I've tried to be fair to the others: what I say about them comes from their own documentation and release notes as of October 2026. If I've got something wrong, let me know and I'll fix it.

A Quick Map of the Landscape¶

Most profilers make two basic choices, and these choices explain most of the differences between tools.

Tracing or sampling. A tracing (deterministic) profiler like cProfile records every function call and return. You get exact call counts, but the overhead changes the program's timing, sometimes a lot. A sampling profiler looks at the call stack at regular intervals and builds a statistical picture from those snapshots. You don't get call counts, but the program runs at close to normal speed. I went into the differences between the two, and how they work under the bonnet, in an earlier post.

In-process or out-of-process. An in-process profiler runs inside your program, usually by starting it for you or being imported. An out-of-process profiler is a separate program that reads the target's memory from outside. Out-of-process tools can attach to something that's already running, like a production server that started misbehaving an hour ago, without restarting it or changing its code.

All four tools in this post are sampling profilers. Tachyon, py-spy and Austin work from outside the process. Scalene works from inside.

Why These Four¶

There are many more Python profilers than I can cover here, so I've picked the ones I think are the most widely known and used, and that have a track record on accuracy and overhead. That track record isn't just my opinion. The most thorough comparison of Python profilers I know of, by the Scalene team at UMass Amherst (Berger, Stern and Altmayer Pizzorno, 2022), measured a dozen of them on the longest-running benchmarks of the pyperformance suite. Its conclusion was that sampling profilers are "both more efficient and often more accurate" than tracing ones.

The two go together. A tracing profiler runs its own code on every function call and return, inside your program, so it slows the program down, and not evenly: code that makes many small calls is slowed down far more than code that makes few, which skews the picture of where time is spent. The study found one tracing profiler reporting that a function took 80% of the run time when it actually took 25%. A sampling profiler stays out of the way of your code, so the timings it reports are much closer to how the program behaves when nobody is watching. In the study, Austin, py-spy and Scalene were among the profilers with the lowest overhead in CPU mode, adding between 0% and 2% to the run time, compared with 73% for cProfile and more than 200% for some others.

Tracing profilers still have their place: they count every call exactly, which sampling can't do. If you need to know that a function was called 1,204 times, use cProfile. If you need to know where the time goes, sample.

Tachyon is too new to have been part of that study, but as the profiler that now ships with Python, it's the one many people will reach for first, so it belongs here.

One notable absence is pyflame, Uber's profiler from 2016. It was one of the first to sample Python stacks from outside the process, and it deserves credit for showing that the approach works. But it was Linux-only, its last changes date from 2019, before Python 3.8 was released, and its repository is now archived and marked as no longer maintained. I mention a few other good tools at the end of the post.

Why Sampling Rate Matters¶

With a sampling profiler, the time attributed to a function is an estimate: the fraction of samples it appears in, multiplied by the duration of the run. Like any statistical estimate, its error shrinks as the number of samples grows, roughly in proportion to 1/√n. Sampling 10 times faster gives you about three times less noise from the same run, so short runs give stable results and short-lived functions are less likely to be missed entirely.

So the maximum rate a profiler can sustain, without disturbing the target, directly limits how accurate it can be.

That's why raw sampling speed has been a priority for Austin from the start, and it's the reason it's written in C and does nothing but sample: every microsecond saved per sample is another sample per second. Each release has pushed this further. Austin 4.0 more than doubled the maximum sampling rate of 3.x, and 4.1 pushes it higher again (more on that below), with a default of 10,000 samples per second.

Tachyon's authors clearly think the same way: the profiler is named after the hypothetical particle that travels faster than light. Its sampling core is written in C too, and while its default rate is a more conservative 1,000 samples per second, it's designed to be raised well beyond that. When a profiler that's been around since 2018 and the one that has just joined the standard library both put speed first, it's a good sign that sampling rate is finally getting the attention it deserves.

How We Got Here¶

Profiling Python from outside the process used to be close to a hack. CPython made no promises about the layout of its internal structures, so tools like Austin and py-spy had to copy large parts of the interpreter's private headers for every Python version, and work out where the interesting fields were from compiler- and platform-specific details. Every new release could break them, and often did.

Those tools showed that the approach was worth it. Being able to attach to a running process with no instrumentation and almost no overhead turned out to be winning proposition in production, and evidently it made a strong case for CPython to support out-of-process tools properly, rather than just tolerate them.

Some of that work came directly from Austin. In 2021, I made three changes to CPython, each to solve a problem I'd run into while working on Austin:

co_qualname (bpo-44530). Code objects only carried a function's bare name, so an external profiler couldn't tell Foo.run from Bar.run, and their samples ended up merged. Since Python 3.11, code objects also carry the qualified name, and every out-of-process profiler can show it.
native_thread_id (bpo-43879). To measure CPU time, a profiler needs the operating system's ID for each thread. On Linux, there was no direct way to find it from outside the process, so tools like Austin had to dig through glibc's internal thread structures to infer it. Since 3.11, the interpreter's thread state simply records it.
Py_Version (bpo-43931). An exported symbol that lets external tools reliably tell which Python version a process is running. py-spy now uses it too.

Tachyon reads both co_qualname and native_thread_id today. In 2023 I moved the fields that out-of-process tools rely on to the top of the interpreter's structures (gh-105271). That change was one of the references Pablo Galindo cited when he proposed the debug offsets table (gh-106597): a structure at the start of the runtime state that tells external tools where to find the data they need, without copying any headers. I later added more fields to it (gh-121311). The debug offsets table is the foundation that PEP 768's remote debugging interface is built on, and that Tachyon uses to read the interpreter's state.

So the profiler that now ships with CPython is, in a sense, the result of years of external tools proving that out-of-process sampling works. CPython made it a supported use case, and the core developers have now built a sampling profiler into the interpreter itself. I find that very satisfying, and it's good news for every external tool too, Austin included.

Tachyon: A Profiler in the Standard Library¶

The biggest news of the year is that CPython 3.15 includes a sampling profiler, profiling.sampling, also known as Tachyon (it was proposed alongside reorganising cProfile into profiling.tracing, in PEP 799). It builds on the remote debugging support added in 3.14 by PEP 768.

python -m profiling.sampling run --flamegraph script.py
python -m profiling.sampling attach 12345
python -m profiling.sampling run --live script.py

Tachyon covers a lot of ground for a first release:

run a script, attach to a running process, or dump its current stacks.
Wall time, CPU time, GIL-holding and exception sampling modes.
An --async-aware mode that rebuilds stacks per asyncio task.
A live, top-style terminal view.
Many output formats: pstats, collapsed stacks, interactive HTML flame graphs and line-level heatmaps, differential flame graphs, the Firefox Profiler's Gecko format, JSONL, and a compact binary format.
Subprocess profiling, and a default sampling rate of 1 kHz that can be raised.

The obvious advantage is that it's already there: nothing to install, and it's maintained alongside the interpreter, so it stays in step with internal changes.

The main limit is version coverage. Tachyon only profiles Python 3.15 and later, and the profiler and the target must run the same minor version (and the same build type: free-threaded or standard). If your services run on 3.12, Tachyon doesn't help you yet. Its --native option adds placeholder <native> frames to show where Python calls into native code, but doesn't unwind the native stack itself.

A Familiar Design¶

If you've used Austin, much of Tachyon will feel familiar. The basic approach of reading the interpreter's memory from outside the process is shared with py-spy, and goes back to pyflame in 2016. But beyond that, many of the features Tachyon has added since its first commit in August 2025 are ones Austin has had for years:

Feature	In Austin since	In Tachyon since
Separate wall time and CPU time modes	0.2 (2018)	September 2025
Profiling child processes	1.0 (2019)	December 2025
Marking samples taken during garbage collection	3.1 (2021)	November 2025
Writing raw samples to a compact binary format, analysed later	3.4 (2022)	December 2025
Column-level source locations in that format	3.5 (2023)	December 2025
Sampling all sub-interpreters	3.6 (2023)	September 2025

Some of these exist in py-spy too, and it goes the other way as well: Tachyon's asyncio support landed before Austin's. That's how open source should work, with good ideas spreading between projects. What's missing is the credit. PEP 799 doesn't mention Austin, py-spy or any of the external profilers whose approach Tachyon adopts. I think that's a missed opportunity to recognise the years of community work that came before the standard library caught up, and that made it possible.

py-spy: The Established Choice¶

py-spy, written in Rust, made out-of-process sampling mainstream for Python, and it's what many people mean when they say "Python sampling profiler".

py-spy record -o profile.svg --pid 12345
py-spy top --pid 12345
py-spy dump --pid 12345

Its record, top and dump commands are a model of a simple interface. It writes SVG flame graphs, Speedscope and Chrome trace files, can follow subprocesses, filter to threads holding the GIL, and include or exclude idle threads. With --native it also unwinds native C, C++ and Rust frames on Linux (x86_64 and, since recently, aarch64) and Windows.

py-spy 0.4.2, released in April 2026, added Python 3.14 support. At the time of writing, free-threaded builds aren't supported yet (there's an open issue tracking it), and there's no task-aware asyncio mode: coroutines appear as ordinary frames on whichever thread runs the event loop.

Scalene: The In-Process All-Rounder¶

Scalene takes a different approach. It runs your program for you and profiles from the inside, which lets it measure things external tools can't easily see.

scalene run script.py

Its strengths are breadth and line-level detail:

CPU time split into Python, native and system time, per line and per function.
Memory profiling through its own allocator, including per-line allocation, likely leaks, and copy volume (for example, data accidentally copied between NumPy and Python objects).
GPU time on NVIDIA GPUs, and on Apple GPUs when using PyTorch's MPS backend.
Wall-clock await time per line for async code, along with how many coroutines were waiting there at once.
Combined Python and native stacks, shown as flame charts and a timeline.
AI-generated optimisation suggestions from the web UI, using a provider of your choice (OpenAI, Bedrock, Azure or a local model through Ollama).

Two of these numbers come with caveats. Scalene doesn't observe native code directly: it uses a timer signal, and treats any delay in handling it as time spent in native code, during which Python can't handle signals. But CPython only handles signals on the main thread, so in a program with several busy threads the delay can simply come from waiting for its turn on the GIL. In 2022, Pablo Galindo showed a multi-threaded, pure-Python program that Scalene reported as spending most of its time in native code (scalene#428). Scalene's author replied that worker threads are profiled in a different way, but the issue is still open, and the current code still estimates the main thread's native time from the signal delay. So treat the Python/native split with some care in multi-threaded programs.

For GPU time on NVIDIA, Scalene needs the GPU's per-process accounting mode, which has to be enabled once with root privileges. Without it, Scalene falls back to the utilisation of the whole GPU, including other processes, and shows a warning saying so.

Scalene 2.4 (October 2026) supports free-threaded Python 3.13 and 3.14. By its own figures, the overhead is typically 10–20%. That's fine during development, but the trade-off of being in-process is that you can't attach Scalene to something already running. You have to start the program under it.

Austin: Sampling from the Outside, at Speed¶

Austin is the tool I maintain. It's a frame stack sampler written in pure C, with no dependencies beyond the C standard library and the operating system. It reads the interpreter's memory from outside the process, writes every sample to a compact binary format called MOJO, and leaves the analysis to other tools. Because it doesn't aggregate anything while it samples, it can sample very fast.

austin -o profile.mojo python script.py
sudo austin -p 12345 -o profile.mojo
austin -w 12345   # print the current stacks of every thread

Austin 4.1 supports CPython 3.10 through 3.15, including free-threaded builds, on Linux, macOS and Windows. On Linux it also runs on less common architectures like i686, armv7 and ppc64le. Each release drops the oldest Python versions, but older releases of Austin are still available if you need to profile something older: Austin 4.0 supports 3.9 through 3.14, 3.7 supports 3.8 through 3.13, and 3.5 goes all the way back to Python 2.3.

Austin 4.1 adds four things worth covering in some detail.

1. Higher Sampling Rates¶

Austin 4.0 already more than doubled the maximum sampling rate of the previous release, by cutting down how much memory Austin reads from the target and how much work it does to write each sample. 4.1 goes further: Austin now detects when a thread's stack hasn't changed since the previous sample. When that happens, it skips walking the stack again and writes a short "same as before" marker instead of the whole stack, which also makes the output files smaller. It also buffers its output instead of writing each sample separately.

Every performance change to Austin is benchmarked in CI against the code it replaces, and the results are posted on the pull request. Here are the two that made the biggest difference, measured on Linux (x86_64) with Python 3.13, at the smallest sampling interval (1 µs), where Austin samples as fast as it can:

Change	Maximum sampling rate (wall time)	Time per sample
4.0 performance work, compared with Austin 3.7 (#295)	54,000 → 111,000 samples/s (×2.1)	32.4 → 12.4 µs
4.1 repeated stack detection (#402)	121,000 → 166,000 samples/s (+37%)	15.0 → 10.4 µs

In CPU time mode, where Austin also has to check whether each thread is on the CPU, the same changes took the maximum rate from 20,000 to 32,000 samples per second (+60%) and from 34,000 to 37,000 (+9%) respectively.

Something to keep in mind is that the gain from repeated stack detection depends on how often stacks repeat between samples. That happens a lot when a program spends time in long-running loops or calls, but it varies with the workload.

As explained earlier, this matters for accuracy: at the default 100 µs interval, Austin collects up to 10,000 samples per second per thread, ten times Tachyon's default rate.

2. Native Mode¶

With -n/--native, Austin collects native call stacks and interleaves them with the Python frames they belong to:

austin -n -i 1ms -o profile.mojo python script.py

This shows, for example, what's actually happening inside NumPy, a Cython module or a Rust extension built with PyO3, not just that time was spent "somewhere in native code". In native mode Austin also samples threads that never run Python code, like worker threads started by native libraries, which are invisible to profilers that only look at Python threads.

Austin users may remember that native stacks used to need a separate tool, austinp, a variant of Austin that uses libunwind to walk native stacks. It was a separate tool for two reasons. It only worked on Linux. And it broke one of Austin's founding principles, minimal impact on the program being profiled: to walk a thread's native stack reliably, libunwind needs the thread to stand still, so austinp had to stop each thread while sampling it. I wrote about that trade-off in Spy on Python down to the Linux kernel level.

The new native mode fixes the first problem. It works on Linux (x86_64 and aarch64), macOS and Windows (x86_64), with no extra dependencies, because Austin now walks native stacks itself instead of relying on libunwind. Its design owes a lot to samply, Markus Stange's excellent native sampling profiler, which showed how fast and portable this kind of unwinding can be. On macOS, Austin captures native stacks the same way samply does. On Linux, both unwind using the compiler-generated unwind tables, though they capture stacks differently, and on Windows samply relies on the system's event tracing while Austin unwinds stacks itself. As far as I know, Austin is currently the only out-of-process Python profiler that unwinds native stacks on macOS. austinp is still available on Linux, for the architectures that native mode doesn't cover yet and for sampling kernel stacks.

There is one other way to get Python and native frames in the same profile. Since 3.12, CPython can make Python function calls visible to native profilers like perf and samply (on macOS from Python 3.15), if you enable its perf support with -X perf or PYTHONPERFSUPPORT=1. You get Python function names in a fully native profile. But it has to be enabled inside the target process, it only goes down to functions, not lines, and it slows Python down: in 2023 Mark Shannon measured an 8% slowdown on a standard build, with more expected in builds that keep frame pointers. So it's a good option for a quick look at a local program, rather than for attaching to something already running.

The second problem is a deliberate compromise. To get a consistent native stack, Austin still has to briefly suspend each thread while sampling it, so native mode has more impact on the program than Austin's normal mode, and the impact grows with the sampling rate. Use a larger interval (1 ms is a good start), and avoid it in production. That's why native mode is an option you turn on, not the default. In its normal mode, Austin still never pauses the program it's profiling, and I explained the reasoning behind that in an earlier post.

3. asyncio Profiling¶

On CPython 3.14 and later, Austin tracks the asyncio task graph: each event loop's tasks, the coroutine chain each task is executing or suspended at, and which tasks are awaiting which. Samples are attributed to the task running on the thread, and the stacks of suspended tasks are recorded together with the await relationships between them.

That's what you need to answer the question that ordinary flame graphs get wrong for async code: not "which frames were on the CPU", but "which task was slow, and was it slow because it was working or because it was waiting on something else?" And austin -w prints the live task tree of a running application, grouped under the thread whose event loop owns it.

There's a further difference that's easy to miss. Most sampling profilers only record what was running, so each sample is anonymous: they can tell you that fetch_user took 30% of the time, but not whether that was one slow task or a thousand quick ones. Tachyon's async-aware mode labels each stack with the name of its task, but still adds the samples up into a single aggregate profile. Austin records which task each sample belongs to, and follows each task from the first time it sees it to the last. That makes its asyncio support closer to a tracer in spirit, while keeping the low overhead of sampling.

The VS Code extension makes use of this. Alongside the usual flame graph, which adds up time across all tasks, its Task Trace panel shows a flame chart for each event loop: every task drawn from when it was first seen to when it was last seen, nested under the task awaiting it, with how much of that time it spent running and how much waiting. So you can see what happened over time, not just where the time went in total, which is often what you need to work out why a request was slow.

4. Making Profiling Approachable¶

Here's an uncomfortable truth: most Python code never gets profiled. Not because the tools are missing, but because profiling feels hard to get into. You have to pick a tool, run it the right way, and then make sense of a flame graph full of frames you've never heard of. Many developers try it once, feel lost, and go back to guessing. That's a shame, because as I argued in How to bust Python performance issues, a single profiling run often turns up easy wins. A big part of what I want Austin to do is lower that barrier, and 4.1 approaches it from two directions.

Let an agent do the work. Coding agents can now profile code with Austin and reason about the results themselves. You ask "why is this endpoint slow?", and the agent runs Austin, finds the hot path, proposes a fix, and profiles again to check that the fix worked. There are two ways to set this up:

The VS Code extension includes an MCP server that exposes the loaded profile to agents. Its data tools let the agent ask for the hottest functions, call stacks, asyncio task traces, GC activity and metadata. It works with GitHub Copilot's agent mode with no setup, and with Claude Code, Cursor or any other MCP client after running a single command.
The austin-python package has a query API that answers the same questions in plain Python, and a command-line front-end for it, austin-query, that prints the results as text tables or JSON. Any agent that can run shell commands can use it, in a terminal, a script or CI. Pointing the agent at austin-query --help is usually all the setup it needs.

The command line is a good place to start, but agents get more out of the Python API behind it. Each query returns typed objects rather than text, so the agent can write a short script that hands them over to other libraries: pandas to compare two profiles, a plotting library to chart the results, or the project's own code to work out which component owns each hot function. An analysis that would take a long chain of CLI calls, each one reading the profile again, becomes a few lines of Python over a profile loaded once. The selectors and filters are the same as on the command line, e.g. q.filter(thread="main").callers("db.py:execute"), so the agent can move from one to the other without learning anything new. For example, this is all it takes to see which functions got faster, or slower, after a fix:

import pandas as pd
from austin.query import ProfileQuery

def hot(path):
    return pd.Series({
        f"{e.location.filename}:{e.location.function}": e.own_pct
        for e in ProfileQuery.from_file(path).top(50)
    })

change = hot("after.mojo").sub(hot("before.mojo"), fill_value=0)
print(change.sort_values().to_string())

A word of caution, though: a single pair of profiles is a shaky basis for a verdict. Sampling, and the system itself, add noise from one run to the next, so part of the change above could be nothing more than that. For a rigorous answer, pyrometry is a small library of mine that implements the ideas of On the Algebraic Properties of Flame Graphs. It treats flame graphs as vectors that can be added, subtracted and averaged, and it compares two sets of profiles, say five runs before a change and five after, with a statistical test (Hotelling's T²), keeping only the stacks whose difference is significant. The result is then split into the stacks that appeared, disappeared, grew or shrank. That's out of reach for a chain of CLI calls, but it's a short script for an agent with the Python API, and pointing it at pyrometry gives it a sound method to follow instead of eyeballing two flame graphs. Nor is this just theory: Austin's own CI uses the same test to validate every pull request. It profiles the same scenarios ten times with the base branch and ten times with the changes, on every supported version of Python, and checks that the new build still reports the same data as the old one.

Or let it teach you. Having an agent fix things for you is useful, but it doesn't make you any better at reading a profile. So the MCP server also has action tools that let the agent search and zoom the flame graph in the editor, while you watch. That turns the extension into a learning tool: instead of "fix this", you can ask your agent to walk you through the analysis. It can zoom into the hot path and explain why it's hot, point out the frame where your code hands over to a library, or show you what time spent waiting looks like compared with time spent computing. You're both looking at the same flame graph, so after a few sessions you start to recognise the patterns yourself, and the next time, you might not need to ask. Busting performance issues, AI edition shows both approaches on a real project, including one of these guided tours.

Other Things in 4.1¶

The rest of the release is about meeting Python where it's going, and where it actually runs. Austin 4.1 supports Python 3.15 on day one, and free-threaded builds too. Free-threading changes a lot for a profiler: without the GIL, threads really do run at the same time, so a sampler has to keep up with all of them at once. The one thing it costs is some accuracy in memory mode, since without the GIL there's no reliable way to tell which thread caused a change in memory usage (see below).

Profiles are also easier to read. Threads now show up under the names you gave them in the threading module, so you see MainThread and worker-3 rather than a list of opaque thread IDs, which makes a big difference in a busy multi-threaded application.

Finally, a lot of Python runs in containers, and profiling it from the host used to be awkward. Austin 4.1 handles containerised processes better, and it now records the target's working directory, virtual environment and PYTHONPATH alongside the samples. That's the information a tool needs to map the file paths in a profile back to the source files on your machine, even when the code ran somewhere else.

Where Austin Falls Short¶

Whilst Austin has a memory mode, users must be aware that it isn't an allocation profiler. Its memory mode measures changes in resident memory (RSS) between samples: how much physical memory the process is actually using, not how much it has asked for. These can be very different numbers. If you allocate a 512 MB array but only write to half of it, the operating system only provides physical memory for the half you've used, so RSS grows by about 256 MB.

That's exactly the test the Scalene study used to compare memory profilers: allocate 512 MB, touch between 0% and 100% of it, and check that the profiler reports 512 MB. Austin mostly reported less, and the study counted that as inaccurate. But reporting only the memory that was touched is the correct answer for what Austin measures. The study was asking an allocation question, and Austin answers a residency question. Both are useful: RSS tells you what your process really costs the machine, which is what matters when you're about to run out of memory. But if you want to know which line allocated what, or hunt for leaks, you want an allocation profiler like Scalene or memray. Austin also has no line-level GUI of its own. It relies on the VS Code extension, the TUI, Speedscope or pprof for visualisation, so out of the box it asks a bit more of you than Scalene or py-spy do. And attaching to a process needs elevated privileges on Linux and macOS, as it does for any out-of-process profiler.

Which One Should You Use?¶

There isn't one best profiler. Here's how I'd choose:

You're on Python 3.15 and want something quick: start with Tachyon. It's already installed, and it's good.
You need to know about memory or GPU time, line by line: use Scalene, as long as you can start the program under it.
You need to profile older Python versions, or attach to production: py-spy and Austin both do this well. Austin covers 3.15 and free-threaded builds today, and samples faster.
You're profiling asyncio code: on 3.14+, Austin gives you task-level profiles that account for waiting time, Tachyon gives you task-aware stacks on 3.15, and Scalene tells you which await lines are slow.
You're working on native extensions, especially on macOS: use Austin's native mode.
You want your coding agent to do the profiling, or to teach you how: use Austin's MCP server or austin-query.

Other tools worth knowing about: pyinstrument for a friendly in-process call tree, memray for serious memory work, and cProfile (now profiling.tracing) when you need exact call counts.

What's Next¶

Python profiling has never been in better shape. Having a sampling profiler in the standard library raises the baseline for everyone, and the remote debugging work in 3.14 and 3.15 makes all external tools more reliable. I expect the next round of progress to come from two directions: better support for concurrency (free-threading and asyncio together), and making profiling accessible to everyone, whether that means an agent doing the work or an agent teaching you to do it yourself. The best profiler is the one people actually use.

Austin 4.1 is available on PyPI, Homebrew, Snap, Chocolatey, Scoop, conda-forge and as standalone binaries from GitHub releases. The documentation covers everything above in more detail. If you try it, I'd love to hear how it goes.

Busting performance issues, AI edition

2026-05-16T20:05:00+01:00

In a previous post we looked at how to find and resolve performance issues lurking in any codebase by simply running some of its code through a profiler. The simplest thing we can do, especially when we come to a new codebase that is totally unknown to us is to run its test suite (or a subset of it) and see what code paths are being exercised. When we look at them through a profiler, the typical approach is to start investigating its plateaux, that is functions at the top (or bottom, depending on the orientation of the flame graph!) that have the largest own time. This is generally where the largest opportunity of improvements are. Of course, there could be cases where a function is simply repeating unnecessary work, and this won't show up through a mere visual plateau scan.

Enter AI¶

Can we automate this process, and perhaps put it on steroids, with the prowess of the most powerful, state-of-the-art coding agents?

Modern coding agents have reached a level of sophistication where they can effectively understand and modify codebases with good practical results. Benchmarks such as SWE-bench (which tests multi-file bug fixes in real GitHub repositories) and LiveCodeBench (which uses fresh competitive programming problems to avoid data contamination) show that frontier models now achieve high pass rates. However, there still is quite the margin for improvement: studies on real-world class-level code generation show a significant gap between synthetic benchmark performance (84-89%) and actual project performance (25-34%) (arxiv-2510.26130), highlighting both the progress made and the room for improvement in practical coding scenarios.

In this post we will explore how well a coding agent can come up with performance improvements when it is provided with focused profiling data. The focus, however, is on the method and the tooling involved, rather than benchmarking a set of models to determine which one is better at finding performance issues.

bytecode: Our Case Study¶

To make the exposition concrete we will focus on an actual project: we will try to improve the performance of the bytecode library.

bytecode is a Python library for generating and modifying Python bytecode. It provides an abstraction over Python's low-level bytecode instructions, allowing you to programmatically create bytecode objects, convert them to actual code objects, and even inspect and modify existing code's bytecode. It is useful for metaprogramming, dynamic code generation, and bytecode analysis.

These are the tools that we will use for our little experiment, all free:

Austin, for Python profiling, easily installed with "pip install austin-dist"
The Austin VS Code extension, easily installed from the VS Code marketplaces. The latest version comes with an MCP server that can feed profiling data and flame graph navigation commands to a coding agent.
Opencode with the MiniMax M2.5 free model as our coding agent setup.
The Opencode VS Code extension to never leave VS Code throughout our experiment

A Bit of Setup Required¶

After having cloned the bytecode repository that we will be working on, and opened it in VS Code, there is a minimum of setup required. We need to allow the coding agent to discover the Austin MCP. If you're using GitHub Copilot instead of Opencode, there is no extra setup required for this, since the Austin MCP server integrates natively with that. Otherwise we need to allow other coding agents to discover the server. This is as simple as running the Austin: Generate .mcp.json command: press Ctrl+Shift+P, type austinmcp and press Enter. That's it. Well, not quite, because we decided to use Opencode for our experiment, and as at the moment of writing, it still does not support a .mcp.json file. So instead we open an Opencode session with Ctrl+Escape and politely ask the agent to convert the .mcp.json configuration to something that opencode understands

The Austin MCP server uses an ephemeral port to run, so every time you re-open VS Code there will be a new port assigned to it. If a .mcp.json was previously created, the Austin extension will update the port automatically so that you don't have to regenerate it every time. However, Opencode won't do this unless you instruct it to.

After Opencode configured itself, you might have to start a new session for it to pick up the connection to the server.

We have asked Opencode to configure itself based on the contents of the .mcp.json file

Finally, let's create a virtual environment where we can install the library in editable mode so that we can quickly test our changes. Pick your favourite version of Python (one that the library supports of course, e.g. 3.13) and run

cd /path/to/bytecode
python3.13 -m venv .venv
source .venv/bin/activate
pip install -e .

After this we're all set to start experimenting!

Establishing the Baseline¶

As a first step, we should probably try to establish a baseline. This will give us a sense of where we're starting from in terms of the performance to make sure that we have a clear signal that the changes that the coding agent is coming up with are actually improving things. For our case at hand, we can use a simple round-trip benchmark run: we take a reference code object, decompile and recompile it in a loop for about 1 second. We can then measure how many times we were able to do this, which will be our throughput metric. Our baseline was 203 iterations per second. Meanwhile, we can also profile the benchmark to see what codepaths are being exercised and what they look like in terms of wall time distributions.

Here we won't bother too much with wall vs. CPU time because we expect the work to be essentially single-threaded CPU-bound anyway.

So this is the simple script that we will be using to drive the analysis

# perf.py

import bytecode
from time import perf_counter as time

code = bytecode.ConcreteBytecode.to_code.__code__

end = time() + 1.0
c = 0
while time() < end:
    c += 1
    bytecode.Bytecode.from_code(code).to_code()
print(c)

The output to screen is the plain throughput, so this will give us an immediate feedback when we run it. So let's run it and take note of the output, and to complete our baseline let's grab a profile as well

austin -o perf.mojo python perf.py

Communicating Our Intent to the Agent¶

Now that we have our baseline, let's state our intent with the agent and ask it to analyse the profiling data we have just collected.

We ask the Opencode agent to load the profile data we have collected with the Austin tools

Because the agent has access to the Austin MCP tools, it can easily figure out that it needs to use the load tool to open the profile. It can then use the other tools to pull meaningful profiling data, such as the top functions in terms of own time (the plateau analysis) and the call stacks. This allows the agent to focus on areas of the code base that the profile has highlighted as requiring attention, instead of getting lost in the entirety of the source code. Immediately the agent comes back to us with some ideas.

The first optimisation is implemented and already giving a small improvement. But we can do better

On a Mission¶

Galvanised by the early success, the agent gets very eager to implement more and more improvements, quickly spiraling into an iteration loop where it autonomously runs and profiles the benchmark script.

The agent gets into an optimisation iteration loop by itself until it can find margin for improvement

We ask the agent to include test runs in the loop to make sure that its changes are not breaking anything and it complies immediately

Tests are now part of the iteration loop to add validation

Now we're not here to advocate for this or that model (I have obtained much better results with more powerful models), but we've already achieved a +17% improvement (203 → 237 iterations per second) in just a few minutes of running this experiment using completely free tools, so this is already a pretty good win. We can maybe push things a bit further with some human input. For example, looking at the Top View from the Austin side panel, we notice that dis shows up at the top in terms of own time and we might wonder if we can bypass its overhead for something more direct for our needs.

We ask the agent to consider some aspects of the coding based on what we can see from the Austin data, e.g. the top view

A Guided Tour of Profiles¶

Reading a profile is generally not an easy task, especially when you jump into a code base that you are not familiar with, and the profile looks "hairy": many spiky peaks and small plateaux that don't really give you an immediate lead for a performance investigation. Because coding agents can understand a code base very quickly and reason about code paths, allowing them to utilise profiling data to drive their analysis means that we can put performance analysis on overdrive with the right set of tools. But while allowing an agent to figure out performance issues is nice and cool, especially because of the satisfying wins, one might still want to be kept in the loop and know what the agent is thinking. The tools exposed by the Austin MCP server allow us to ask the agent to show us around the profile in a sort of guided tour. So we could, for instance, ask it to show us the relevant part in a profile where it is considering making performance optimisations, and why. This might help the user develop a better intuition of how to use profiles, essentially using the coding agent as a tutor that can instruct how to read a profile.

We ask the agent to show us around the profile so that we can also get an idea of what is going on

A Bonus: Learning to Read Profiles¶

So beyond just fixing issues, there's a hidden gem in this workflow: you can use the agent as a learning tool. Instead of just handing off the problem, try asking it to explain what it's looking at:

"What does this plateau tell us?"
"Why is this function taking so long compared to others?"
"What's the call path to this hot function?"
"What would you look for next in this profile?"

Over time, this might help you build intuition for reading profiles yourself, if you have never dived into one before, or you have only skimmed some briefly. The agent doesn't get tired of explaining, and you get a personalised tutorial every time you run a new profile. This turns a powerful but esoteric tool like a profiler into something more accessible, an assistant that can teach you at your own pace.

Is Profiling Data Really Necessary?¶

One observation that can be made at this point is how relevant profiling data is in this process. Would the agent been able to identify the same performance issues in the code without consulting any profiling data and by just looking at the code? To try answer this question, I have repeated the experiment, starting from the same commit, and I have asked the agent to identify any optimisation opportunities just by looking at the code

The agent is now tasked to find optimisations without any profiling data

Perhaps unsurprisingly, after a first scan of the code, the agent feels the need to reach out for a profiler (cProfile from the Python standard library in this case) to guide its analysis.

The agent feels the need for profiling data

However, even with that data now at its disposal, it struggles quite a bit to decide what needs to be optimised. It looks as if it had identified the repeated validation issue, but it's not quite sure it is the best path to take. After 15 minutes the agent is still debating on what to do.

The agent is not quite sure what to do exactly

When asked where we are in terms of gains after some attempts at reducing some repeated operations, the agent replies that the gains are still quite marginal and that it needs to think about the code more.

The agent is getting somewhere, but the progress is pretty slow by its own admission

Half an hour in we still haven't breached the 5% barrier, and when I ask the agent to validate its changes by running the test suite we discover a segmentation fault.

The tests are failing hard with a segmentation fault

One fair conclusion we can draw from this is that indeed profiling data matters, as well as the quality of the data itself. Quite likely, the Austin MCP server was able to feed the agent with more fine-grained, detailed profiling data that allowed it to focus on the areas of the codebase that really needed attention to find those performance optimisation opportunities. With lower quality data, or with the lack of it, the agent feels a bit lost and tries to reach for that missing information by itself.

The Takeaway¶

Our experiment has highlighted that a coding agent, when equipped with good quality profiling data, can find genuine performance improvements without much hand-holding. Our baseline of 203 iterations per second jumped to 237 with about 15 minutes of agent time and no manual profiling or code inspection on our part. The agent identified an O(n²) algorithm, inlined hot paths, and eliminated unnecessary validation. Instead, when profiling data was missing or not so fine-grained, the agent struggled to find any solid and correct improvements.

So take that project you've been meaning to optimise, fire up a good profiler, and ask your favourite coding agent to take a look. You might be surprised how far a little profiling data goes.

Running C unit tests with pytest

2022-02-11T14:01:00+00:00

Why?¶

That's probably what you might be asking right now. Why use a Python testing framework to test C code? Why don't just use C testing frameworks, like Google Test, or Check. I can give you an answer with all the reasons that led me to adopt pytest as the testing framework of choice for one of my C projects, Austin. The first is that I spend most time coding in Python these days and I have more familiarity with pytest than any other testing framework. Secondly, whilst Austin is a C project, it actually targets Python programs, so Python is already one of the testing dependencies that ends up being installed in CI anyway. Hence, instead of spending time learning an entire new testing framework, I could quickly write them in Python, and leverage all the features of pytest, as well as all the packages that are available for Python, should I ever need to. But these are not all the reasons for adopting pytest for running C tests. If you keep reading you will discover a few more that might convince you to use pytest for your C unit tests too!

Calling native code from Python¶

Testing C code with Python would only make sense if it were easy to call native code from the interpreter. Thankfully, the Python standard library comes with the ctypes module that allows us to do just that! So let's start looking at some C code, for instance,

# file: fact.c

long fact(long n) {
    if (n < 1)
        return 1;
    return n * fact(n - 1);
}

which we want to compile as a shared object, e.g. with

gcc -shared -o fact.so fact.c

How do we test the fact function from Python? Easy peasy!

# file: fact.py

from ctypes import CDLL

libfact = CDLL("./fact.so")

assert libfact.fact(6) == 720
assert libfact.fact(0) == 1
assert libfact.fact(-42) == 1

Assuming we are in the directory where both fact.so and fact.py reside, we can test the fact function inside fact.c simply with

python3 fact.py

If the test succeeds, the script's return code will be 0.

Congratulations! You have now tested some C code with Python! 🎉

Enter pytest¶

We are not here to just play around with bare asserts. I promised you the full power of Python and pytest, so we can't settle with just this simple example. Let's add pytest to our test dependencies and do this instead

# file: test_fact.py

from ctypes import CDLL

import pytest

@pytest.fixture
def libfact():
    yield CDLL("./fact.so")


def test_fact(libfact):
    assert libfact.fact(6) == 720
    assert libfact.fact(0) == 1
    assert libfact.fact(-42) == 1

Now run pytest to get

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0
rootdir: /tmp
collected 1 item

test_fact.py .                                                       [100%]

============================ 1 passed in 0.00s =============================

That's some more informative output than what a plain Python test script would give us! How about starting to leverage some of the other pytest features, like parametrised tests? Let's rewrite our test case like so

# file: test_fact.py

from ctypes import CDLL

import pytest


@pytest.fixture
def libfact():
    yield CDLL("./fact.so")


@pytest.mark.parametrize("n,e", [(6, 720), (0, 1), (-42, 1)])
def test_fact(libfact, n, e):
    assert libfact.fact(n) == e

Let's run this again with pytest, this time with a more verbose output:

pytest -vv
=========================== test session starts ============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /tmp/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /tmp
collected 3 items

test_fact.py::test_fact[6-720] PASSED                                [ 33%]
test_fact.py::test_fact[0-1] PASSED                                  [ 66%]
test_fact.py::test_fact[-42-1] PASSED                                [100%]

============================ 3 passed in 0.01s =============================

Sweet! 🍯

In the wild¶

Thus far we've got an idea of how to invoke C from Python and how to write some simple tests that we can run with pytest while also leveraging features like fixtures and parametrised tests. Let us now step this up a notch and consider the organisation of sources within an actual C project, for instance

my-c-project/
├── docs/
├── src/    <- All *.c and *.h sources, perhaps organised into sub-folders
├── tests/  <- Our test sources, obviously!
├── ChangeLog
├── configure.ac
├── LICENCE
├── Makefile.am
├── README
...

In the previous example, we built the shared object fact.so by hand, but in a CI environment we would probably want to automate that step too. What should we use for that? A bash script? A makefile? Python, of course! What else?!? 😀

Let's make our sample C sources slightly more interesting. For example, we could borrow a few parts of the cache.c and cache.h sources from Austin, which implement a simple LRU cache. This is part of the spec

// file: src/cache.h

#ifndef CACHE_H
#define CACHE_H

#include <stdint.h>
#include <stdlib.h>

typedef uintptr_t key_dt;
typedef void *value_t;

typedef struct queue_item_t
{
    struct queue_item_t *prev, *next;
    key_dt key;
    value_t value; // Takes ownership of a free-able object
} queue_item_t;

typedef struct queue_t
{
    unsigned count;
    unsigned capacity;
    queue_item_t *front, *rear;
    void (*deallocator)(value_t);
} queue_t;

queue_item_t *
queue_item_new(value_t, key_dt);

void
queue_item__destroy(queue_item_t *, void (*)(value_t));

queue_t *
queue_new(int, void (*)(value_t));

int
queue__is_full(queue_t *);

int
queue__is_empty(queue_t *);

value_t
queue__dequeue(queue_t *);

queue_item_t *
queue__enqueue(queue_t *, value_t, key_dt);

void
queue__destroy(queue_t *);

and this is the corresponding part of the implementation

// file: src/cache.c

#include <stdbool.h>
#include <stdio.h>

#include "cache.h"

#define isvalid(x) ((x) != NULL)

// ----------------------------------------------------------------------------
queue_item_t *
queue_item_new(value_t value, key_dt key)
{
    queue_item_t *item = (queue_item_t *)calloc(1, sizeof(queue_item_t));

    item->value = value;
    item->key = key;

    return item;
}

// ----------------------------------------------------------------------------
void
queue_item__destroy(queue_item_t *self, void (*deallocator)(value_t))
{
    if (!isvalid(self))
        return;

    deallocator(self->value);

    free(self);
}

// ----------------------------------------------------------------------------
queue_t *
queue_new(int capacity, void (*deallocator)(value_t))
{
    queue_t *queue = (queue_t *)calloc(1, sizeof(queue_t));

    queue->capacity = capacity;
    queue->deallocator = deallocator;

    return queue;
}

// ----------------------------------------------------------------------------
int
queue__is_full(queue_t *queue)
{
    return queue->count == queue->capacity;
}

// ----------------------------------------------------------------------------
int
queue__is_empty(queue_t *queue)
{
    return queue->rear == NULL;
}

// ----------------------------------------------------------------------------
value_t
queue__dequeue(queue_t *queue)
{
    if (queue__is_empty(queue))
        return NULL;

    if (queue->front == queue->rear)
        queue->front = NULL;

    queue_item_t *temp = queue->rear;
    queue->rear = queue->rear->prev;

    if (queue->rear)
        queue->rear->next = NULL;

    void *value = temp->value;
    free(temp);

    queue->count--;

    return value;
}

// ----------------------------------------------------------------------------
queue_item_t *
queue__enqueue(queue_t *self, value_t value, key_dt key)
{
    if (queue__is_full(self))
        return NULL;

    queue_item_t *temp = queue_item_new(value, key);
    temp->next = self->front;

    if (queue__is_empty(self))
        self->rear = self->front = temp;
    else
    {
        self->front->prev = temp;
        self->front = temp;
    }

    self->count++;

    return temp;
}

// ----------------------------------------------------------------------------
void
queue__destroy(queue_t *self)
{
    if (!isvalid(self))
        return;

    queue_item_t *next = NULL;
    for (queue_item_t *item = self->front; isvalid(item); item = next)
    {
        next = item->next;
        queue_item__destroy(item, self->deallocator);
    }

    free(self);
}

It's quite a fair bit of code; however, we are not interested in how the data structures are implemented, but rather to what it actually implements. This already gives us plenty to play with.

The important detail here is that our C application has a component implemented in cache.c and we want to unit-test it. Before we can run any actual tests, we need to build a binary object that we can invoke from Python. So let's put this code in tests/cunit/__init__.py

from pathlib import Path
from subprocess import PIPE, STDOUT, run

HERE = Path(__file__).resolve().parent
TEST = HERE.parent
ROOT = TEST.parent
SRC = ROOT / "src"


class CompilationError(Exception):
    pass


def compile(source: Path, cflags=[], ldadd=[]):
    binary = source.with_suffix(".so")

    result = run(
        ["gcc", "-shared", *cflags, "-o", str(binary), str(source), *ldadd],
        stdout=PIPE,
        stderr=STDOUT,
        cwd=SRC,
    )

    if result.returncode == 0:
        return

    raise CompilationError(result.stdout.decode())

This simply defines the compile utility that allows us to invoke gcc to compile a source and generate the .so shared object. We can then use it in our test source this way

from ctypes import CDLL

import pytest
from tests.cunit import SRC, compile

C = CDLL("libc.so.6")


@pytest.fixture
def cache():
    source = SRC / "cache.c"
    compile(source)
    yield CDLL(str(source.with_suffix(".so")))


def test_cache(cache):
    lru_cache = cache.lru_cache_new(10, C.free)
    assert lru_cache
    cache.lru_cache__destroy(lru_cache)

At this point, your project folder should have the following structure

my-c-project/
...
├── src/
|   ├── cache.c
|   └── cache.h
├── tests/
|   ├── cunit/
|   |   ├── __init__.py
|   |   └── test_cache.py
|   └── __init__.py
...

and when you run pytest again, this time the C source would be compiled at runtime using gcc. The tests then run as before, which should produce the same output we saw earlier.

Dead end?¶

If you're still with me, then things are probably looking interesting to you too. So let's test a bit more of the functions exported by the caching component. Let's make a test case for the queue_item_t and queue_t objects, like so

from ctypes import CDLL

import pytest
from tests.cunit import SRC, compile

C = CDLL("libc.so.6")


@pytest.fixture
def cache():
    source = SRC / "cache.c"
    compile(source)
    yield CDLL(str(source.with_suffix(".so")))


NULL = 0


def test_queue_item(cache):
    value = 1
    queue_item = cache.queue_item_new(value, 42)
    assert queue_item

    cache.queue_item__destroy(queue_item, C.free)


@pytest.mark.parametrize("qsize", [0, 10, 100, 1000])
def test_queue(cache, qsize):
    q = cache.queue_new(qsize, C.free)

    assert cache.queue__is_empty(q)
    assert qsize == 0 or not cache.queue__is_full(q)

    assert cache.queue__dequeue(q) is NULL

    values = [C.malloc(16) for _ in range(qsize)]
    assert all(values)

    for k, v in enumerate(values):
        assert cache.queue__enqueue(q, v, k)

    assert qsize == 0 or not cache.queue__is_empty(q)
    assert cache.queue__is_full(q)
    assert cache.queue__enqueue(q, 42, 42) is NULL

    assert values == [cache.queue__dequeue(q) for _ in range(qsize)]

Let's run the new tests with pytest -vv and

=============================== test session starts ===============================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/gabriele/Projects/cunit
collected 5 items

tests/cunit/test_cache.py::test_queue_item Fatal Python error: Segmentation fault

Current thread 0x00007f4016e4f740 (most recent call first):
  File "/home/gabriele/Projects/cunit/tests/cunit/test_cache.py", line 24 in test_queue_item
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/python.py", line 1718 in runtest
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 261 in <lambda>
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 340 in from_call
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 322 in _main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/home/gabriele/Projects/cunit/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/home/gabriele/Projects/cunit/.venv/bin/pytest", line 8 in <module>
[1]    337951 segmentation fault  .venv/bin/pytest -vv

Wait, what?! Where are our tests? A segmentation fault?!? Where did that come from? Well, there goes all this pytest hype! 😡

Now, do you think I would have written this post if this was really the end of the story?

If you want to figure out for yourself where the problem is, pause here. When you are ready to carry on, change line 20 to

value = C.malloc(16)

and now the tests will all be happy! However, we really want to avoid crashing the pytest process when we run into a segmentation fault, which is not so rare when running arbitrary C code. Not only that, but we would like to get some useful information, like a traceback, that could give us insight as to where the problem might be! One of the many strengths of pytest is its extensive configuration API. How do we use it to not crash the test runner? The idea is to spawn another pytest process that runs just a test. Now, if that test causes a segmentation fault, the parent process will keep running the other tests. Let's put this into tests/cunit/conftest.py

# file: tests/cunit/conftest.py

import os
import sys
from subprocess import PIPE, run
from types import FunctionType


class SegmentationFault(Exception):
    pass


class CUnitTestFailure(Exception):
    pass


def pytest_pycollect_makeitem(collector, name, obj):
    if (
        not os.getenv("PYTEST_CUNIT")
        and isinstance(obj, FunctionType)
        and name.startswith("test_")
    ):
        obj.__cunit__ = (str(collector.fspath), name)


def cunit(module: str, name: str, full_name: str):
    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        if result.returncode == 0:
            return

        raise CUnitTestFailure("\n" + result.stdout.decode())

    return _


def pytest_collection_modifyitems(session, config, items) -> None:
    if test_name := os.getenv("PYTEST_CUNIT"):
        # We are inside the sandbox process. We select the only test we care
        items[:] = [_ for _ in items if _.name == test_name]
        return

    for item in items:
        if hasattr(item._obj, "__cunit__"):
            item._obj = cunit(*item._obj.__cunit__, full_name=item.name)

Let's re-run our broken test suite and see what happens this time:

================================ test session starts =================================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/gabriele/Projects/cunit
collected 5 items

tests/cunit/test_cache.py::test_queue_item <- tests/cunit/conftest.py FAILED   [ 20%]
tests/cunit/test_cache.py::test_queue[0] <- tests/cunit/conftest.py PASSED     [ 40%]
tests/cunit/test_cache.py::test_queue[10] <- tests/cunit/conftest.py PASSED    [ 60%]
tests/cunit/test_cache.py::test_queue[100] <- tests/cunit/conftest.py PASSED   [ 80%]
tests/cunit/test_cache.py::test_queue[1000] <- tests/cunit/conftest.py PASSED  [100%]

====================================== FAILURES ======================================
__________________________________ test_queue_item ___________________________________

_ = ()
__ = {'cache': <CDLL '/home/gabriele/Projects/cunit/src/cache.so', handle 25c8400 at 0x7efd5b5d83d0>}
test = '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queue_item'
env = {'ANDROID_HOME': '/home/gabriele/.android/sdk', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}
result = CompletedProcess(args=['.venv/bin/pytest', '-svv', '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queu...__init__.py", line 188 in console_main\n  File "/home/gabriele/Projects/cunit/.venv/bin/pytest", line 8 in <module>\n')

    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        if result.returncode == 0:
            return

>       raise CUnitTestFailure("\n" + result.stdout.decode())
E       tests.cunit.conftest.CUnitTestFailure: 
E       ============================= test session starts ==============================
E       platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
E       cachedir: .pytest_cache
E       rootdir: /home/gabriele/Projects/cunit
E       collecting ... collected 1 item
E       
E       tests/cunit/test_cache.py::test_queue_item

tests/cunit/conftest.py:49: CUnitTestFailure
============================== short test summary info ===============================
FAILED tests/cunit/test_cache.py::test_queue_item - tests.cunit.conftest.CUnitTestF...
============================ 1 failed, 4 passed in 1.21s =============================

How do we like this better? Now the first test fails with the segmentation fault, but the rest of the test suite still runs and we can see the reason of the failure for the first test in the report, i.e. the segmentation fault.

But what is the conftest.py code actually doing? Let's have a look. The pytest_pycollect_makeitem hook gets invoked when the tests inside tests\cunit are being collected by pytest. At this stage we "mark" them as C unit tests by giving each collected item the __cunit__ attribute. The value is a tuple containing the information of where the item came from (collection.fspath is the path of the module that defined the test, e.g. tests/cunit/test_cache.py) and the test name (e.g. test_queue_item). In our case we only care about items that are of FunctionType type and that start with test_. The environment variable PYTEST_CUNIT is used to detect whether we are running in the parent pytest process or the "sandbox" child. In the latter case we don't care of marking tests because we know exactly what we want to run.

Once all the tests have been collected, we use the pytest_collection_modifyitems hook to actually modify the tests that we previously marked as C unit tests. Again, the behaviour depends on whether we are in the parent pytest process, or in the sandbox. If PYTEST_CUNIT is set, that's the signal that we are in the child pytest process. The value, as we shall see shortly, contains the information needed to pick the test that we want to run. So we use it to modify the list items to just the test that matches the information stored in PYTEST_CUNIT. In the parent pytest process we actually modify what the items that we marked as C unit tests do. Obviously, we don't want them to run the actual test, but rather a new instance of pytest that will then run the test on behalf of the parent process. The magic happens inside the "decorator" cunit, which we use to build a closure around the test that we want to run. As you can see, it returns a function (with a bit of a funny and unusual signature) which, when called, will set the PYTEST_CUNIT variable with the full name of the test (this is to support parametrised tests) and then run sys.argv[0] (which should be pytest if we run the test suite with pytest), followed by the switches -svv and the path of the module that provides the test we are wrapping around. This way, when the process terminates, either because the test passed or something really bad happened, we can inspect the return code and the streams, and act accordingly.

So now we have a test runner that can handle segmentation faults gracefully but still doesn't tell us where they actually happened. Can we get some more detailed information in the output? When a Python test fails we get a nice traceback that tells us where things went wrong. Can we do the same with C unit tests? The answer is yes, provided we collect core dumps while tests run. If you are running on Ubuntu, you can do ulimit -c unlimited and a core dump will be generated in the working directory every time a segmentation fault occurs. We can then run gdb in batch mode to print a nice traceback that will hopefully help us investigate the problem. So let's add these helpers to the conftest.py file

from pathlib import Path
from subprocess import STDOUT, check_output


def gdb(cmds: list[str], *args: str) -> str:
    return check_output(
        ["gdb", "-q", "-batch"]
        + [_ for cs in (("-ex", _) for _ in cmds) for _ in cs]
        + list(args),
        stderr=STDOUT,
    ).decode()


def bt(binary: Path) -> str:
    if Path("core").is_file():
        return gdb(["bt full", "q"], str(binary), "core")
    return "No core dump available."

and improve the cunit decorator like so

from tests.cunit import SRC


def cunit(module: str, name: str, full_name: str):
    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        match result.returncode:
            case 0:
                return

            case -11:
                binary_name = Path(module).stem.replace("test_", "")
                raise SegmentationFault(bt((SRC / binary_name).with_suffix(".so")))

        raise CUnitTestFailure("\n" + result.stdout.decode())

    return _

Now, when we run our broken test suite we should get this more verbose output

================================ test session starts =================================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0 -- /home/gabriele/Projects/cunit/.venv/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/gabriele/Projects/cunit
collected 5 items

tests/cunit/test_cache.py::test_queue_item <- tests/cunit/conftest.py FAILED   [ 20%]
tests/cunit/test_cache.py::test_queue[0] <- tests/cunit/conftest.py PASSED     [ 40%]
tests/cunit/test_cache.py::test_queue[10] <- tests/cunit/conftest.py PASSED    [ 60%]
tests/cunit/test_cache.py::test_queue[100] <- tests/cunit/conftest.py PASSED   [ 80%]
tests/cunit/test_cache.py::test_queue[1000] <- tests/cunit/conftest.py PASSED  [100%]

====================================== FAILURES ======================================
__________________________________ test_queue_item ___________________________________

_ = ()
__ = {'cache': <CDLL '/home/gabriele/Projects/cunit/src/cache.so', handle fc4ba0 at 0x7f5ec8d50460>}
test = '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queue_item'
env = {'ANDROID_HOME': '/home/gabriele/.android/sdk', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}
result = CompletedProcess(args=['.venv/bin/pytest', '-svv', '/home/gabriele/Projects/cunit/tests/cunit/test_cache.py::test_queu...__init__.py", line 188 in console_main\n  File "/home/gabriele/Projects/cunit/.venv/bin/pytest", line 8 in <module>\n')
binary_name = 'cache'

    def _(*_, **__):
        test = f"{module}::{name}"
        env = os.environ.copy()
        env["PYTEST_CUNIT"] = full_name

        result = run([sys.argv[0], "-svv", test], stdout=PIPE, stderr=PIPE, env=env)

        match result.returncode:
            case 0:
                return

            case -11:
                binary_name = Path(module).stem.replace("test_", "")
>               raise SegmentationFault(bt((SRC / binary_name).with_suffix(".so")))
E               tests.cunit.conftest.SegmentationFault: 
E               warning: core file may not match specified executable file.
E               [New LWP 548824]
E               [Thread debugging using libthread_db enabled]
E               Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
E               Core was generated by `/home/gabriele/Projects/cunit/.venv/bin/python3.10 .venv/bin/pytest -svv /home/'.
E               Program terminated with signal SIGSEGV, Segmentation fault.
E               #0  raise (sig=<optimised out>) at ../sysdeps/unix/sysv/linux/raise.c:50
E               50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
E               #0  raise (sig=<optimised out>) at ../sysdeps/unix/sysv/linux/raise.c:50
E                       set = {
E                         __val = {[0] = 0, [1] = 1, [2] = 3, [3] = 140049677169600, [4] = 8, [5] = 4714713, [6] = 140049688588951, [7] = 140049678146624, [8] = 3, [9] = 5586517, [10] = 17917168, [11] = 3, [12] = 3, [13] = 140049678074368, [14] = 140049678074368, [15] = 4663610}
E                       }
E                       pid = <optimised out>
E                       tid = <optimised out>
E                       ret = <optimised out>
E               #1  <signal handler called>
E               No locals.
E               #2  __GI___libc_free (mem=0x1) at malloc.c:3102
E                       ar_ptr = <optimised out>
E                       p = <optimised out>
E                       hook = 0x0
E               #3  0x00007f5fdbf1b40a in queue_item.destroy () from /home/gabriele/Projects/cunit/src/cache.so
E               No symbol table info available.
E               #4  0x00007f5fdb188ff5 in ?? () from /usr/lib/x86_64-linux-gnu/libffi.so.7
E               No symbol table info available.
E               #5  0x00007f5fdb18840a in ?? () from /usr/lib/x86_64-linux-gnu/libffi.so.7
E               No symbol table info available.
E               #6  0x00007f5fdb1a2286 in ?? () from /usr/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu.so
E               No symbol table info available.
E               #7  0x00007f5fdb196eba in ?? () from /usr/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu.so
E               No symbol table info available.
E               #8  0x0000000000512f83 in ?? ()
E               No symbol table info available.
E               #9  0x00007f5fda9fd160 in ?? ()
E               No symbol table info available.
E               #10 0x00007f5fdb1a12d0 in ?? () from /usr/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu.so
E               No symbol table info available.
E               #11 0x00007f5fdaa1b658 in ?? ()
E               No symbol table info available.
E               #12 0x00007f5fda9f9690 in ?? ()
E               No symbol table info available.
E               #13 0x00007f5fdabd1f20 in ?? ()
E               No symbol table info available.
E               #14 0x00007f5fdaa1b680 in ?? ()
E               No symbol table info available.
E               #15 0x00000000011325a0 in ?? ()
E               No symbol table info available.
E               #16 0x00007f5fda9f214a in ?? ()
E               No symbol table info available.
E               #17 0x00007f5fdaa1b820 in ?? ()
E               No symbol table info available.
E               #18 0x00007f5fda9f20f0 in ?? ()
E               No symbol table info available.
E               #19 0x00007f5fda9fd160 in ?? ()
E               No symbol table info available.
E               #20 0x000000000057b1ec in ?? ()
E               No symbol table info available.
E               #21 0x59586f2afaac8ab6 in ?? ()
E               No symbol table info available.
E               #22 0x00007f5fdaa1b7e0 in ?? ()
E               No symbol table info available.
E               #23 0x0000000001116534 in ?? ()
E               No symbol table info available.
E               #24 0x00007f5fda9d50c0 in ?? ()
E               No symbol table info available.
E               #25 0x00007f5fdaa1b808 in ?? ()
E               No symbol table info available.
E               #26 0x8000000000000002 in ?? ()
E               No symbol table info available.
E               #27 0x00007f5fda949480 in ?? ()
E               No symbol table info available.
E               #28 0x00007f5fdaa1b810 in ?? ()
E               No symbol table info available.
E               #29 0x00007f5fda96aec0 in ?? ()
E               No symbol table info available.
E               #30 0x00007f5fdaa1b800 in ?? ()
E               No symbol table info available.
E               #31 0x00000000011164f0 in ?? ()
E               No symbol table info available.
E               #32 0x0000000000575f1d in ?? ()
E               No symbol table info available.
E               #33 0x42439dfdd749cffb in ?? ()
E               No symbol table info available.
E               #34 0x00007f5fdaa1b620 in ?? ()
E               No symbol table info available.
E               #35 0x0000000001116534 in ?? ()
E               No symbol table info available.
E               #36 0x00007f5fdb4ec070 in ?? ()
E               No symbol table info available.
E               #37 0x00007f5fda9fbb80 in ?? ()
E               No symbol table info available.
E               #38 0x0000000000000000 in ?? ()
E               No symbol table info available.

tests/cunit/conftest.py:56: SegmentationFault
============================== short test summary info ===============================
FAILED tests/cunit/test_cache.py::test_queue_item - tests.cunit.conftest.Segmentati...
============================ 1 failed, 4 passed in 1.73s =============================

Hmm, Still not that useful. What if we compile with debug symbols? Let's change the cache fixture in test_cache.py so that it compiles the caching sources with the -g option

@pytest.fixture
def cache():
    source = SRC / "cache.c"
    compile(source, cflags=["-g"])
    yield CDLL(str(source.with_suffix(".so")))

Now that's much, much better!

E               #3  0x00007ff51cf8b40a in queue_item__destroy (self=0x104ac60, deallocator=0x7ff51cc68850 <__GI___libc_free>) at /home/gabriele/Projects/cunit/src/cache.c:37

This tells us exactly where the problem occurred. Now that we have this information we can fix the test and make the test suite happy again! 🎉

Pythonic C unit testing¶

OK, running C unit tests is nice and fun, but it doesn't save us much in terms of typing. In fact, the tests we wrote so far not only feel quite verbose, considering we are writing Python code, but don't feel Pythonic at all. Whilst there is, in principle, no reason why non-Python tests written in Python should look and feel Pythonic, can we somehow do something perhaps more elegant? The idea of using a fixture to wrap around a binary object is perhaps interesting, but can we maybe pretend that cache is a Python module instead so that we can do things like from cache import queue_item_new etc... and sweep all this ctypes business under the carpet? Well, let's give this a try, shall we?

Back in tests/cunit/__init__.py, let's add the following subtype of ModuleType:

from ctypes import CDLL
from types import ModuleType


class CModule(ModuleType):
    def __init__(self, source):
        super().__init__(source.name, f"Generated from {source.with_suffix('.c')}")
        self.__binary__ = CDLL(source.with_suffix(".so"))
        print(self.__binary__.__dict__)

    def __getattr__(self, name):
        return getattr(self.__binary__, name)

    @classmethod
    def compile(cls, source, cflags=[], ldadd=[]):
        compile(source.with_suffix(".c"), cflags, ldadd)
        return cls(source)

Now create tests/cunit/cache.py with the following content

import sys
from pathlib import Path

from tests.cunit import SRC, CModule

CFLAGS = ["-g"]

sys.modules[__name__] = CModule.compile(SRC / Path(__file__).stem, cflags=CFLAGS)

and get rid of the cache fixture in test_cache.py (make sure to remove it also from the test arguments!). Instead, add

import tests.cunit.cache as cache

et voilà! Now cache feels like a Python module that exports ordinary functions that we can call like any other Python function.

Now, what about this code

def test_queue_item():
    value = C.malloc(16)
    queue_item = cache.queue_item_new(value, 42)
    assert queue_item

    cache.queue_item__destroy(queue_item, C.free)

Clearly cache.queue_item_new is creating a new object. The Pythonic way of writing something like this would be

def test_queue_item():
    value = C.malloc(16)
    queue_item = cache.QueueItem(value, 42)
    assert queue_item

and we don't even care about destroying the object as we'd love the garbage collector to take care of that for us. Now that's a more Pythonic way of going about our C unit tests! Can we achieve something like this? The answer is once again yes, but ... we can make this work provided we make some further assumptions, like some naming conventions. You might have noticed that the data structures defined in cache.h have the naming <adt>_t, in an OOP flavour. Methods follow the naming convention <adt>_<staticmethod> and <adt>__<emethod>. Some of the methods have special names, like <adt>_new, <adt>__destroy etc.... So, provided we adhere to some naming conventions, like this one, we could dynamically create Python types at runtime. How? With the secret art of metaprogramming. But before we can start creating new types at runtime, we need to be able to parse C header files to infer their definitions. That's why our next step is to add pycparser to our test dependencies and implement a C AST visitor that can collect all the relevant type and method declarations that we can use to build the spec for Python types. Here is what it might look like

from ctypes import c_char_p

from pycparser import c_ast, c_parser
from pycparser.plyparser import ParseError

class DeclCollector(c_ast.NodeVisitor):
    def __init__(self):
        self.types = {}
        self.functions = []

    def _get_type(self, node):
        print(node)
        return self.types[" ".join(node.type.type.names)]

    def visit_Typedef(self, node):
        if isinstance(node.type.type, c_ast.Struct) and node.type.declname.endswith(
            "_t"
        ):
            struct = node.type.type
            self.types[node.type.declname[:-2]] = CTypeDef(
                node.type.declname,
                [decl.name for decl in struct.decls],
            )

    def visit_Decl(self, node):
        if "extern" in node.storage:
            return

        if isinstance(node.type, c_ast.FuncDecl):
            func_name = node.name
            ret_type = node.type.type
            rtype = None
            if isinstance(ret_type, c_ast.PtrDecl):
                if "".join(ret_type.type.type.names) == "char":
                    rtype = c_char_p
            args = (
                [_.name if hasattr(_, "name") else None for _ in node.type.args.params]
                if node.type.args is not None
                else []
            )
            if func_name.endswith("_new"):
                self.types[f"{func_name[:-4]}"].constructor = CFunctionDef(
                    "new", args, rtype
                )
            elif "__" in func_name:
                type_name, _, method_name = func_name.partition("__")
                if not type_name:
                    return
                self.types[type_name].methods.append(
                    CFunctionDef(method_name, args, rtype)
                )
            else:
                self.functions.append(CFunctionDef(func_name, args, rtype))

    def collect(self, decl):
        parser = c_parser.CParser()
        try:
            ast = parser.parse(decl, filename="<preprocessed>")
        except ParseError as e:
            lines = decl.splitlines()
            line, col = (
                int(_) - 1 for _ in e.args[0].partition(" ")[0].split(":")[1:3]
            )
            for i in range(max(0, line - 4), min(line + 5, len(lines))):
                if i != line:
                    print(f"{i+1:5d}  {lines[i]}")
                else:
                    print(f"{i+1:5d}  \033[33;1m{lines[line]}\033[0m")
                    print(" " * (col + 5) + "\033[31;1m<<^\033[0m")
            raise

        self.visit(ast)
        return {
            k: v
            for k, v in self.types.items()
            if isinstance(v, CTypeDef) and v.constructor
        }

This is also a handful, but the behaviour is very simple. We define an AST visitor that listens to typedefs and declarations. For the former, we single out structure declarations and store the relevant information inside an instance of the custom CTypeDef dataclass; as for the latter, we only trap function declaration (plus a special handling for those functions that return char *). The two intermediate dataclasses CTypeDef and CFunctionDef are merely defined as (up to you to use the more elegant @dataclass decorator)

class CFunctionDef:
    def __init__(self, name, args, rtype):
        self.name = name
        self.args = args
        self.rtype = rtype


class CTypeDef:
    def __init__(self, name, fields):
        self.name = name
        self.fields = fields
        self.methods = []
        self.constructor = False

Armed with these definitions we can enhance our CModule class like so

class CModule(ModuleType):
    def __init__(self, source):
        super().__init__(source.name, f"Generated from {source.with_suffix('.c')}")
        self.__binary__ = CDLL(source.with_suffix(".so"))

        collector = DeclCollector()

        for name, ctypedef in collector.collect(
            preprocess(source.with_suffix(".h"))
        ).items():
            parts = name.split("_")
            py_name = "".join((_.capitalize() for _ in parts))
            setattr(self, py_name, CMetaType(self, ctypedef, None))

        for cfuncdef in collector.functions:
            name = cfuncdef.name
            try:
                cfunc = CFunction(cfuncdef, getattr(self.__binary__, name))
                setattr(self, name, cfunc)
            except AttributeError:
                # Not part of the binary
                pass

    @classmethod
    def compile(cls, source, cflags=[], ldadd=[]):
        compile(source.with_suffix(".c"), cflags, ldadd)
        return cls(source)

That is, we add the collected types and functions as attributes to the C module. This is the C type metaclass that we use as a C type factory

class CMetaType(type(Structure)):
    def __new__(cls, cmodule, ctypedef, _=None):
        ctype = super().__new__(
            cls,
            ctypedef.name,
            (CType,),
            {"__cmodule__": cmodule},
        )

        constructor = getattr(cmodule.__binary__, f"{ctypedef.name[:-2]}_new")
        ctype.new = CStaticMethod(ctypedef.constructor, constructor, ctype)

        for method_def in ctypedef.methods:
            method_name = method_def.name
            method = getattr(cmodule.__binary__, f"{ctypedef.name[:-2]}__{method_name}")
            setattr(ctype, method_name, CMethod(method_def, method, ctype))

        ctype.__cname__ = ctypedef.name

        return ctype

This is responsible for creating a new C type as a Python type, with all the methods added as appropriate instances of wrappers around the C functions (note the special handling of the _new method!):

class CFunction:
    def __init__(self, cfuncdef, cfunc):
        self.__name__ = cfuncdef.name
        self.__args__ = cfuncdef.args
        self.__cfunc__ = cfunc
        if cfuncdef.rtype is not None:
            self.__cfunc__.restype = cfuncdef.rtype

        self._posonly = all(_ is None for _ in self.__args__)

    def check_args(self, args, kwargs):
        if self._posonly and kwargs:
            raise ValueError(f"{self} takes only positional arguments")

        nargs = len(args) + len(kwargs)
        if nargs != len(self.__args__):
            raise TypeError(
                f"{self} takes exactly {len(self.__args__)} arguments ({nargs} given)"
            )

    def __call__(self, *args, **kwargs):
        self.check_args(args, kwargs)
        return self.__cfunc__(*args, **kwargs)

    def __repr__(self):
        return f"<CFunction '{self.__name__}'>"


class CMethod(CFunction):
    def __init__(self, cfuncdef, cfunc, ctype):
        super().__init__(cfuncdef, cfunc)
        self.__ctype__ = ctype

    def __get__(self, obj, objtype=None):
        def _(*args, **kwargs):
            cargs = [obj.__cself__, *args]
            self.check_args(cargs, kwargs)

            return self.__cfunc__(*cargs, **kwargs)

        _.__cmethod__ = self

        return _

    def __repr__(self):
        return f"<CMethod '{self.__name__}' of CType '{self.__ctype__.__name__}'>"


class CStaticMethod(CFunction):
    def __init__(self, cfuncdef, cfunc, ctype):
        super().__init__(cfuncdef, cfunc)
        self.__ctype__ = ctype

    def __repr__(self):
        return f"<CStaticMethod '{self.__name__}' of CType '{self.__ctype__.__name__}'>"

The CType class implementation tries to mimic the behaviour of Python classes, with __cself__ playing the role of the C analogue of self:

class CType(Structure):
    def __init__(self, *args, **kwargs):
        self.__cself__ = self.new(*args, **kwargs)

    def __del__(self):
        if len(self.destroy.__cmethod__.__args__) == 1:
            self.destroy()

    def __repr__(self):
        return f"<{self.name} CObject at {self.__cself__}>"

We also handle the destructor by overriding the __del__ special method so that the garbage collector can take care of freeing memory for us.

In general, we would need to preprocess sources, especially headers, before we can parse them concretely with pycparser. That's why we also need to define something like

restrict_re = re.compile(r"__restrict \w+")

_header_head = r"""
#define __attribute__(x)
#define __extension__
#define __inline inline
#define __asm__(x)
#define __const=const
#define __inline__ inline
#define __inline inline
#define __restrict
#define __signed__ signed
#define __GNUC_VA_LIST
#define __gnuc_va_list char
#define __thread
"""


def preprocess(source: Path) -> str:
    with source.open() as fin:
        code = _header_head + fin.read()
        return restrict_re.sub(
            "",
            run(
                ["gcc", "-E", "-P", "-"],
                stdout=PIPE,
                input=code.encode(),
                cwd=SRC,
            ).stdout.decode(),
        )

The _header_head and the restrict_re are needed to take care of GCC extensions, which are not supported by pycparser. But apart from them, all we do is invoke the gcc preprocessor on the C sources. Putting everything together inside tests/cunit/__init__.py we would then have something that looks like

import ctypes
import re
from ctypes import CDLL, POINTER, Structure, c_char_p, cast
from pathlib import Path
from subprocess import PIPE, STDOUT, run
from types import ModuleType
from typing import Any, Optional

from pycparser import c_ast, c_parser
from pycparser.plyparser import ParseError

HERE = Path(__file__).resolve().parent
TEST = HERE.parent
ROOT = TEST.parent
SRC = ROOT / "src"


restrict_re = re.compile(r"__restrict \w+")

_header_head = r"""
#define __attribute__(x)
#define __extension__
#define __inline inline
#define __asm__(x)
#define __const=const
#define __inline__ inline
#define __inline inline
#define __restrict
#define __signed__ signed
#define __GNUC_VA_LIST
#define __gnuc_va_list char
#define __thread
"""


def preprocess(source: Path) -> str:
    with source.open() as fin:
        code = _header_head + fin.read()
        return restrict_re.sub(
            "",
            run(
                ["gcc", "-E", "-P", "-"],
                stdout=PIPE,
                input=code.encode(),
                cwd=SRC,
            ).stdout.decode(),
        )


def compile(source: Path, cflags=[], ldadd=[]):
    binary = source.with_suffix(".so")

    result = run(
        ["gcc", "-shared", *cflags, "-o", str(binary), str(source), *ldadd],
        stdout=PIPE,
        stderr=STDOUT,
        cwd=SRC,
    )

    if result.returncode == 0:
        return

    raise RuntimeError(result.stdout.decode())


C = CDLL("libc.so.6")


class CFunctionDef:
    def __init__(self, name, args, rtype):
        self.name = name
        self.args = args
        self.rtype = rtype


class CTypeDef:
    def __init__(self, name, fields):
        self.name = name
        self.fields = fields
        self.methods = []
        self.constructor = False


class CType(Structure):
    def __init__(self, *args, **kwargs):
        self.__cself__ = self.new(*args, **kwargs)

    def __del__(self):
        if len(self.destroy.__cmethod__.__args__) == 1:
            self.destroy()

    def __repr__(self):
        return f"<{self.name} CObject at {self.__cself__}>"


class CFunction:
    def __init__(self, cfuncdef, cfunc):
        self.__name__ = cfuncdef.name
        self.__args__ = cfuncdef.args
        self.__cfunc__ = cfunc
        if cfuncdef.rtype is not None:
            self.__cfunc__.restype = cfuncdef.rtype

        self._posonly = all(_ is None for _ in self.__args__)

    def check_args(self, args, kwargs):
        if self._posonly and kwargs:
            raise ValueError(f"{self} takes only positional arguments")

        nargs = len(args) + len(kwargs)
        if nargs != len(self.__args__):
            raise TypeError(
                f"{self} takes exactly {len(self.__args__)} arguments ({nargs} given)"
            )

    def __call__(self, *args, **kwargs):
        self.check_args(args, kwargs)
        return self.__cfunc__(*args, **kwargs)

    def __repr__(self):
        return f"<CFunction '{self.__name__}'>"


class CMethod(CFunction):
    def __init__(self, cfuncdef, cfunc, ctype):
        super().__init__(cfuncdef, cfunc)
        self.__ctype__ = ctype

    def __get__(self, obj, objtype=None):
        def _(*args, **kwargs):
            cargs = [obj.__cself__, *args]
            self.check_args(cargs, kwargs)

            return self.__cfunc__(*cargs, **kwargs)

        _.__cmethod__ = self

        return _

    def __repr__(self):
        return f"<CMethod '{self.__name__}' of CType '{self.__ctype__.__name__}'>"


class CStaticMethod(CFunction):
    def __init__(self, cfuncdef, cfunc, ctype):
        super().__init__(cfuncdef, cfunc)
        self.__ctype__ = ctype

    def __repr__(self):
        return f"<CStaticMethod '{self.__name__}' of CType '{self.__ctype__.__name__}'>"


class CMetaType(type(Structure)):
    def __new__(cls, cmodule, ctypedef, _=None):
        ctype = super().__new__(
            cls,
            ctypedef.name,
            (CType,),
            {"__cmodule__": cmodule},
        )

        constructor = getattr(cmodule.__binary__, f"{ctypedef.name[:-2]}_new")
        ctype.new = CStaticMethod(ctypedef.constructor, constructor, ctype)

        for method_def in ctypedef.methods:
            method_name = method_def.name
            method = getattr(cmodule.__binary__, f"{ctypedef.name[:-2]}__{method_name}")
            setattr(ctype, method_name, CMethod(method_def, method, ctype))

        ctype.__cname__ = ctypedef.name

        return ctype


class DeclCollector(c_ast.NodeVisitor):
    def __init__(self):
        self.types = {}
        self.functions = []

    def _get_type(self, node):
        print(node)
        return self.types[" ".join(node.type.type.names)]

    def visit_Typedef(self, node):
        if isinstance(node.type.type, c_ast.Struct) and node.type.declname.endswith(
            "_t"
        ):
            struct = node.type.type
            self.types[node.type.declname[:-2]] = CTypeDef(
                node.type.declname,
                [decl.name for decl in struct.decls],
            )

    def visit_Decl(self, node):
        if "extern" in node.storage:
            return

        if isinstance(node.type, c_ast.FuncDecl):
            func_name = node.name
            ret_type = node.type.type
            rtype = None
            if isinstance(ret_type, c_ast.PtrDecl):
                if "".join(ret_type.type.type.names) == "char":
                    rtype = c_char_p
            args = (
                [_.name if hasattr(_, "name") else None for _ in node.type.args.params]
                if node.type.args is not None
                else []
            )
            if func_name.endswith("_new"):
                self.types[f"{func_name[:-4]}"].constructor = CFunctionDef(
                    "new", args, rtype
                )
            elif "__" in func_name:
                type_name, _, method_name = func_name.partition("__")
                if not type_name:
                    return
                self.types[type_name].methods.append(
                    CFunctionDef(method_name, args, rtype)
                )
            else:
                self.functions.append(CFunctionDef(func_name, args, rtype))

    def collect(self, decl):
        parser = c_parser.CParser()
        try:
            ast = parser.parse(decl, filename="<preprocessed>")
        except ParseError as e:
            lines = decl.splitlines()
            line, col = (
                int(_) - 1 for _ in e.args[0].partition(" ")[0].split(":")[1:3]
            )
            for i in range(max(0, line - 4), min(line + 5, len(lines))):
                if i != line:
                    print(f"{i+1:5d}  {lines[i]}")
                else:
                    print(f"{i+1:5d}  \033[33;1m{lines[line]}\033[0m")
                    print(" " * (col + 5) + "\033[31;1m<<^\033[0m")
            raise

        self.visit(ast)
        return {
            k: v
            for k, v in self.types.items()
            if isinstance(v, CTypeDef) and v.constructor
        }


class CModule(ModuleType):
    def __init__(self, source):
        super().__init__(source.name, f"Generated from {source.with_suffix('.c')}")
        self.__binary__ = CDLL(source.with_suffix(".so"))

        collector = DeclCollector()

        for name, ctypedef in collector.collect(
            preprocess(source.with_suffix(".h"))
        ).items():
            parts = name.split("_")
            py_name = "".join((_.capitalize() for _ in parts))
            setattr(self, py_name, CMetaType(self, ctypedef, None))

        for cfuncdef in collector.functions:
            name = cfuncdef.name
            try:
                cfunc = CFunction(cfuncdef, getattr(self.__binary__, name))
                setattr(self, name, cfunc)
            except AttributeError:
                # Not part of the binary
                pass

    @classmethod
    def compile(cls, source, cflags=[], ldadd=[]):
        compile(source.with_suffix(".c"), cflags, ldadd)
        return cls(source)

So let's use this new technology to make our C unit tests more Pythonesque. This is what our new test_cache.py can look like

import pytest
from tests.cunit import C
from tests.cunit.cache import Queue, QueueItem

NULL = 0


def test_queue_item():
    value = C.malloc(16)
    queue_item = QueueItem(value, 42)
    assert queue_item.__cself__

    queue_item.destroy(C.free)


@pytest.mark.parametrize("qsize", [0, 10, 100, 1000])
def test_queue(qsize):
    q = Queue(qsize, C.free)

    assert q.is_empty()
    assert qsize == 0 or not q.is_full()

    assert q.dequeue() is NULL

    values = [C.malloc(16) for _ in range(qsize)]
    assert all(values)

    for k, v in enumerate(values):
        assert q.enqueue(v, k)

    assert qsize == 0 or not q.is_empty()
    assert q.is_full()
    assert q.enqueue(42, 42) is NULL

    assert values == [q.dequeue() for _ in range(qsize)]

We can practically treat our C binary object as if it were an actual Python module and import the Queue and QueueItem types from it. We can then use them as if they were actual Python classes and call methods on them. How cool is that? 😄

Coverage, please!¶

What about test coverage? If you've used pytest before, you are probably familiar with the pytest-cov plugin, which is a handy way of collecting and reporting test coverage. GCC supports the -fprofile-arcs and -ftest-coverage to emit coverage data. So if we also wanted to get test coverage data while running C unit tests with pytest, all we would have to do is add these flags to the CFLAGS list in the cache.py module. The gcovr tool is actually inspired by the Python counterpart coverage.py and can be used to generate some nice reports. In particular, it could be used to generate Cobertura XML reports that could be uploaded to services like codecov.io.

My first close encounter with the Linux kernel source

2022-01-15T14:39:00+00:00

Background¶

As you probably know if you've been on my blog before, I develop and maintain Austin, a frame stack sampler for CPython which, in essence, is the main material for building a zero-instrumentation, minimal impact sampling profiler for Python. Recently I worked on a variant, austinp, which samples not only the Python stacks, but also native ones (optionally including Linux kernel stacks as well). The reason why this is just a variant and not part of Austin itself is because it violates two of its fundamental principles: dependencies and impact. The austinp variant relies on libunwind to perform native stack unwinding, which means that a. we have a dependency on a third-party library and b. since libunwind relies on ptrace, austinp no longer has the guarantee of minimal impact.

With the rapid and highly active development of eBPF in the Linux kernel, ptrace-based stack unwinding might as well be considered a thing of the past. One can exploit the BPF support for perf_event to implement a simple native profiler in just a few lines of C code with the support of libbpf. But for Austin to one day have a BPF variant, native stacks alone are not enough. We would need a way to unwind the Python frame stack from a BPF program, and at the moment of writing this is not quite yet possible. Why? Because the BPF part of the Linux kernel lacks support for accessing the VM space of a remote process. Indeed, the process_vm_readv system call is the core functionality that makes Austin possible on Linux, and unfortunately, as I discovered, there is no counterpart exposed to BPF at the time of writing.

How do I know there is no such functionality in BPF-land yet? Back in September 2021 I sent an email to the BPF mailing list, asking if it was possible to do what process_vm_readv does inside a BPF program. The answer I received from the maintainer was negative, but it turned out I asked the question at the right time, because the feature that was required to make this possible, i.e. sleepable BPF programs, had just been introduced in the kernel. Unfortunately, I coulnd't pick up the invite from the maintainers to prepare a patch back then. I still had not experience with the Linux kernel, nor the time to fully commit to the task, so I had to postpone this. The chance to actually look into this came with the January R&D week. This post is a recollection of the events of that week.

Getting things started¶

So January came and along with it the R&D Week. I decided to use this time to finally have a play with the Linux kernel, something that I wanted to do in a long time. Now I had the chance to do that for fun, but also for profit as I actually had a pretty concrete use case. The first step was of course to get the development environment up. Since I didn't want to risk bricking my Linux partition, I decided the best approach was to spin up a virtual machine.

I had a Ubuntu VM lying around from previous experiments with BPF and I thought of using that. Soon I discovered that, whilst many tutorials use or recommend a Ubuntu system for Linux kernel development, things are much smoother on Debian. So my personal recommendation for anyone starting on the Linux kernel for the first time, like me, is to use Debian.

But let's go in order. The first thing we want to do is get our hands on the latest version of the Linux kernel source code. So install git along with these other dependencies

sudo apt-get install vim libncurses5-dev gcc make git libssl-dev bison flex libelf-dev bc dwarves

and clone the repository with

mkdir kernel
cd kernel
git clone -b staging-testing git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git
cd staging

These steps are adapted from this KernelNewbies page, which is the guide that I have used to get started.

The Linux kernel is highly (like, very highly) configurable, depending your system and your needs, but of course the source code comes with none. The next step then is to make one. The guide recommends starting from the current system's configuration as a base, which we can get with

cp /boot/config-`uname -r`* .config

If the VM is running a not-so-recent build of the Linux kernel, it's perhaps worth running make olddefconfig to get any new configuration options in. And to speed the compilation process up a bit, we can use the current module configuration with make localmodconfig to get rid of some of the defaults that are not applicable to the current system.

This is where things get a bit annoying if you are using Ubuntu instead of Debian. It turns out there are some options that involve security certificates that are best set to the empty string to allow the build steps to regenerate whatever is needed. So if you are using Ubuntu to experiment on the Linux kernel, you might want to set CONFIG_SYSTEM_TRUSTED_KEYS, CONFIG_MODULE_SIG_KEY and CONFIG_SYSTEM_REVOCATION_KEYS to "".

Before moving on, it might be a good idea to change the value of CONFIG_LOCALVERSION to something else other than the empty string, in case we end up overwriting the current Linux kernel. For my experiments I have used CONFIG_LOCALVERSION=p403n1x87. This will show up as a prefix in the kernel image name in the bootloader menu

To make sure that everything is in order, and that you get GRUB menu entries similar to the ones shown above we can try to compile the sources with

make -j2 > /dev/null && echo "OK"

Feel free to replace the 2 with any other number that might be appropriate to your system. This represents the number of parallel compilation processes that are spawned, so it correlates with the number of cores that are available to your VM. The /dev/null is there to suppress any output that goes to stdout, which is not very instructive. This leaves us with more useful warnings and errors that are spit out to stderr. If the compilation succeeeded we would then see OK at the end of the output. If this is the case, we can proceed with the installation with

sudo make modules_install install

To boot into the newly compiled Linux kernel, simply reboot the VM with

sudo reboot

We're now all set to start hacking the Linux kernel 🎉.

Adding new bells and whistles¶

I spent the Monday of R&D Week getting my environment up and running and having a first look at the structure of my local copy of the Linux kernel source repository. The next day I started studying the parts that were relevant to my goal, that is find out where the process_vm_readv system call is defined and how it works internally. The Linux kernel source comes with ctags support for easy navigation. However, if you do not want to install additional tools you can make use of the Elixir project to find where symbols are defined and referenced.

The process_vm_readv system call is defined in mm/process_vm_access.c. Studying the content of this source carefully, we discover that memory is read page-by-page using copy_page_to_iter. The system call is also generic in the sense that it performs multiple operations, according to the number of iovec elements that are passed to it. For my use case, I'm seeking to implement a function with the signature

ssize_t process_vm_read(pid_t pid, void * dst, ssize_t size, void * user_ptr);

so that a possible implementation might have looked something like

ssize_t process_vm_read(pid_t pid, void * dst, ssize_t size, void * user_ptr) {
  struct iovec lvec, rvec;

  if (unlikely(size == 0))
    return 0;

  lvec = (struct iovec) {
    .iov_base = dst,
    .iov_len = size,
  }
  rvec = (struct iovec) {
    .iov_base = user_ptr,
    .iov_len = size,
  }

  return process_vm_rw(pid, &lvec, 1, &rvec, 1, 0, 0);
}

This is indeed what I have started experimenting with. I added this function to mm/process_vm_access.c and exposed it to BPF with the helper

BPF_CALL_4(bpf_copy_from_user_remote, void *, dst, u32, size,
    const void __user *, user_ptr, pid_t, pid)
{
  if (unlikely(size == 0))
    return 0;

  return process_vm_read(pid, dst, size, user_ptr);
}

const struct bpf_func_proto bpf_copy_from_user_remote_proto = {
  .func     = bpf_copy_from_user_remote,
  .gpl_only = false,
  .ret_type = RET_INTEGER,
  .arg1_type    = ARG_PTR_TO_UNINIT_MEM,
  .arg2_type    = ARG_CONST_SIZE_OR_ZERO,
  .arg3_type    = ARG_ANYTHING,
  .arg4_type    = ARG_ANYTHING,
};

To test this I wrote a simple C application that takes an integer from the command line, stores it safely somewhere in its VM space and is kind enough to tell us its PID and the VM address where we can find the "secret"

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    if (argc != 2)
    {
        fprintf(stderr, "usage: secret <SECRET>\n");
        return -1;
    }

    int secret = atoi(argv[1]);

    printf("I'm process %d and my secret is at %p\n", getpid(), &secret);
    printf("%d %p\n", getpid(), &secret);

    for (;;)
        sleep(1);

    return 0;
}

We can then write a simple BPF program that takes the PID and the address to pass them to the new helper in the attempt to retrieve the secret value from the remote target process

int secret;

...

SEC("fentry.s/__x64_sys_write")
int BPF_PROG(test_sys_write, int fd, const void *buf, size_t count)
{
    int pid = bpf_get_current_pid_tgid() >> 32;

    if (pid != my_pid)
        return 0;

    long bytes;

    bytes = bpf_copy_from_user_remote(&secret, sizeof(secret), user_ptr, target_pid);
    bpf_printk("bpf_copy_from_user_remote: copied %d bytes", bytes);

    return 0;
}

We are using fentry because it has a sleepable variant, fentry.s, and using sys_write as our target system call. Every time this system call is called our BPF program is executed. In this case we just run only if it was the associated user-space process that made the call.

Unfortunately, when checking the result of the bpf_printk call from /sys/kernel/debug/tracing/trace_pipe, I would always see -14 (-EFAULT) instead of the expected 4. This is an indication that we are passing the wrong memory addresses around. Further investigation led me to the conclusion that the problem was with how I was using process_vm_rw. Indeed, the manpage of the process_vm_readv system call clearly states that

These system calls transfer data between the address space of the calling process ("the local process") and the process identified by pid ("the remote process"). The data moves directly between the address spaces of the two processes, without passing through kernel space.

The problem here is that the destination is a variable with a kernel-space address, which would then fail the checks in import_iovec in the attempt to copy the struct iovec data over from user-space into kernel-space. Looking at how process_vm_rw is actually implemented, one can see that the local vectors are turned into a struct iov_iter, so I though that all that was needed was to create one from a struct kvec instead. My next attempt has then been

ssize_t process_vm_read(pid_t pid, void * dst, ssize_t size, void * user_ptr) {
  struct kvec lvec;
  struct iovec rvec;
  struct iov_iter;

  if (unlikely(size == 0))
    return 0;

  lvec = (struct kvec) {
    .iov_base = dst,
    .iov_len = size,
  };
  rvec = (struct iovec) {
    .iov_base = user_ptr,
    .iov_len = size,
  };

  iov_iter_kvec(&iter, READ, &lvec, 1, size);
  return process_vm_rw_core(pid, &iter, &rvec, 1, 0, 0);
}

Progress! The return value of the BPF helper is now 4 as expected, but for some reason the value read into int secret was always 0. 💥

While in the middle of my experiments, I sent a new message to the BPF mailing list on Wednesday, asking for feedback on my approach. I just wanted to see if, generally, I was on the right track and what I was doing made any sense. Clearly, my initial use of the details of process_vm_readv did not. On Thurday evening a get the reply of Kenny from Meta, who showed me their patch with a similar change. They used access_process_vm from mm/memory.c instead, with the added bonus of requring a reference to a process descriptor struct task_struct instead of a pid. This has the benefit, as Kenny explained, to remove the ambiguity around pid which comes from the use of namespaces. However, for my particular use case, I really need a pid (the namespace information might also be given at some point). Looking back at the notes that I have taken throughout the previous three days, I actually realised that I had come across access_process_vm while poking around to find what else was there. What threw me off was a misinterpretation of the comment that preceded its definition

/*
* Access another process' address space.
* Source/target buffer must be kernel space,
* Do not walk the page table directly, use get_user_pages
*/

Being a total Linux kernel newbie, I interpreted this as meaning that both target and source should have been kernel-space addresses, which of course was not my case. However, that comment refers to the buf argument, which makes a lot of sense when you think of it, for the kernel-to-kernel interpretation does not really fit with the name of the function. So I ditched my previous attempts inside mm/process_vm_access.c and refactored my new BPF helper into

BPF_CALL_4(bpf_copy_from_user_remote, void *, dst, u32, size,
    const void __user *, user_ptr, pid_t, pid)
{
  struct task_struct * task;

  if (unlikely(size == 0))
    return 0;

  task = find_get_task_by_vpid(pid);
  if (!task)
    return -ESRCH;

  return access_process_vm(task, (unsigned long) user_ptr, dst, size, 0);
}

and success! The simple BPF program that we saw earlier is now able to correctly read the secret from the target process! 🎉

For the full BPF program sources, head over to my bpf repository.

Use cases beyond Austin¶

Is this helper useful only for Austin? Not at all. Whilst there clearly is a use case for Austin in the way that I described at the beginning of this post, the new BPF helper opens up for many interesting observability applications. At the end of the day, what this does is to allow BPF programs to read the VM of any process on the system. Debugging of production systems should then be an idea popping in your mind right now, and indeed this is the use case of Kenny and the reason why they are proposing a similar BPF helper.

Think of a Python application, for example, running with CPython and native extensions. Getting any sort of meaningful observability into them is generally hard. This new helper might be able to, say, get the arguments of a (native) function, interpret them as Python object, and return a meaningful representation. The possibilities are virtually endless.

Increasing Austin accuracy with a dobule-heap trick

2021-12-17T13:35:00+00:00

It has been argued that, in order to collect reliable data with a sampling profiler that peeks at the private VM address space of a process, like Austin, it is necessary to pause the tracee. However, one of the fundamental principles that Austin is based on is: keep perturbations to a minimum. Therefore, pausing the tracee to sample its private VM is not an option on the table. Does this mean that Austin is bound to report unreliable data? My aim with this post it to convince you that halting the tracee is a sufficient condition, but not a necessary one. That is, we can still get pretty accurate results without pausing the tracee every time we want to sample it, provided we are willing to trade in some physical memory.

So how do we leverage some extra memory consumption for increased accuracy? To understand this we need to review how Austin works, which in turns forces us to review how Python works when it comes to the frame stack management. When Python evaluates some bytecode, it creates a frame object that references the code that is being executed and that holds some useful information. These frames are referenced by a linked list, which means that, in general, there are no guarantees as to where in the VM space the frame objects might get allocated. However, the coding of frame_alloc seems to suggest that entries in the list might get reused when the become free. So if we keep track of the addresses as we discover them we could in principle get all the frames with just a single system call, instead of making a call for each one of them. But in order to make a single call to the system call to read the private VM of the tracee we need an address range, which means we need a buffer large enough to receive the content. The VM address space is huge on a 64-bit architecture so we cannot really afford to dump an arbitrary range, and therefore we need a compromise: start getting frame objects and increase the range up to a given threshold. The hope here is that, with a bit of luck, we start tracking a range where frame objects are likely to be allocated.

In practical terms, the above solution might still not be good enough. Depending on the allocator that is being used, memory could be allocated anywhere in the massive VM address space. However, some patterns generally emerge: sometimes the address is close to the image in memory of the process, sometimes it is closer to the upper boundary of the allowed user-space VM range. Since the gap in between these two regions could be very big, we could double up the above idea and have two local buffers, one close to the process image in memory, the other closer to the upper boundary of the VM address space. This effectively doubles our chances of dumping frames with just two system calls. When we need to resolve a frame that is not within that range we bite the bullet and make another system call. But, if we are lucky, the overall rate of calls we make should drop considerably with the double-heap trick above, and result accuracy should benefit from it.

So let's see if our theoretical speculations work in reality. The latest release 3.2.0 of Austin comes with a new option, -h/--heap, which allows us to cap the total amount of double-heap that Austin can allocate to dump frame objects in one go. Let's try profiling

def sum_up_to(n):
    if n <= 1:
        return 1

    result = n + sum_up_to(n - 1)

    return result


for _ in range(1_000_000):
    N = 16
    assert sum_up_to(N) == (N * (N + 1)) >> 1

with -h 0, i.e. with no heap. This is the picture that we get if we run the above with the Austin VS Code extension:

Whilst it is true that most of the CPU time is spent in sum_up_to, we would have expected a call stack at most 16 frame high in this case. This flame graph is then misleading, although it still conveys the correct information that sum_up_to is the largest consumer of CPU time. Let's try again, but with the default heap size this time (which is 256 MB):

This time the recursive call stack have the expected height, and Austin is able to tell which samples didn't look quite right. But instead of throwing them away, they are displayed starting from a parent INVALID frame. Why do we want to do that? Two reasons: we still account for the total CPU time and we still have useful information. Even if invalid, those where the frames that got sampled, so some of them must have effectively been running when we sampled. Collecting that data is therefore useful for other aggregations, like a source code heat map, like the one produced by the Austin VS Code extension:

But before we get too hopeful about the merits of this solution, let's have a look at some other examples. The following flame graph comes from profiling my solution for Day 12 of the Advent of Code 2021 challenge with the default heap size:

There are clearly invalid frames (those in green that start directly from dfs, in the middle of the graph) that are not reported as such. For comparison, this is the resulting graph with no heap:

The conclusion that we draw from these experiments is that the double-heap trick implemented in Austin 3.2 increases the accuracy of the results, in the sense that Austin is better able to detect invalid samples, but there still is the chance of getting invalid samples as valid in the output. However, one thing to notice is that these are quite some artificial examples that involve code that runs pretty quickly. In many practical situations you would be using a profiler to detect unexpected slow paths, where Austin has good chances of producing fairly accurate results. This is the profiling data generated by this simple test script:

import threading


def keep_cpu_busy():
    a = []
    for i in range(2000000):
        a.append(i)


if __name__ == "__main__":
    threading.Thread(target=keep_cpu_busy).start()
    keep_cpu_busy()

with the heap:

and without:

i.e. they're almost identical.

The default value of 256 MB for the maximum combined heap size seems a reasonable compromise for getting even more accurate results, but on system with limited resources it is perhaps advisable to run Austin with a lower value, if not with the more drastic option --heap 0, which gives the pre-3.2 behaviour.

How I completed the Hacktoberfest 2021 challenge with a profiler

2021-12-16T15:18:00+00:00

Remember my post about how to bust performace issues? My claim there was that if you picked a project at random from e.g. GitHub, you'd find something that would catch your eye if you ran the code through a profiler. Iterating this process then seemed like a good strategy to generate PRs, which is what you need to do if you want to complete the Hacktoberfest challenge when that time of the year comes around.

But let's not get the wrong idea. You shouldn't walk away from here thinking that performance analysis is as trivial as turning the profiler on during test runs. What my previous post was trying to show is that, in many cases, code is not profiled and therefore it is easy to find some (rather) low-hanging fruits that can be fixed easily just as simply as looking at profiling data from the test suite. Once these are out of the way, that's when the performance analysis becomes a challenge itself, and some more serious and structured methodologies are required to make further progress.

So how did I actually use a profiler to complete the Hacktoberfest? I started by looking at all the Python projects with the hacktoberfest topic on GitHub and picked some that looked interesting to me. The profiler of choice was (surprise, surprise) Austin, since it requires no instrumentation and has practically no impact on the tracee, meaning that I could just sneak a austin in the command line used to start the tests to get the data that I needed.

As a concrete example, let's look at how I was able to detect and fix a performance regression in pyWhat. I forked the repository, made a local clone and looked at how the test suite is run. Peeking at the GitHub Actions I could see the test suite was triggered with nox

python -m nox

Inside the noxfile.py we can find the tests session, which is the one we are interested in

@nox.session
def tests(session: Session) -> None:
    """Run the test suite."""
    session.run("poetry", "install", "--no-dev", external=True)
    install_with_constraints(
        session,
        "pytest",
        "pytest-black",
        "pytest-cov",
        "pytest-isort",
        "pytest-flake8",
        "pytest-mypy",
        "types-requests",
        "types-orjson",
    )
    session.run("pytest", "-vv", "--cov=./", "--cov-report=xml")

So let's create a profile session where we run the test suite through Austin. All we have to do is add austin at the right place in the arguments to session.run, plus some additional options, e.g.:

@nox.session
def profile(session: Session) -> None:
    """Profile the test suite."""
    session.run("poetry", "install", "--no-dev", external=True)
    profile_file = os.environ.get("AUSTIN_FILE", "tests.austin")
    install_with_constraints(
        session,
        "pytest",
        "pytest-black",
        "pytest-cov",
        "pytest-isort",
        "pytest-flake8",
        "pytest-mypy",
        "types-requests",
        "types-orjson",
    )
    session.run("austin", "-so", profile_file, "-i", "1ms", "pytest")

Here I've actually removed options to pytest which I don't care about, like code coverage, as it's not what I want to profile this time. The -s option tells Austin to give us non-idle samples only, effectively giving us a profile of CPU time. I'm also allowing the Austin output file to be specified from the environment via the AUSTIN_FILE variable. This means that, if I want to profile the tests and save the results to tests.austin, all I have to do is invoke

pipx install nox  # if not installed already
AUSTIN_FILE=tests.austin nox -rs profile

Once this completes, the profiling data will be sitting in tests.austin, ready to be analysed. With VS Code open on my local copy of pyWhat, I've used the Austin VS Code extension to visualise the data in the form of a flame graph and, by poking around, this is what caught my eye

The suspect here is the chunky deepcopy frame stack which is quite noticeable. The question, of course, is whether the deepcopy is really needed. Clicking on the check frame takes us straight into the part of the code where the deepcopy is triggered. By inspecting the lines around I couldn't really see the need of making deepcopy of objects. So I turned that back (it was originally a shallow copy, that was later turned into a deep copy) into a shallow copy with this PR, ran the test and checked for the expected output. All was looking find. In fact, things now looked much, much better! Rerunning the profile session with the change produced the following picture:

The deepcopy stacks have disappeared and the check frame is overall much slimmer! And so, just like that, a performance regression has been found and fixed in just a few minutes :).

Spy on Python down to the Linux kernel level

2021-09-27T11:56:00+01:00

When I conceived the design of Austin for the first time, I've sworn to always adhere to two guiding principles:

no dependencies other than the standard C library (and whatever system calls the OS provides);
minimal impact on the tracee, even under high sampling frequency.

Let me elaborate on why I decided to stick to these two rules. The first one is more of a choice of simplicity. The power horse of Austin is the capability of reading the private memory of any process, be it a child process or not. Many platforms provide the API or system calls to do that, some with more security gotchas than others. Once Austin has access to that information, the rest is plain C code that makes sense of that data and provides a meaningful representation to the user by merely calling libc's fprintf on a loop.

The second guiding principle is what everybody desires from observability tools. We want to be able to extract as much information as possible from a running program, perturbing it as little as possible as to avoid skewed data. Austin can make this guarantee because reading VM memory does not require the tracee to be halted. Furthermore, the fact that Python has a GIL implies that a simple Python application will run on at most one physical core. To be more precise, a normal, pure-Python application would not spend more CPU time than wall-clock time. Therefore, on machines with multiple cores, even if Austin ends up acting like a busy loop at high sampling frequencies and hogging a physical core, there would still be plenty of other cores to run the Python application unperturbed and unaware that is being spied on. Even for multiprocess applications, the expected impact is minimal, for if you are running, say, a uWSGI server on a 64-core machine, you wouldn't lose much if Austin hogs one of them. Besides, you probably don't need to sample at very high frequences (like once every 50 microseconds), but you could be happy with, e.g. 1000 Hz, which is still pretty high, but would not cause Austin to require an entire core for itself.

When you put these two principles together you get a tool that compiles down to a single tiny binary and that has minimal impact on the tracee at runtime. The added bonus is that it doesn't even require any instrumentation! These are surely ideal features for an observability tool that make Austin very well suited for running in a production environment.

But Austin strengths are also its limitations unfortunately. What if our application has parts written as Python extensions, e.g. native C/C++ extensions, Cython, Rust, or even assembly? By reading a process private VM, Austin can only reconstruct the pure-Python call stacks. To unwind the native call stacks, Austin would need to use some heavier machinery. Forget about using a third-party library for doing that, which would violate the first principle, the more serious issue here is that there are currently no ways of avoiding the use of system calls like ptrace(2) from user-space. This would be a serious violation of the second principle. Why? Because stack unwinding using ptrace requires threads to be halted, thus causing a non-negligible impact on the tracee. Besides, stack unwinding is not exactly straight-forward on every platform to implement.

The compromise is austinp, a variant of Austin that can do native stack unwinding, just on Linux, using libunwind and ptrace. This tool is to be used when you really need to have observability into native call stacks, as the use of ptrace implies that the tracee will be impacted to some extent. This is why, be default, austinp samples at a much lower rate. This doesn't mean that you cannot use this tool in a production environment, but that you should be aware of the potential penalties that come with it. Many observability tools from the past relied on ptrace or similar to achieve their goal, and austinp is just a (relatively) new entry into that list. More modern solutions rely on technologies like eBPF to provide efficient observability into the Linux kernel, as well as into user-space.

Speaking of the Linux kernel, eBPF is not the only way to retrieve kernel stacks. In the future we might have a variant of Austin that relies on eBPF for some heavy lifting, but for now austinp leverages the information exposed by procfs to push stack unwinding down to the Linux kernel level. The austinp variant has the same CLI of Austin, but with the extra option -k, which can be used to sample kernel stacks alongside native ones. I am still to find a valid use-case for wanting to obtain kernel observability from a Python program, but I think this could be an interesting way to see how the interpreter interacts with the kernel; and perhaps someone might find ways of inspecting the Linux kernel performance by coding a simple Python script rather than a more verbose C equivalent.

You can find some examples of austinp in action on my Twitter account. This, for example, is what you'd get for a simple scikit-learn classification model, when you open the collected samples via the Austin VS Code extension:

The latest development builds of @AustinSampler, including the austinp variant for native stack sampling on Linux are now available from @github releases https://t.co/nBfzm3mDng. pic.twitter.com/IjVfAm1hRk
— Gabriele Tornetta 🇪🇺 🇮🇹 🇬🇧 (@p403n1x87) September 8, 2021

If you want to give austinp a try you can follow the instructions on the README for compiling from sources, or download the pre-built binary from the the Development build. In the future, austinp will be available from ordinary releases too!

How to bust Python performance issues

2021-07-02T17:43:00+01:00

In my experience as a software engineer, I think it's still way to common to see production-ready code being shipped without having been profiled at least once. With the current computing power and the ever increasing number of available cores per machine, it feels like a lot of preference is generally given to readable and maintainable code at the cost of those extra microseconds. Whilst this might make sense for extremely complex code-bases in low-level languages, this is perhaps more of an issue with technologies like Python, where in general you can still make some substantial optimisations while still retaining readability and maintainability.

To further prove to myself that profiling is still an important step in the development process that gets overlooked, I did the following experiment. I grabbed a Python project at random, the first one that popped up on my GitHub feed, looked at its test suite and profiled the test runs. The day I did this, Rich was sitting at the top of my GitHub feed, so what follows is a trace-back of the steps that led me to contribute this performance PR to the project. Besides Python, the other tools that I have used are Austin 3 and VS Code with the Austin extension installed.

So first of all, let's make sure that our test environment is fully set up. If you want to follow along, make sure that you have Austin and VS Code installed. I was using Windows the day I made this experiment, so I had Austin installed with choco, and the VS Code extension installed from the Visual Studio Marketplace. Let's get our hands on some code now by cloning Rich and checking out that commit that was master for me at that time. Open up a terminal and type

git clone https://github.com/willmcgugan/rich.git
git checkout ce4f18c

The project uses poetry so running the test suite is as easy as invoking

poetry install
poetry run python -m pytest

Once we are sure that all the tests pass we are ready to start getting some profiling data to see what's actually running. Version 3 of Austin comes with a reworked sleepless mode that can be used to get an estimate of CPU time instead of wall time. One big advantage of using a tool like Austin is that we do not have to make any changes to the code in order to get profiling data out of it. Besides, Austin runs out-of-process, which means that it won't have any effects on the code. Getting profiling data is as easy as invoking Austin just before the test run

poetry run austin -so profile_master.austin python -m pytest -vv tests

WARNING Here we can let Austin sample the whole pytest run because we have checked beforehand that it only takes a few seconds to complete. DO NOT try the same exact thing with long-running test suites or you would end up with a massive sample file that would be hard to process. In such cases you can either select a few tests, or run Austin with the -x,--exposure option to limit sampling to just a few seconds, and adjust the sampling interval with the -i\--interval option as best suited.

The -s option turns the sleepless mode on, which gives us only the on-CPU samples, whereas the -o option specifies the output file. Once the test run terminates, our profiling data will be in profile_master.austin, ready to be analysed with the Austin VS Code extension. We get the best experience if we start VS Code from within the project's root directory as this allows us to browse the code while we look at the flame graph. So fire up VS Code from the terminal with

code .

and activate the Austin extension by clicking on the FLAME GRAPH tab in the bottom panel. Sometimes you would have to right click inside the panel and click the menu entry to fully activate the extension.

At this point we are ready to load the profiling data that we have collected. Click on the OPEN button or press CTRL + SHIFT + A to bring up the open dialog and select profile_master.austin. The Austin VS Code extension will analyse all the collected sample and generate the flame graph.

The picture we get is certainly overwhelming at first, especially for those that are not familiar with how pytest works internally. This is perhaps a nice way to actually find out how pytest collects and run tests. By poking around we discover that some of the test runs are under the pytest_pyfunc_call frame. If you are struggling to find it, press F inside the flame graph to reveal the search input box, type pytest_pyfunc_call and hit ENTER. The frames that match the search string will be highlighted in purple. Let's scroll until we find the largest one. When we click on it, it will expand to the full width of the panel to give us a better idea of what lies underneath it, and the corresponding portion of the source code will also appear in VS Code!

We now have a better view of the tests that are being executed under this path. At this point we can start looking for the largest leaf frames and see if we can make any sense of them. When I first looked at this graph, one thing that quickly caught my eye was this particular stack.

Clicking on the test_log frame reveals the test code in VS Code. Surprisingly, the test has just a single call to Console.log, and the percent timing annotation generated by the Austin extension tell us that, of the whole on-CPU time for the test suite, about 4.9% is spent on that single call!

Looking back at the flame graph, we realise that all the time in the log frame is spent calling stack. Clicking on the log frame reveals the source code for the Console.log method and we can inspect how the information from the stack is used to generate the log entry. The line we are interested in is 1685, where we have

            caller = inspect.stack()[_stack_offset]

So inspect.stack() is called, which according to the flame graph does a lot of resolutions for each frame (see those calls to findsource, getmodule etc...), none of which seems to be used in Console.log, and besides we just pick one of the frames close to the top of the stack and chuck the rest away. That's pretty expensive for a simple log call. Since I had some familiarity with the inspect module with my work on Austin and other stuff, I knew there is (at least for CPython) the lower level method currentframe that would give you essential information about the currently executing frame. From there you can navigate down the stack and stop at the frame of interest. In this case we just need to take the parent frame of the current one, and we already find all the information needed by the Console.log method. I made the changes as part of the already mentioned PR #1253, so if you check that code out and re-run the tests with

poetry run austin -so profile_pr.austin python -m pytest -vv tests

and open the new profiling data in profile_pr.austin you will see that the test case test_log has pretty much disappeared as it basically takes almost zero CPU time now.

Instead we see test_log_caller_frame_info, which is the test case for the compatibility utility for those Python implementations that do not implement currentframe. But with CPython, calling Console.log is now inexpensive compared to the original implementation.

See how easy it's been to find a performance issue. With the right tool we didn't have to add any instrumentation to the code, especially in one we probably had no familiarity with. In many cases you only understand your code truly if you see it in action. So no more excuses for not profiling your code before you ship it! ;P

The Austin TUI Way to Resourceful Text-based User Interfaces

2020-10-26T10:08:00+00:00

Introduction¶

Due to the increasing popularity of the sample tools that were included with earlier versions of Austin, I have decided to move them into their own dedicated repositories. The TUI, for example, can now be found at Austin TUI.

Being, as I said, sample tools, the original coding wasn't very pleasant as the main focus was on how to embed Austin into your application, rather than the application itself. So the first step was to come up with a good design so that the code would be tidy and easy to maintain. Austin TUI has also been my very first attempt at a serious TUI application. The standard approach in Python is with the curses module, but one thing that you learn quite quickly is that such a low-level API tends to make for untidy-looking code pretty easily if you're not careful. There are some way out of this, in the form of higher level frameworks which offer you many cool features and abstractions like widgets. None of them, as nice as they are, were to my taste though.

My previous experiences with UI have almost always revolved around the notion of resource files. That is, the various bits of the user interface, like the main window, configuration and about dialogs, were all described by some DSL living in the project folder as resource files. You can take Glade as an example, which is also the tool that inspired the solution that I ended up developing and adopting for Austin TUI. With Glade, a GTK UI is described by an XML document, which is then parsed at runtime to produce the actual UI. What I wanted for Austin TUI was something similar so that I could de-clutter the Python code from all the UI definition logic, and focus only on the other aspects of a UI project, like event handling. I also wanted something that played nicely with the MVC design pattern and, as we shall see briefly, the main concept behind Austin TUI is indeed that of a view.

The View Object¶

The central object of the Austin TUI is the View object. This is responsible for refreshing the interface as well as exposing the event handlers for events like key presses. The view itself is not the UI though, but it contains a reference to it via the root_widget. The novelty in Austin TUI is that we want to partially build a view, or at least the UI part, using a resource file.

So take a look at the following minimalist example.

minimal-view.xml

<?xml version="1.0" encoding="UTF-8" ?>

<aui:MinimalView xmlns:aui="http://austin.p403n1x87.com/ui" name="minimal_view">
  <aui:Window name="main">
    <aui:Label name="label"
      text="Hello World"
      align="center" />
  </aui:Window>

  <!-- Signal mappings -->

  <aui:signal key="q" handler="on_quit" />

</aui:MinimalView>

The above XML document describes a view with two main components: a Window element, which provides the root node for the UI, and a signal to bind the method on_quit to the key q. The UI itself contains a single label that will display the text Hello World. When we run an application that uses this UI, we expect to see the text Hello World centred on the screen and we also expect to exit as soon as we press Q.

So what do we need to make the above UI work? With the framework included in Austin TUI, all that we need to do is declare a subclass of View with the same name as the root node of the XML document, that is MinimalView in this case, and then build the actual view object using the ViewBuilder class, like so

import asyncio

from austin_tui.view import View, ViewBuilder


class MinimalView(View):
    def on_quit(self, data=None):
        raise KeyboardInterrupt("quit signal")


def main():
    with open("minimal-view.xml") as view_stream:
        view = ViewBuilder.from_stream(view_stream)
        view.open()

    try:
        asyncio.get_event_loop().run_forever()
    except KeyboardInterrupt:
        view.close()
        print("Bye!")


if __name__ == "__main__":
    main()

This is all the Python code required to build a minimalist TUI that displays a label on the screen and that quits whenever the user presses Q. Things we notice from this example are

we are using asyncio to handle user events on the UI; this means that we can schedule our own asynchronous task without making the UI unresponsive;
we use the static method from_stream of the ViewBuilder class to build a UI from file; if the file resided inside a Python module we could have used the from_resource static method instead for convenience;
we call the open method on the view object to display the UI;
we call the close method on the view object to close the UI and restore the terminal to its original status.

In this particular example, we make the on_quit event handler simulate a keyboard interrupt and we handle KeyboardInterrupt to quit nicely.

The Widgets¶

Austin TUI uses the widget abstraction too. Elements like Window and Label that we have seen above are all exposed by the Austin TUI library via the austin_tui.widgets.catalog sub-module. A window is a simple logical container that can hold a single child, spanning the full content of the window. In the example above, the only child of the window is the Label widget identified by the name label. If you want to add multiple children to the window, you would want to include an intermediate Box container widget which acts as an HTML5 flex container. Let's see how we can build a simple UI for a minimalist top utility.

mini-top.xml

<?xml version="1.0" encoding="UTF-8" ?>

<aui:MiniTop xmlns:aui="http://austin.p403n1x87.com/ui" name="tui">
  <aui:Window name="main">
    <aui:Box name="main_box" flow="v">
      <aui:Box name="overview_box" flow="h">
        <aui:Label name="nprocs_label"
          text="No. of procs."
          width="16" />

        <aui:Label name="nprocs"
          align="center"
          width="8"
          bold="true" />

        <aui:Label name="cpu_label"
          text="Total %CPU"
          width="16" />

        <aui:Label name="cpu"
          align="right"
          width="6"
          bold="true" />
      </aui:Box>

      <aui:Box name="table_header" flow="h">
        <aui:Label name="own"
          text="PID"
          align="right"
          width="8"
          bold="true"
          reverse="true" />
        <aui:Label name="proc_cpu"
          text="%CPU"
          align="center"
          width="10"
          bold="true"
          reverse="true" />
        <aui:Label name="cmdline"
          text="COMMAND LINE"
          bold="true"
          reverse="true" />
      </aui:Box>

      <aui:ScrollView name="proc_view">
        <aui:Table name="table" columns="3" />
      </aui:ScrollView>

      <aui:Label name="footer"
        text="Press Q to exit."
        align="center"
        bold="true" />
    </aui:Box>
  </aui:Window>

  <!-- Signal mappings -->

  <aui:signal key="q"         handler="on_quit"   />
  <aui:signal key="KEY_UP"    handler="on_up"     />
  <aui:signal key="KEY_DOWN"  handler="on_down"   />
  <aui:signal key="KEY_PPAGE" handler="on_pgup"   />
  <aui:signal key="KEY_NPAGE" handler="on_pgdown" />

  <!-- Palette -->

  <aui:palette>
    <aui:color name="pid"  fg="3"   />
    <aui:color name="opt"  fg="4"   />
    <aui:color name="cmd"  fg="10"  />
    <aui:color name="args" fg="246" />
  </aui:palette>

</aui:MiniTop>

This is a slightly bigger XML document where we have a few more nested widgets, as well as a new feature: the palette element. Let's start by looking at the UI part. We see that the child of the Window element is now a Box with vertical flow. This means that all the children that we add to this box will span the whole width and pile up vertically. We use this element to divide the window into three parts: the top one will hold some summary stats; the middle will hold the process table; the bottom part is just a label telling us how to quit the application.

We see that the top part is just another Box, this time with horizontal flow. Inside we have four labels, two of which have fixed content and act as actual labels, describing the values that we will update. In this case we will keep track of the number of active process and the total CPU load.

The middle section of the UI is a ScrollView, which allows us create widgets within it that overflow the actual terminal size. This is an abstraction that makes for easy scrolling of overflowing content. In this example, inside the ScrollView we have a Table object with three columns; these will be the process ID, the CPU load for the process and its command line.

To make the UI appealing to the eye, we shall make use of colours, and this is where the new palette element in the XML document steps in. This is used to give a name to curses colour pairs. In this particular example we are changing the foreground colour only, but in principle we could change the background as well by setting the bg attribute. We'll see with the Python code below how to easily reference the colours in the palette.

Before moving on though, I appreciate that there isn't much of an official documentation of the UI framework used by the Austin TUI project, and this post is a way to make up for that as much as possible for now. Looking at all the sample XML above you might be wondering where all those attributes come from. For example, when we look at a Label element, we see

<aui:Label name="label"
  text="Hello World"
  align="center" />

The attributes name, text and align are precisely the arguments of the __init__ method of the Label class. Hence, if you want to find out which attributes are available for a certain widget you will have to find it in the widget collection and look at its constructor.

Every widget has at least the name attribute. Every other widget requires its own set of attributes.

Widgets are discovered by the view builder in a dynamic way, which means that you could sub-class Widget and make your own widgets. If you do so and want to reference your custom widget in the XML document, all you have to do is use the class name as element name. For example, if you have something like

from austin_tui.widgets import Widget


class MyWidget(Widget):
    def __init__(self, name, some_attribute):
        super().__init__(name)
        ...

then in the XML document you would have something like

<aui:MyWidget name="mywidget_instance" some-attribute="42" >
  <!-- any potential children here -->
</aui:MyWidget>

Updating the UI¶

For the model part of the MVC pattern we don't have much to say here as that will depend upon your application. In this post I will show you how to make a minimalist top application, so we can have a look what the model code could look like in this case a move over to the more interesting bit, which is the C in MVC.

import psutil


data = sorted(
    [
        (
            p.info["pid"],
            p.info["cpu_percent"],
            p.info["cmdline"],
        )
        for p in psutil.process_iter(["pid", "cpu_percent", "cmdline"])
    ],
    key=lambda x: x[1],
    reverse=True,
)

If you are familiar with the psutil module, you will see that we are iterating over all active processes to extract some information from them. In this case we are interested in the PIDs, the CPU usage and the command line. These three values will be used to fill in the three columns of the Table widgets that we introduced in mini-top.xml. As we will see, there is some more code that we could put into the model part of our design, but for simplicity we will embed that into the controller. The code below shows you how to update the UI every 2 seconds with fresh system data.

import asyncio

import psutil
from austin_tui.view import View, ViewBuilder


def format_cmdline(cmdline):
    if not cmdline:
        return ""
    cmd, *args = cmdline
    args = " ".join(
        [arg if arg.startswith("-") else f"<b><opt>{arg}</opt></b>" for arg in args]
    )
    return f"<cmd>{cmd}</cmd> <args>{args}</args>"


class MiniTop(View):
    def on_quit(self, data=None):
        raise KeyboardInterrupt("quit signal")

    def on_pgdown(self, data=None):
        self.proc_view.scroll_down(self.table.height - 1)
        self.proc_view.refresh()
        return False

    def on_pgup(self, data=None):
        self.proc_view.scroll_up(self.table.height - 1)
        self.proc_view.refresh()
        return False

    def on_up(self, data=None):
        self.proc_view.scroll_up()
        self.proc_view.refresh()
        return False

    def on_down(self, data=None):
        self.proc_view.scroll_down()
        self.proc_view.refresh()
        return False

    async def update(self):
        while True:
            data = sorted(
                [
                    (
                        p.info["pid"],
                        p.info["cpu_percent"],
                        format_cmdline(p.info["cmdline"]),
                    )
                    for p in psutil.process_iter(["pid", "cpu_percent", "cmdline"])
                ],
                key=lambda x: x[1],
                reverse=True,
            )
            self.table.set_data(
                [
                    (
                        self.markup(f"<pid>{pid:8d}</pid>"),
                        self.markup(f"<b>{cpu:^10.2f}</b>"),
                        self.markup(format_cmdline(cmdline)),
                    )
                    for pid, cpu, cmdline in data
                ]
            )

            self.nprocs.set_text(str(len(data)))
            self.cpu.set_text(f"{psutil.cpu_percent():3.2f}")

            self.table.draw()
            self.root_widget.refresh()

            await asyncio.sleep(2)


def main():
    with open("mini-top.austinui") as austinui:
        view = ViewBuilder.from_stream(austinui)
        view.open()

    loop = asyncio.get_event_loop()

    try:
        loop.create_task(view.update())
        loop.run_forever()
    except KeyboardInterrupt:
        view.close()
        print("Bye!")


if __name__ == "__main__":
    main()

There are quite a few new things that we need to explan here. First of all we see that we implement MiniTop as a sub-class of View and we declare all the required event handlers. We now have an update asynchronous method which we use to update the UI every 2 seconds. For this to work, we need to create a task using this method before starting the asyncio event loop (line 83). On line 43 we have the model logic that we just saw above and the interesting part starts after that. On line 55 we set the collected tabular data to the table widget. You would have guessed at this point that the way to reference a widget declared by the XML document is via attribute access on the view object. In this case, the Table element has the name attribute set to table. Therefore we can reference it inside the MiniTop instance via self, i.e. self.table. On lines 66-67 we do a similar thing, i.e. we update the value of the nprocs and cpu labels with the number of process and the total CPU usage respectively. At this point, no update is displayed on screen and this is for efficiency reasons. Once we have modified all the widgets that needed to be updated we can force a redraw by calling the draw method. This merely updates some buffers in memory, so in order to flush the changes to screen we have to make a call to the refresh method on a window-like object. In this case the simplest thing is to just refresh the whole root widget (line 70).

You might have also noticed that most of the event handlers are explicitly returning False. That's because we are manually forcing a refresh of the ScrollView widget (lines 23, 28, 33, 38) and therefore we return False to avoid propagating the refresh request further up the widget hierarchy for performance. Here we only needed refreshing the ScrollView; we could have omitted the manual refresh and returned True with the same result, but the window that would get refreshed would be the root one.

Spicing Things up with Colours¶

The last thing we need to have a look at are those mysterious calls to the markup method and the format_cmdline helper function. This is we we find references to the palette element in the XML document. Let's have a closer look at line 58 for example.

self.markup(f"<pid>{pid:8d}</pid>")

The markup method of a View object is a convenience method for creating strings with multiple attributes, like foreground/background colour, boldface, reversed, using an XML-like markup syntax. In this particular case, we want to write the PID on screen using the color pair with name pid declared inside the palette element of the XML document. On line 59 we use the <b> tag to make the CPU usage bold. For the command line we do something more complex with the format_cmdline function. If the command line is non-empty, we use the cmd colour for the actual command, and the colour args for the rest of the arguments, with the exception of option values, which are highlighted with the colour opt and with boldface. This spares us from having to manually split a string into chunks with different formatting.

When you run the code above you should see something similar to the screenshot below appearing in your terminal.

To quit the application, simply press Q as suggested at the bottom.

You can find a working example of this minimalistic top utility on GitHub if you want to play around with it and familiarise a bit more with the Austin TUI way to resourceful text-based user interfaces.

An Overview of Monads in Haskell

2020-09-17T16:18:00+01:00

Introduction¶

If you are learning a functional programming language like Haskell, sooner or later you will find yourself dealing with the concept of monad. You probably knew already that there is a fair bit of category theory embedded in the language, and this comes as no surprise at all. Indeed, functional programming languages were developed with the goal of making lambda calculus practical and, as it turns out, type theory has many relations to category theory.

In this post we make our way to the concept of monads in Haskell starting from the notion of functor. The ultimate goal is to motivate the use of the term monad by showing that we indeed have a monadic structure, but also give a somewhat rigurous justification of the fact that "all told, monads are just monoids in the category of endofunctors".

Functors¶

A typical pattern of functional programming is that of applying a function to a list element-wise. There are many ways of doing this, but the most general one is that of using the map function

map :: (a -> b) -> [a] -> [b]

That is, map takes a function a -> b and a list of type [a] to produce a list of type [b] by simply applying the function to each entry of the first list. Two important properties of the function map follow from this definition. The first involves its interaction with the identity function id :: a -> a which simply evaluates to its sole argument. Indeed,

map id = id

in the sense that both sides represent the same function. Now, if we have two functions, f :: a -> b and g :: b -> c, it doesn't matter whether we first compute the composition g . f and apply it to map, or compose map g with map f. The net result is that we obtain a function that takes a list of type [a] that is sent to a list of type [c] whose elements are precisely given by the image of each element of the initial list through the composition g . f. We can express this fact with the identity

map (f . g) = map f . map g

Let's look back at what we have here. In more abstract terms, we can think of the operation of putting square brackets around a type as some sort of map on the set of Haskell types into itself, namely $[\ \cdot\ ] : a \mapsto [a]$, for any Haskell type $a$. Observe further that the type of the function map is indeed map :: (a -> b) -> ([a] -> [b]). That is to say that map sends a function a -> b to a function [a] -> [b], together with the two properties above.

We may be tempted, at this point, to introduce the Haskell category $\mathsf{Hask}$, whose objects are the Haskell types, and whose arrows are the Haskell functions, i.e. those types that can be put into the form a -> b for some Haskell types a and b. If we do so, we then recognise that the pair ([], map) defines a functor from $\mathsf{Hask}$ into itself. That's because [] maps objects to objects (lists) and map maps functions to functions (between lists). That these mappings are functorial follows immediately from the two properties that we have observed earlier.

The next abstraction step is in realising that there is nothing special in the ([], map) pair. For if all we want are endofunctors on $\mathsf{Hask}$, all we really need is a parametrised type T and a map fmap :: (a -> b) -> T a -> T b that plays nicely with T, in the sense that it satisfies the functoriality properties

fmap id = id

and

fmap (f . g) = fmap f . fmap g

for any functions f and g. We then see that the list example we have analyised above is just the special case where T a = [a] and fmap = map.

To get a feel of what functors mean in Haskell (or more generally in functional programming), we should regard the parametrised type T as a sort of container type. A list, of course, is an example of a container, as it contains multiple instances of a certan type in an ordered fashon. But T can also be, e.g., a rooted tree with values of type a on each of its nodes. So, generally, T is some sort of container structure that accomodates for multiple values of the parameter type a.

Notice now how the definition of fmap depends on the parametrised type T. This means that the first step in defining an endofunctor over $\mathsf{Hask}$ is to produce one such parametrised type T. But once we have one, can we find an fmap such that (T, fmap) is a functor? The answer to this question very much depends on the nature of T, but what we can be certain of is that once we have found one such fmap, then it is unique. Why? This is a consequence of the so-called parametricity result, which derives from parametric polymorphism. The function fmap :: (a -> b) -> T a -> T b implies universal quantifiers for the types a and b. That is to say, for any types a and b, fmap sends a function a -> b to a function T a -> T b. The key observation is that, because of this parametric dependency on the types a and b, the function fmap cannot act in a way that depends on a particular choice of the types, but can depend at most on the container structure described by the parametrised type T.

In order to understand this concept, let's take a step back and consider again the special case of lists. Suppose that we have another candidate fmap' for an fmap other than map. Since the types are arbitrary, such new candidate can only interact with the list structure, i.e. apply the first argument, that is, the function f :: a -> b to each element of the list of type [a] to produce [b], and perhaps do a reshuffling. Now, if we apply this new candidate to the identity id :: a -> a, we get precisely the reshuffling bit s = fmap' id. On the other hand, fmap' must satisfy the functor properties, and in particular fmap' id = id, whence s = id. That is to say that fmap' cannot actually do any reshuffling, so it must coincide with map.

The general argument works in the same way, we only need to replace the list structure with a general one T and argue that by the functor properties fmap can only map a function to each element of type a inside T a while preserving the container structure described by T.

More elegantly, this result could have been obtained using the free theorem associated to the type (a -> b) -> (T a -> T b); see (Wadler, 1989) for the details.

Applicative Functors¶

We have seen that functors are a generalisation of the common pattern of mapping a function over the elements of a list. Can we generalise to mapping of functions with more than one argument? That is to say, given that (T, fmap) describes a functor, can we find, e.g., fmap2 :: (a -> b -> c) -> (T a -> T b -> T c), that satisfies some reasonable coherence properties? A first observation is that the type a -> b -> c is identical to a -> (b -> c), just by definition. If we apply fmap to any instances of this type we would get something of type T a -> T (b -> c), which isn't quite what we want. But, if we had a function <*> :: T (a -> b) -> (T a -> T b) we would get closer, but not quite there yet. Indeed, we could now define fmap2 like so

fmap2 :: (a -> b -> c) -> (T a -> T b -> T c)
fmap2 f x y = <*> (fmap f x) y

that is, we curry the result of fmap f with the first argument x and then apply <*> to the result to obtain a function that we can compute on y to get a result of type T c. Of course, we can now repeat these steps to define fmap3 :: (a -> b -> c -> d) -> (T a -> T b -> T c -> Td) by using fmap2 as a starting point, that is

fmap3 f x y z = <*> (fmap2 f x y) z

and carry on, to get a whole family of maps fmapn. Using infix notation for <*> and assuming associativity to the left, we could then write the general case as

fmapn :: (a1 -> ... -> an -> b) -> (T a1 -> ... -> T an -> b)
fmapn f x1 ... xn = fmap f x1 <*> ... <*> xn

This is now starting to look just like function application, except that we are not making that very explicit with the first argument x1 of type T a1. If we wanted to fix that, we would have to find, e.g., a function p such that fmap f x1 = (p f) <*> x. What's the type of p? We see that p :: a -> T a would work well here, so if we could find such a function we could push the base of the recursion back and define fmap itself in terms of p and <*>. Then we could easily generalise functors to functions of an arbitrary number of arguments by simply mapping it through p and applying arguments with <*>.

In Haskell, it is customary to call the function p with the name pure; any functor for which one can define both pure and <*> is then called an applicative functor. The importance of these special objects does not come solely from the fact that they generalise the notion of mapping to functions of more than one argument. Indeed, they represent an important step toward effectful programming inside a purely functional programming language. Perhaps we can appreciate this better with a concrete example based on the functor Maybe. Its values can be used to represent the success or failure of an operation. For exmaple, when an operation failed, we can simply return Nothing, else we return Just x, where x is the valid result of the operation. Haskell makes Maybe into an applicative functor by default by defining

pure = Just

Nothing <*> _ = Nothing
(Just f) <*> mx = fmap f mx

We see from the above definitions and the recursive nature of applicative functors that the propagation of the occurrence of an invalid value like Nothing during the computation is automatically propagated to the end result, with no need to put checks in place for each argument of the function f.

Before moving on, we should spend some more time looking back at the identity (pure f) <*> x = fmap f x and recalling that fmap must satisfy some coherence properties that make it part of the description of a functor. As we have said earlier, we expect pure and <*> to satisfy some coherence properties themselves if they are to work as they are supposed to, and we see why that is. One of such properties comes for free, viz.

(pure id) <*> x = x

since fmap id x = x. The other properties that are required of pure and <*> are

pure (f x) = pure f <*> pure x
x <*> pure y = pure ($ y) <*> x
pure (.) <*> x <*> y <*> z = x <*> (y <*> z)

and all ensure that pure pretty much delivers what it promises. For example, the first of the above three properties is just a way of ensuring that pure embeds ordinary function application into the effectful programming realm in an unsurprising way.

Deeper ties with Category Theory are presented in details in the original paper (McBride & Paterson, 2008). There it is shown how one can give a symmetric definition of <*> and show that an applicative functor is just a lax monoidal functor.

Monads¶

Monads have been introduced to crystallise yet another common pattern in effectful programming that is not quite captured by both the Functor and the Applicative "patterns".

Consider the case of a program that consists of a series of steps that need to be executed in order, and such that the output of one is used as input for the next one. A classical example is a function f that requires the result of a division g x y = x / y. It is clear that if g receives a 0 as second argument during the execution of the program, we are in an exceptional situation that somehow we need to handle. But we can hardly do so if g is of type, say, g :: Int -> Int -> Int. Lacking support for catching and reacting to exceptions, we need a way to signal that something went wrong and propagate that to the end.

The first obvious thing to do is to rewrite g in such a way that its type is g :: Int -> Int -> Maybe Int, so that g _ 0 = Nothing and g x y = Just (x / y). Then, for every function f that receives the output of g as input, we would have to go through the tedious process of checking whether the value it received is valid or not. If those functions have been coded already, then we have a problem that is not much fun to solve.

How about we check the arguments we are passing to a function that is coming from another function, detecting any failures to propagate, else continue with the normal execution? With this approach, all we need is a binding function >>= :: Maybe a -> (a -> Maybe b) -> Maybe b that checks the first argument, and only apply it to the second argument if it makes sense to do so, propagating any "bad" value otherwise. It is clear, just by looking at the involved types, that applicative functors are of no much use in this case, since a function like g is not pure, given that it can produce a side effect like Nothing. Hence, what we have here is a different pattern: the monad pattern.

The notion of monad comes from category theory and arises from adjunction. But before we can make contact with the Haskell notion of monads that was given above we need to replace the bind operator >>= :: M a -> (a -> M b) -> M b (here M is a parametrised type that we want to turn into a monad; hence M is a functor) with the function join :: M (Μ a) -> M a. Together with pure (or return we should say), it provides the structure that makes the endofunctor M a monad. In a nutshell, what we want to do is prove that (M, pure, join) has the monad structure. And we shall also see in what sense a monad can be defined as a "monoid in the category of endofunctors", as Mac Lane put it.

First things first, let's see how the join function relates to the bind operator >>=. You can convince yourself that the definition

join x = x >>= id

has the correct type. As to what this is good for from a practical point of view, recall the example of the function g :: Int -> Int -> Maybe Int that we saw before. We argued that we couldn't apply the applicative pattern because g is not a pure function and that the issue boiled down to g having the "wrong" type. For if we used fmap2 :: (a -> b -> c) -> (Maybe a -> Maybe b -> Maybe c), we would end up with fmap2 g :: Maybe Int -> Maybe Int -> Maybe (Maybe Int). But now, if we curry and apply join we can cure that double boxing Maybe (Maybe Int) to get back a Maybe Int instead.

Let's now look at the coherence properties that must be satisfied by join. The first one we note is its interaction with itself

join . (fmap join) = join . join

which is quite easily understood if we look at the case of lists. Here the join function does what we would expect, considering its name: it flattens the list of lists [[a]] into a single list [a] obtained by concatenation. The second coherence property that we look at involves the interaction between join and pure (or return), viz.

join . pure = join . (fmap pure) = id

which can be understood as saying that when we embed a value of type M a into the "pure" part of M (M a), then join "unboxes" the structure and gives us the initial value back.

Now, thanks to the two above properties, we can easily recognise a monad structure for the triplet (M, pure, join), with T the endofunctor and pure and join representing the required natural transformations.

In what sense can we regard a monad as a monoid? The general qualitative answer is that the map pure is akin to a unit and join is akin to the binary operation on a monoid. But if we look at the type of join, which is M (M a) -> M a, we don't see any sign of a Cartesian product. Instead, we have some sort of composition of functors, whence the second part of the qualitative answer, i.e. that the Cartesian product should be replaced with "composition". But what kind of composition? Let's clarify these points a bit.

Let'sabandon Haskell functors for now, since it is pretty clear that the last
"pattern" that we have described has all the rights to be called a monad, and
look at the matter from the perspective of category theory. Given functors $F, G: C \to D$, $J, K : D \to E$ and natural transformations $\alpha : F \Rightarrow G$ and $\beta : J \Rightarrow K$, we can construct a new natural transformation $\beta \circ \alpha$ between the functors $J \circ F$ and $K \circ G$ according to the following commutative diagram

$$\require{AMScd} \begin{CD} F(X) @>\alpha_X>> G(X)\ @VJVV @VJVV\ (J\circ F)(X) @>J(\alpha_X)>> (J\circ G)(X) @>\beta_{G(X)}>> (K\circ G)(X) \end{CD}$$

That is, we define the natural transformation $\beta\circ\alpha : J \circ F \Rightarrow K \circ G$ as having components

$$(\beta \circ \alpha)X = \beta \circ J(\alpha_X)$$

Using $\eta$ and $\mu$ as short-hand notation for the natural transformations pure and join, we see that the two coherence properties above can be stated in the form of commutative diagram

$$\require{AMScd} \begin{CD} M \circ M \circ M @>\iota_M\circ \mu>> M \circ M\ @V\mu \circ \iota_MVV @V\mu VV\ M \circ M @>\mu>> M \end{CD}$$

and

$$\require{AMScd} \begin{CD} M \circ I @> \iota_M \circ \eta >> M \circ M @< \eta\circ\iota_M << I \circ M\ @| @V \mu VV @|\ M @= M @= M \end{CD}$$

where $\iota_M$ denotes of course the identity natural transformation. Note that, in terms of components, the two diagrams are equivalent to the relations

$$\forall X,\ \mu_X(M(\mu_X)) = \mu_X(\mu_{M(X)})$$

and

$$\forall X,\ \mu(M(\eta_X)) = \mu(\eta_{M(X)}),$$

which we can easily translate back into the original properties for the Haskell pure and join.

The above diagrams should now make more precise the meaning of Mac Lane's sentence

All told, a monad in $X$ is just a monoid in the category of endofunctors of $X$, with product $\times$ replaced by composition of endofunctors and unit set by the identity endofunctor.

Indeed we see that $\mu$ and $\eta$ have diagrams similar to the analogous concepts in monoids, with the difference that functor composition $\circ$ is now everywhere we would expect a Cartesian product $\times$.

Deterministic and Statistical Python Profiling

2019-05-05T17:35:00+01:00

Brief Introduction to Profiling¶

Let's start with a quick introduction to what profiling is. Profiling is a run-time program analysis technique. Generally, a certain level of instrumentation is required to retrieve some kind of tracing information while the program is running. This is usually in the form of tracing instructions interleaved with the line of your source code, like debug statements, for instance, usually enriched with timestamp information or other relevant details, like memory usage, etc... .

One normally distinguishes between two main categories of profilers:

event-based (or deterministic)
statistical (or sampling).

Profilers in the first category make use of hooks that allow registering event callbacks. At the lowest level, these hooks are provided directly by the operating system and allow you to trace events like function calls and returns. Virtual machines and interpreters, like JVM and CPython, provide software hooks instead, for generally the same events, but also for language-specific features, like class loading for instance. The reason why profilers in this category are called deterministic is that, by listening to the various events, you can get a deterministic view of what is happening inside your application.

In contrast, statistical profilers tend to provide approximate figures only, obtained by, e.g., sampling the call stack at regular interval of times. These samples can then be analysed statistically to provide meaningful metrics for the profiled target.

One might get the impression that deterministic profilers are a better choice than statistical profilers. However, both categories come with pros and cons. For example, statistical profilers usually require less instrumentation, if none at all, and introduce less overhead in the profiled target program. Therefore, if a statistical profiler can guarantee a certain accuracy on the metrics that can be derived from them, then it is usually a better choice over a more accurate deterministic profiler that can introduce higher overhead.

Python Profiling¶

There are quite a plethora of profiling tools available for Python, either deterministic or statistical. The official documentation describes the use of the Python profiling interface through two different implementations:

profile,
cProfile.

The former is a pure Python module and, as such, introduces more overhead than the latter, which is a C extension that implements the same interface as profile. They both fit into the category of deterministic profilers and make use of the Python C API PyEval_SetProfile to register event hooks.

Standard Python Profiling¶

Let's have a look at how to use cProfile, as this will be the standard choice for a deterministic profiler. Here is an example that will profile the call-stack of psutil.process_iter.

# File: process_iter.py

import cProfile
import psutil

cProfile.run(
  '[list(psutil.process_iter()) for i in range(1_000)]',
  'process_iter'
)

The above code runs psutil.process_iter for 1000 times through cProfile and sends the output to the process_iter file in the current working directory. A good reason to save the result to a file is that one can then use a tool like gprof2dot to provide a graphical representation of the collected data. This tool turns the output of cProfile into a dot graph which can then be visualised to make better sense of it. E.g., these are the commands required to collect the data and visualise it in the form of a DOT graph inside a PDF document:

python3 process_iter.py
gprof2dot -f pstats process_iter | dot -Tpdf -o process_iter.pdf

This is what the result will look like. The colours help us identify the branches of execution where most of the time is spent.

A Look Under the Bonnet¶

The output of a tool like gprof2dot can be quite intuitive to understand, especially if you have had some prior experience with profilers. However, in order to better appreciate what is still to come it is best if we have a quick look at some of the basics of the Python execution model.

Python is an interpreted language and the reference implementation of its interpreter is CPython. As the name suggests, it is written in C, and it offers a C API that can be used to write C extensions.

One of the fundamental objects of CPython is the interpreter itself, which has a data structure associated with it, namely PyInterpreterState. In principle, there can be many instances of PyInterpreterState within the same process, but for the sake of simplicity, we shall ignore this possibility here. One of the fields of this C data structure is tstate_head, which points to the first element of a doubly-linked list of instances of the PyThreadState structure. As you can imagine, this other data structure represents the state of a thread of execution associated with the referring interpreter instance. We can navigate this list by following the references of its field next (and navigate back with prev).

Each instance of PyThreadState points to the current execution frame, which is the object that bears the information about the execution of a code block via the field frame. This is described by the PyFrameObject structure, which is also a list. In fact, this is the stack that we are after. Each frame will have, in general, a parent frame that can be retrieved by means of the f_back pointer on the PyFrameObject structure. The picture produced by gprof2dot of the previous section is the graphical representation of this stack of frames. The information contained in the first row of each box comes from the PyCodeObject structure, which can be obtained from every instance of PyFrameObject via the f_code field. In particular, PyCodeObject allows you to retrieve the name of the file that contains the Python code being executed in that frame as well as its line number and the name of the context (e.g. the current function).

Sometimes the C API changes between releases, but the following image is a fairly stable representation of the relations between the above-mentioned structures that are common among many of the major CPython releases.

The loop around PyFrameObject, which represents its field f_back, creates the structure of a singly-linked list of frame objects. This is precisely the frame stack.

The Python profiling API can be demonstrated with some simple Python code. The following example declares a decorator, @profile, that can be used to extract the frame stack generated by the execution of a function. In this case, we define the factorial function

import sys


def profile(f):
    def profiler(frame, event, arg):
        if "c_" in event:
            return

        stack = []
        while frame:
            code = frame.f_code
            stack.append(f"{code.co_name}@{frame.f_lineno}")
            frame = frame.f_back
        print("{:12} {}".format(event.upper(), " -> ".join(stack[::-1])))

    def wrapper(*args, **kwargs):
        old_profiler = sys.getprofile()

        sys.setprofile(profiler)
        r = f(*args, **kwargs)
        sys.setprofile(old_profiler)

        return r

    return wrapper


@profile
def factorial(n):
    if n == 0:
        return 1

    return n * factorial(n-1)


if __name__ == "__main__":
    factorial(3)

Note that the coding of the profiler function can be simplified considerably by using the inspect module:

import inspect

...

  def profiler(frame, event, arg):
      if "c_" in event:
          return

      stack = [f"{f.function}@{f.lineno}" for f in inspect.stack()[1:]]
      print("{:8} {}".format(event.upper(), " -> ".join(stack[::-1])))

...

Statistical Profiling¶

For a profiler from the statistical category, we have to look for external tools. In this case, The "standard" approach is to make use of a system call like setitimer, which is used to register a signal handler that gets called at regular intervals of time. The general idea is to register a callback that gets a snapshot of the current frame stack when triggered. An example of a profiler that works like this is vmprof.

Some drawbacks of this approach are: 1. the signal handler runs in the same process as the Python interpreter, and generally the main thread; 2. signals can interrupt system calls, which can cause stalls in the running program.

There are other approaches that can be taken in order to implement a statistical profiler, though. An example is pyflame, which is more in the spirit of a debugging tool and uses ptrace-like system calls. The situation is a bit more involved here since the profiler is now an external process. The general idea is to use ptrace to pause the running Python program, read its virtual memory and reconstruct the frame stack from it. Here, the main challenges are 1. to find the location of the relevant CPython data structures in memory and 2. parse them to extract the frame stack information. The differences between Python 2 and Python 3 and the occasional changes of the CPython ABI within the same minor release add up to the complexity of the task.

Once all has been taken care of, though, a statistical profiler of this kind has the potential of lowering the overhead caused by source instrumentation even further so that the payoff is generally worth the effort.

Enter Austin¶

We just saw that with a tool like pyflame we can get away with no instrumentation. An objection that can be raised against it, though, is that it still halts the profiled program in order to read the interpreter state. System calls like ptrace were designed for debugging tools, for which it is desirable to stop the execution at some point, inspect memory, step over one instruction or a whole line of source code at a time etc.... Ideally, we would like our profiler to interfere as little as possible with the profiled program.

This is where a tool like Austin comes into play. Austin is, strictly speaking, not a full-fledged profiler on its own. In fact, Austin is merely a frame stack sampler for CPython. Concretely, this means that all Austin does is to sample the frame stack of a running Python program at (almost) regular intervals of time.

A similar approach is followed by py-spy, another Python profiler written in Rust and inspired by rbspy. However, Austin tends to provide higher performance in general for two main reasons. One is that it is written in pure C, with no external dependencies other than the standard C library. The other is that Austin is just a frame stack sampler. It focuses on dumping the relevant parts of the Python interpreter state as quickly as possible and delegates any data aggregations and analysis to external tools. In theory, Austin offers you higher sampling rates at virtually no cost at the expenses of the profiled process. This makes Austin the ideal choice for profiling production code at run-time, with not even a single line of instrumentation required!

So, how does Austin read the virtual memory of another process without halting it? Many platforms offer system calls to do just that. On Linux, for example, the system call is process_vm_readv. Once one has located the instance of PyInterpreterState, everything else follows automatically, as we saw with the discussion on some of the details of the CPython execution model.

On Your Marks¶

At this point you might have started, quite understandably, to be a bit concerned with concurrency issues. What can we actually make of a memory dump from a running process we have no control over? What guarantees do we have that the moment we decide to peek at the Python interpreter state, we find all the relevant data structures in a consistent state? The answer to this question lies in the difference in execution speed between C and Python code, the latter being, on average, order of 10 times faster than the former. So what we have here is a race between Austin (which is written in C) and the Python target. When Austin samples the Python interpreter memory, it does so quite quickly compared to the scale of execution of a Python code block. On the other hand, CPython is also written in C, can refresh its state pretty quickly too. As a cinematic analogy, think that we are trying to create an animation by taking snapshots of a moving subject in quick succession. If the motion we are trying to capture is not too abrupt (compared to the time it takes to take a snapshot, that is), then we won't spot any motion blur and our images will be perfectly clear. This video of the Cassini flyby over Jupiter, Europa and Io, for instance, been made from still images, visualises this idea clearly.

With Austin, each frame stack sample is the analogue of a snapshot and the moving subject is the Python code being executed by the interpreter. Of course, Austin could be unlucky and decide to sample precisely during the moment CPython is in the middle of updating the frame stack. However, based on our previous considerations, we can expect this to be a rather rare event. Sometimes a picture is worth a thousand words, so here is an idealistic "CPython vs Austin" execution timeline comparison.

Now one could argue that, in order to decrease the error rate, an approach similar to pyflame, where we halt the execution before taking a snapshot, would be a better solution. In fact, it makes practically no difference. Indeed it could happen that the profiler decides to call ptrace while CPython is in the middle of refreshing the frame stack. In this case, it doesn't really matter whether CPython has been halted or not, the frame stack will be in an inconsistent state anyway.

As a final wrap-up comment to this digression, statistical profilers for Python like Austin can produce reliable output, as the error rate tends to be very low. This is possible because Austin is written in pure C and therefore offers optimal sampling performance.

Flame Graphs with Austin¶

The simplest way to turn Austin into a basic profiler is to pipe it to a tool like Brendan Gregg's FlameGraph. For example, assuming that austin is in your PATH variable (e.g. because you have installed it from the Snap Store with sudo snap install austin --beta --classic) and that flamegraph.pl is installed in /opt/flamegraph, we can do

austin python3 process_iter.py | /opt/flamegraph/flamegraph.pl --countname=usec > process_iter.svg

We are using --countname=usec because Austin samples frame stacks in microseconds and this information will then be part of the output of the flame graph tool. The following image is the result that I have got from running the above command.

Austin is now included in the official Debian repositories. This means that you can install it with

apt install austin

on Linux distributions that are derived from Debian. On Windows, Austin can be installed from Chocolatey with the command

choco install austin --pre

Alternatively, you can just head to the release page on GitHub and download the appropriate binary release for your platform.

The TUI¶

The GitHub repository of Austin comes with a TUI application written in Python and based on curses. It provides an example of an application that uses the output from Austin to display live top-like profiling statistics of a running Python program.

If you want to try it, you can install it with

pip install git+https://github.com/P403n1x87/austin.git

and run it with

austin-tui python3 /path/to/process_iter.py

By default, the TUI shows only the current frame being executed in the selected thread. You can navigate through the different threads with ⇞ Page Up and ⇟ Page Down. You can also view all the collected samples with the Full Mode, which can be toggled with F. The currently executing frame will be highlighted and a tree representation of the current frame stack will be available on the right-hand side of the terminal.

If you are a statistician or a data scientist working with Python, you can use the TUI to peek at your model while it is training to see what is going on and to identify areas of your code that could potentially be optimised to run faster. For example, let's assume that you are training a model on Linux in a single process using the command

python3 my_model.py

You can attach the TUI to your model with the command (as superuser)

austin-tui -p `pgrep -f my_model.py | head -n 1` -i 10000

The pgrep part is there to select the PID of the Python process that is running your model, while -i 10000 sets the sampling interval to 10 ms.

Web Austin¶

Web Austin is another example of how to use Austin to build a profiling tool. In this case, we make use of the d3-flame-graph plugin for D3 to produce a live flame graph visualisation of the collected samples inside a web browser. This opens up to remote profiling, as the web application can be served on an arbitrary IPv4 address.

Like the TUI, Web Austin can be installed from GitHub with

pip install git+https://github.com/P403n1x87/austin.git

Assuming you are still interested to see what is happening inside your statistical model while it is training, you can use the command

austin-web -p `pgrep -f my-model.py | head -n 1` -i 10000

As for the TUI, the command line arguments are the same as Austin's. When Web Austin starts up, it creates a simple HTTP server that serves on localhost at an ephemeral port.

# austin-web -p `pgrep -f my-model.py | head -n 1` -i 10000
_____      ___       __    ______       _______              __________             _____
____/|_    __ |     / /_______  /_      ___    |___  __________  /___(_)______      ____/|_
_|    /    __ | /| / /_  _ \_  __ \     __  /| |  / / /_  ___/  __/_  /__  __ \     _|    /
/_ __|     __ |/ |/ / /  __/  /_/ /     _  ___ / /_/ /_(__  )/ /_ _  / _  / / /     /_ __|
 |/        ____/|__/  \___//_.___/      /_/  |_\__,_/ /____/ \__/ /_/  /_/ /_/       |/


* Sampling process with PID 3711 (python3 my_model.py)
* Web Austin is running on http://localhost:34257. Press Ctrl+C to stop.

If you then open http://localhost:34257 in your browser you will then see a web application that looks like the following

Note that an active internet connection is required for the application to work, as the d3-flame-graph plugin, as well as some fonts, are retrieved from remote sources.

If you want to change the host and the port of the HTTP server created by Web Austin you can set the environment variables WEBAUSTIN_HOST and WEBAUSTIN_PORT. If you want to run the Web Austin web application on, e.g., 0.0.0.0:8080, so that it can be accessed from everywhere, use the command

# WEBAUSTIN_HOST="0.0.0.0" WEBAUSTIN_PORT=8080 austin-web -p `pgrep -f my-model.py | head -n 1` -i 10000
_____      ___       __    ______       _______              __________             _____
____/|_    __ |     / /_______  /_      ___    |___  __________  /___(_)______      ____/|_
_|    /    __ | /| / /_  _ \_  __ \     __  /| |  / / /_  ___/  __/_  /__  __ \     _|    /
/_ __|     __ |/ |/ / /  __/  /_/ /     _  ___ / /_/ /_(__  )/ /_ _  / _  / / /     /_ __|
 |/        ____/|__/  \___//_.___/      /_/  |_\__,_/ /____/ \__/ /_/  /_/ /_/       |/


* Sampling process with PID 3711 (python3 my_model.py)
* Web Austin is running on http://0.0.0.0:8080. Press Ctrl+C to stop.

Write Your Own¶

Austin's Powers (!) reside in its very simplicity. The "hard" problem of sampling the Python frame stack has been solved for you so that you can focus on processing the samples to produce the required metrics.

If you decide to write a tool in Python, the Austin project on GitHub comes with a Python wrapper. Depending on your preferences, you can choose between a thread-based approach or an asyncio one. Just as an example, let's see how to use the AsyncAustin class to make a custom profiler based on the samples collected by Austin.

import sys

from austin import AsyncAustin
from austin.stats import parse_line


class MyAustin(AsyncAustin):

    # Subclass AsyncAustin and implement this callback. This will be called
    # every time Austin generates a sample. The convenience method parse_line
    # will parse the sample and produce the thread name, the stack of contexts
    # with the corresponding line numbers and the measured duration for the
    # sample.

    def on_sample_received(self, sample):
        print(parse_line(sample.encode()))


if __name__ == "__main__":
    my_austin = MyAustin()

    my_austin.start(sys.argv[1:])
    if not my_austin.wait():
       raise RuntimeError("Austin failed to start")
    try:
        print("MyAustin is starting...")
        my_austin.join()
        print("The profiled target has terminated.")
    except KeyboardInterrupt:
        print("MyAustin has been terminated from keyboard.")

As the example above shows, it is enough to inherit from AsyncAustin and define the on_sample_received callback. This will get called every time Austin produces a sample. You can then do whatever you like with it. Here we simply pass the sample, which is just a binary string in the format Thread [tid];[func] ([mod]);#[line no];[func] ...;L[line no] [usec] to the parse_line function, which conveniently split the string into its main components, i.e. the thread identifier, the stack of frames and the sample duration. We then print the resulting triple to screen.

The rest of the code is there to create an instance of this custom Austin application. We call wait to ensure that Austin has been started successfully. The optional argument is a timeout, which defaults to 1. If Austin is not started within 1 second, wait returns False. If we do not wish to do anything else with the event loop, we can then simply call the join methods which schedules the main read loop that calls the on_sample_received callback whenever a sample is read from Austin's stdout file descriptor.

Conclusions¶

In this post, we have seen a few profiling options for Python. We have argued that some statistical profilers, like Austin, can prove valuable tools. Whilst providing approximate figures, the accuracy is in general quite high and the error rate very low. Furthermore, no instrumentation is required and the overhead introduced is very minimal, all aspects that make a tool like Austin a perfect choice for many Python profiling needs.

A feature that distinguishes Austin from the rest is its extreme simplicity which implies great flexibility. By just sampling the frame stack of the Python interpreter, the user is left with the option of using the collected samples to derive the metrics that best suit the problem at hand.

Extending Python with Assembly

2018-03-24T00:32:00+01:00

Introduction¶

If you have landed on this page, you must have had one between two only possible reactions to the title of this post, either "Hmm, this sounds interesting" or "Just, why?". The straight answer is, well, "just, because". And perhaps a bit more articulated answer is: because the people in the first category probably enjoy this kind of things :).

Reactions aside, the subject of this post is the coding of an extension for Python written in pure Assembly for the Intel x86-64 architecture on a Linux-based operating system. If you are familiar with general assembly but have never coded for the architecture that we are targeting, it is perhaps worth reading through my previous post "Getting Started with x86-64 Assembly on Linux".

I will also assume that you are somewhat familiar with extending Python with C. If not, then it probably is a good idea to go through the official documentation before reading on, or some things might not make too much sense. The approach of this post is by example and builds on knowledge about C to transition to Assembly. My favourite assembler on Linux is NASM, since it supports the Intel syntax, the one that I am more comfortable with. Therefore the only dependencies for following along are the NASM assembler and the GNU linker ld. Optionally, we can make use of a Makefile to assemble and link our code, and perhaps docker to test it in a clean environment. You will find all the relevant files in the linked GitHub repository.

Now it's time to jump straight into the code.

The Code¶

There isn't much more to say before we can see the code really, so here it is. This is the content of my asm.asm source file.

DEFAULT                 rel

%include                "asm/python.inc"

GLOBAL                  PyInit_asm:function


;; ---------------------------------------------------------------------------
SECTION                 .rodata
;; ---------------------------------------------------------------------------

l_sayit_name            db "sayit", 0
l_sayit_doc             db "This method has something important to say.", 0
l_sayit_msg             db "Assembly is great fun! :)", 10
l_sayit_msg_len         equ $ - l_sayit_msg

l_module_name           db "asm", 0


;; ---------------------------------------------------------------------------
SECTION                 .data
;; ---------------------------------------------------------------------------

l_asm_methods:              ;; struct PyMethodDef[] *
ISTRUC PyMethodDef
  at PyMethodDef.ml_name    , dq l_sayit_name
  at PyMethodDef.ml_meth    , dq asm_sayit
  at PyMethodDef.ml_flags   , dq METH_NOARGS
  at PyMethodDef.ml_doc     , dq l_sayit_doc
IEND
NullMethodDef

l_asm_module:                ;; struct PyModuleDef *
ISTRUC PyModuleDef
  at PyModuleDef.m_base     , PyModuleDef_HEAD_INIT
  at PyModuleDef.m_name     , dq l_module_name
  at PyModuleDef.m_doc      , dq NULL
  at PyModuleDef.m_size     , dq -1
  at PyModuleDef.m_methods  , dq l_asm_methods
  at PyModuleDef.m_slots    , dq NULL
  at PyModuleDef.m_traverse , dq NULL
  at PyModuleDef.m_clear    , dq 0
  at PyModuleDef.m_free     , dq NULL
IEND


;; ---------------------------------------------------------------------------
SECTION                 .text
;; ---------------------------------------------------------------------------

asm_sayit: ;; ----------------------------------------------------------------
                        push  rbp
                        mov   rbp, rsp

                        mov   rax, 1                  ; SYS_WRITE
                        mov   rdi, 1                  ; STDOUT
                        mov   rsi, l_sayit_msg
                        mov   rdx, l_sayit_msg_len
                        syscall

                        mov   rax, Py_None
                        inc   QWORD [rax + PyObject.ob_refcnt]

                        pop   rbp
                        ret
;; end asm_sayit


PyInit_asm: ;; --------------------------------------------------------------
                        push  rbp
                        mov   rbp, rsp

                        mov   rsi, PYTHON_API_VERSION
                        mov   rdi, l_asm_module
                        call  PyModule_Create2 WRT ..plt

                        pop   rbp
                        ret
;; end PyInit_asm

If you have never written a C extension for Python before, this might look a bit mysterious to you, although the general structure, at least, should be quite clear after you've glimpsed through the official Python documentation on extending Python with C.

We shall now analyse every single part of the above code sample in details to see what each block of code does.

Shared Object¶

On the very first line of the source we see the line

DEFAULT                 rel

Our goal is to assemble and link our code into an ELF64 shared object file. Contrary to ordinary program code, shared object files are dynamically loaded into random memory addresses. It is therefore important that all our code is position-independent. One way of doing this is to make sure that any memory reference is not absolute, but relative to the value of the RIP register, which points to the current instruction being executed. This guarantees that, no matter where the shared object is loaded into memory, references to local variables are correct. In 64-bit mode, NASM defaults to absolute addresses, therefore the above line is necessary to switch to RIP-relative addresses.

The CPython Headers¶

On line 3 we include a file to our main Assembly source. Given the simplicity of this example, we could have included the content of the python.inc file within asm.asm itself. However, for larger projects it is perhaps good practice to separate declarations and actual code, like it is usually done in C, with .h and .c files. In fact, the python.inc file includes the equivalent of structures and macros as declared in the CPython header files. As far as I'm aware, there are no assembly-specific include files provided by the maintainers of CPython, so we have to go through the extra effort of typing them ourselves. We will get back to the content of this file later on.

Exporting Global Symbols¶

Line 5 is an important one. It exports the symbol PyInit_asm, of type function, and makes it available for external programs. This is the function that CPython calls the moment we load the asm module with import asm from the Python interpreter. If we do not export this symbol, then CPython won't be able to find the code necessary to initialise the module. In analogy with C, this is equivalent to declaring a non-static function.

Immutable Strings¶

Next we have the read-only data section

;; ---------------------------------------------------------------------------
SECTION                 .rodata
;; ---------------------------------------------------------------------------

l_sayit_name            db "sayit", 0
l_sayit_doc             db "This method has something important to say.", 0
l_sayit_msg             db "Assembly is great fun! :)", 10
l_sayit_msg_len         equ $ - l_sayit_msg

l_module_name           db "asm", 0

Here we initialise the strings that we will need later on. As they won't change during the course of the code execution, we put them in a read-only section of the shared object. The GNU C compiler does just the same thing with every literal string that you use in C code. You will notice references to their address in the following section, that of (read-write) initialised data.

CPython Data Structures¶

Next is the section of initialised data.

;; ---------------------------------------------------------------------------
SECTION                 .data
;; ---------------------------------------------------------------------------

l_asm_methods:              ;; struct PyMethodDef[] *
ISTRUC PyMethodDef
  at PyMethodDef.ml_name    , dq l_sayit_name
  at PyMethodDef.ml_meth    , dq asm_sayit
  at PyMethodDef.ml_flags   , dq METH_NOARGS
  at PyMethodDef.ml_doc     , dq l_sayit_doc
IEND
NullMethodDef

l_asm_module:               ;; struct PyModuleDef *
ISTRUC PyModuleDef
  at PyModuleDef.m_base     , PyModuleDef_HEAD_INIT
  at PyModuleDef.m_name     , dq l_module_name
  at PyModuleDef.m_doc      , dq NULL
  at PyModuleDef.m_size     , dq -1
  at PyModuleDef.m_methods  , dq l_asm_methods
  at PyModuleDef.m_slots    , dq NULL
  at PyModuleDef.m_traverse , dq NULL
  at PyModuleDef.m_clear    , dq 0
  at PyModuleDef.m_free     , dq NULL
IEND

Here is where things start to get interesting, and the content of the python.inc file comes into play. The first two labels point to the beginning of CPython-specific structures. The first is an array of PyMethodDef structures. As the name suggests, each instance of this structure is used to hold information about a method that should be made available to the Python interpreter from within our module. To find out in which header file it is defined, we can use the command

grep -nr /usr/include/python3.6 -e "struct PyMethodDef"

In my case, I get that the structure is defined in /usr/include/python3.6/methodobject.h, starting from line 54. Inside the python.inc we then have the equivalent structure declaration

STRUC PyMethodDef
  .ml_name              resq 1    ; const char *
  .ml_meth              resq 1    ; PyCFunction
  .ml_flags             resq 1    ; int
  .ml_doc               resq 1    ; const char *
ENDSTRUC

The NullMethodDef is a NASM macro that conveniently defines the sentinel PyMethodDef structure, which is used to mark the end of the PyMethodDef array pointed by l_asm_methods. Its definition is also in the python.inc file and, as you can see, simply initialises a new instance of the structure with all the fields set to NULL or 0, depending on their semantics, i.e. whether they are memory pointers or general integers.

%define NullMethodDef         dq NULL, NULL, 0, NULL

Note that NULL is not a native NASM value. To align the coding conventions with C, I have defined NULL as a constant in python.inc and assigned the value of 0 to it. The idea is that, like in C, it makes the intent of the code clearer, since any occurrence of NULL indicates a null pointer rather than just the literal value 0.

The next label, l_asm_module, points to an instance of the PyModuleDef structure, which is pretty much the core data structure of our Python module. It contains all the relevant metadata that is then passed to CPython for correct initialisation and use of the module. Its definition is in the moduleobject.h header file and, at first sight, looks a bit complicated, with some references to other structures and C macros.

typedef struct PyModuleDef{
  PyModuleDef_Base m_base;
  const char* m_name;
  const char* m_doc;
  Py_ssize_t m_size;
  PyMethodDef *m_methods;
  struct PyModuleDef_Slot* m_slots;
  traverseproc m_traverse;
  inquiry m_clear;
  freefunc m_free;
} PyModuleDef;

So lets take our time to figure out what its byte content looks like. The first field is an instance of the PyModuleDef_Base structure, which is defined in the same header file, just a few lines above. The non-trivial bit in this new structure is the first part, PyObject_HEAD, which looks like a C macro. As the name suggest, its definition is quite likely to be found in object.h. Indeed, there we find

#define PyObject_HEAD                   PyObject ob_base;

so our chase continues. The definition of the PyObject structure can be found a few lines below. Again, all the fields are quite simple, i.e. just integers value or memory pointers, except for the macro _PyObject_HEAD_EXTRA. We then have to jump back up a few lines, to find that this macro is conditionally defined as either nothing or 0, 0. By default, the macro Py_TRACE_REFS is not defined, so in our case _PyObject_HEAD_EXTRA evaluates to nothing. Backtracking from our macro chase in CPython headers, we see that we can define the following structures in python.inc

STRUC PyObject
  .ob_refcnt            resq 1    ; Py_ssize_t
  .ob_type              resq 1    ; struct _typeobject *
ENDSTRUC

STRUC PyModuleDef_Base
  .ob_base              resb PyObject_size
  .m_init               resq 1    ; PyObject *
  .m_index              resq 1    ; Py_ssize_t
  .m_copy               resq 1    ; PyObject *
ENDSTRUC

STRUC PyModuleDef
  .m_base               resb PyModuleDef_Base_size
  .m_name               resq 1    ; const char *
  .m_doc                resq 1    ; const char *
  .m_size               resq 1    ; Py_ssize_t
  .m_methods            resq 1    ; PyMethodDef *
  .m_slots              resq 1    ; struct PyModuleDef_Slot *
  .m_traverse           resq 1    ; traverseproc
  .m_clear              resq 1    ; inquiry
  .m_free               resq 1    ; freefunc
ENDSTRUC

As you can easily guess, NASM generates the constants PyObject_size etc... automatically so that they can be used to reserve enough memory to hold the entire structure in the definition of other structures. This makes nesting quite easy to implement in NASM.

Local and Global Functions¶

Finally we get to the actual code that will get executed by CPython when our module is loaded and initialised, and when the methods that it provides are called.

;; ---------------------------------------------------------------------------
SECTION                 .text
;; ---------------------------------------------------------------------------

asm_sayit: ;; ----------------------------------------------------------------
                        push  rbp
                        mov   rbp, rsp

                        mov   rax, 1                  ; SYS_WRITE
                        mov   rdi, 1                  ; STDOUT
                        mov   rsi, l_sayit_msg
                        mov   rdx, l_sayit_msg_len
                        syscall

                        mov   rax, Py_None
                        inc   QWORD [rax + PyObject.ob_refcnt]

                        pop   rbp
                        ret
;; end asm_sayit


PyInit_asm: ;; --------------------------------------------------------------
                        push  rbp
                        mov   rbp, rsp

                        mov   rsi, PYTHON_API_VERSION
                        mov   rdi, l_asm_module
                        call  PyModule_Create2 WRT ..plt

                        pop   rbp
                        ret
;; end PyInit_asm

In fact, we have a total of two functions, one local and one global. The first one, asm_sayit, is the only method contained in our module. All it does is to write a string, l_sayit_msg, to standard output by invoking the SYS_WRITE system call. Perhaps the most interesting bit of this function is the code on lines 61-62. This is the idiom for any function that wishes to return None in Python. Recall that, in Python, None is an object instantiated by CPython. As such, our shared library needs to import it as an external symbol. This is why you will find the macro PyNone defined as

%define Py_None               _Py_NoneStruct

together with the line

EXTERN    _Py_NoneStruct

in the python.inc file. This is equivalent to the two lines

PyAPI_DATA(PyObject) _Py_NoneStruct; /* Don't use this directly */
#define Py_None (&_Py_NoneStruct)

in the object.h header file, where the None object is defined. All of this explains line 61, but what about line 62? This has to do with Reference Counting. In a nutshell, every object created in Python comes with a counter that keeps track of all the references attached to it. When the counter gets down to 0, the object can be de-allocated from memory and resources freed for other objects to use. This is how Python, which heavily relies on malloc, can keep memory leaks at bait. It is therefore very important to properly maintain reference counts in Python extensions. As None is a Python object like any others, when we return a reference to it, we have to bump its reference count. In C, this is conveniently done with the Py_INCREF macro. Its definition is in the object.h and, as it is easy to guess, it just increases the ob_refcnt field of the PyObject structure. This is precisely what we do on line 62.

Stack Frames Matter! You might be wondering why we are taking care of creating a stack frame on function entry, and cleaning up after ourself on leave. The reason is a pretty obvious one: we don't know what code will call ours, so it is safe to make sure that stack alignment is preserved across calls by doing what every function is expected to do. When I was lying down the code for this post, I was getting a SIGSEGV exception, and the debugger revealed that the instruction movaps was trying to store the value of the xmm0 register on a memory location that was not a multiple of 16. The problem was solved by the extra 8 bytes from push rbp.

The second and last function is our global exported symbol PyInit_asm. It gets called by CPython as soon as we import the module with import asm. In this simple case, we don't have to do much here. In fact, all we have to do is call a standard CPython function and pass it the instance of PyModuleDef allocated at l_asm_module. As we have briefly seen, this contains all the information about our module, from the documentation to the list of methods.

Now, if you have read through the official documentation on how to extend Python with C, you might be wondering why we are calling PyModule_Create2 instead of PyModule_Create (is there a typo?), and why we are passing it two arguments instead of one. If you are starting to smell a C macro, then you are correct! Long story short, PyModule_Create is a macro defined as

#define PyModule_Create(module) PyModule_Create2(module, PYTHON_API_VERSION)

with PYTHON_API_VERSION defined as the literal 1013. So the actual function to call is indeed PyModule_Create2, and it takes two arguments.

Did you notice that weird WRT ..plt? Remember the discussion about ensuring position-independent code? Since we have no clue of where the PyModule_Create2 function resides in memory, we have to rely on some sort of indirection. This is provided by the so-called Procedure Linkage Table, or PLT for short, which is some code that is part of our shared library. When we call PyModule_Create2 WRT ..plt, we are jumping to the PLT section of our object file in memory, which contains the necessary code to make the actual jump to the function that we want to call.

Installation¶

Once our assembly code is ready, it needs to be assembled and linked into a shared object file. We will now see how to perform these steps, and how to test and install our Python extension.

Assembling and Linking¶

Once the code is ready, it needs to be assembled and linked into the final shared object file. The NASM assembler is invoked with minimal arguments as

nasm -f elf64 -o asm/asm.o asm/asm.asm

This creates an intermediate object file asm.o. To create the final shared object file, we use the GNU linker with the following arguments

ld -shared -o asm/asm.so asm/asm.o -I/lib64/ld-linux-x86-64.so.2

Note the use of the -shared switch, which instructs the linker to create a shared object file.

How to Test the Module¶

The first thing that you might want to do is to manually test that the shared object file works fine with Python. For CPython to be able to find the module, we need to ensure that its location is included in the search path. One way is to add it to the PYTHONPATH environment variable. For example, from within the project folder, we can launch Python with

PYTHONPATH=./asm python3

and from the interactive session we should be able to import the module with

>>> import asm
>>>

Alternatively, we can add the search path to sys.path with these few lines of Python code

>>> import sys
>>> sys.path.append("./asm")
>>> import asm
>>>

Once we have successfully imported our module in Python, we can test that its method sayit works as expected

>>> asm.sayit.__doc__
'This method has something important to say.'
>>> asm.sayit()
Assembly is great fun! :)
>>>

I hope that you would agree :).

Distributing the Module¶

The simplicity of our sample module wouldn't justify the use of setuptools for distribution. In this case, a simple, old-fashioned Makefile is the simplest solution to go for. Even for larger projects, you would probably still delegate the build job of your code to a Makefile anyway, which would then get called from your setup.py at some phase, perhaps during build. However, the recommended standard is that you build wheels instead of eggs, and the requirement is that you provide pre-built binaries with your package.

This being said, let's see how to distribute the module. As we have seen in the previous section, the shared object needs to resides in one of the Python search paths. The easiest way to find out what these paths are on the platform that you are targeting is to launch the Python interpreter and print the value of sys.path. On my platform, I get the following output

Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.6/dist-packages']
>>>

The Makefile could then contain the following line inside the install rule:

install: default
    cp asm/asm.so /usr/lib/python${PYTHON_TARGET}/asm.so

with the environment variable PYTHON_TARGET set to 3.6.

To automate the building and testing process of the module, we could use Docker to build an image out of the target platform and trigger a build, and perhaps execute some unit tests too. A simple Dockerfile that does the minimum work to build and test would look something like the following

FROM  ubuntu:latest
USER  root
ADD   . asm
RUN   apt-get update              &&\
      apt-get install -y            \
        nasm                        \
        python3-pytest              \
        build-essential
ENV   PYTHON_TARGET=3.5
RUN   cd asm                      &&\
      make                        &&\
      make install                &&\
      python3 -m pytest -s

As you can see, we are targeting the latest stable version of Ubuntu, which comes with python3.5. We make sure we install all the required dependencies, the assembler and the standard build tools, along with python3-pytest to perform unit testing once our module builds successfully.

The bare minimum that we can test is that the import of the module works fine and that we can call its method. So a possible test_asm.py test script would look like

import asm


def test_asm():
    asm.sayit()

Conclusions¶

Whilst I appreciate that the cases where you'd want to seriously consider extending Python with Assembly code are rare, it is undoubtedly the case that, if you enjoy experimenting with code, this could be a fun and instructing experience. In my case, this has forced me to go look into the CPython header files, which I probably wouldn't have if I were using C. I now know more about the internal workings of Python and a clearer idea of how CPython is structured.

As always, I hope you have enjoyed the read. Happy Assembly coding! :)

Android Development from the Command Line

2017-10-14T22:42:00+01:00

Introduction¶

Yes you guessed it right, I'm not a huge fan of IDEs. Don't get me wrong though, I am fully aware of how powerful modern IDEs are, and all the magic that they can do for you to assist you while you are coding your application. But this is also why I don't like them, especially when I'm picking up something new.

To the date of writing, I am not an experienced Android developer. When I started developing applications (there was a time when I believed that no matter how big a project could be, you could always find the time to code it in assembly language), a plain old text editor was my friend, together with some command line tools, like assemblers, linkers, compilers, debuggers etc.... With no support at all, you had to know exactly what you were doing, you had to know the syntax, the APIs and where they were located. You also needed a minimum knowledge of what a linker is for and when and why you need one.

Whenever I tried an IDE, e.g. Android Studio, I always felt like I didn't really need to know much about the frameworks I was using, as all the built-in tools would come to my rescue. As a consequence, I started feeling like I wasn't really mastering anything and put the project I was working on aside for future me to one day resume working on it. Rather than me using the IDE, it kind of was the other way around: the IDE was using me to magically generate code.

IDEs also tend to hide all the machinery involved in the build process from the developer as well. In most cases, everything goes well, but what would you do if you suddenly come across a problem and you have no clue at which stage of the build process it is happening?

Surely, if you work on a big company project, it would be crazy to renounce entirely to IDEs, as your life might be a bit harder in everyday maintenance of your code, but for smaller projects this argument is somewhat weak, and opting for a plain text editor might have its many advantages. For once, you are in total control of the code that is going into your final product. And then again, there is also the educational aspect, which can give you the right amount of experience to tackle unexpected issues that could pop up during any stage of the development life-cycle.

All this being said, in this post we shall see how to develop an Android application by only relying on a text editor of your choice and the standard CLI tool provided by the Android SDK. The focus is on the steps required to install the Android SDK command line tools and how to organise your source code, rather than on the details of the application itself.

As with a standard Android project created with Android Studio, we are going to rely on Gradle and the Android Gradle plugin for the build process. You may rightfully think that this somehow partly defeats the point of this post, but, hey, in the end Gradle is just a command line tool, and quite a standard way to build and deploy Java projects these days.

This post is targeted to Linux users, but there is a good chance that the steps that we will go through have an equivalent on other platforms, like Windows. I'm afraid this is something that you will have to find out on your own.

Code very similar to the one presented in this post can be found in the GitHub repository androtest.

Pre-requisites¶

Before embarking on an adventure, it is wise to check that we are taking all that we need along the way with us. As we are trying to keep things as simple as possible, we won't need much, but there are a few preliminary steps that we need to perform in order to set up our development environment.

The first few steps couldn't be simpler: pick your favourite text editor (my laptop can still handle an application like Atom) and terminal application, and we already have almost half of what we need! The rest of the tools is provided by the Java Development Kit, the Android SDK Tools and Gradle. More details in due time.

The JDK is usually available from your distro's repositories. On Ubuntu, it can be installed with, e.g.

sudo apt install openjdk-9-jdk

The Android SDK might not be available from the official repositories. In principle the could be installed along with Android Studio, but if you are not going to use Google's official IDE then it is a bit of waste of space. The cleaner alternative is to just download the Android SDK Tools from the Android Studio download page. For simplicity, I will split the installation process of the Android SDK into different steps.

Android SDK Tools¶

Download the zip archive and extract it somewhere, e.g. ~/.android/sdk, then update your .bashrc file to define the ANDROID_HOME environment variable and include the SDK tools binaries in the PATH variable by adding the following lines

# Android SDK Tools
export ANDROID_HOME=$HOME/.android/sdk
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/tools/bin

Note that most of the tools installed in $ANDROID_HOME/tools are deprecated and one should use the dedicated ones provided in the $ANDROID_HOME/tools/bin folder. These include the fundamental sdkmanager and the avdmanager tools for respectively creating and managing different SDK versions (and other packages too) and virtual devices (the emulators).

Android SDK Platform Tools¶

Throughout the Android development life-cycle, you are likely to need to interface with the Android platform for testing your progress. In concrete terms this means that you might want to compile your project as you develop it for testing on an actual Android device. In order to connect to the device and look at the log you will need the Android Debug Bridge, which is provided with the Android Platform Tools. To install them we can use the sdkmanager CLI tool to pull the latest released version with the following command

sdkmanager "platform-tools"

This will install the Platform Tools into the $ANDROID_HOME/platform-tools folder. We can then add it to the PATH variable for easy invocation by simply adding the following lines to .bashrc

# Android Platform Tools
export PATH=$PATH:$ANDROID_HOME/platform-tools

Android SDK Platforms¶

In order to compile a project we need a certain API revision to be installed. This provides all the functionalities that our application can use and are provided in the form of Java packages and classes and other useful components. For example, if our project targets Marshmallow, then we need to install the Android SDK Platform API Level 23. You can find out more at this page. To find out about all the packages available to download we can use the command

sdkmanager --list --verbose

At the moment we are interested in Android platforms, i.e. those packages that are prefixed with platforms;, so we can filter the output of the above command as follows

sdkmanager --list --verbose | grep -A 3 platforms\;

Among the matching results, you should see something similar to the following

platforms;android-23
    Description:        Android SDK Platform 23
    Version:            3

We can then proceed to installing the Android SDK Platform API Level 23 with the command

sdkmanager "platforms;android-23"

When you use the sdkmanager tool, you might see the following warning message

Warning: File /home/user/.android/repositories.cfg could not be loaded.

In order to get rid of it you can simply create this file with no content.

Android SDK Build Tools¶

Now that we have the API to compile against, we need the tools to actually be able to build a project: the build tools. They provide utilities like apksigner, Jack and Jill etc..., but for the moment we don't have to worry about the details of this package, as they will be invoked behind the scenes by Gradle.

The Android Studio User Guide recommends that you keep the build tools updated to the latest version. To find out all the versions available for download, run the following command

sdkmanager --list --verbose | grep -A 3 build-tools\;

and locate the latest version. At the moment of writing this is 26.0.1, so the command to use in this case is

sdkmanager "build-tools;26.0.1"

Android Emulator¶

The installation of the Android Emulator package is not mandatory, as you can decide to test your application on an actual Android device. However, there are many reasons why you might want to use an emulator: you probably don't own a huge variety of Android devices, differing not only in physical size, but also in the API version (Kit Kat, Lollipop, Marshmallow, just to name a few of the most recent code-names). The package can be installed through the sdkmanger with the command

sdkmanager "emulator"

Again, we can add the emulator folder to the PATH variable for easy access. However, the Android SDK Tools provides a deprecated set of tools, emulator and emulator-check, that would collide with the ones we have just installed. To solve this problem we can rename the deprecated executables with

chmod -x $ANDROID_HOME/tools/emulator
chmod -x $ANDROID_HOME/tools/emulator-check
mv $ANDROID_HOME/tools/emulator $ANDROID_HOME/tools/emulator.dep
mv $ANDROID_HOME/tools/emulator-check $ANDROID_HOME/tools/emulator-check.dep

and then add the others to the PATH variables by appending the following lines to the ~/.bashrc file

# Android Emulator
export PATH=$PATH:$ANDROID_HOME/emulator

Gradle¶

Let's now proceed to the installation of Gradle, a build automation system that is also the default in Android Studio. Google has developed a dedicated Android plugin to assist with the most common tasks. The ones that I personally tend to run more frequently are collected in the following table

Task	Description
`compileDebugJavaWithJavac`	Compiles the debug version of the java sources. Useful to check for syntax errors while coding
`installDebug`	Compiles and installs the debug version on all the devices discovered by the ADB.
`lint`	Runs a lint on the sources, producing a report in `build/reports`.

Gradle is quite popular so it should be available from your distro's official repositories. On Ubuntu 17.04 though, a quite old version of Gradle is available from them, so I would recommend that you add Cheng-Wei Chien's PPA to your software sources and install Gradle from there

sudo add-apt-repository ppa:cwchien/gradle
sudo apt update && sudo apt install gradle

Gradle is quite a powerful tool, but you might find that it has a rather steep learning curve to master all of its features, especially if you are not familiar with Groovy. In this post we shall only scratch the very surface and look at only the closures that we need for this project, as a discussion on Gradle surely deserves a dedicated post on its own.

Getting back on business, the installation of Gradle was the last step that we needed to perform in order to set up the development environment, and we can now move on to creating an Android project from scratch.

Creating the Gradle Project¶

The first thing to do is to create a Gradle project of Java type. This involves setting up a directory structure in the project's parent folder and creating the Gradle build script build.gradle. With Gradle installed, these steps can be automated with the init task

gradle init --type java-library

If you are contributing to a project that is the work of many hands, it will probably be the case that everybody is using the same build tools. As everybody can have a different version of Gradle on their local machine, the Gradle project recommends that one uses a wrapper to build a project, rather than invoke the local installation of Gradle directly. By sharing the wrapper along with your project, every other developer working on the same project will be able to use the same version of Gradle as everybody else, thus getting rid of problems caused by switching between different versions. Even though this is a sample project, we will nonetheless create and use a Gradle wrapper to build our project. The previous command should have created a gradlew shell script in the project's folder. If not, run the following command

gradle wrapper

You should also have a folder src/ containing all the sub-folders where Gradle expects the sources and the resources that make up your project. But, most importantly, you should also have the build.gradle and settings.gradle files containing some sample build settings. This structure is slightly different from the one generated by Android Studio, and documented on the developer portal, where you can notice a nested Gradle project, with the topmost one used to import the actual project as a module, and configure the global build settings. For the case at hand, we could to without this nested structure and only define a single build script, since our project is made up of only one module.

The following is the content of the build.gradle file.

buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.3.3'
    }
}

apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion '26.0.1'
}

You can find a detailed explanation of the meaning of each closure in the Configure Your Build page of the Android Developer portal. Briefly, the buildscript closure is used to configure Gradle itself so that it knows where to find the Android-specific Gradle tools that we want to use. We can then import the Android Gradle plugin and use the extensions to the DSL that it provides to configure the Android-specific build process. In this case we specify that the compilation SDK version that we want to use is Level 23 (Marshmallow), and that we want to use the version 26.0.1 of the build tools.

The gradle.properties file is used to configure the project-wide Gradle settings, such as the Gradle daemon's maximum heap size. In fact, this is all we will use it for in our case. Open it with your text editor and put the following content in it

org.gradle.jvmargs=-Xmx1536m

This is our Gradle project created and configured! All the magic of the building process is hidden from us by the Android Gradle plugin so that we don't have to worry about anything else and just focus on our application code.

Writing the Application¶

Two essential ingredients for an Android application are the Main Activity and the App Manifest. Let's start with the latter first.

The Application Manifest¶

The Android Application Manifest is an XML manifest file that is used to provide the Android system with essential information of an application, like the name of the Java package, the activities provided, the permissions required, the minimum SDK version supported etc... . Within a Gradle project, this file must be located in the src/main folder and named rigorously AndroidManifest.xml. In our case, this is what such manifest file would look like

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.p403n1x87.androtest">

  <uses-sdk
    android:minSdkVersion="21"
    android:targetSdkVersion="23" />

  <application
    android:label="AndroTest"
    android:theme="@android:style/Theme.Material">

    <activity android:name="MainActivity">
      <intent-filter>
        <action android:name="android.intent.action.MAIN" />
        <category android:name="android.intent.category.LAUNCHER" />
      </intent-filter>
    </activity>
  </application>
</manifest>

The structure of the Android Application Manifest file is described on the developer portal in the form of a skeleton, as it is not based on any XML schema like DTD or XSD.

The root element is manifest, which accepts the package attribute. Here we specify the name of the Java package of our application. We also specify our target API level to 23. The minimum level supported is set to 21 for reasons that relate to the output of the lint Android Gradle task that we will look at later on. It is recommended that you set this attribute to a reasonable value. For example, if you do not set it, Android will automatically add the legacy overflow three-dot button, even though there are no actions to show. Starting from level 11, Android does not add this button by default.

The manifest element must contain a unique application element that describes our application in may aspects, from the theme to use, to the activity content. In this case we set the application name to AndroTest and the theme to Material and we expose only one activity, i.e. the MainActivity, which is the entry point of our application, and the one that would be fired up when we launch it.

The Main Activity¶

This takes us now to the main activity, i.e. the Java class that contains the code to be executed when our application is launched. Within a Gradle project, the main Java code should reside in the src/main/java folder. Since the package name is com.p403n1x87.androtest, the MainActivity.java source file should be created within the src/main/java/com/p403n1x87/androtest folder. Here is the code

package com.p403n1x87.androtest;

import android.app.Activity;
import android.os.Bundle;

import android.content.Context;

import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorEventListener;
import android.hardware.SensorManager;

import android.view.ViewGroup.LayoutParams;

import android.widget.TextView;
import android.widget.LinearLayout;

import java.util.List;

import static java.lang.Math.sqrt;


public class MainActivity extends Activity
{
  private SensorManager mSensorManager;
  private Sensor        mSensor;

  private TextView      text       = null;
  private LinearLayout  layoutMain = null;

  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    // UI
    setContentView(R.layout.main_layout);
    if (text == null)       text       = (TextView)     findViewById(R.id.text);
    if (layoutMain == null) layoutMain = (LinearLayout) findViewById(R.id.layout_main);

    // Sensors
    mSensorManager = (SensorManager) getSystemService(Context.SENSOR_SERVICE);
    if (mSensorManager.getDefaultSensor(Sensor.TYPE_GRAVITY) != null){
      List<Sensor> gravSensors = mSensorManager.getSensorList(Sensor.TYPE_GRAVITY);
      int nSensors = gravSensors.size();
      text.setText("Detected gravity sensors: " + Integer.toString(nSensors));

      for (int i = 0; i < nSensors; i++) {
        final TextView tvSensor = new TextView(this);
        final int      j        = i + 1;

        tvSensor.setLayoutParams(new LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT));

        mSensor = gravSensors.get(i);
        mSensorManager.registerListener(new SensorEventListener() {
          @Override
          public void onSensorChanged(final SensorEvent event) {
                float x = event.values[0];
                float y = event.values[1];
                float z = event.values[2];
                float g = (float) sqrt(x*x + y*y + z*z);

            tvSensor.setText(String.format("Sensor %d: %f m/s^2", j, g));
          }

          @Override
          public void onAccuracyChanged(Sensor sensor, int a) {}
        }, mSensor, SensorManager.SENSOR_DELAY_NORMAL);

        layoutMain.addView(tvSensor);
      }
    }
    else {
      text.setText("We DO NOT have gravity! :(");
    }
  }

}

The code is fairly simple and quite self-explanatory. We override the onCreate method of the Activity class, from which MainActivity inherits, to set up the UI. We use a layout resource as a basis, which we then dynamically extend with extra TextView widgets to hold the value of each gravity sensor that gets discovered at run-time.

The Layout Resource¶

In this project we have a mixture of static layout resources and dynamic creation of TextView elements. This offers us the chance to see how to make resources available to the Java code, i.e. by placing them in the place where Gradle, and the Android Gradle plugin, would expect them. Files placed in the src/main/res folder are treated as resources and can be referenced in the way described at the Android Developer portal. In the previous code block, on line 36 we have

    setContentView(R.layout.main_layout);

The expression R.layout.main_layout refers to the resource main_layout.xml located in the layout sub-folder of src/main/res. The various grouping of resources is detailed at the page Providing Resources.

Later on, in dealing with the output of the lint task, we will have the chance to look at the strings.xml resources in the values sub-folder. For the time being, let's have a look at what the main_layout.xml looks like in this case:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
              android:id="@+id/layout_main"
              android:layout_width="match_parent"
              android:layout_height="match_parent"
              android:padding="32px"
              android:gravity="center"
              android:orientation="vertical" >

  <TextView android:id="@+id/text"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:paddingBottom="12px"
            android:text="Gravity sensors" />

  <!--
  A new TextView element will be added programmatically for each on-board
  gravity sensor detected.
  -->

</LinearLayout>

This is a rather simple layout. We create a vertical LinearLayout container to display a vertical stack of TextView elements. The first one, statically included in the XML resource file, will hold the counter of the discovered gravity sensors. More TextView elements are added at run-time, one for each of the sensors, to display their updated value.

Running the Application¶

Now that we have set up the Gradle project and coded our application, it is time to build it and install it on an Android device to run it. In this section we shall see how to invoke the Gradle build task to generate the APK (the Android Package Kit) package, and how to install it on our devices, being them either virtual or physical, in two different ways. Finally, we will give a final touch to our sample application by fixing a few of the issue reported by the lint task.

Building with Gradle¶

The process of building an Android application involves many steps and tools. If we rely on a Gradle project, as we have done in this case, and as it would be if you were using Android Studio, all of the details of this process are hidden to us. In a post where we are trying to make use of only simple tools, you may think that Gradle would defeat the point. Whilst I'd agree with you, I also believe that we will have to draw a line at some point, and Gradle feels like the right place. If, at any point, you feel the need to manually build your project, you can refer to these two references for more details:

This being said, let's see how to build our sample project with Gradle. We can list all the available tasks with

./gradlew tasks --all

Among all the tasks listed by the previous command, we should see the following ones

Build tasks
-----------
assemble - Assembles all variants of all applications and secondary packages.
assembleAndroidTest - Assembles all the Test applications.
assembleDebug - Assembles all Debug builds.
assembleRelease - Assembles all Release builds.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
clean - Deletes the build directory.
cleanBuildCache - Deletes the build cache directory.
compileDebugAndroidTestSources
compileDebugSources
compileDebugUnitTestSources
compileReleaseSources
compileReleaseUnitTestSources
mockableAndroidJar - Creates a version of android.jar that's suitable for unit tests.

We could decide to assemble our project, or build it in case we also wrote tests and want them built too. As it doesn't make much of a difference in our case, we can run the build task with

./gradlew build

The build task actually does a bit more than assemble and execute tests. It also runs the lint task. More on this later.

If the build task was successful, you should now have the androtest-debug.apk into the build/outputs/apk folder, ready to be deployed on an Android device.

Creating a Virtual Device¶

In case you want to install the application on a physical device, you can skip this section and go straight to the next one. Under some circumstances, you might want to test your application on diverse hardware settings, and the best way is probably to make use of a virtual device. If you have installed the emulator package as previously described, you can create a virtual machine with the following commands. But first we need to install a system image, for example the Intel x86 Atom_64 System Image. Since our application doesn't make use of the Google APIs we can opt for the default image:

sdkmanager "system-images;android-23;default;x86_64"

The download and installation might take some time, so just wait for sdkmanager to complete. You can then proceed to creating a virtual device with

avdmanager create avd -n test -k "system-images;android-23;default;x86_64" -d 8

This will create a virtual device named test with the system image that we have just downloaded. The -d switch specifies which device we want to emulate. In this case the value 8 represents a "Nexus 5" device. A complete list of the devices that can be emulated may be found with

avdmanager list device

Finally, to run the newly created virtual device we use the emulator tool, just like this:

emulator @test -skin 768x1280

where @test specified the name of the virtual device we want to run, and the -skin switch defines the resolution we want to run the emulator at. Note that a bug in the emulator tools prevents you from running this command from any working directory. Instead, you need to navigate to $ANDROID_HOME/emulator in order to start it without errors.

For more details on how to manage virtual devices, have a look at the avdmanager documentation page.

Installing the APK¶

There are at least to ways of installing the APK to an Android device. One is via a Gradle task, and the other is more manual and involves the Android Debug Bridge tool adb. Let's have a look at both.

From the previous run of the ./gradlew tasks --all, you should have seen the following tasks

Install tasks
-------------
installDebug - Installs the Debug build.
installDebugAndroidTest - Installs the android (on device) tests for the Debug build.
uninstallAll - Uninstall all applications.
uninstallDebug - Uninstalls the Debug build.
uninstallDebugAndroidTest - Uninstalls the android (on device) tests for the Debug build.
uninstallRelease - Uninstalls the Release build.

The installDebug task will look for all the Android devices in debugging mode connected to your machine and attempt to install the APK on all of them. For instance, if you have a physical device in debug mode connected to your PC/laptop while also running a virtual device, this Gradle task will install the APK on both of them.

If you want to install the APK on only one of the currently connected devices, you can do so with the adb tool. First of all, determine the identifier of each of the connected devices with

$ adb devices
List of devices attached
* daemon not running. starting it now at tcp:5037 *
* daemon started successfully *
05426e2409d30434    device
emulator-5554   device

The device with ID 05426e2409d30434 is a physical Nexus 5 in debug mode connected via USB to my laptop, while emulator-5554, as the name indicates, is a running instance of the test virtual device we created before. We can install the APK on e.g. the emulator with

adb -s "emulator-5554" install build/outputs/apk/androtest-debug.apk

which will now be ready to be executed on the virtual device.

The lint tasks¶

If you have ever used Android Studio you might wonder if there is the chance to get all the useful code analysis and hints that the IDE brings up while you code your application. While this feature is probably not supported by the text editor of your choice, we can remedy by using the lint tool provided by the Android SDK. A convenient way to invoke it is through a dedicated Gradle task. From the output of ./gradlew tasks --all you should also see the following tasks

Verification tasks
------------------
check - Runs all checks.
connectedAndroidTest - Installs and runs instrumentation tests for all flavors on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
connectedDebugAndroidTest - Installs and runs the tests for debug on connected devices.
deviceAndroidTest - Installs and runs instrumentation tests using all Device Providers.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on all variants.
lintDebug - Runs lint on the Debug build.
lintRelease - Runs lint on the Release build.
test - Run unit tests for all variants.
testDebugUnitTest - Run unit tests for the debug build.
testReleaseUnitTest - Run unit tests for the release build.

Notice how we can invoke the lint tool in multiple ways. As we saw before, the build task, as opposed to assemble, will also run the lint tool. But we can also choose to run it as part of a full check with the check task, or just on its own, on every variant of our project or simply on the debug build. As an example, let's run the lint task with

./gradlew lint

The output should contain something that looks like the following

...
> Task :lint
Ran lint on variant debug: 13 issues found
Ran lint on variant release: 13 issues found
Wrote HTML report to file:///home/gabriele/Projects/androtest/build/reports/lint-results.html
Wrote XML report to file:///home/gabriele/Projects/androtest/build/reports/lint-results.xml


BUILD SUCCESSFUL in 1s
25 actionable tasks: 7 executed, 18 up-to-date

This tells us that the lint tool has found 13 issues on both the debug and the release build of our project, and that a report is available as both an HTML and an XML file in build/reports.

Let's open the HTML report and have a look at the reported issues. As predictable, the largest number of issues is about internationalisation, as we have used many hard-coded strings in our source code. So let's try to fix some of these, starting from the HardcodedText one. By clicking on it, we get sent to a more detailed part of the page that suggests us to use a @string resource instead of the hard-coded string Gravity sensors. Since our code will change the string at run-time in any case, the simplest solution is to replace it with an empty string. If we now run the lint task again, we should see only 12 issues. Good! :).

Let's now tackle a couple of TextView Internationalization issues. The first one is about the string "Detected gravity sensors: " + Integer.toString(nSensors) that we have used in the MainActivity.java source file. The problem here is two-fold. We should use a resource string and favour placeholders rather than concatenation. So let's create a string resource file. Recall from before that this should go in an XML file, say strings.xml, within the src/main/res/values folder.

<?xml version="1.0" encoding="utf-8"?>
<resources>
    <string name="sensors_no">Detected gravity sensors: %d</string>
</resources>

We can now replace line 44 in MainActivity.java with

      text.setText(getString(R.string.sensors_no, nSensors));

Run the lint task again, and the count should now be down to only 9 issues. This should give you the idea of how to continue fixing the remaining findings.

Conclusions¶

In this post we saw that, if we want to develop Android applications by simply relying on a text editor and some command line tools, the initial set up of the development environment requires a few preliminary steps, some of which are performed when you install Android Studio. However, some of the other steps need to be performed regardless, with the only difference that, in Android Studio, you would have a GUI to assist you. The bonus of going IDE-free, however, is in the fact that we now know what operations are being performed by the IDE when, e.g., we install an Android SDK that targets a certain API level.

Creating a Gradle project is quite easy, and in this we are assisted by Gradle itself. The Android Gradle plugin takes care of invoking the right tools to perform the most common tasks, like building our project, or even installing it on an Android device. One handy feature that we might miss out with a plain text editor is the live linting of our code. However, at any stage of the development, we can run the lint Gradle task to generate a report with all the issues that have been discovered within the whole project.

This post should have also showed you that, whilst Android Studio is a very useful tool, there is practically nothing that we would miss out by using the command line tools of the SDK instead. Development might at first be less smooth, and the learning curve steeper, but surely has its positives too.

Prime Numbers, Algorithms and Computer Architectures

2017-03-07T00:15:00+01:00

A natural number $p\in\mathbb N$ is said to be prime if its only divisors are 1 and $p$ itself. Any other number that does not have this property is sometimes called composite. The discovery that there are infinitely many prime numbers dates back to c. 300 BC and is due to Euclid. His argument by contradiction is very simple: suppose that, indeed, there are only finitely many primes, say $p_1,\ldots,p_n$. The natural number

$$m=p_1p_2\cdots p_n + 1$$

is larger than and evidently not divisible by any of the primes by construction, and therefore $m$ must be prime. However, being larger than any of the $p_k$s, $m$ cannot be one of the finitely many primes, thus reaching to a contradiction.

Prime numbers play a fundamental role in Number Theory, a branch of Mathematics that deals with the properties of the natural numbers. Everybody gets to know about the prime factorisation of the natural numbers, a result so important that has been given the name of Fundamental Theorem of Arithmetic.

Counting Primes¶

Even though we saw that prime numbers are infinite, one might still want to know how many prime numbers are there within a certain upper bound. As numbers become bigger, the help of a calculator becomes crucial to tackle this problem and therefore it makes sense to think of algorithms that would get us to the answer efficiently.

The fastest way to count all the primes less than a given upper bound $n$ is by means of an ancient algorithm known as the Sieve of Eratosthenes. The idea is to start with the sequence of all the numbers from 0 up to $n$ and discard/mark the composite numbers as they are discovered. By definition, 0 and 1 are not prime, so they are removed. The number 2 is prime, but all its multiples are not, so we proceed by removing all the multiples of 2. We keep the first number that remains after 2, 3 in this case, and proceed to remove its multiples (starting from its square, since smaller multiples have already been removed at the previous steps). The process is repeated until it is no longer possible to proceed beyond the assigned bound $n$. It is clear that it is enough to get up to at most $\lceil\sqrt n\rceil$.

The following is a simple implementation in C++.

#include <iostream>
#include <vector>
#include <cmath>
#include <cassert>

using namespace std;

class Sieve {

private:
  vector<bool> * sieve;
  vector<int>  * primes;

public:
  Sieve(int n) : primes(NULL) {
    sieve = new vector<bool>(n + 1, true);

    (*sieve)[0] = false;
    (*sieve)[1] = false;
    for (int i = 0; i <= ceil(sqrt(n)); i++)
      if ((*sieve)[i])
        for (unsigned int j = i * i; j < sieve->size(); j += i)
          (*sieve)[j] = false;
  }

  ~Sieve() {
    delete sieve ;
    delete primes;
  }

  vector<int> * get_primes() {
    if (primes == NULL) {
      primes = new vector<int>();

      for (unsigned int i = 0; i < sieve->size(); i++)
        if ((*sieve)[i]) primes->push_back(i);
    }
    return primes;
  }

  int get(int i) {
    assert(i < count());
    return (*primes)[i];
  }

  int count() {
    if (primes == NULL) get_primes();

    return primes->size();
  }
};

int main() {
  int n;

  cin >> n;

  Sieve s = Sieve(n);

  cout << "There are " << s.count() << " primes between 0 and " << n << endl;

  #ifdef VERBOSE
  for (int i = 0; i < s.count(); i++) cout << s.get(i) << endl;
  #endif

  return 0;
}

A vector of booleans is implemented in C++ by an arry of bits instead of single bytes. Apart from turning all the possible complier optimisations, at the hardware level, this more compact data structure is more cache-friendly. Here is a first link between a software implementation of a prime search and the computer architecture the code runs on.

With an input of the order of $10^6$ the sieve is still quite fast. However the memory requirements are substantial: up to $10^9$ we are able to still use integers, but the memory consumption is of the order of the GB. The amount of memory on the system then can pose a serious limitation to the input parameter.

Segmented Sieve¶

If we want to list and/or count all the primes between two given (and possibly quite large) integers, we need a Segmented Sieve. If we are interested in all the primes between $a$ and $b$ we could, in principle, use the sieve of Eratosthenes to find all the primes up to $b$ and then list/count all the primes larger than $a$. But with $b$ of the order, say, $10^{15}$, a lot of memory is required to hold the result. Instead we can split the interval $[a,b]$ into chunks and process them separately.

The two main questions that we need to answer are: how do we adapt the sieve algorithm to start from $a$ rather than 0, and how do we fix the chunk size. Let us deal with the latter question first. The reason why we need a segmented sieve in the first place is because of memory limitations. So an upper bound for the chunck size is given by the available memory. However, for large values of the inputs, the sieve might need to jump to memory location which are further apart. But how do we quantify this "further apart"? The answer, again, is in the system architecture, which quite likely include a system of cache memory. In order not to violate the locality principle we should choose a chunk size which is comparable to the cache size. Assuming this to be of the order of the MB, and recalling that vector<bool> is an array of bits, a possible chunk size is of the order of $10^7$.

Coming to the question of how to implemente a segmented sieve, all we need to do is mark/remove all the composite number in range. Of course we would need to start by removing all the even numbers, then all the multiples of 3, then of 5 and so on. Therefore we still need the knowledge of the primes starting from 2 and going above. But how much above? Since our upper limit is $b$, we need all the prime numbers up to $\lceil\sqrt b\rceil$, which can be obtained with the standard sieve discussed earlier. These prime numbers can then be used to discover all the primes in the range $[a,b]$. We start by removing the first even number greater than or equal to $a$, together with all the numbers obtained by repeatedly adding 2 to it until we are out of bound. More generally, to find the first multiple of the prime $p$ in $[a,b]$ we use the formula

$$s = \left\lceil\frac ap\right\rceil\cdot p$$

However, recall that, for the standard sieve we really have to start from $p^2$, since lower multiples of $p$ have already been removed at the previous iteration. Therefore, as our starting point we pick the maximum between $s$ and $p^2$ (actually between $\lceil a/p\rceil$ and $p$).

The following is a simple implementation of the Segmented Sieve in C++.

#define CHUNK 10000000 // 10e7

class SSieve {

private:
  Sieve        * sieve;   // Sieve
  vector<bool> * ssieve;  // Segmented Sieve
  vector<int>  * seg_c;   // primes in each segment
  long long      a, b;    // Bounds
  int            c;       // Cached primes count
  int            seg;     // Current segment
  long long      size;    // Total numbers in interval
  int            max_seg; // Total number of segments

  void do_segment(int s) {
    if (seg == s) return; // Do not regenerate the current segment

    assert(s < max_seg);

    seg = s;

    // Determine segment bounds
    unsigned int l = a + s * CHUNK;
    unsigned int h = l + min(CHUNK, (int)size - s * CHUNK) - 1;

    // Allocate the new segmented sieve
    if (ssieve) delete ssieve;
    ssieve = new vector<bool>(h - l + 1, true);

    // Remove composite numbers in segment
    for (int p : (*sieve->get_primes())) {
      if (p * p > h) break;
      for (int i = max((int)l/p + (l % p == 0 ? 0 : 1), p) * p - l; i < ssieve->size(); i += p) //{
        (*ssieve)[i] = false;
    }
  }

public:
  SSieve(long long low, long long high) : c(-1), ssieve(NULL), seg(-1), seg_c(NULL) {
    assert(low <= high);

    seg_c   = new vector<int>(1, 0);

    if (high < 2) {
      c = 0;
      return;
    }

    if (low < 2) low = 2;

    a       = low;
    b       = high;
    size    = b - a + 1;
    sieve   = new Sieve(ceil(sqrt(b))); // The standard sieve
    max_seg = size / CHUNK + (size % CHUNK > 0 ? 1 : 0);
  }

  ~SSieve() {
    delete sieve;
    delete ssieve;
    delete seg_c;
  }

  unsigned int count() {
    if (c == -1) {
      c = 0;
      for (unsigned int i = 0; i < max_seg; i++) {
        do_segment(i);
        for (bool p : (*ssieve)) if (p) c++;
        // Keep track of the number of primes in segments
        // This is used by SSieve::get to retrieve the primes
        seg_c->push_back(c);
      }
    }
    return c;
  }

  long long get(int i) {
    assert(i >= 0 && i < count());

    int s, k, n = 0;

    // Determine which segment the requested prime belongs to
    for (s = 0; s < seg_c->size() - 1; s++)
      if ((*seg_c)[s + 1] > i) break;

    // Reconstruct the segmented sieve if necessary
    do_segment(s);

    // Translate into the actual prime
    int j = i - (*seg_c)[s];
    for (k = 0; k < ssieve->size(); k++) {
      if ((*ssieve)[k]) n++;
      if (n > j) break;
    }

    return k + s * CHUNK + a;
  }
};

This can be tested with a slightly modified main procedure, for example

int main() {
  long long n, m;

  cin >> n >> m;

  SSieve ss = SSieve(n, m);

  cout << "There are " << ss.count() << " primes between " << n << " and " << m << endl;

  #ifdef VERBOSE
  for (int i = 0; i < ss.count(); i++) cout << ss.get(i) << endl;
  #endif

  return 0;
}

Getting Started with x86-64 Assembly on Linux

2016-08-10T15:48:37+01:00

In this post we will learn how to assemble and link a simple "Hello World" application written in x86-64 assembly for the Linux operating system. If you have experience with Intel IA-32 assembly and you want to quickly get adjusted to the x86-64 world then this post is for you. If you're trying to learn the assembly language from scratch then I'm afraid this post is not for you. There are many great resources online on 32-bit assembly. One of my favourite documents is Paul Carter's PC Assembly Language, which I highly recommend if you're moving your first steps into the assembly language. If you then decide to come back to this post, you should be able to read it with no problems, since the tools that I will employ here are the same used in Carter's book.

Overview¶

This post is organised as follows. In the next section, I gather some details about the tools that we will use to code, assemble, link and execute the applications. As already mentioned above, most of the tools are the same as those used in Carter's book. Our assembler (and hence the syntax) will be NASM. I will make use of two linkers, ld and the one that comes with gcc, the GNU C Compiler, for reasons that will be explained later. The first x64 application that we will code will give us the chance to get familiar with the new system calls and how they differ from the 32-bit architecture. With the second one we will make use of the Standard C Library. Both examples will give us the chance to explore the x86-64 calling convention as set out in the System V Application Binary Interface.

All the code shown in this post will also be available from the dedicated asm GitHub repository.

Tools¶

The Netwide Assembler is arguably the most popular assembler for the Linux Operating System and it is an open-source project. Its documentation is nicely written and explains all the features of the language and of the (dis)assembler. This post will try to be as much self-contained as possible, but whenever you feel the need to explore something a bit more, the NASM documentation will probably be the right place. To assemble a 64-bit application we will need to use the command

nasm -f elf64 -o myapp.o myapp.asm

The flag -f elf64 instructs NASM that we want to create a 64-bit ELF object file. The flag -o myapp.o tells the assembler that we want the output object file to be myapp.o in the current directory, whereas myapp.asm specifies the name of the source file containing the NASM code to be assembled.

When an application calls functions from shared libraries it is necessary to link our object file to them so that it knows where to find them. Even if we are not using any external libraries, we still need to invoke the linker in order to obtain a valid executable file. The typical usage of ld that we will encounter in this post is

ld -o myapp.o myapp

This is enough to produce a valid executable when we are not linking our object file myapp.o against any external shared library or any other object file. Occasionally, depending on your distribution, you will have to specify which interpreter you want to use. This is a library which, for ELF executables, acts as a loader. It loads the application in memory, as well as the required linked shared libraries. On Ubuntu 16.04, the right 64-bit interpreter is at /lib64/ld-linux-x86-64.so.2 and therefore my invocation of ld will look like

ld -o myapp myapp.o -I/lib64/ld-linux-x86-64.so.2

Some external shared libraries are designed to work with C. It is then advisable to include a main function in the assembly source code since the Standard C Library will take care of some essential cleanup steps when the execution returns from it. Cases where one might want to opt for this approach are when the application works with file descriptors and/or spawns child processes. We will see an example of this situation in a future tutorial on assembly and Gtk+. For the time being, we shall limit ourselves to see how to use the GNU C Compiler to link our object file with other object files (and in particular with libc). The typical usage of gcc will be something like

gcc -o myapp.o myapp

which is very much similar to ld.

Hello Syscalls!¶

In this first example we will make use of the Linux system calls to print the string Hello World! to the screen. Here is where we encounter a major difference between the 32-bit and the 64-bit Linux world.

But before we get to that, let's have a look at what is probably the most important difference between the 32-bit and the 64-bit architecture: the registers. The number of the general purpose registers (GPRs for short) has doubled and now have a maximum size of ... well ... 64-bit. The old EAX, EBX, ECX etc... are now the low 32-bit of the larger RAX, RBX, RCX etc... respectively, while the new 8 GPRs are named R8 to R15. The prefix R stands for, surprise, surprise, register. This seems like a sensible decision, since this is in line with many other CPU manufacturers. Further details can be found in Chapter 11 of the NASM documentation and in this Intel white.

Let's now move back to system calls. Unix systems and derivatives do not make use of software interrupts, with the only exception of INT 0x80, which on 32-bit systems is used to make system calls. A system call is a way to request a service from the kernel of the operating system. Most C programmers don't need to worry about them, as the Standard C Library provides wrappers around them. The x86_64 architecture introduced the dedicated instruction syscall in order to make system calls. You can still use interrupts to make system calls, but syscall will be faster as it does not access the interrupt descriptor table.

The purpose of this section is to explore this new opcode with an example. Without further ado, let's dive into some assembly code. The following is the content of my hello64.asm file.

hello64.asm

global _start

;
; CONSTANTS
;
SYS_WRITE   equ 1
SYS_EXIT    equ 60
STDOUT      equ 1

;
; Initialised data goes here
;
SECTION .data
hello           db  "Hello World!", 10      ; char *
hello_len       equ $-hello                 ; size_t

;
; Code goes here
;
SECTION .text

_start:
    ; syscall(SYS_WRITE, STDOUT, hello, hello_len);
    mov     rax, SYS_WRITE
    mov     rdi, STDOUT
    mov     rsi, hello
    mov     rdx, hello_len
    syscall
    push    rax

    ; syscall(SYS_EXIT, <sys_write return value> - hello_len);
    mov     rax, SYS_EXIT
    pop     rdi
    sub     rdi, hello_len
    syscall

Lines 1, 13, 20 and 22 are part of the skeleton of any NASM source code. With line 1 we export the symbol _start, which defines the entry point for the application, i.e. the point in where the execution starts from. The actual symbol is declared on line 22, and line 24 will be the fist one to be executed.

In lines 6 to 8 we define some constants to increase the readability of the code. The price to pay is that NASM will export these symbols as well, thus increasing the size of the final executable file. I will discuss how to deal with this later on in this post. For the time being, let's focus on the rest of the code.

Line 13 marks the beginning of the initialised data section. Here we define strings and other immediate values. In this case we only need to define the "Hello World!\n" string (10 is the ASCII code for the newline character \n) and label it hello. Line 15 defines a constant equals to the length of the string, and this is accomplished by subtracting the address of the label hello from the current address, given by $ in NASM syntax.

The .text section, where the actual code resides, starts at line 20. Here we declare the _start symbol, i.e. the entry point of the application, followed by the code to be executed. In this simple example, all we need to do is print the string to screen and then terminate the application. This means that we need to call the sys_write system call, followed by a call to sys_exit, perhaps with an exit code that will tell us whether the call to sys_write has been successful or not.

Here is our first encounter with the new syscall opcode and the x86_64 calling convention. There isn't much to say about syscall. It does what you would expect it to do, i.e. make a system call. The system call to make is specified by the value of the rax register, whereas the parameters are passed according to the already mentioned x86_64 calling convention. It is recommended that you have a look at the official documentation to fully grasp it, especially when it comes to complex calls. In a nutshell, some of the parameters are passed through registers and the rest go to the stack.

The order of the registers is: rdi, rsi, rdx, r10, r8, r9.

We shall see in the next code example that, when we call a C function, we should use rcx instead of r10. Indeed, the latter is only used for the Linux kernel interface, while the former is used in all the other cases.

On line 23 we have a comment that shows us the equivalent C code for a call to sys_write. Its "signature" is the following.

1. SYS_WRITE

Parameters
----------

unsigned int    file descriptor
const char *    pointer to the string
size_t              number of bytes to write


Return value
------------
The number of bytes of the pointed string written on the file descriptor.

The number that appears on the top right corner is the code associated to the system call (compare this with line 6 above), and by convention this goes into the rax register (see line 24). Since sys_write requires 3 integer parameters we only need the registers rdi, rsi and rdx, in this order. Therefore, the file descriptor, the standard output in this case, will go in rdi, the address of the first byte of the string will go in rsi while its length will be loaded into rdx (lines 25 to 27).

In order to make the actual system call we can now use the new opcode syscall. The return value, namely the number of bytes written by sys_write in this case, is returned in the rax register. With line 29 we save the return value in the stack in order to use it as an exit code to be passed to sys_exit.

Since the application has done everything that needed to be done, i.e. print a string to standard output, we are ready to terminate the execution of the main process. This is achieved by making the exit system call, whose "signature" is the following.

60. SYS_EXIT

Parameters
----------

int     error code


Return value
------------
This system call does not return.

With line 32 we load the code of sys_exit into the rax register in preparation for the system call. As error code, we might want to return 0 if sys_write has done its job properly, i.e. if it has written all the expected number of bytes, and something else otherwise. The simplest way to achieve this is by subtracting the string length from the return value of sys_write. Remember that we stored the latter in the stack, so it is now time to retrieve it. The first and only argument of sys_exit must go in rdi, so we might as well pop the sys_write return value in there directly, and this is precisely what line 33 does. On line 34 we subtract the length of the string from rdi, so that if sys_write has written all the expected number of bytes, rdi will now be 0. The last instruction on line 35 is the syscall opcode that will make the system call and terminate the execution.

All right, time now to assemble, link and execute the above code.

nasm -f elf64 -o hello64.o hello64.asm
ld -o hello64 hello64.o -I/lib64/ld-linux-x86-64.so.2

This will assemble the source code of hello64.asm into the object file hello64.o, while the linker will finish off the job by linking the interpreter to the object file and produce the ELF64 executable. To run the application, simply type

./hello64

If you also want to display the exit code to make sure the executable is behaving as expected we could use

./hello64; echo "exit code:" $?

and, on screen, we should now see

Hello World!
exit code: 0

Apart from the fun, another reason to write assembly code is that you can shrink the size of the executable file. Let's check how big hello64 is at this stage

$ wc -c < hello64
1048

A kilobyte seems a bit excessive for an assembly application that only prints a short string on screen. The reason of such a bloated executable is in the symbol table created by NASM. This plays an important role inside our ELF file in case we'd need to link it with other object files. You can see all the symbols stored in the elf file with

$ objdump -t hello64

hello64:     file format elf64-x86-64

SYMBOL TABLE:
00000000004000b0 l    d  .text 0000000000000000 .text
00000000006000d8 l    d  .data 0000000000000000 .data
0000000000000000 l    df *ABS* 0000000000000000 hello64.asm
0000000000000001 l       *ABS* 0000000000000000 SYS_WRITE
000000000000003c l       *ABS* 0000000000000000 SYS_EXIT
0000000000000001 l       *ABS* 0000000000000000 STDOUT
00000000006000d8 l       .data 0000000000000000 hello
000000000000000d l       *ABS* 0000000000000000 hello_len
00000000004000b0 g       .text 0000000000000000 _start
00000000006000e5 g       .data 0000000000000000 __bss_start
00000000006000e5 g       .data 0000000000000000 _edata
00000000006000e8 g       .data 0000000000000000 _end

Assuming that we are not planning of doing this with our simple Hello World example, we strip the symbol table off hello64 with

strip -s hello64

If we now check the file size again, this is what we get

$ wc -c < hello64
512

i.e. less than half the original size. Looking at the symbol table again, this is what we get now:

$ objdump -t hello64

hello64: file format elf64-x86-64

SYMBOL TABLE:
no symbols

Observe that we can obtain the same result with the -s switch to the linker we decide to use, that is, either ld or gcc. Thus, for example,

ld -s -o hello64 hello64.o

will produce an ELF executable that lacks the symbol table completely.

The possibility of removing symbols from an ELF file gives us the chance of defining the constants for the system calls once and for all. In my GitHub repository you can find the file syscalls.inc where I have defined all the system calls together with their associated ID, and the "signature" of each on a comment line. With the help of this file, our source code would look like this

hello64_inc.asm

global _start

%include "../syscalls.inc"

;
; CONSTANTS
;
STDOUT      equ 1

;
; Initialised data goes here
;
SECTION .data
hello           db  "Hello World!", 10      ; char *
hello_len       equ $-hello                 ; size_t

;
; Code goes here
;
SECTION .text

_start:
    ; syscall(SYS_WRITE, STDOUT, hello, hello_len);
    mov     rax, SYS_WRITE
    mov     rdi, STDOUT
    lea     rsi, [hello]
    mov     rdx, hello_len
    syscall
    push    rax

    ; syscall(SYS_EXIT, <sys_write return value> - hello_len);
    mov     rax, SYS_EXIT
    pop     rdi
    sub     rdi, hello_len
    syscall

Note the inclusion of the file syscalls.inc at line 3, assumed to be stored in the parent folder of the one containing the assembly source code, and the only constant STDOUT at line 8.

If you do not need symbols in the final ELF file, you can just remove the symbol table completely with the previous command. However, if you want to retain some, but get rid of the one associated to constants that are meaningful to just your source code, you can add a -N <symbol name> (e.g. strip -N STDOUT hello64) switch to strip for each symbol you want dropped. To automate this when using syscalls.inc, one can execute the following (rather long) command

strip `while IFS='' read -r line || [[ -n "$line" ]]; do read s _ <<< $line; echo -n "-N $s "; done < <(tail -n +5 ../syscalls.inc)` hello64

on the ELF executable.

Finally, let's verify that all we really have is pure assembly code, i.e. that our application doesn't depend on external shared objects:

$ ldd hello64
        not a dynamic executable

In this case, this output is telling us that hello64 is not linked to any other shared object files.

Hello libc!¶

We shall now rewrite the above Hello World! example and let the Standard C Library take care of the output operation. That is, we won't deal with system calls directly, we shall instead delegate a higher abstraction layer, the Standard C Library, do that for us. Furthermore, with this approach, we will also delegate some basic clean-up involving, e.g., open file descriptor, child processes etc..., which we would have to deal with otherwise. For a simple application like a Hello World! this last point is pretty much immaterial, but we will see in another post on GUIs with Gtk+ 3 the importance of waiting for child processes to terminate an application gracefully.

So the code we want to write is the assembly analogue of the following C code

#include <stdio.h>

int main()
{
  return printf("Hello World!\n") - 13;
}

Inside the main function, we call printf to print the string on screen and then use its return value, decreased by the string length, as exit code. Thus, if printf writes all the bytes of our string, we get 0 as exit code, meaning that the call has been successful.

The didactic importance of this example resides in the use of the variadic function printf. The System V ABI specifies that, when calling a variadic function, the register rax should hold the number of XMM registers used for parameter passing. In this case, since we are just printing a string, we are not passing any other arguments apart from the location of the first character of the string, and therefore we need to set rax to zero. With all these considerations, the assembly analogue of the above C code will look like this

hello64_libc.asm

global main

extern printf

;
; Initialised data goes here
;
SECTION .data
hello           db  "Hello World!", 10, 0   ; const char *
hello_len       equ $ - hello               ; size_t
;
; Code goes here
;
SECTION .text

; int main ()
main:
    ; return printf(hello) - hello_len;
    lea     rdi, [hello]
    xor     rax, rax
    call    printf
    sub     rax, hello_len

On line 1 we export the main symbol, which will get called by the libc framework. On line 3 we instruct NASM that our application uses an external symbol, i.e. the variadic function printf. There is nothing new to say about the .data section, that starts at line 8. The code, however, is quite different. On line 17 we declare the label main, which marks the entry point of the C main function. We do not need local variables no access the standard argument of main, namely argc and argv, so we do not create a local stack frame. Instead, we go straight to calling printf. We load the string address in the rdi register (line 19), set the rax register to zero (line 20), since we are not passing any arguments by the XMM registers, and finally call the printf function. On the last line we subtract the string length, hello_len, from the return value of printf.

Assuming the above code resides in the file hello64_libc.asm, we can assemble and link it with

nasm -f elf64 -o hello64_libc.o hello64_libc
gcc -o hello64_libc hello64_libc.o

The ELF executable I get on my machine is 8696 bytes in size, and 6328 without the symbol table. If we thought 1048 was too much for a simple Hello World application, the libc example is 8 times bigger. And without symbols, you can see that we are wasting about 8K by relying on the Standard C Library.

A somewhat intermediate approach is to drop the main function and only use the printf function from libc. The advantage is a reduced file size, since our executable only depends on the Standard C Library. However, as discussed above, we lose an important clean-up process that can be convenient, if not necessary, at times.

hello64_libc2.asm

global _start

%include "../syscalls.inc"

extern printf

;
; Initialised data goes here
;
SECTION .data
hello           db  "Hello World!", 10, 0   ; const char *
hello_len       equ $ - hello - 1           ; size_t
;
; Code goes here
;
SECTION .text

_start:
    ; printf(hello) - hello_len;
    lea     rdi, [hello]
    xor     rax, rax
    call    printf
    sub     rax, hello_len

    ; syscall(SYS_EXIT, rax - hello_len)
    push    rax
    mov     rax, SYS_EXIT
    pop     rdi
    syscall

Note how, on lines 1 and 18, we removed the main function and reintroduced the _start symbol to tell NASM where the entry point is. Thus, execution of our application now starts at line 20. Here we prepare to call the printf function from libc (lines 20 to 22), we compute the exit code (line 23) and we store it in the stack. Now there is no Standard C Library framework to terminate the execution for us, since we cannot return from the non-existent main function, and therefore we have to make a call to SYS_EXIT ourselves (lines 26 to 29).

Assuming this code resides in the file hello64_libc2, we assemble and link with the commands

nasm -f elf64 -o hello64_libc2.o hello64_libc2
ld -s -o hello64_libc2 hello64_libc.o -I/lib64/ld-linux-x86-64.so.2

Checking the file size, this is what I get on my machine now

$ wc -c < hello64_libc2
2056

i.e. about a third of the "full" libc example above. There is something we can still do with strip, namely determine which sections are not needed. After linking with ld, the ELF I get has the following sections

$ readelf -S hello64_libc2 | grep [.]
  [ 1] .interp           PROGBITS         0000000000400158  00000158
  [ 2] .hash             HASH             0000000000400178  00000178
  [ 3] .dynsym           DYNSYM           0000000000400190  00000190
  [ 4] .dynstr           STRTAB           00000000004001c0  000001c0
  [ 5] .gnu.version      VERSYM           00000000004001de  000001de
  [ 6] .gnu.version_r    VERNEED          00000000004001e8  000001e8
  [ 7] .rela.plt         RELA             0000000000400208  00000208
  [ 8] .plt              PROGBITS         0000000000400220  00000220
  [ 9] .text             PROGBITS         0000000000400240  00000240
  [10] .eh_frame         PROGBITS         0000000000400260  00000260
  [11] .dynamic          DYNAMIC          0000000000600260  00000260
  [12] .got.plt          PROGBITS         00000000006003a0  000003a0
  [13] .data             PROGBITS         00000000006003c0  000003c0
  [14] .shstrtab         STRTAB           0000000000000000  000003ce

By trials and errors, I have discovered that I can get rid of .hash, .gnu.version and .eh_frame while still getting a valid ELF executable that does its job. To get rid of these sections one can use the command

strip -R .hash -R .gnu.version -R .eh_frame hello64_libc2

which yields an executable of 1832 bytes.

Conclusions¶

With the above examples, we have seen that, if our real goal is that of coding a Hello World application meant to run on an architecture with the x86_64 instruction set, assembly is the best shot we have. Chances are, if you are coding an application, it is more complex than just printing a string on screen. Even pretending for a moment that you don't care about the portability of your code, there are certainly some benefits from linking your application with gcc and letting the Standard C Library do some clean-up work for you. We will have the chance to see this last point from a close-up perspective in a future post. So take this current post as a reference point where you can look back when you need to recall the basics of writing a 64-bit assembly application for the Linux OS.