grack.com

This week I came in far-to-close contact with a fake-interview scam designed to backdoor my machine, and from the context of the emails, I assume my packages on crates.io.

Note: I’m calling it the “PinpinRAT” because of some of the internal strings, but it’s possible this has another name out there. I couldn’t find any other references to it online.

A week and a half ago I received an email from “D█████ S████” claiming to be from Lua Ventures, a (unbeknownst to me at the time) defunct Singapore-based VC in the DeFi space. To be clear: this is a fabricated persona, and the name was likely chosen to be easily mistaken for one of a number of real people with the name.

It looked like a real email, including a link to a somewhat boring, but legitimate-looking LinkedIn profile.

The attacker even name-dropped two of their investments that were specifically looking for advisory work: Lyrasing and Roadpay. Searching for either of the companies wasn’t really a flag - they both had some very basic web presense, but nothing that would indicate they were fake rather than just early stage. (archive.org snapshot of roadpay.cc).

We went back and forth on a meeting time and eventually settled on a time we were going to chat. There was nothing odd about the call itself, either. A somewhat-difficult-to-understand man with a German accent was on the other line. He said he was taking the call while travelling which was a bit odd, but again, not necessarily a flag.

After the call came the bait. A follow-up email that offered up a “test”.

At this point I was mildly annoyed, but not suspicious. I cloned the repo, but the first true red flag only fired here.

Where I got lucky: they sent me a TypeScript repo. It didn’t make sense to me. The instructions looked more like a TypeScript job interview than any sort of architecture analysis. I decided to zip up the repo and toss it into the Claude to get a quick scan - a combination of caution and laziness.

A few moments later it had identified some oddities:

Notably, the root package.json has no postinstall/preinstall hook — interesting, because they use patch-package, which is normally wired to a postinstall. Let me check every package.json for lifecycle scripts, then audit the patch files (the real risk surface — patches can inject arbitrary code into node_modules).

That piqued my interest. I scanned the directories myself and noticed a fairly unreasonable number of patches/ directories. The first few I checked seemed innocent enough, but as you might have guessed, they were just trying to add noise so the real payload wouldn’t get caught.

packages/electron-benchmarks/patches/sumchecker+3.0.1.patch
packages/electron-benchmarks/patches/@electron+get+2.0.3.patch
packages/electron-benchmarks/patches/extract-zip+2.0.1.patch

While I was checking slowly by hand, Claude identified first signs of PinpinRAT:

I found it. This is a malicious payload. Embedded in the typescript+5.9.2.patch file — inside what is presented as a routine “module specifier” patch to TypeScript — there is a base64-blob self-executing obfuscation stub injected at the very top of both _tsc.js and typescript.js:

;;(function(r,k){const d=Buffer.from(r,'base64');for(let i=0;i<d.length;i+=1)d[i]^=k;return new Function('require','Buffer','WebAssembly','process','__dirname',d.toString('utf8'))(require,Buffer,WebAssembly,process,__dirname)})("YWFg...",73)/*12ff4b51*/
void "ticket-harbor-tsc-shim-anchor";

This decodes a base64 string, XOR-decrypts every byte with the key 73, and runs the result through new Function(…) with require, process, Buffer, etc. handed in. That is a hidden code-execution payload that fires every time tsc or anything importing typescript.js runs — i.e. on the very first npm run typecheck / build / dev.

… and that’s the point where I decided to stop poking the bear on my own machine. I zipped it up with a password to stop myself from accidentally detonating it and kept running the analysis in the sandbox.

The Trap

The repo is themed as a ferry-ticketing app named “Ticket Harbor”. The task.txt included in the bundle was a plausible set of boring tasks, but ended with:

Run the repo typecheck, test suite, and relevant desktop/server build commands before submitting.

That instruction is the trap that gets you.

The chain works like this:

1. Four separate postinstall hooks run patch-package. But one of them also runs git update-index --skip-worktree on the patch files, which hides them from git status.

2. The typescript+5.9.2.patch injects a self-executing stub at the top of typescript.js and _tsc.js. This is a lightly-obfuscated blob fed into new Function(...) (avoiding eval, presumably to avoid malware detection).

3. That loader reads a hidden chunk appended to a file named operators/3.png, runs a small embedded WASM stub (in a custom wAsm chunk), then spawns a detached, silent Node process carrying a 1.68 MB obfuscated second-stage payload.

4. It cleans up after itself at three layers: the git skip-worktree trick, the dropper rewrites the patch to delete its own injected lines after first run, and the stage-2 temp directory self-deletes on execution.

The actual payload is a RAT (a remote-access trojan). I was originally worried this was a credential stealer but that’s a lot worse. PinpinRAT is nested in three obfuscated layers which were a pain to unwrap: obfuscator.io (which claims LLM protection, hah), and two further base64 layers.

What it drops

In the interest of 1) quickly sharing this info and 2) not accidentally detonating malware on my own machines, I let Claude tear apart the actual trojan in its sandbox and had it describe it to me.

The drop is a full remote-access trojan that seems to have been put together by someone who knows what they are doing. It sets up an RSA key locally and uses AES-256-CBC as a session key.

On startup it calls a checkin routine that harvests and exfiltrates a host fingerprint:

• primary IP address (enumerates all non-internal interfaces), plus all IPs
• username (os.userInfo().username)
• hostname
• OS type + release + platform + architecture
• process PID and full process.argv
• Node version

It generates an RSA-2048 keypair and a random AES-256 session key (aes_psk), then all subsequent traffic is AES-256-CBC encrypted with an HMAC-SHA256 integrity tag.

It supports the following commands:

• env — JSON.stringify(process.env) dumped and sent back.
• upload — reads an arbitrary file path and exfiltrates it.
• download — writes attacker-supplied bytes to any writable path.
• spawn — runs an arbitrary process with optional shell expansion.
• ls / cd / pwd / cp / mv — general filesystem primitives.
• dns — makes the host resolve arbitrary names through a specified resolver (for DNS tunneling?).
• dismantle — self-removal.

Indicators of Compromise

If you ended up running one of these, you should immediately disconnect your system from the network and rotate your credentials from another machine. Remediation should be straightforward, but consider your credentials (including cookies and password-protected secrets) compromised.

These are some indicators of compromise found in the PinpinRAT malware:

• C2: 89.124.107.161:80
• Scheduled task (Windows): PinpinWrappedJs
• Process masquerade (macOS): com.apple.WebKit.Networking
• Env vars: NODT_PAYLOAD_PATH, NODT_PAYLOAD_ARGS
• PNG chunk guard: WASMPACK (wAsm)
• PINPIN_NO_AUTOSTART=1: stops persistence
• cronjob with mutex.js (only if the RAT had permission, may not exist on macOS)
• Anchor strings in typescript.js: 12ff4b51, ticket-harbor-tsc-shim-anchor
• typescript+5.9.2.patch with the payload
• Artifact dirs: ~/Library/Caches/runtime-cache/.cache-<randomhex>/ (macOS), /tmp/.cache-<randomhex>/ (Linux), %TEMP%\.cache-<randomhex>\ (Windows)
  • .. containing payload.js and mutex.js

Where I should have seen flags

There’s a few places I should have seen flags earlier on. The goal of the campaign is to keep the flags subtle enough they don’t trigger your defences, but you need to be vigilent enough to see when enough yellow flags stack up to a red one.

The messages have some LLM tells to them when you look closer. That’s probably a sign that you should approach anything with extreme skepticism.

The LinkedIn profile looks real at first glance, but it’s filled with gibberish (“BSc(Hons), MA (Dist), PGDipFM, CEng”?) that should at least trigger sort of salad vibes. No real activity.

The social media link on their website has a real history, but the name was changed in November 2025. The posts are all pretty vacuous, vague praise for companies that aren’t really described in any detail.

None of the companies that had websites had a real presence beyond their flashy websites.

They never sent a proper invite - it was just a time and a Google Meet. What VC doesn’t use a calendar? Their camera was off the whole time and they were “travelling”.

And the overall approach, a VC fund based in Singapore, operating out of CEST, reaching out to a developer in Canada, with domains targeting American customers but ending in .cc. It’s far more difficult to check credentials of an organization that’s so far away.

Nothing was obvious without hindsight, but the missing pieces are there if you look at the whole thing together.

So who was this?

It’s impossible to say for sure, but this was targeted, had a pretty convincing cover story with a fake persona, multiple fake websites with stolen history, and a patient timeline. The git trap was sophisticated. This “fake-interview scam” has been a theme for a number of actors in 2026.

Who is actually behind this is the responsibility of the agencies now. What is worth noting is that this was targeted to developers like you and I, and that I was lucky enough to see a red flag right before springing the trap.

And to be honest, what’s terrifying and sobering to me is that if this had been a Rust repository with a booby-trapped build.rs script, I might have even fallen for it.

Every Rust binary has one thing in common: fn main(). If you come from the C world, that might be more familiar as int main(argc, argv). Some platforms might obfuscate it a bit more, but under the hood, every binary has an entrypoint.

We’re going to discuss what happens before main and what interesting things we can do there. In addition, we’ll be showing some novel techniques for mutable data that aren’t in common use in the Rust ecosystem today.

This post is a deep dive into some technical details of how Rust source becomes a Rust binary. Some background knowledge may be helpful to the reader, including:

• References in Rust
• Unsafe Rust

Before main

What might not be familiar to most developers is how you get into the main function. You see, under the hood for every language is the runtime. C has one: the C runtime that you might recognize as libc. Rust also has its own runtime: the Rust standard library. And because C is the lingua franca of runtimes for most executable code ¹, Rust builds its own runtime atop of C’s, effectively building its own higher-level abstraction encapsulating C’s.

A runtime is a bit fuzzy to define. It’s both the executable code that lives on disk and compilable headers and libraries used at compile time. But the purpose of a runtime is always the same: integrating developer code with the platform’s operating system.

There’s an entire ecosystem of processing that happens before the function you declared as main starts up. C uses this to configure allocation, file access, thread-local storage and other C runtime services. Rust uses this time to configure parts of its own language and runtime. Specifically, Rust has infrastructure to handle panics and unwinding. Rust also needs to translate the C-style program arguments ² into its own std::env::args interface. The machinery for all this is visible in the Rust compiler project.

Runtimes make use of this pre-main phase because it guarantees (1) running before user code, and (2) a single-threaded, highly-consistent and predictably-ordered environment, which allow for reliable and deterministic initialization.

By not taking advantage of this environment, you are missing out on a very useful bootstrapping phase. We’ll see later on in this post how we can build some useful primitives making use of life before main.

Entry Points

A binary starts when the operating system’s loader ³ - the part of the OS that loads the binary into memory and sets up the environment - hands off control. The runtime is responsible for accepting the hand-off from the loader. There’s a platform-specific hook on every OS that accepts the hand-off - to some extent this is the real main. On Linux, the entry point is stored in the e_entry field of the ELF header, and by default, the linker places the address of a symbol named _start there. A similar hook exists on Windows, and boots the executable in a function named _WinMainCRTStartup. At this point the C runtime has a chance to configure itself, and the way that all runtimes do this is via initialization functions.

In early iterations of runtimes, bootstrapping was a static tree of function calls: initialize file I/O, initialize the allocator, etc. As runtimes became more complex, this tree of function calls became more complex, and binary sizes increased to absorb more C runtime functionality that they may or may not need.

Over time, linkers developed the ability to discard unused code before even writing the binary to disk (including unused parts of the C runtime), and with that came a need for a replacement for the static init call trees.

The most popular method ⁴ of declaring init code came from GCC: __attribute__((constructor)). The way this worked was to place a list of init functions into a contiguous chunk of the binary on disk. When the C runtime started, it could walk through each of these functions and call them, allowing various bits of the C runtime to request initialization without strongly coupling subsystems, and allowing the linker to jettison unused subsystems, init code and all.

Eventually the need for constructor ordering became important enough that constructors could be given a priority and run in a specific order, allowing the runtime to initialize subsystems before and after each other. E.g., the memory allocation (malloc) subsystem might be needed for buffered file I/O.

On most platforms ⁵, the linker was called in to do the priority work: each platform ended up with a way to prioritize the order in which data gets written to sections, which allowed for the C runtime to end up with a well-ordered list of function pointers ⁶.

We can even build an example of this by hand in Rust using the #[unsafe(link_section = "...")] attribute (try it in the Rust Playground):

/// Linux example: the modern glibc runtime uses `.init_array` to hold function
/// pointers, and a numeric suffix allows them to be ordered. Note that priorities
/// less than or equal to 100 are reserved for the runtime itself, so any code that
/// wants to use the C runtime must use a priority of 101 or higher.

// On Linux, `.init_array` holds _function pointers_, not functions.
// We can convert a function to a function pointer with one of the below
// blocks which is equivalent to this:
//
// #[used] // <-- without this, Rust might decide the init function is unused and remove it
// #[unsafe(link_section = ".init_array.NNNNN")] // <-- the section where we place the function pointer
// static INIT_ARRAY_FN_PTR: extern "C" fn() 
//     = function; // <-- the function pointer data: we assign the function to it
//
// extern "C" fn function() { ... } // <-- the function itself

#[used]
#[unsafe(link_section = ".init_array.101")]
static INIT_FN_FIRST: extern "C" fn() = const {
    extern "C" fn init() {
        println!("Initializing (first!)");
    }
    init
};

#[used]
#[unsafe(link_section = ".init_array.201")]
static INIT_FN_SECOND: extern "C" fn() = const {
    extern "C" fn init() {
        println!("Initializing (second!)");
    }
    init
};

fn main() {
    println!("Main!")
}

linktime: ctor, link-section and more

The ctor crate is designed to handle all of the boilerplate of registering constructors in a cross-platform way. This allows us to simplify our examples above to:

use ctor::ctor;

#[ctor(unsafe, priority = 101)]
fn init1() {
    println!("Initializing (first)!");
}

#[ctor(unsafe, priority = 201)]
fn init2() {
    println!("Initializing (second)!");
}

fn main() {
    println!("Main!")
}

Note that neither example explicitly calls the init functions. The linker organized them in a way that the C runtime called them for us!

Sections and Linker Scripts

The process in which constructors are linked isn’t mysterious, though. In fact, compilers allow you to name the location in the binary (on most platforms called a “section”) you want to place any of your data and/or code. And by extension, and as we saw above, Rust allows this as well. The challenge, as we will see, is making use of this organizational feature.

Linkers have been the key to C’s ability to target any form of binary for some time. Most linkers allow for developers to provide linker scripts - text files that live alongside your source code (which is compiled to object files) and instruct the linker on how those object files are assembled. Using a linker script, a single C file might become a Linux executable, or a block of raw assembly that lives in the boot sector of a hard drive.

Linker scripts also allow for defining virtual symbols - that is, symbols that don’t exist in any source file but can be used by C code to access pointers to the underlying data in the loaded binary.

Linker scripts are a complex topic and beyond the scope of this post, but we can easily find examples of them in the wild:

// Adapted from https://wiki.osdev.org/Linker_Scripts
SECTIONS
{
  .text.start (_KERNEL_BASE_) : {
    startup.o( .text )
  }

  .text : ALIGN(CONSTANT(MAXPAGESIZE)) {
_TEXT_START_ = .;
    *(.text)
_TEXT_END_ = .;
  }

  .data : ALIGN(CONSTANT(MAXPAGESIZE)) {
_DATA_START_ = .;
    *(.data)
_DATA_END_ = .;
  }
}

In the above example, the virtual symbols _TEXT_START_ and _TEXT_END_ are explicitly defined to point to the beginning and end of the .text section, respectively. The period in _TEXT_START_ = .; is a special syntax that refers to a location counter that resolves roughly to the current output address in the binary.

Linker Symbols

This trips up most developers that encounter it for the first time, but the linker is setting the address of the start and end symbols, and therefore where the static with the same name is placed, and not setting the value of symbols that are pointers. That is to say: the start and stop symbols aren’t a *const Type. The start and stop symbols carry no data themselves and are used for their addresses only! The section consists of the range of data between the start (inclusive) and stop (exclusive) symbols.

Specifying start and end symbols for every section can be complex and tedious in linker scripts, so many linkers ⁹ eventually gained a feature where they could automatically define symbols bounding all sections in the executable. E.g., for GNU toolchains, a section named MY_SECTION will automatically have symbols __start_MY_SECTION and __stop_MY_SECTION defined. macOS has a similar pattern where it synthesizes a section$start and section$end symbol for each section.

In the GNU linker, those sections not explicitly defined in the linker script are called “orphan sections” ¹⁰. One important thing to note: if (and only if!) a section’s name is compatible with a C symbol name, the linker will automatically define a _start- and _stop-prefixed symbol for the section. In the example you’ll see below, the section name our_strings that we used works, but if we had chosen our.strings or .our_strings it would not have!

Try it in the Rust Playground:

use std::mem::MaybeUninit;

#[used]
#[unsafe(link_section = "our_strings")]
static FIRST_STRING: &'static str = "Hello, ";

#[used]
#[unsafe(link_section = "our_strings")]
static SECOND_STRING: &'static str = "world!";

// Note: these are not pointers. Instead, the linker has placed
// the boundary symbols STATIC_STRING_START and STATIC_STRING_END at
// the start and end of the section!
unsafe extern "C" {
    #[link_name = "__start_our_strings"]
    static STATIC_STRING_START: MaybeUninit<()>;
    #[link_name = "__stop_our_strings"]
    static STATIC_STRING_END: MaybeUninit<()>;
}

fn main() {
    let strings: &'static [&'static str] = unsafe {
        // SAFETY: get the addresses of the start and end symbols without
        // reading them.
        let start = &raw const STATIC_STRING_START as *const &'static str;
        let end = &raw const STATIC_STRING_END as *const &'static str;
        std::slice::from_raw_parts(start, end.offset_from(start) as usize)
    };
    
    // "Hello, world!"
    println!("String: {}", strings.join(""));
}

The link-section crate is designed to abstract away the details of these linker sections and convert them into traditional Rust slices with all standard slice operations available. We can use it to simplify the example above to:

use link_section::{in_section, section};

#[section(typed)]
static OUR_STRINGS: link_section::TypedSection<&'static str>;

#[in_section(OUR_STRINGS)]
static FIRST_STRING: &'static str = "Hello, ";

#[in_section(OUR_STRINGS)]
static SECOND_STRING: &'static str = "world!";

fn main() {
    println!("String: {}", OUR_STRINGS.join(""));
}

In these examples we’re submitting items to the link section in a single module within a single crate, but that’s not a requirement. In fact, the power of link sections is that you can submit items to a link section from any crate that contributes code to a binary - the linker will gather them all together just before writing the final binary.

Dependency Injection

The registration pattern we’re about to build is Dependency Injection by another name. This is a well-known pattern: frameworks like Dagger and Spring are built on the same principle that consumers of registration data should not be coupled to the providers of that data. A provider registers data at its definition site, a consumer simply reads the registry.

What’s somewhat different with linker sections versus traditional DI is that in DI the framework often needs to walk the module graph or scan loaded classes at startup to discover both providers and consumer sites. With linker sections, this magic is handled when the binary is written. The linker is the one that gathers all of the provider data and makes it trivially available to the consumer.

The example below uses a link_section::section to register CLI subcommands and is an instance of this pattern. More complex projects like Turbopack use this pattern to register string-pool constants, and the registration machinery used for serialization/deserialization and turbotask incremental compilation functions. A hypothetical webserver could make use of this pattern to register routes and middleware that are automatically collected at build time. The core mechanism is the same: the contributors place data into a shared registration system from any crate in the dependency tree, and the consumer reads the collected data without having to know where it was provided from.

Using Sections for Registration

One advantage we have in doing work before main is that it is well-behaved. No threads are running unless we start them. This means we are able to avoid the complexity of locks and other synchronization primitives in many cases, and that we can explicitly split our writable and immutable phase of our data’s lifecycle clearly: before and after main. And because of that, accessing data in the running program can become both simpler and more efficient by avoiding the need to acquire and release locks.

First, we’ll define our subcommand, a const constructor function, and a #[section] to collect them:

use std::collections::VecDeque;
use std::path::Path;

use link_section::{in_section, section};

struct CliSubcommand {
    is_default: bool,
    name: &'static str,
    description: &'static str,
    f: fn(&Path, &[String]),
}

impl CliSubcommand {
    const fn new(name: &'static str,
                 description: &'static str, 
                 f: fn(&Path, &[String])) -> Self {
        Self { is_default: false, name, description, f }
    }

    const fn new_default(name: &'static str,
                        description: &'static str, 
                        f: fn(&Path, &[String])) -> Self {
        Self { is_default: true, name, description, f }
    }
}

#[section(typed)]
static CLI_SUBCOMMANDS: link_section::TypedSection<CliSubcommand>;

Then we’ll register subcommands - these can live anywhere in your code:

mod list {
    #[in_section(CLI_SUBCOMMANDS)]
    static CLI_SUBCOMMAND_LIST: CliSubcommand = 
        CliSubcommand::new("list", "List all items", |_exe, _args| {
            println!("Listing all items");
        });
}

mod add {
    #[in_section(CLI_SUBCOMMANDS)]
    static CLI_SUBCOMMAND_ADD: CliSubcommand = 
        CliSubcommand::new("add", "Add a new item", |_exe, _args| {
            println!("Adding a new item");
        });
}

mod help {
    #[in_section(CLI_SUBCOMMANDS)]
    static CLI_SUBCOMMAND_HELP: CliSubcommand = 
        CliSubcommand::new_default("help", "Show help", |exe, _args| {
            println!("Usage: {} <subcommand> [options]", exe.display());
            println!();
            println!("Subcommands:");
            for subcommand in CLI_SUBCOMMANDS {
                println!("  {}: {}", subcommand.name, subcommand.description);
            }
        });
}

And then in our main function we can dynamically dispatch to any registered subcommand without ever having to know what they are or where they live. It only needs to be able to see the CLI_SUBCOMMANDS section definition:

fn main() {
    let mut args: VecDeque<String> = std::env::args().collect();
    let exe = args.pop_front().expect("No executable name provided");
    let exe = Path::new(&exe);
    let subcommand_name = args.pop_front().unwrap_or_default();
    let rest: Vec<String> = args.into();

    // Try to find the subcommand by name
    for cmd in CLI_SUBCOMMANDS {
        if cmd.name == subcommand_name {
            (cmd.f)(exe, &rest);
            return;
        }
    }
    // If no subcommand was found, fall back to the default subcommand
    for cmd in CLI_SUBCOMMANDS {
        if cmd.is_default {
            (cmd.f)(exe, &rest);
            return;
        }
    }
}

Running the code above works as you’d expect:

$ ./cli
Usage: ./cli <subcommand> [options]

Subcommands:
  list: List all items
  add: Add a new item
  help: Show help

$ ./cli list
Listing all items

Beyond Immutable Data

The example above assumes that the linked data is immutable. But that’s only half the power of using linker organization for data. Mutability in global static data is a common problem with well-known solutions in standard Rust. We could potentially use Rust’s built-in tools for interior mutability like mutexes, or atomic types, for example. Each of those comes with some runtime cost. If they are “uncontended” they aren’t expensive, but they are not necessarily free. ¹¹

But what if we want to minimize the overhead of runtime data access? Immutable data is trivial: Rust allows safe concurrent access to immutable data by default ¹². Rust has strict requirements for mutable data, however. There are two requirements to safely mutate data: (1) the modifications must be done in a thread-safe manner, and (2), there must never be more than one reference to the data if a mutable reference exists.

At the beginning of this post, we mentioned that life-before-main is a useful place to bootstrap because no threads are running unless we start them. And the solution to (1) is trivial if the data is currently accessible to a single thread only! We don’t need to do anything atomically; we only need to ensure that all of the changes we make to that data “happen before” any reads to the data. In a single-threaded environment, “happens before” is automatic ¹³. This means that we can mutate data in a link-section before main and it will be safe to access, lock free, from any thread after main.

The resolution for (2) is similar: as long as we only ever take a mutable reference (and only a mutable reference!) before main, there will never be more than one reference to the data when a mutable reference exists.

The pre-main environment satisfies both (1) and (2), without needing to reach for locks or other synchronization primitives.

There’s also one additional gotcha with linker sections we need to be very careful with: the slice that contains all of the items in a section is an alias to the static item that lives in the section. The rules about aliasing apply to both the slice and the static item, and you must ensure that static items are placed in UnsafeCell to safely mutate them from the slice ¹⁴. Rust does not allow a static item to be modified through other means. With static items that aren’t wrapped in an UnsafeCell, LLVM may consider itself free to cache, reorder or otherwise make assumptions about the data itself. UnsafeCell itself is not Sync, so you’ll need to add your own wrapper types on top of this!

Let’s walk through an example of how we might otherwise do something like this. We’re going to build a string interning pool defined entirely at link-time, and add a wrinkle: we want to be able to sort the slice of interned strings at runtime so we can quickly intern a string by value if needed via binary search (try it in the Rust Playground):

use std::cell::UnsafeCell;
use std::mem::MaybeUninit;
#[cfg(debug_assertions)]
use std::sync::atomic::{AtomicBool, Ordering};

/// Nightly Rust offers a built-in `SyncUnsafeCell`. This is a minimal
/// reimplementation of that: 
/// <https://doc.rust-lang.org/std/cell/struct.SyncUnsafeCell.html>
#[repr(transparent)]
struct SyncUnsafeCell<T: ?Sized>(UnsafeCell<T>);
// SAFETY: safety burden of UnsafeCell is placed entirely on the user
unsafe impl<T: ?Sized + Sync> Sync for SyncUnsafeCell<T> {}

macro_rules! intern_string {
    ($name:ident, $string:literal) => {
        #[allow(unused)]
        const $name: &'static str = const {
            // This is not a common pattern, but it's entirely valid
            // to nest static items inside of const blocks. 
            // You can think of this as a way to hide the symbols
            // in a completely anonymous namespace.
            const VALUE: &str = $string;
            // Safety note: this static must _never_ be used. This is
            // purely a submission to the linker and _any_ access to it
            // may be UB.
            #[used]
            #[unsafe(link_section = "our_strings")]
            static ITEM: SyncUnsafeCell<&'static str> = 
                SyncUnsafeCell(UnsafeCell::new(VALUE));
            VALUE
        };
    };
}

intern_string!(WORLD, "world");
intern_string!(EXCLAMATION, "!");
intern_string!(HELLO, "hello");
intern_string!(FROM, "from");
intern_string!(RUST, "Rust");

unsafe extern "C" {
    #[link_name = "__start_our_strings"]
    static STATIC_STRING_START: MaybeUninit<SyncUnsafeCell<()>>;
    #[link_name = "__stop_our_strings"]
    static STATIC_STRING_END: MaybeUninit<SyncUnsafeCell<()>>;
}

/// Debug check to make sure the slice is sorted once and only once. This _could_
/// be enabled in release mode without any major performance impact, but we have enough
/// guarantees in place. Note that atomic access _does_ establish some memory ordering
/// guarantees, but the soundness guarantees are upheld with or without this atomic check.
#[cfg(debug_assertions)]
static SLICE_IS_SORTED: AtomicBool = AtomicBool::new(false);

// Implementation note: this function must not be called before `SORT_STRINGS_CTOR` has
// run.
fn interned_strings() -> &'static [&'static str] {
    // We use Acquire/Release pairing as a double-initialization check
    #[cfg(debug_assertions)]
    debug_assert!(SLICE_IS_SORTED.load(Ordering::Acquire), "Oh no! Slice was not sorted!");

    // SAFETY: we are calling this after main and we can guarantee that no
    // mutable reference is still alive. Since we know that no other code
    // is running before main, and that `SORT_STRINGS_CTOR` will run before main,
    // we can guarantee creating these slices is safe as 1) the sort "happens-before"
    // any access and 2) the mutable reference has been closed before any read-reference
    // access (satisfying aliasing XOR mutability requirement).
    let strings: &'static [&'static str] = unsafe {
        let start = &raw const STATIC_STRING_START as *const &'static str;
        let end = &raw const STATIC_STRING_END as *const &'static str;
        std::slice::from_raw_parts(start, end.offset_from(start) as usize)
    };
    strings
}

// Implementation note: this function assumes the slice has been sorted. See
// the guarantee above on `interned_strings` for reasoning.
fn maybe_intern_string(s: impl AsRef<str>) -> Option<&'static str> {
    let s = s.as_ref();
    let strings = interned_strings();
    strings.binary_search(&s).ok().map(|index| strings[index])
}

// SAFETY: We use the reserved `.init_array.0` priority because we do not
// access any C runtime functions (sort_unstable does not allocate) and we
// want to run before all other code. `.init_array.101` would work in our
// case, but this prevents other early-init code from accidentally running
// in the wrong order. `SLICE_IS_SORTED` is a debug check to make sure that
// doesn't happen. Note that all early-init code is tagged with `unsafe` so
// it always needs to be aware of safety guarantees of all APIs it touches.
#[used]
#[unsafe(link_section = ".init_array.0")]
static SORT_STRINGS_CTOR: extern "C" fn() = const {
    extern "C" fn sort_strings() {
        // We use Acquire/Release pairing as a double-initialization check
        #[cfg(debug_assertions)]
        debug_assert!(!SLICE_IS_SORTED.load(Ordering::Acquire), "Oh no! Sorted twice?!?");
        // SAFETY: we are calling this before main and we can guarantee that no
        // reference from `interned_strings` exists yet because we know no other
        // threads will be running, and we're not calling `interned_strings` yet.
        let strings: &mut [&'static str] = unsafe {
            // SAFETY: the bounds markers are not mutable, but we can safely
            // cast them to mutable pointers because we know the data behind
            // them is stored within `UnsafeCell` which is Rust's way of 
            // giving us interior mutability.
            let start = &raw const STATIC_STRING_START as *mut &'static str;
            let end = &raw const STATIC_STRING_END as *mut &'static str;
            std::slice::from_raw_parts_mut(start, end.offset_from(start) as usize)
        };
        strings.sort_unstable();
        #[cfg(debug_assertions)]
        SLICE_IS_SORTED.store(true, Ordering::Release);
    }
    sort_strings
};

fn main() {
    for (i, s) in interned_strings().iter().enumerate() {
        println!("[{i}]: {s}");
    }

    println!("{}, {}{}", HELLO, WORLD, EXCLAMATION);
    println!(
        "{}, {}{}",
        maybe_intern_string("hello").unwrap(),
        maybe_intern_string("world").unwrap(),
        maybe_intern_string("!").unwrap()
    );
}

The above example is pretty heavy (and a bit thicker thanks to the generous commentary), but it’s a good example of how much boilerplate crates like ctor and link-section can save you.

The equivalent using those crates can make use of the TypedMutableSection and a ctor to ensure the items are sorted before main. Note that the requirements for TypedMutableSection are that the items must be const - the reason is that the mutable section uses a similar style of code to the manually-implemented example above.

//! String interning pool using `ctor` and `link-section`.
use ctor::ctor;
use link_section::{in_section, section};

#[section(mutable)]
static INTERNED_STRINGS: link_section::TypedMutableSection<&'static str>;

#[in_section(INTERNED_STRINGS)]
const WORLD: &'static str = "world";

#[in_section(INTERNED_STRINGS)]
const EXCLAMATION: &'static str = "!";

#[in_section(INTERNED_STRINGS)]
const HELLO: &'static str = "hello";

#[in_section(INTERNED_STRINGS)]
const FROM: &'static str = "from";

#[in_section(INTERNED_STRINGS)]
const RUST: &'static str = "Rust";

#[ctor(unsafe)]
fn sort_strings() {
    let strings: &mut [&'static str] = unsafe { INTERNED_STRINGS.as_mut_slice() };
    strings.sort_unstable();
}

fn maybe_intern_string(s: impl AsRef<str>) -> Option<&'static str> {
    let s = s.as_ref();
    let strings = INTERNED_STRINGS.as_slice();
    strings.binary_search(&s).ok().map(|index| strings[index])
}

fn main() {
    for (i, s) in INTERNED_STRINGS.iter().enumerate() {
        println!("[{i}]: {s}");
    }

    println!("{}, {}{}", HELLO, WORLD, EXCLAMATION);
    println!(
        "{}, {}{}",
        maybe_intern_string("hello").unwrap(),
        maybe_intern_string("world").unwrap(),
        maybe_intern_string("!").unwrap()
    );
}

This particular example isn’t impossible without link sections, of course. What we get from the patterns we’ve discussed in this post are three things: (1) the guaranteed aggregation of tagged items, with all data pre-allocated and contiguous in memory; (2) the ability to distribute registrations anywhere in the code; and (3) a guaranteed count of the items in the section.

One major benefit that falls out of the three advantages above is that link sections require no allocations. If we were to rewrite this without link sections we’d be allocating a HashMap, Vec or other data structure, and potentially resizing it a number of times as we gather items (because we don’t actually know how many items we’ll have until runtime!).

The second major benefit that falls out is the Inversion of Control. The dependency graph for a traditional “collection” approach looks like below, with shared types deeply nested in the dependency graph, modules depending on that shared types module, and then a collector module that depends on all those modules to collect their types:

The change doesn’t seem large, but there is a large impact: the collector can now live anywhere and no longer needs to care what modules are contributing data:

But Seriously: When Not to Use This

Link-time computation is fun and powerful, and it’s not always the right tool for the job. There is often a non-link-time equivalent: manually collecting data in crates that have visibility into each crate that wishes to contribute data. This can be inconvenient at times - instead of the contributors seeing a single contribution point “upstream” in a core crate, a “collector” crate with lots of crate references is required to collect them all.

Dead-code elimination becomes challenging: the link-section crate (and the linkme equivalent) both decorate all items using #[used], so the linker is disallowed from pruning unused data. Figuring out how to make link-time collection and dead-code elimination work well together is a complex problem beyond the scope of this post. For smaller bits of data like interned string atoms this might not be a problem, but if a program wants to intern larger chunks of data like chunks of raw JSON/JavaScript, or extensive data structures, this may add up to a lot of dead-code that may be difficult to identify.

Pre-main constructor functions have limitations: they cannot panic ¹⁵, Rust does not guarantee that all stdlib functions are available, and the order that the initialization functions are called within a given priority level is not guaranteed and highly platform-dependent. With careful planning, these limitations may be worked around but life-before-main may not be correct for subtle and difficult-to-debug reasons.

At this time, Miri is not fully compatible with all pre-main constructors and link-section constructions: it has a very basic view of pre-main execution, and does not model link sections at all. This may improve over time, but as of the time of writing, LLVM sanitizers (ASan, TSan, and others) are the recommended way to test for undefined behaviour.

The Inversion of Control pattern also has a cost: it makes it potentially harder to audit all the places that contribute data to a link section.

In reality, many widely-deployed and heavily-used Rust programs already rely on pre-main functionality: the ctor, link-section, inventory and linkme crates are used by many downstream crates today.

Conclusions

You can do a lot before main, and the benefits of doing so are pretty significant for certain cases. It’s a highly-ordered, highly-controllable environment that lets you more confidently do a lot of work without locks, atomics and other synchronization primitives. Link sections give you arbitrary aggregation and co-location of related data across your whole binary without awkward crate dependency order. In a lot of cases you can even avoid allocations completely which helps keep you away from one of the worst allocator sins: churn of allocations leading to fragmentation.

For further reading, check out the various crates discussed in this post:

• ctor: Module initialization functions that run before main
• dtor: Not discussed in this post, but the shutdown analogue to ctor
• link-section: Linker-managed typed (slices) and untyped sections, with mutability support.
• scattered-collect: Linker-managed higher-level collections: slices, sorted slices, maps

Thanks

Thanks to my lovely wife Mia, Benjamin Woodruff and Luke Sandberg, as well as @ssokolow for their feedback and review. This post would not have been possible without their help.

As a thin layer on top of assembly language, C’s integer arithmetic APIs have always been minimal, effectively just mapping the underlying assembly opcodes to the C arithmetic operators. In addition, while unsigned arithmetic can safely overflow in C, signed arithmetic overflow is considered undefined behaviour, and UB can end up in heartache for C developers.

More modern languages like Rust have a much richer integer API, however. By default, using the standard addition and subtraction operators in debug mode, integer APIs will panic! on overflow or underflow. The same operators will wrap in two’s-complement mode in release mode, though the behaviour is considered defined. If the developer wants to specify carrying, wrapping, checked, or saturating operations, APIs for each of these modes are available.

We don’t have these convenient APIs available in C yet (see the epilogue for some nuance), but it would be great to have them. Given that signed arithmetic overflow is undefined behaviour, can we build a function with the following C signature that works?

bool will_add_overflow(int32_t a, int32_t b);

All modern machines use two’s complement representation for negative integers, but C was developed when computing was experimenting with other forms of representing signed integers. In this post, we will safely assume that all processors we’re targetting on are not using one of the alternative forms. If you find yourself programming for esoteric machines, you may wish to consult your local manual or guru.

A valid, quick-and-dirty solution to this would be to use integer promotion and add 64-bit signed integers instead, checking to see if the result is within range of an int16_t:

bool will_add_overflow_64bit(int32_t a, int32_t b) {
    // a and b are promoted to 64-bit signed integers
    int64_t result = (int64_t)a + b;
    if (result < INT32_MIN || result > INT32_MAX) {
        return true;
    }
    return false;
}

No undefined behaviour! It also has the advantage of being easily read and obviously correct. But this required sign-extending two 32-bit numbers and performing two 64-bit additions:

will_add_overflow_64bit:
        movsxd  rax, edi
        movsxd  rcx, esi
        add     rcx, rax
        movsxd  rax, ecx
        cmp     rax, rcx
        setne   al
        ret

We can do better by taking advantage of the fact that on two’s-complement machines, addition is bitwise-identical between signed and unsigned numbers so long as you ignore carry, overflow, underflow and any other flags. In addition, the C specification (C99 6.3.1.3 ¶2) guarantees that the bit pattern will be preserved on a two’s-complement system.

We know that unsigned overflow is not UB, and we know that we can only overflow if a > 0 and b > 0, and we can only underflow if a < 0 and b < 0. If either a or b is zero, we’re safe. We also know that adding two positive integers must result in a positive result if no overflow occurred. For two negative integers, the result must also be negative. If we find that the sign of the sum does not match the sign expected, we’ve wrapped around!

bool will_add_overflow_if(int32_t a, int32_t b) {
    // Explicitly convert to uint32_t and then back
    int32_t c = (int32_t)((uint32_t)a + (uint32_t)b);
    if (a > 0 && b > 0 && c < 0) {
        return true;
    }
    if (a < 0 && b < 0 && c >= 0) {
        return true;
    }
    return false;
}

And we get a fairly hefty assembly representation:

will_add_overflow_if:
        lea     ecx, [rsi + rdi]
        test    edi, edi
        jle     .LBB2_3
        test    esi, esi
        jle     .LBB2_3
        mov     al, 1
        test    ecx, ecx
        jns     .LBB2_3
        ret
.LBB2_3:
        test    esi, edi
        sets    dl
        test    ecx, ecx
        setns   al
        and     al, dl
        ret

This is arguably a bit worse, as now we have a branch in the mix. But we can start to see a pattern here:

In two’s-complement, the expression x < 0 is equivalent to the expression (x & 0x80000000) == 0x80000000. Similarly, x >= 0 is equivalent to (x & 0x80000000) == 0.

Let’s create a NEG macro with the above expression and reproduce our pseudo-truth table in code. Note that we’ll also collapse the if statements into a single boolean expression so we can eliminate those branches:

bool will_add_overflow_expression(int32_t a_, int32_t b_) {
    // Explicitly work with uint32_t in this function
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    #define NEG(x) (((uint32_t)(x) & 0x80000000) == 0x80000000)
    return ((!NEG(a) && !NEG(b) && NEG(c)) 
        || (NEG(a) && NEG(b) && !NEG(c)));
    #undef NEG
}

This is looking better, but because we’re using short-circuiting logic, those branches are still there: we still have a jump!

will_add_overflow_expression:
        mov     eax, esi
        or      eax, edi
        setns   dl
        mov     ecx, esi
        add     ecx, edi
        sets    al
        and     al, dl
        test    edi, edi
        jns     .LBB3_3
        test    al, al
        jne     .LBB3_3
        test    esi, esi
        sets    dl
        test    ecx, ecx
        setns   al
        and     al, dl
.LBB3_3:
        ret

We can get rid of the branches by using non-short-circuiting bitwise operators:

bool will_add_overflow_bitwise(int32_t a_, int32_t b_) {
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    #define NEG(x) (((uint32_t)(x) & 0x80000000) == 0x80000000)
    return ((!NEG(a) & !NEG(b) & NEG(c)) 
        | (NEG(a) & NEG(b) & !NEG(c)));
    #undef NEG
}

And now it’s starting to look pretty compact (though we can do better):

will_add_overflow_bitwise:
        lea     ecx, [rsi + rdi]
        mov     eax, esi
        or      eax, edi
        and     esi, edi
        xor     eax, esi
        not     eax
        and     eax, ecx
        xor     eax, esi
        shr     eax, 31
        ret

Notice that the assembly gives us a bit of a hint here that repeated use of our macro isn’t actually necessary. The sign bit we’re interested in isn’t tested until the end of the function! Because we’re testing the same bit in every part of the expression, and bits in a given position only interact with other bits in the same position, we can pull that bit test out of the whole expression:

bool will_add_overflow_bitwise_2(int32_t a_, int32_t b_) {
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    #define NEG(x) (((uint32_t)(x) & 0x80000000) == 0x80000000)
    return NEG((~a & ~b & c) | (a & b & ~c));
    #undef NEG
}

We can also make use of the knowledge that testing the sign bit is the same as an unsigned shift right:

bool will_add_overflow_bitwise_3(int32_t a_, int32_t b_) {
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    return ((~a & ~b & c) | (a & b & ~c)) >> 31;
}

Not too bad! But let’s revisit the truth table and instead use the value of the sign bit directly. What we see is that a and b need to be the same value, and c needs to be the opposite value:

This truth table shows that what we ultimately want to test is this:

(a == 1 && b == 1 && c == 0) || (a == 0 && b == 0 && c == 1)

… but with a bit of work, we can simplify this down to two shorter expression candidates:

(a == b) && (a == !c)
(c == !a) && (c == !b)

For bit twiddling like we’re doing here, xor (^) can work like a “not-equals” operator (outputs 1 iff the inputs are 0,1 or 1,0), which means we can re-write our two expressions like so:

~(a ^ b) & (c ^ a)
(c ^ a) & (c ^ b)

By looking at those two options, is there a hint that one might be cheaper to implement? Let’s plug both into the compiler and see what we get!

bool will_add_overflow_optimized_a(int32_t a_, int32_t b_) {
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    return (~(a ^ b) & (c ^ a)) >> 31;
}

bool will_add_overflow_optimized_b(int32_t a_, int32_t b_) {
    uint32_t a = (uint32_t)a_, b = (uint32_t)b_;
    uint32_t c = (uint32_t)a + (uint32_t)b;
    return ((c ^ a) & (c ^ b)) >> 31;
}

And the resulting compiled versions:

will_add_overflow_optimized_a:
        lea     eax, [rsi + rdi]
        xor     eax, edi
        mov     ecx, edi
        xor     ecx, esi
        not     ecx
        and     eax, ecx
        shr     eax, 31
        ret
will_add_overflow_optimized_b:
        lea     eax, [rsi + rdi]
        xor     edi, eax
        xor     eax, esi
        and     eax, edi
        shr     eax, 31
        ret

We have a clear winner here: the compiler can do a much better job with (c ^ a) & (c ^ b). This is most likely because of the common sub-expression and the removal of the bitwise-not operator.

We can also confirm that there’s no known undefined behaviour by compiling it with clang’s -fsanitize=undefined feature. No UB warnings are printed, which means no UB was detected!

Epilogue

While this is the fastest we can get with bog-standard C99, this isn’t necessarily the best we can do.

Rust makes use of the compiler intrinsics to access the overflow flag of the processor directly:

pub fn add(a: i32, b: i32) -> bool {
    a.checked_add(b).is_none()
}

example::add:
    add     edi, esi
    seto    al
    ret

It turns out that both GCC and LLVM have C intrinsics that you can use. While they are non-portable to some compilers, they drastically simplify the assembly output!

bool will_add_overflow_intrinsic(int32_t a, int32_t b) {
    int32_t result;
    return __builtin_add_overflow(a, b, &result);
}

And, just like with the Rust compiler above, this generates optimal assembly!

will_add_overflow_intrinsic:
        add     edi, esi
        seto    al
        ret

Not to worry about this being so deeply compiler-specific for now, however. This will be standardized in C23 with the addition of the functions in the stdckdint.h header.

A full suite of tests to explore the solutions is available on Godbolt or as a Gist.

I was hoping to make more progress on self-hosting my scripting language (Kalos) this week, but I’m running out of steam because I think I coded myself into a corner. This is not a post where I’ve got everything figured out, but instead I’m taking a few moments to re-hash where things are at and figure out a plan for the future.

My original plan for this language was to offer a Python-like experience with minimal resource requirements: it should be able to run on an AVR, on bare metal, or even as part of a DOS executable. I had originally planned to support compilation on those devices, and even built a zero-allocation parser. The runtime is lightweight, integers are a configurable size, and strings are even optional. I believe it’ll be a great option on lower-end devices where Python is just too heavy.

Over the last week I started work to extract a small piece of the parser that deals with KIDL (example of KIDLE below), the part of the language that glues it to your C code. This is currently part of the existing parser, and the only piece that I could think of to carve off on the slow march to true self-hosting.

idl {
    prefix "kalos_module_idl_";
    dispatch name;

    module builtin {
        fn println(s: string) = kalos_idl_compiler_println;
        fn print(s: string) = kalos_idl_compiler_print;
        fn log(s: string) = kalos_idl_compiler_log;
    }
    ...
}

Where I’m stuck now is that the language is almost good enough to parse itself, but I’m finding lots of corner cases and papercut bugs that make it less-than-ideal. For example, I’m finding that the parser as-is is not quite good enough to handle function calls that are deeply nested in expressions.

I also ended up hacking in some dynamic-dispatch objects that help with not having classes, but that’s not a long-term thing that I want to support in lieu of a proper object/class system.

Eventually I’ll have to commit to rewriting the whole parser in Kalos, but as the title of the post suggests, I’m stuck in a potential energy well where the next steps are going to be difficult. The current parser is written in C and – while the code is pretty clean – it’s a lot of work to make changes to it. It’s going to take some time and effort to add support for things like classes, tear-off functions, etc, and being allocation-free doesn’t make any of this easy.

My mistake was being too ambitious and going right for C as the bootstrap, rather than something higher level. I should have started with Python as the bootstrap!

So I need to gather enough energy to choose and work on one of the following paths:

1. Commit to rewriting the parser in Kalos, maybe after adding support for hashtables/dicts to the language. The language spec is still small enough that I could port the current parser. Debugging is very difficult, and you need to ensure that you’re running the code while developing it to discover if you’ve accidentally stepped on one of the many landmines. Once I have the parser/compiler in a higher-level language like Kalos itself, these landmines will be much easier to fix!
2. Rewrite the parser in Python, knowing that I’ll have to rewrite it in Kalos later. This might not be terrible because the language is supposed to be python-like and there might be a mechanical translation route available. The thought of writing more code to throw away doesn’t fill me with a lot of joy, but it might just be what I have to do.
3. Scrap the hand-rolled parser and switch to something like Lemon. We’re already using the amazing re2c to write the lexer, so adding another tool isn’t a bad idea. Again, we’re putting in a bunch of effort knowing that this will be tossed away later, but maybe there’s a middle ground like just having Lemon build an AST, then have Kalos script generate the bytecode?

This is the last part of a three-part series covering the odyssey of getting a new coffeemaker, learning BTLE and how it works, reverse-engineering the Bluetooth interface and Android applications for the coffeemaker, writing a Rust-based CLI interface, and finally, hooking it all up to a GitHub actions bot that lets you brew a coffee just by filing an issue!

• Part 1: Introduction
• Part 2: Reverse Engineering
• Part 3: GitHub Actions
  • Getting Started
  • Extracting the Command-Line
  • Building the Workflow

In part 2 we got the command-line application working, and now it’s time to connect the dots and build a secure, yet web-accessible interface.

We could choose a standard web host, add some sort of authentication on top of it, build the right web tooling to integrate with the nice command-line application we built, and all the associated security so random people can’t brew coffee. But as you’ve guessed from the title of these posts, we’re going hook this command-line app into a private GitHub repo as our “interface”.

Making use of GitHub issues for automating weird things isn’t new, but I think this is the first time you can make coffee from it!

Getting Started

Here’s our goal:

1. We want to allow users to brew a coffee from a GitHub issue, which will be pre-populated from a number of pre-defined templates
2. The issue will contain part of the command-line that we want to run, and we’ll need to validate that it’s reasonable and correct, and that nobody is trying to inject any sort of “funny business” to break/backdoor the runner
3. We don’t want coffee brewers to have to chase down the status of the brewing operation, so we’re going to make use of issue comments as our basic UI. The user will be able to follow the progress of their coffee inside of the issue, and get a notification when it’s done.

This is what the user will see just before they brew the coffee:

The first question you might have is how we’re going to talk to a Bluetooth coffeemaker from GitHub’s system. This part turns out to be pretty easy: we can use GitHub self-hosted runners as a backdoor into the coffeemaker’s physical location! By running this on a computing device that has a Bluetooth radio in proximity to the coffeemaker, we can send commands to it in response to events occurring in a repo. Conveniently the Raspberry Pi 3 Model B and Pi 4 both support Bluetooth, but in our case we’re going to be using a spare MacBook that’s kicking around.

First thing, we need to create a new runner on GitHub for our project, and then set up the runner on the MacBook:

curl -O -L https://github.com/actions/runner/releases/download/${version}/actions-runner-osx-x64-${version}.tar.gz
tar xzf ./actions-runner-osx-x64-${version}.tar.gz
./config.sh --url https://github.com/mmastrac/brew-a-coffee --token ${token}
./run.sh

GitHub actions are pretty flexible and we have a huge number of events that can trigger them. In our case, we want the creation of a new issue to trigger a run, so our trigger becomes:

on:
  issues:
    types: [opened]

We’ll pull in the create-or-update-comment action from peter-evans for updating the user about the status of their coffee:

steps:
  - name: Add comment
    uses: peter-evans/create-or-update-comment@v2

And once the coffee is brewed or the process has failed for some other reason, we’ll want to close that issue, so we’re going to pull in peter-evans/close-issue for this:

  - name: Close issue
    if: always()
    uses: peter-evans/close-issue@v2

The actual brewing part will be pretty easy as well, but it’s going to require use to fetch the text of the issue and use that to create the command-line to run.

Let’s take a look at the event information that GitHub provides to us in $GITHUB_EVENT_PATH. There’s a lot that GitHub provides for us in this file, and this particular one is trimmed down significantly:

{
    "action": "opened",
    "issue": {
        ...
        "body": "This is my issue body comment\r\n",
        ...
        "title": "This is my issue title!",
        ...
    }
    ...

jq is one the best tools for integrating JSON APIs with shell scripts, so we’ll make use of that. We’ll create a small test JSON file called test.json that contains just the interesting subset of what’s available in the file at $GITHUB_EVENT_PATH:

{
    "action": "opened",
    "issue": {
        "body": "This is my issue body comment\r\n",
        "title": "This is my issue title!"
    }
}

First, we can test extraction of the issue body:

$ jq -r '.issue.body' < test.json
This is my issue body comment

$

That worked, but we’ve got some extra whitespace there. We can trim that with another jq command, gsub. By replacing leading or trailing whitespace (gsub("^\\s+|\\s+$";"")) with nothing, we can get just the text of the comment:

$ jq -r '.issue.body|gsub("^\\s+|\\s+$";"")' < test.json
This is my issue body comment
$

Better!

Extracting the Command-Line

Now what we want to do is allow the user to specify the command-line in the issue, but ensure that they can’t run anything nefarious on the runner. We developed a command-line cappuccino recipe in part 2 that we ran like this:

cargo run -- brew --beverage cappuccino --coffee 40 --milk 100 --taste extrastrong

So let’s extract out everything past the hyphens and make that the required input in our newly-filed issues:

brew --beverage cappuccino --coffee 40 --milk 100 --taste extrastrong

To work on extraction, we’ll update the issue.body field in our test.json file to this partial command-line:

{
    "action": "opened",
    "issue": {
        "body": "brew --beverage cappuccino --coffee 40 --milk 100 --taste extrastrong\r\n"
    }
}

Since we are creating a valid partial command-line for our brewing app, we can make use of that fact that we know the exact structure. In this case, we know we want it to:

1. Start with the subcommand brew
2. Next, contain the beverage to brew with --beverage <something>
3. Finally, contain a list of beverage parameters which are limited to coffee, milk, hotwater, taste, and temperature. Each parameter is separated from its value by a space (ie: --coffee 100), and is either a number or an enumeration value (ie: --taste strong).

We can then build a regular expression that will be limited to just the arguments we’re allowing here. We’ll use the \w character class as it’s a close match to the values required by our parameters.

We could go further in validating the --beverage parameter, or the values for the ingredients, but we know that those are carefully checked in the application and we’ll let the application handle the validation:

^brew --beverage \w+( --(coffee|milk|taste|hotwater|temperature) \w+)*$

Now we can put it all together and extract the command-line like so (note that we have to escape the \w patterns in regular expression):

CMDLINE=$(jq -r '.issue.body|gsub("^\\s+|\\s+$";"")|select(test("^brew --beverage \\w+( --(coffee|milk|taste|hotwater|temperature) \\w+)*$"))' < $GITHUB_EVENT_PATH)

And that’s probably the only tricky part of the process. Now we can build our GitHub action, piece-by-piece.

Building the Workflow

First, the preamble that tells GitHub where and when to run the action, and what permissions it has:

name: Brew

on:
  issues:
    types: [opened]

jobs:
  build:
    runs-on: self-hosted
    permissions:
      issues: write

steps:

Our first step will drop a comment into the issue so the user knows things are happening

      - name: Add initial comment
        uses: peter-evans/create-or-update-comment@v2
        id: comment
        with:
          issue-number: $
          body: ' - [X] Getting ready to brew your ☕️!'

We’ll then install the longshot executable from cargo and let them know it was done:

      - name: Install longshot
        run: cargo install --root /tmp/longshot -- longshot 
      - name: Update comment
        uses: peter-evans/create-or-update-comment@v2
        with:
          issue-number: $
          comment-id: $
          body: ' - [X] Installed the `longshot` executable'

Next, we’ll process the requested brew operation using the jq incantation from earlier. This step will create a body.md that we’ll use to update the comment, as well as cmdline.txt that will be used to execute our brewing operation later on:

      - name: Process the request
        run: |
          OUTFILE=body.md
          CMDLINE=$(
            jq -r '
              .issue.body |
              gsub("^\\s+|\\s+$";"") |
              select(
                test("^brew --beverage \\w+( --(coffee|milk|taste|hotwater|temperature) \\w+)*$")
              )' < $GITHUB_EVENT_PATH
          )
          echo Command-line we parsed was: $CMDLINE
          if [[ "$CMDLINE" == "" ]]; then
            echo " - [X] Couldn't parse the command line from your comment? 🤔" > $OUTFILE
            exit 1
          fi
          echo -n ' - [X]' Running brew command: \`$CMDLINE\` > $OUTFILE
          echo ' [(Log here)](https://github.com/'${GITHUB_REPOSITORY}'/actions/runs/'${GITHUB_RUN_ID}')' >> $OUTFILE
          echo "/tmp/longshot/bin/longshot $CMDLINE --device-name $" > cmdline.txt

We then update the comment with body.md:

      - name: Update comment
        uses: peter-evans/create-or-update-comment@v2
        with:
          issue-number: $
          comment-id: $
          body-file: body.md

And run the brewing command:

      - name: Brew coffee
        run: |
          echo '<details><summary>Log</summary><pre>' > log.md
          sh -c "`cat cmdline.txt`" | tee -a log.md
          echo '</pre></details>' >> log.md
          echo '✅ Success!' >> log.md
      - name: Update comment on success
        uses: peter-evans/create-or-update-comment@v2
        with:
          issue-number: $
          comment-id: $
          body-file: log.md

Finally, we’ll log a message to the comment on an error, and close the issue unconditionally:

      - name: Update comment on failure
        if: failure()
        uses: peter-evans/create-or-update-comment@v2
        with:
          issue-number: $
          comment-id: $
          body: |
            <br>
            ❌ Failed! Please check the log for the reason.
      - name: Close issue
        if: always()
        uses: peter-evans/close-issue@v2

And with all those steps, we can get ourselves a coffee from GitHub!

While you can’t access my private repository that I’m using to brew us coffee at home, you can definitely try out the example repo that I’ve set up here which uses the command-line interface’s simulator and runs on GitHub’s action runners instead:

https://github.com/mmastrac/brew-a-coffee-demo

To recap, we:

• Figured out how to talk to the coffeemaker over Bluetooth and sniffed some packets
• Reverse engineered the application, figured out how to construct packets, and implemented a command-line applications
• And finally, hooked the entire system up to GitHub actions!

Follow me on Mastadon for more updates on this adventure!