audio

Audio Processors

Render sound yourself, one block at a time, on the audio thread.

An audio processor is a class that extends AudioProcessor<A>, exported under its own name. A is the type of the one argument it starts with. Audio.start builds the processor in its own instance of the module, and mixes what it renders into the audio output. A processor is one of the four lanes.

A processor has two hooks:

init(sampleRate, maxFrames, channels, arg) runs once before the first block. maxFrames is the most frames a block asks for, and channels is how many channels interleave in each frame. arg is a copy of the argument given to Audio.start.
process(out, frames, startFrame) renders one block of frames interleaved frames into out, which arrives zeroed. startFrame is the block’s first frame on the engine’s clock, the one Audio.timeFrame reads.

process runs on the audio thread, which plays every sound. So it must never wait, and should never allocate. Read rings with receiveInto and snapshots with latestInto, into values made in init.

A first processor

This processor plays a 110 Hz hum. The main lane sets its level through a Snapshot, which holds the latest value published to it.

hum.as

import { Layout, Context } from "rive/host";
import { Audio, AudioHandle, AudioProcessor } from "rive/audio";
import { Snapshot } from "rive/shared";

export class Level {
    gain: f64 = 0.1;
}

export class Hum extends AudioProcessor<Snapshot<Level>> {
    level: Snapshot<Level>? = null;
    current: Level = new Level();
    channels: i32 = 2;
    phase: f64 = 0;
    step: f64 = 0;

    override init(sampleRate: f64, maxFrames: i32, channels: i32, level: Snapshot<Level>): void {
        this.level = level;
        this.channels = channels;
        this.step = 2 * Math.PI * 110 / sampleRate;
    }

    override process(out: StaticArray<f32>, frames: i32, startFrame: f64): void {
        this.level!.latestInto(this.current);
        let channels = this.channels;
        for (let i = 0; i < frames; i++) {
            let sample = <f32>(Math.sin(this.phase) * this.current.gain);
            for (let c = 0; c < channels; c++) {
                out[i * channels + c] = sample;
            }
            this.phase += this.step;
        }
    }
}

export class Humming extends Layout {
    level: Snapshot<Level> = Snapshot.create<Level>();
    hum: AudioHandle? = null;

    override init(context: Context): bool {
        this.level.publish(new Level());
        this.hum = Audio.start(() => new Hum(), this.level);
        return true;
    }
}

The snapshot is the processor’s argument, and a shared object, so it crosses by id: both lanes use the same snapshot. Publish a new Level from the main lane at any time, and the next block picks it up. Keep the AudioHandle in a field, since a handle that is collected stops its processor.

Rendering to a file

rive <dir> --render-audio runs the project with no window and no audio device. It writes what the processors render to a float wav file, for as long as the --advance steps add up to:

shell

rive . --render-audio=out.wav --advance=5s

The same run always writes the same file, so a processor can be checked against a reference. A render has all the time it wants, so no block is ever dropped for running long. --audio-rate=<Hz> and --audio-channels=<N> pick the format, 48000 Hz and 2 channels by default. Sounds played from audio assets are not in the file.

In a render, Audio.timeFrame follows the capture instead of a device. That is the same clock a processor’s startFrame reads, so a note the main lane stamps with a frame from Audio.timeFrame plays on that frame.

A processor renders the same samples on every native platform and in the audio worklets of Chrome, Firefox and Safari, with SIMD or without. That makes a rendered file a dependable reference for a test. The one exception is a processor that calls the fused multiply-adds that wasmSimd: relaxed allows (see Build Settings). Those can round differently by platform.

When a processor fails

A throw, a trap, or a block that runs past the hard limit ends a processor for good. A trap is an error the engine raises, such as for a float cast out of range. The hard limit is 2 blocks of audio by default. The processor leaves the mix and plays silence from then on, and the log says so once, naming its class:

log

processor Chorus failed: integer overflow

failed on the handle says why: the message a throw carried, or for a trap a text from the engine. That text differs between hosts, so check whether failed is null rather than matching its text.

To keep playing, check failed in advance and call Audio.start again. That makes a new processor and runs its init from scratch. A cause that lasts, such as a bad value in a snapshot, fails each new processor on its first block, and every start logs another failure line. So restart only a few times.

player.as

import { Layout, Context } from "rive/host";
import { Audio, AudioHandle, AudioProcessor } from "rive/audio";

export class Tone extends AudioProcessor<f64> {
    channels: i32 = 2;
    phase: f64 = 0;
    step: f64 = 0;

    override init(sampleRate: f64, maxFrames: i32, channels: i32, pitch: f64): void {
        this.channels = channels;
        this.step = 2 * Math.PI * pitch / sampleRate;
    }

    override process(out: StaticArray<f32>, frames: i32, startFrame: f64): void {
        let channels = this.channels;
        for (let i = 0; i < frames; i++) {
            let sample = <f32>(Math.sin(this.phase) * 0.1);
            for (let c = 0; c < channels; c++) {
                out[i * channels + c] = sample;
            }
            this.phase += this.step;
        }
    }
}

export class Player extends Layout {
    tone: AudioHandle? = null;
    restarts: i32 = 0;

    override init(context: Context): bool {
        this.tone = Audio.start(() => new Tone(), 220.0);
        return true;
    }

    override advance(seconds: f64): bool {
        let tone = this.tone!;
        let failed = tone.failed;
        if (failed != null && this.restarts < 3) {
            console.warn("tone failed, restarting: " + failed);
            this.restarts++;
            tone.release();
            this.tone = Audio.start(() => new Tone(), 220.0);
        }
        return true;
    }
}

Writing a real instrument

A processor that does real work needs compiled code. The log names the tier each lane module runs on, such as processors module tier: interp. On the interpreter, signal processing runs many times slower and soon overruns its blocks, which silences the processor. Pass --aot to rive to compile scripts ahead of time. As a measure, a physical model piano holding a ten note chord takes about 3 percent of a core compiled, and about half a core interpreted.

Debug bakes, which the live window plays, also give every lane fuel: the checks that let a browser end a lane. Fuel makes a processor’s blocks about a fifth to a third slower. An optimized bake, such as --optimize or a publish makes, gives lanes fuel only when the file sets wasmFuel.

Use SIMD, with a scalar twin

Mark the hot functions @unchecked, so their element accesses skip bounds checks in optimized builds. Give the inner loops a v128 path, where an f32x4 operation works on four f32 values at once. Guard that path with ASC_FEATURE_SIMD, and keep a scalar path beside it that sums in the same order, so both render the same samples. The raw pointer here needs an unsafe function.

fader.as

import { Layout, Context } from "rive/host";
import { Audio, AudioHandle, AudioProcessor } from "rive/audio";

// Scales samples in place, four at a time when the bake has SIMD.
@unchecked
unsafe function scale(out: StaticArray<f32>, count: i32, gain: f32): void {
    let i = 0;
    if (ASC_FEATURE_SIMD) {
        // A raw pointer is good for this call only.
        let base = changetype<usize>(out);
        let wide = f32x4.splat(gain);
        for (; i + 4 <= count; i += 4) {
            let at = base + (<usize>i << 2);
            v128.store(at, f32x4.mul(v128.load(at), wide));
        }
    }
    // The scalar twin finishes the block, or does all of it without SIMD,
    // with the same multiply, so both paths render the same samples.
    for (; i < count; i++) {
        out[i] = out[i] * gain;
    }
}

export class Fader extends AudioProcessor<f64> {
    channels: i32 = 2;
    phase: f64 = 0;
    step: f64 = 0;

    override init(sampleRate: f64, maxFrames: i32, channels: i32, pitch: f64): void {
        this.channels = channels;
        this.step = 2 * Math.PI * pitch / sampleRate;
    }

    override process(out: StaticArray<f32>, frames: i32, startFrame: f64): void {
        let channels = this.channels;
        for (let i = 0; i < frames; i++) {
            let sample = <f32>Math.sin(this.phase);
            for (let c = 0; c < channels; c++) {
                out[i * channels + c] = sample;
            }
            this.phase += this.step;
        }
        scale(out, frames * channels, 0.1);
    }
}

export class Fading extends Layout {
    fader: AudioHandle? = null;

    override init(context: Context): bool {
        this.fader = Audio.start(() => new Fader(), 220.0);
        return true;
    }
}

Our compiler has only part of wasm SIMD. SIMD covers how to write it.

Has	Lacks
`v128` loads, stores, bitwise operations, shuffles and `any_true`	shifts and rounding
splats, and lane reads and writes, on `f32x4` and `i32x4`	conversions between float and integer lanes
arithmetic, comparisons, `min`, `max`, `pmin`, `pmax` and `sqrt` on `f32x4`	`lt`, `gt`, `le` and `ge` on `i32x4`
`add`, `sub`, `mul`, `eq`, `ne`, `all_true` and `bitmask` on `i32x4`	loads that splat or widen
only shuffles on `i16x8`, and only shuffles and swizzles on `i8x16`	`f64x2` and `i64x2`

Rules for the audio thread

A float to integer cast can trap. A cast such as <u32>x traps when x is NaN or out of range, and the trap fails the processor. A cast to an 8 or 16 bit type checks the range of the 32 bit type of the same sign, then wraps, so <u16>x still traps when x is -1 or less. A delay line is the usual case, since its read position, the write position less the delay, goes negative. For a line whose length is a power of two, <i32>Mathf.floor(pos) & (length - 1) wraps a negative position instead, though a NaN or infinite delay still traps. Bring such values into range with a comparison before the cast, since min and max pass a NaN through.
A raw pointer is good for one call only. Objects may move between calls, so take changetype<usize>(array) at the top of process, and never keep it in a field.
The block size is the host’s. frames changes from call to call, up to the maxFrames that init was given. A core that wants fixed blocks renders whole blocks of its own, and hands out what each call asks for.
The sample rate is the output’s. That is usually the device’s own. Most devices run at 44100 or 48000 Hz, but a headset during a call can drop to 8000, and an audio interface can run at 192000. Read it from init’s sampleRate rather than assuming one.
A heavy processor wants a bigger buffer. On a loaded machine, a bigger device buffer keeps it from dropping out, at the cost of latency. Pass rive <dir> --audio-buffer=2048, or set audioBuffer: 2048 in rive.yaml. Events stamped against Audio.timeFrame then need to be sent more than a buffer ahead.
Make everything up front. Make every object in the constructor or in init, and nothing in process.
Pass large data as a Blob. Send a sample bank or a model as a shared Blob from rive/shared. It crosses by id, where an ArrayBuffer in the start argument is copied. Read it with readInto, which allocates nothing. For samples, await source.toBlob(Audio.processorSampleRate(), Audio.processorChannels()) decodes an audio asset on a job thread straight into one, in the format the processor renders. For audio too long to hold whole, await source.stream(rate, channels) decodes as it plays, and the processor reads it through the stream's reader. A blob asset is not a Blob yet; Large data says how to load a large one.

Cheap math

Wasm has no instructions for sines, exponentials or logarithms. So Mathf.sin, exp, log, pow and tanh, and their Math versions, run full precision code inside the module. Compiled, Mathf.sin takes about 3 ns on an Apple M4 Pro, and pow and tanh about twice that. On the interpreter each takes about 15 to 20 times as long.

One call per sample costs little, but a bank of voices, or a curve worked out for every sample, adds up:

Keep oscillator phases in turns, where 1 is a whole cycle, and use a polynomial such as the one below.
Work out smoothing coefficients when a knob changes, rather than on every block.
Skip a compressor’s log and pow while the signal stays under its threshold.

Mathf.sqrt, abs, floor, min and max, by contrast, are single instructions.

dsp.as

// A sine of a phase in turns, where 1 is a whole cycle, within 7.4e-7 of
// Math.sin for about a quarter of what Mathf.sin costs compiled.
export function sinTurn(turns: f32): f32 {
    let x = turns - floor<f32>(turns + 0.5);
    let a = abs<f32>(x);
    x = copysign<f32>(min<f32>(a, 0.5 - a), x);
    let x2 = x * x;
    return x * (6.28316402 + x2 * (-41.3371429 + x2 * (81.3407669 + x2 * -70.9933929)));
}

Sleeping when silent

A processor that is silent most of the time, such as an instrument between notes, still has process called for every block. It runs its whole loop unless it puts itself to sleep:

While asleep, return from process at once. out arrives zeroed, so the block plays silence.
Wake when the ring has records, which its used shows without taking them. Or wake when the snapshot’s seq has moved and its values differ from the ones in use. seq moves on every publish, even of the same values. So when the main lane publishes every frame, compare the values too. When it publishes only when a knob changes, a moved seq is enough.
Go to sleep once the tail is too quiet to hear.
Publish meters as zero before going to sleep, or the main lane keeps showing the last level.
process is still called while the processor sleeps, with the right startFrame. So a processor that keeps its own clock, such as a sequencer, takes its time from startFrame, and wakes when its next event is due.

Asleep, a block reads used and seq, and decodes the knobs to compare them only after a publish. So it costs close to what an empty processor does. This pluck imports sinTurn from the dsp.as above:

pluck.as

import { Layout, Context } from "rive/host";
import { Audio, AudioHandle, AudioProcessor } from "rive/audio";
import { Ring, Snapshot } from "rive/shared";
import { sinTurn } from "./dsp";

export class Note {
    gain: f32 = 0;
}

export class Knobs {
    decay: f32 = 0.9998;
}

export class Meter {
    level: f32 = 0;
}

export class Inputs {
    notes: Ring<Note> = Ring.create<Note>(256);
    knobs: Snapshot<Knobs> = Snapshot.create<Knobs>();
    meter: Snapshot<Meter> = Snapshot.create<Meter>();
}

export class Pluck extends AudioProcessor<Inputs> {
    inputs: Inputs? = null;
    note: Note = new Note();
    knobs: Knobs = new Knobs();
    next: Knobs = new Knobs();
    meter: Meter = new Meter();
    knobsSeq: f64 = -1;
    asleep: bool = false;
    level: f32 = 0;
    phase: f32 = 0;
    step: f32 = 0;
    channels: i32 = 2;

    override init(sampleRate: f64, maxFrames: i32, channels: i32, inputs: Inputs): void {
        this.inputs = inputs;
        this.channels = channels;
        this.step = <f32>(220 / sampleRate);
    }

    woken(inputs: Inputs): bool {
        if (inputs.notes.used != 0) {
            return true;
        }
        let seq = inputs.knobs.seq;
        if (seq == this.knobsSeq) {
            return false;
        }
        this.knobsSeq = seq;
        inputs.knobs.latestInto(this.next);
        return this.next.decay != this.knobs.decay;
    }

    override process(out: StaticArray<f32>, frames: i32, startFrame: f64): void {
        let inputs = this.inputs!;
        if (this.asleep && !this.woken(inputs)) {
            return;
        }
        this.asleep = false;
        inputs.knobs.latestInto(this.knobs);
        while (inputs.notes.receiveInto(this.note)) {
            this.level = this.note.gain;
        }
        let channels = this.channels;
        for (let i = 0; i < frames; i++) {
            let sample = sinTurn(this.phase) * this.level;
            for (let c = 0; c < channels; c++) {
                out[i * channels + c] = sample;
            }
            this.phase += this.step;
            this.phase -= floor<f32>(this.phase);
            this.level *= this.knobs.decay;
        }
        this.asleep = this.level < 1e-5;
        this.meter.level = this.asleep ? 0 : this.level;
        inputs.meter.publish(this.meter);
    }
}

export class Plucking extends Layout {
    inputs: Inputs = new Inputs();
    knobs: Knobs = new Knobs();
    pluck: AudioHandle? = null;
    level: f32 = 0;
    frame: i32 = 0;

    override init(context: Context): bool {
        this.pluck = Audio.start(() => new Pluck(), this.inputs);
        return true;
    }

    override advance(seconds: f64): bool {
        // Published every frame, as a tuning panel might.
        this.inputs.knobs.publish(this.knobs);
        this.frame++;
        if (this.frame % 200 == 2) {
            let note = new Note();
            note.gain = 0.3;
            this.inputs.notes.send(note);
        }
        let meter = this.inputs.meter.latest();
        if (meter != null) {
            this.level = meter.value.level;
        }
        return true;
    }
}

Processors in browsers

Processors play in real time only in a web build with threads, in an audio worklet. Without threads, a processor fails as it starts, and failed on its handle says why. A processor’s init runs in the worklet too, on the audio render thread, so heavy setup there delays the mix.

In a browser, startFrame and the sample rate that init gets come from the page’s audio output. Audio.timeFrame and Audio.sampleRate do not report that output’s frames and rate. So there, a note the main lane stamps with a frame from Audio.timeFrame does not play on that frame. A file baked with SIMD the browser lacks runs none of its scripts, and says so on the console; wasmSimd: off bakes a file without SIMD. The other platform differences are on the Threads page.

For every detail of the API, run rive docs animascript/api/audio. The full guide is rive docs animascript/threading.