Audio

Intro

MotorCortex provides Audio capabilities out of the box. The way Audio has been implemented in MotorCortex is based on the Web Audio API. All Clips and AudioClips that belong to the same Incidents’ tree, all have Audio capabilities, have their own Audio Nodes Set and all Audio Node Sets are interconnected, to form an audio routing graph targeting the speakers of the user.

Each of these Audio Node Sets consists of a chain of AudioNodes:

• A gain Node
• A pan Node
• The master Node

The output of the one Set is connected to the input of the other to form a tree of Audio Nodes with 1:1 correspondence to the tree formed by their Incidents.

The master gain of this audio routing graph is controlled by the root Clip and you can control it via the setVolume(level) method it provides.

Keep in mind that for each audioSource loaded on any of the Nodes of the diagram an Audio Node Set will also be created and it will only handle the pan, gain and master volume of the audioSource alone.

AudioClip

As you should already know HTMLClip has audio capabilities as it has an Audio Node Set of its own. Same stands even for the BrowserClip, so all custom Clips also support audio. AudioClip is a type of Clip that has no visual context at all but it only has audio capabilities, owning an Audio Node Set.

If you want to create a plugin that:

• Exposes easy-to-use audio Incidents
• Exposes ready to use sounds
• Extends the audio capabilities (e.g. by adding more audio effects), working on the Audio Node Set
• Provides an easier interface on top of the native audio Incidents of MotorCortex
• Provides complex sound compilations based on user preferences then extending AudioClip is a great way to start it.

buildTree

The only method that you need to overwrite when extending AudioClip is the buildTree method, exactly as when you’re extending an HTMLClip to develop an Animation, only this time on buildTree you will place strictly Audio Incidents on your clip’s timeline as these are the only compatible with it.

audioSources getter

As on HTMLClip same with AudioClips, you can overwrite the get audioSources to define (and load in your context) the full list of audio sources you want to use.

this.audioNodeSet

this.audioNodeSet available anywhere in the AudioClip Class, represents exactly the Audio Node Set of the Clip.

this.audioNodeSet is a Class providing the following properties and methods:

• input: represents the input Node of the Set. You can you it to connect other Audio Nodes to it
• output: represents the output of the Audio Set which be default gets connected to the parent Audio Node Set of the Clip
• pannerNode: the Audio Node handling the pan
• gainNode: the Audio Node handling the gain Effects applied
• connect(AudioNode): a method that will connect the full Audio Node Set’s output to the specified AudioNode
• disconnect(): disconnects the output of the Audio Node Set from the AudioNode it’s connected with

Effects

First some facts. By extending AudioClip the class that will be produced will still support the Audio Effects (pan, gain), targeting audioSources of it. Developing new Audio Effects follows the exact same logic with the simple - non Audio Effects. A selector will select the audioSources, the animatedAttrs will define which of the attributes to affect and under the hood MotorCortex will create one MonoIncident per element and attribute. Keep in mind that when working with Audio, extending Effect will have on the this.element property the audioSource itself, which has the following properties:

• soundLoaded: boolean, indicates if the sound has been loaded or not
• buffer: the AudioBuffer of the sound
• audioNodeSet: the Audio Node Set of the specific sound
• startValues: an object holding the start values for gain and pan (if provided by the user)

Custom Audio Plugins

The built-in audio system described above works with audio files (mp3, wav, etc.) played through the Web Audio API. But there are cases where a plugin needs its own audio engine — for example, real-time synthesis, MIDI playback, or integration with third-party audio libraries.

For these cases, MotorCortex provides the audio: "custom" plugin type. Instead of using the internal AudioClip, MC instantiates the plugin's own Clip class as the audio clip. The plugin takes full ownership of audio creation, routing, and playback.

How it works

A custom audio plugin:

1. Exports a Clip that extends ExtendableClip (not BrowserClip or AudioClip)
2. Creates its own ExtendableContextHandler subclass for managing audio sources and selector resolution
3. Sets audio: "custom" in the plugin manifest
4. Uses MC's shared audioContext for Web Audio node creation so all nodes live on the same audio graph

import { ExtendableClip, ExtendableContextHandler, audioContext } from "@donkeyclip/motorcortex";

class MyAudioClip extends ExtendableClip {
  constructor(attrs, props) {
    super(attrs, props);
    const handler = new MyContextHandler(props.audioSources);
    handler.context.initParams = props.initParams;
    this.ownContext = handler.context;
  }
}

class MyContextHandler extends ExtendableContextHandler {
  constructor(sources) {
    super();
    // Create and register audio sources
    // Resolve selectors (~#id, ~.class)
    this.setContext({ contextLoaded: true, audio: true /* ... */ });
  }
}

The plugin manifest uses audio: "custom":

export default {
  npm_name: "@my-org/my-audio-plugin",
  version: "1.0.0",
  incidents: [
    { exportable: MyPlayback, name: "Playback" },
    { exportable: MyEffect, name: "MyEffect" },
  ],
  Clip: { exportable: MyAudioClip },
  audio: "custom",
};

audioContext

MC creates a single, shared AudioContext used by the internal audio system. Custom audio plugins must use this same context — creating a separate AudioContext will cause InvalidAccessError when nodes from different contexts try to connect.

Import it directly:

import { audioContext } from "@donkeyclip/motorcortex";

// Use it for creating Web Audio nodes
const gainNode = audioContext.createGain();

// Or pass it to third-party libraries that accept an AudioContext
Tone.setContext(audioContext);

Example

mc-tone is a custom audio plugin that integrates Tone.js for real-time music synthesis. It demonstrates the full pattern: ExtendableClip, ExtendableContextHandler, custom audio routing, Playback and Effect incidents, and use of MC's shared audioContext.

For a deeper understanding of ExtendableClip and ExtendableContextHandler — the foundation on which all Clip types are built — see Exotic Clips.

Limitations

As the full Audio capability is handled directly by the Web Audio API and the audio routing graph explained above any audio source that can not be directly connected to it should not exist within the context of a Clip.

For example if a Video plugin attaches a video on the Clip the audio output of this video must get attached to the audio routing graph otherwise the gain of it will not be controlled by the master volume control of the Clip.