midi au slider

AUv3 MIDI Plugins for iOS Developers

Abstract: this text provides tips, background information and considerations for iOS developers wishing to create AUv3 MIDI plugins or to support MIDI plugins in AU host software. It assumes familiarity with ‘normal’ AUv3 plugin development.

Introduction

One of the most interesting music related novelties quietly introduced in iOS11 is the ability for AUv3 Extension plugins to send MIDI messages to their host. It was first demonstrated in Apple’s WWDC’17 CoreAudio video: 

https://developer.apple.com/videos/play/wwdc2017/501/

(Jump to 39:00 to hear the explanation and 44:00 to see a demo)

Apparently, Apple’s main use case for the feature was to support AU instruments that have their own virtual keyboard (or other performance UI). Users can then record their performance in any compatible AU host.

However, the feature also opens doors for custom sequencers and other MIDI generators that don’t necessarily need their own sound source or synth engine. In my opinion an even richer application angle than the one shown in the WWDC video as it supports my personal belief that musicians should be able to mix and match sequencers and sound sources – rather than having a sequencer tied to a sound source.

The Rozeta sequencer suite is one first practical example of where I see lots of potential for MIDI AU plugins.

A. For Host Developers

AUv3 MIDI plugins are essentially identical to normal AU instrument plugins. They share the ‘aumu‘ type, provide audio I/O buses (which they can ignore) and expect render calls at frequent intervals. So if you already have a working infrastructure for AU instruments, you’re 80% done.

On top of the regular AU functionality, AU MIDI plugins are given a ‘block’ by the host which they can invoke from the regular render call. This block, called midiOutputEventBlock, is used by the plugin to send a list of MIDI packets back to the host.

Loading AU MIDI plugins

Hosts typically take the following steps to load and handle AU MIDI extensions:

  1. instantiate the AU extension
  2. check for the existence of “midiOutputNames“. This is an array of NSStrings. If this field is nil there will not be MIDI output from this plugin. If it is set it contains the name(s) of the virtual MIDI output cable(s). This is your indication that the plugin intends to send MIDI to the host. Note: Rozeta only uses one output cable.
  3. before any call to allocateRenderResourcesAndReturnError on the plugin the host must provide these three important blocks:
    • midiOutputEventBlock
    • musicalContextBlock
    • transportStateBlock
  4. during each call to the plugin’s renderBlock, the plugin can call the cached midiOutputEventBlock to send MIDI events back to the host. Only one call will be made per virtual MIDI cable, so all MIDI events for this buffer duration will be combined into a single MIDI packet list. This list has the same format as the MIDI packet list sent from the host to the AU.
  5. important: information passed to the plugin using the transportStateBlock and musicalContextBlock must be complete and as accurate as possible, as these will be used by the plugin to determine the host’s sequencer state and how to sync MIDI events to the host’s timeline! I recommend using double precision floats for any time related calculation and to be extra careful with rounding.

Detecting AU MIDI plugins

As described above, AU MIDI plugins can be easily identified after the AUv3 has been instantiated. Simply look if the field “midiOutputNames” is present. If it is, you’re dealing with an AU MIDI plugin.

However, for hosts it can also be interesting to know before instantiating if an extension is capable of sending MIDI. E.g. for creating lists of available MIDI plugins for their users to choose from.

There is an official method for identifying MIDI-capable plugins: each AU extension has an optional boolean field named “hasMIDIOutput“. Unfortunately this mechanism is useless as it defaults to YES, meaning virtually all AU instrument plugins in the field today report YES when queried.

Proposal: as a workaround I am adding the tag “MIDIOnly” to my plugins’ Component Description tags, which can be checked using the array field “allTagNames“. I am aware that this is neither an optimal nor an official solution, but it’s a temporary stopgap I’m proposing until Apple give us a robust alternative.

 

B. For plugin developers

AU Instrument vs. AU MIDI Processor

In theory there are 2 valid AU MIDI formats. Apple presented the “aumu” type in its WWDC’17 CoreAudio video. Also known as “kAudioUnitType_MusicDevice“, this is the regular AU instrument plugin type we already knew. It allows plugins to output audio, MIDI or both (Moog Model 15 is an example of a plugin that does both). This is also the format implemented in Apple’s Garageband, and most other AU MIDI compatible hosts.

Additionally, there exists another format: “aumi“, or “kAudioUnitType_MIDIProcessor“, which can be found in the CoreAudio header files. While this sounds like a more suitable format for MIDI plugins, it comes with several potential problems and risks. First off, it’s unofficial, unsupported and completely undocumented by Apple. That means it could be disabled, deprecated and disappear at any moment.

Besides that, it also likely doesn’t allow any audio buses. While pure MIDI plugins don’t need audio, these I/O buses are (to my knowledge) the sole way for plugins to detect the host’s samplerate. Since AU MIDI timing is calculated in frames this means that MIDI Processor plugins can not safely do any time-based MIDI processing. Perhaps there is a workaround for this, but due to its undocumented nature this is currently unknown.

For these key reasons, it was decided to base Rozeta and the first AU MIDI hosts on the Apple-endorsed “aumu” type.

MIDI AU Basics

As explained above, AU MIDI plugins are quite simply stripped AU Instrument plugins with some added functionality for sending MIDI back to the AU host. If you already have a working AUv3 instrument you have a good starting point. You can ignore the audio processing part of the code, but you still need to set up the I/O buses to be able to access the current samplerate of your host. 

  • Set up the midiOutputNames array property of your AudioUnit: this is an array of strings and is used by hosts to determine if your plugin intends to send MIDI. Every item in the array represents the name of a virtual MIDI cable (Rozeta plugins only use a single virtual cable).
  • during any call to “allocateRenderResourcesAndReturnError” you must cache (or re-cache) three blocks provided to you by the host:
    • midiOutputEventBlock
    • musicalContextBlock
    • transportStateBlock
  • whenever the host calls your renderBlock you can build a MIDI packet list (using the same format as the MIDI messages sent from the host to the AU). When you’re done, send them to the host – in a single call per virtual cable – using the cached midiOutputEventBlock
  • Proposal: to help hosts identify MIDI-only AU plugins, I add the tag “MIDIOnly” to my plugins’ tag array in their plist (in the ComponentDescription). Note that this is an unofficial workaround for the current lack of official methods for identifying MIDI AU plugins.  

That’s it in a nutshell. As you can see it’s only a small extra step to make an AU MIDI plugin if you’re already familiar with the AUv3 instrument architecture.

 

Acknowledgement

I would like to thank Jonatan Liljedahl (of Kymatica/AUM fame) for pioneering the MIDI AU format with me. Not only was AUM the first host to support AU MIDI plugins with full MIDI routing, but he also put considerable time and brainpower into thinking about the best (and most futureproof) ways to turn this mostly undocumented format into something meaningful and useful for iOS.