--- a/StreamProcessing/StreamProcessing.html Fri Oct 28 18:27:39 2011 +1300
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,778 +0,0 @@
-<!DOCTYPE HTML>
-<html>
-<head>
-<title>MediaStream Processing API</title>
-<link rel="stylesheet" href="main.css">
-</head>
-<body>
-
-<div class="head">
- <h1>MediaStream Processing API</h1>
- <h2>Draft Proposal</h2>
- <dl><dt>Editor:</dt><dd>Robert O'Callahan, Mozilla Corporation <robert@ocallahan.org></dd>
-</div>
-
-<h2>Status of this Document</h2>
-<p>This document is a draft specification proposal with no official status. Send comments to the <a href="mailto:public-audio@w3.org">W3C audio mailing list</a>, or <a href="mailto:robert@ocallahan.org">Robert O'Callahan</a>. It is inappropriate to cite this document except as a work in progress.
-
-<h2>Abstract</h2>
-
-<p>A number of existing or proposed features for the Web platform deal with continuous real-time media:
-<ul>
-<li>HTML media elements
-<li>Synchronization of multiple HTML media elements (e.g. proposed HTML MediaController)
-<li>Capture and recording of local audio and video input (e.g. proposed HTML Streams)
-<li>Peer-to-peer streaming of audio and video streams (e.g. proposed WebRTC and HTML Streams)
-<li>Advanced audio APIs that allow complex mixing and effects processing (e.g. Mozilla's AudioData, Chrome's AudioNode)
-</ul>
-Many use-cases require these features to work together. This proposal makes HTML Streams the foundation for integrated Web media processing by creating a mixing and effects processing API for HTML Streams.
-
-<h2>Table of Contents</h2>
-
-<ol id="toc">
- <li><a href="#introduction">1. Introduction</a>
- <ol>
- <li><a href="#scenarios">1.1. Scenarios</a>
- </ol>
- <li><a href="#mediastreams">2. MediaStreams</a>
- <ol>
- <li><a href="#scenarios">2.1. The Semantics Of MediaStreams</a>
- <li><a href="#media-formats">2.2. Media Formats</a>
- <li><a href="#mediastream-extensions">2.3. MediaStream Extensions</a>
- </ol>
- <li><a href="#media-elements">3. Media Elements</a>
- <ol>
- <li><a href="#media-element-extensions">3.1. Media Element Extensions</a>
- <li><a href="#audio-element-extensions">3.2. Audio Element Extensions</a>
- </ol>
- <li><a href="#stream-mixing-and-processing">4. Stream Mixing And Processing</a>
- <ol>
- <li><a href="#time-varying-attributes">4.1. Time-varying Attributes</a>
- <li><a href="#processedmediastream">4.2. ProcessedMediaStream</a>
- <li><a href="#mediainput">4.3. MediaInput</a>
- <li><a href="#worker-processing">4.4. Worker Processing</a>
- </ol>
- <li><a href="#built-in-processing-engines">5. Built-In Processing Engines</a>
- <ol>
- <li><a href="#default-processing-engine">5.1. Default Processing Engine</a>
- <li><a href="#lastinput-processing-engine">5.2. "LastInput" Processing Engine</a>
- </ol>
- <li><a href="#media-graph-considerations">6. Media Graph Considerations</a>
- <li><a href="#canvas-recording">7. Canvas Recording</a>
- <li><a href="#implementation-considerations">8. Implementation Considerations</a>
- <li><a href="#examples">9. Examples</a>
-</ol>
-
-<h2 id="introduction">1. Introduction</h2>
-
-<p>The ideas here build on <a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html">Ian Hickson's proposal for HTML Streams</a>, adding features partly inspired by <a href="https://wiki.mozilla.org/Audio_Data_API"> the Mozilla audio API</a> and <a href="http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html">the Chrome audio API</a>. Unlike previous audio API proposals, the API presented here integrates with proposed API for media capture from local devices, integrates with proposed API for peer-to-peer media streaming, handles audio and video in a unified framework, incorporates Worker-based Javascript audio processing, and specifies synchronization across multiple media sources and effects. The API presented here does not include a library of "native" effects; those should be added as new "named effects" (described below), perhaps as a "level 2" spec.
-
-<p>The work here is nascent. Until a prototype implementation exists, this proposal is likely to be incomplete and possibly not even implementable.
-
-<h3 id="scenarios">1.1. Scenarios</h3>
-
-<p>These are concrete usage scenarios that have helped guide the design of the API. They are higher-level than use-cases.
-
-<ol>
-<li>Play video with processing effect applied to the audio track (e.g. high-pass filter)
-<li>Play video with processing effects mixing in out-of-band audio tracks (in sync) (e.g. mixing in an audio commentary with audio ducking)
-<li>Capture microphone input and stream it out to a peer with a processing effect applied to the audio (e.g. XBox 360 chat with voice distortion)
-<li>Capture microphone input and visualize it as it is being streamed out to a peer and recorded (e.g. Internet radio broadcast)
-<li>Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record (e.g. Internet radio broadcast)
-<li>Receive audio streams from peers, mix them with spatialization effects, and play (e.g. live chat with spatial feature)
-<li>Seamlessly chain from the end of one input stream to another (e.g. playlists, audio/video editing)
-<li>Seamlessly switch from one input stream to another (e.g. adaptive streaming)
-<li>Synthesize samples from JS data (e.g. game emulators or MIDI synthesizer)
-<li>Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking (e.g. game sound effects)
-<li>Trigger a sound sample to be played through the effects graph at a given time (e.g. game sound effects)
-<li>Capture video from a camera and analyze it (e.g. face recognition)
-<li>Capture video and audio, record it to a file and upload the file (e.g. Youtube upload)
-<li>Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast", or composite multiple video sources with effects into a single canvas then record)
-</ol>
-
-<h2 id="mediastreams">2. MediaStreams</h2>
-
-<h3 id="the-semantics-of-mediastreams">2.1. The Semantics Of MediaStreams</h3>
-
-<p>The description of MediaStreams here extends and must remain compatible with
-<a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html#stream-api">HTML MediaStreams</a>.
-
-<p>Each MediaStream DOM object has an underlying media stream. The underlying media streams form a graph;
-some streams (represented by ProcessedMediaStream DOM objects) can take other streams as inputs and compute an output
-stream.
-
-<p>To avoid interruptions due to script execution, script execution can overlap with media stream processing;
-media streams continue to play and change state during script execution. However, to simplify the DOM programming model,
-we limit the interaction of MediaStream DOM objects with their underlying media streams. Specifically:
-<ul>
-<li>Changes to the MediaStream and MediaInput DOM objects are batched together between <em>stable states</em> (as defined by the HTML spec, while script is not running), and propagate as an atomic change to the media stream graph to take effect
-after some delay.
-This ensures that incomplete changes to the media stream graph during script execution do not result in transient glitches in the output, and that scripted graph changes do not interfere with concurrent media processing.
-<div class="example">Thus the following script would never cause an interruption in audio output, since no stable state occurs between the two volume changes:
-<pre><code> stream.inputs[0].volume = 0;
- if (needToPlay()) {
- stream.inputs[0].volume = 1.0;
- }</code></pre>
-</div>
-
-<p class="todo">Specify exactly which attributes (and methods) are subject to this regime, possibly extending to
-attributes and methods already defined in HTML for media elements etc.
-<li>State changes from the media stream graph are only propagated back to the MediaStream DOM objects during a <em>stable state</em>. This ensures that between stable states (usually, during the execution of a script event handler), MediaStream DOM APIs always reflect a consistent snapshot of the state of the media stream graph.
-</ul>
-In this spec, references to <code>MediaStream</code>s and <code>MediaInput</code>s refer to the DOM-visible state, and references to <em>media streams</em> and <em>input ports</em> refer to the underlying real-time media stream graph.
-
-<p>A stream is an abstraction of a time-varying video and/or audio signal. At a given point in time, a media stream can be <em>blocked</em>, that is, not playing for some reason. All non-blocked streams play at the same constant rate --- real time. Streams cannot be seeked or played at a rate other than real time. For convenience we define a stream's "current time" as the duration it has played since it was created, but (unless blocking is involved) streams do not have independent timelines; they are synchronized.
-
-<p>At the implementation level, and when processing media data with Workers, we assume that each stream has a buffered window of media data available, containing the sample it is currently playing (or that it will play the next time it unblocks). This buffer defines the stream's contents into the future.
-
-<p>A stream can be in the <em>finished</em> state. A finished stream is always blocked and can
-never leave the finished state --- it will never produce any more content.
-
-<p>A stream that has no consumers must block. Stream consumers defined in this specification are media elements and <code>ProcessedMediaStream</code>s (see below). This avoids situations where streams that aren't connected
-to anything and could be garbage-collected have to be kept running because their stopping could possibly be observed (e.g. because a Worker is being used to generate data for the stream and the Worker can observe whether the <code>ProcessMediaEvent</code> is being sent). A muted audio element can be used as a dummy sink if necessary.
-
-<div class="note">
-<p>We do not allow streams to have independent timelines (e.g. no adjustable playback
-rate or seeking within an arbitrary stream), because that can lead to a single stream being
-consumed at multiple different "current times" simultaneously, which requires either unbounded buffering
-or multiple internal decoders and buffers for a single stream. It seems simpler and more
-predictable for performance to require authors to create multiple streams (if necessary) and change
-the playback rate in the original stream sources to handle such situations.
-<p>For example, consider this hard case:
-<ul>
-<li>Three media element input streams: http://slow, http://fast, and http://fast2
-<li>http://slow is mixed with http://fast
-<li>http://fast is mixed with http://fast2
-</ul>
-Does the http://fast stream have to provide data at two different offsets? This spec's answer is no.
-This leads us to the conclusion that if a stream feeds into a blocked mixer, then it itself must be
-blocked. Since obviously a mixer with a blocked input must also be blocked, the entire graph of
-connected streams block as a unit. This means that the mixing of http://fast and http://fast2 will
-be blocked by delays in http://slow in the above scenario.
-<p>Authors can avoid this by explicitly splitting streams that may need to progress at
-different rates --- in the above case, by using two separate media elements each loading
-http://fast. The HTML spec encourages implementations to share cached media data between
-media elements loading the same URI.
-</div>
-
-<p>A media stream contains video and audio tracks. Tracks can start and end at any time. Each track
-contains a stream of audio or video data.
-
-<h3 id="media-formats">2.2 Media Formats</h3>
-
-<p>This spec mostly treats the formats used for stream audio and video data
-as an implementation detail. In particular, whether stream buffers are compressed or uncompressed, what compression
-formats might be used, or what uncompressed formats might be used (e.g. audio sample rates, channels, and sample
-representation)
-are not specified, and are not directly observable. An implementation might even support changing formats over
-time within a single stream. Media data is implicitly resampled as necessary, e.g. when mixing streams with different formats. Non-normative suggestions for resampling algorithms will be provided in section 7.
-
-<p>Built-in audio processing filters guarantee that if all the audio inputs constantly have the same uncompressed format
-(same audio sample rate and channel configuration), the audio output will have the same format and there will be no unnecessary resampling.
-
-<p>When samples are exposed to a Worker for processing, the user-agent chooses a fixed uncompressed audio format (sample rate and channel configuration) for its inputs and outputs; see section 4.4.
-
-<p class="todo">However, suggested resampling algorithms will be provided in an appendix.
-
-<h3 id="mediastream-extensions">2.3 MediaStream Extensions</h3>
-
-<pre><code>partial interface MediaStream {
- readonly attribute double currentTime;
-
- ProcessedMediaStream createProcessor(optional in DOMString namedEffect);
- ProcessedMediaStream createWorkerProcessor(in Worker worker);
-};</code></pre>
-
-<p>The <code>currentTime</code> attribute returns the amount of time that this <code>MediaStream</code> has played since it was created.
-
-<p>The <code>createProcessor(namedEffect)</code> method returns a new <code>ProcessedMediaStream</code> with this <code>MediaStream</code> as its sole input.
-The <code>ProcessedMediaStream</code> is configured with a built-in processing engine named by <code>namedEffect</code>,
-or the default processing engine if <code>namedEffect</code> is omitted. If <code>namedEffect</code> is not supported
-by this user-agent, <code>createProcessor</code> returns null. User-agents adding nonstandard named effects should use
-vendor prefixing, e.g. "MozUnderwaterBubbles". The stream's <code>autofinish</code> attribute
-is set to true.
-
-<p>The <code>createWorkerProcessor(worker)</code> method returns a new <code>ProcessedMediaStream</code> with this <code>MediaStream</code> as its sole input.
-The stream is configured with <code>worker</code> as its processing engine.
-The stream's <code>autofinish</code> flag is set to true.
-
-<p class="todo">Add event handlers or callback functions for all finished and blocking state changes?
-
-<h2 id="media-elements">3. Media Elements</h2>
-
-<h3 id="media-element-extensions">3.1 Media Element Extensions</h3>
-
-<p>We extend HTML media elements to produce and consume streams. When an HTML media element
-produces a stream, it acts as a resource loader and control mechanism; the stream consists of whatever the
-media element is currently playing. When a media element consumes a stream, it acts a playback
-mechanism for the stream.
-
-<pre><code>partial interface HTMLMediaElement {
- readonly attribute MediaStream stream;
-
- MediaStream captureStream();
- MediaStream captureStreamUntilEnded();
- attribute boolean captureAudio;
-
- attribute any src;
-};</pre></code>
-
-<p>The <code>stream</code> attribute returns a stream which always plays whatever the element is playing. The stream is blocked while the media element is not playing. It is never finished, even when playback ends
-(since the element might load a new resource or seek within the current resource). The <code>stream</code> attribute for a given element always returns
-the same stream. When the stream changes to blocked, we fire the <code>waiting</code> event for the media element,
-and when it changes to unblocked we fire the <code>playing</code> event for the media element.
-
-<p class="XXX">Currently the HTML media element spec says that <code>playing</code> would fire on an element
-that is able to play except that a downstream <code>MediaController</code> is blocked. This is incompatible
-with the above. I think that part of the HTML media spec should be changed so that only elements that are actually
-going to play fire <code>playing</code>.
-
-<p>The <code>captureStream()</code> method returns a new <code>MediaStream</code> that plays the same audio and video as <code>stream</code>.
-<code>captureStream()</code> sets the <code>captureAudio</code> attribute to true.
-
-<p>The <code>captureStreamUntilEnded()</code> method returns a new <code>MediaStream</code> that plays the same audio and video as <code>stream</code>, until the element next reaches
-the "ended playback" state, at which point this stream will enter the finished state.
-<code>captureStreamUntilEnded()</code> sets the <code>captureAudio</code> attribute to true.
-
-<p>While the <code>captureAudio</code> attribute is true, the element does not produce direct audio output.
-Audio output is still sent to <code>stream</code> and the resource stream. This attribute is NOT reflected into the DOM. It
-is initially false.
-
-<p>The <code>src</code> attribute is extended to allow it to be set to a <code>MediaStream</code>. The element
-will play the contents of the given stream. This is treated as a live stream; seeking and <code>playbackRate</code>
-are not supported.
-
-<p>The <code>URL.createObjectURL(stream)</code> method defined for HTML MediaStreams can create a URL to be
-used as a source for a media element. Setting a media element to use this source URL is equivalent to setting
-the media element <code>src</code> to the given stream.
-
-<h3 id="audio-element-extensions">3.2 Audio Element Extensions</h3>
-
-<p>We add an <code>Audio</code> constructor taking a <code>MediaStream</code> as a parameter.
-This sets the initial <code>src</code> to the stream.
-
-<pre><code>[NamedConstructor=Audio(MediaStream src)]
-partial interface HTMLAudioElement {
-}</code></pre>
-
-<h2 id="stream-mixing-and-processing">4. Stream Mixing And Processing</h2>
-
-<h3 id="time-varying-attributes">4.1 Time-varying Attributes</h3>
-
-<p>To enable precise control over the timing of attribute changes, many attributes can be set using a
-"timed setter" method taking a <code>startTime</code> parameter. The user-agent will attempt to make the change take
-effect when the subject stream's "current time" is exactly the given <code>startTime</code> --- certainly no earlier, but possibly later if the change request is processed after the stream's current time has reached <code>startTime</code>. <code>startTime</code> is optional; if ommitted, the change takes effect as soon as possible.
-
-<p>Using a timed setter method never changes the observed attribute value immediately. Setter method changes always take effect after the next <em>stable state</em>, as described in section 2.1. Setting the attribute value changes the observed attribute value immediately, but the change to the underlying media stream will still not take effect until after
-the next stable state.
-
-<p>Multiple pending changes to an attribute are allowed. Calling the setter method with
-<code>startTime</code> T sets the value of the attribute for all times T' >= T to the desired value (wiping out the effects of previous calls to the setter method with a time greater than or equal to <code>startTime</code>). Therefore
-by calling the setter method multiple times with increasing <code>startTime</code>, a series of change requests
-can be built up. Setting the attribute directly sets the value of the attribute for all future times, wiping
-out any pending setter method requests.
-
-<h3 id="processedmediastream">4.2 ProcessedMediaStream</h3>
-
-<p>A <code>ProcessedMediaStream</code> combines zero or more input streams and applies some processing to
-combine them into a single output stream.
-
-<pre><code>[Constructor(),
- Constructor(in Worker worker, in optional long audioSampleRate, in optional short audioChannels)]
-interface ProcessedMediaStream : MediaStream {
- readonly attribute MediaInput[] inputs;
- MediaInput addInput(in MediaStream input, in optional double outputStartTime, in optional double inputStartTime);
-
- attribute any params;
- void setParams(in any params, in optional double startTime);
-
- readonly attribute boolean autofinish;
-};</pre></code>
-
-<p>The constructors create a new <code>ProcessedMediaStream</code> with no inputs.
-The second constructor creates a new <code>ProcessedMediaStream</code> with a Worker
-processing engine, setting the
-audio sample rate to <code>audioSampleRate</code> and setting the number of
-audio channels to <code>audioChannels</code> (defaulting to 2). These parameters control the audio sample
-format used by the Worker (see below). Both constructors initialize <code>autofinish</code> to false.
-
-<p class="todo">Specify valid values for <code>audioChannels</code> and <code>audioSampleRate</code>.
-
-<p>The <code>inputs</code> attribute returns an array of <code>MediaInput</code>s, one for
-each stream currently configured as an input to the <code>ProcessedMediaStream</code>. (A stream can be used as multiple inputs to the same <code>ProcessedMediaStream</code>.) It is
-initially empty if constructed via the <code>ProcessedMediaStream()</code> constructor, or
-contains a single element if constructed via <code>MediaStream.createProcessor</code>.
-
-<p>The <code>addInput(input, outputStartTime, inputStartTime)</code> method adds a new <code>MediaInput</code> to the end of the
-<code>inputs</code> array, whose input stream is <code>input</code>. The <code>outputStartTime</code>
-and <code>inputStartTime</code> attributes control when an input port is enabled and helps
-synchronize inputs with outputs. The input port is enabled when the input stream's current time is
-<code>inputStartTime</code> and the output stream's current time is <code>outputStartTime</code>.
-
-<p>More precisely, when the <code>addInput</code> call takes effect (see section 2.1),
-the user-agent runs the following steps:
-<ol>
-<li>If the <code>outputStartTime</code> was omitted, set it to the output stream's current time.
-<li>If the <code>inputStartTime</code> was omitted, set it to the input stream's current time.
-<li>Compute the <em>deadline miss delay</em>: the maximum of
- <ul>
- <li>The input stream's curent time minus <code>inputStartTime</code>
- <li>The output stream's current time minus <code>outputStartTime</code>
- </ul>
-<li>If the deadline miss delay is greater than zero, add it to the <code>inputStartTime</code> and <code>outputStartTime</code>. (This would be a good place for user-agents to emit a developer-accessible warning.)
-<li>While the input stream's current time is less than <code>inputStartTime</code>, or the output stream's current time is less than <code>outputStartTime</code>:
- <ul>
- <li>Whenever the input stream's current time is equal to <code>inputStartTime</code>, block the input stream.
- <li>Whenever the output stream's current time is equal to <code>outputStartTime</code>, block the output stream.
- </ul>
-<li>Enable the input port.
-</ol>
-
-<div class="note">It is easy to (mis)use start times to cause deadlocks in media processing, i.e. a
-graph configuration that will continually block until it is modified. For example,
-<pre><code> var p = inputStream.createProcessor();
- p.addInput(inputStream, 5);</pre></code>
-In this example, <code>inputStream</code> is used as an input to <code>p</code> twice. <code>inputStream</code> must block until <code>p</code> has played 5s of output, but also <code>p</code> cannot play anything until <code>inputStream</code> unblocks. It seems hard to design an API that's hard to deadlock; even creating a cycle will cause deadlock.</div>
-
-<p>A <code>MediaInput</code> represents an input port. An input port is <em>active</em> while it is enabled (see below) and the input stream is not finished.
-
-<p>The <code>params</code> attribute and the <code>setParams(params, startTime)</code> timed setter method set the paramters for this stream. On setting, a <em>structured clone</em> of this object is made. The clone is sent to
-the worker (if there is one) during media processing. On getting, a fresh clone is returned.
-
-<p>When an input stream finishes, at the next stable state any <code>MediaInput</code>s for that
-stream are automatically removed.
-
-<p>When the <code>autofinish</code> attribute is true, then when all stream inputs are finished (including if there are no inputs), the stream will automatically enter the finished state and never produce any more output (even if new inputs are attached).
-
-<h3 id="mediainput">4.3 MediaInput</h3>
-
-<p>A <code>MediaInput</code> object controls how an input stream contributes to the combined stream.
-
-<pre><code>interface MediaInput {
- readonly attribute MediaStream stream;
-
- attribute double volume;
- void setVolume(in double volume, in optional double startTime, in optional double fadeTime);
-
- attribute any params;
- void setParams(in any params, in optional double startTime);
-
- attribute boolean blockInput;
- attribute boolean blockOutput;
-
- void remove(in optional double time);
-};</pre></code>
-
-<p>The <code>stream</code> attribute returns the <code>MediaStream</code> connected to this input.
-The input stream is treated as having at most one audio and/or video track; all enabled audio tracks are mixed
-together and the rest are dropped, and all video tracks other than the selected video track are dropped.
-
-<p class="todo">Add additional API to select particular tracks.
-
-<p>The <code>volume</code> volume attribute and the <code>setVolume</code> timed setter method
-control the input volume; the input stream's audio is multiplied by this volume before
-being processed. The <code>setVolume</code> method takes an additional <code>fadeTime</code> parameter; when greater
-than zero, the volume is changed gradually from the value just before <code>startTime</code> to
-the new value over the given fade time. The transition function is chosen so that if one stream changes from V1 to V2
-and another stream changes from V2 to V1 over the same interval, the sum of the volumes at each point in
-time is V1 + V2. This attribute is initially 1.0.
-
-<p class="todo">Specify the exact transition function. Tim says "w=cos((pi/2)*t)^2 for t=0...1".
-
-<p>The <code>params</code> attribute and the <code>setParams(params, startTime)</code> timed setter method set the paramters for this input. On setting, a <em>structured clone</em> of this object is made. The clone is sent to
-the worker (if there is one) during media processing. On getting, a fresh clone is returned.
-
-<p>For the timed setter methods of <code>MediaInput</code>, the subject stream is the output stream, so changes take effect when the output stream's current time is equal to <code>startTime</code>.
-
-<p>The <code>blockInput</code> and <code>blockOutput</code> attributes control
-how the blocking status of the input stream is related to the blocking status of the output stream.
-When <code>blockOutput</code> is true and the input port is active, if the input stream is blocked then the output stream must be blocked. While an active input is blocked and the output is not blocked, the input is treated as having no tracks. When <code>blockInput</code> is true and the input port is active, if the output is blocked,
-then the input stream must be blocked. When false, while the output is blocked and an active input is not, the input will simply be discarded. These attributes are initially true.
-
-<p class="XXX">Need to look again at these. It's not clear we have use cases for both attributes, and I haven't implemented them yet and they could be hard to implement.
-
-<p>The <code>remove(time)</code> method removes this <code>MediaInput</code> from the inputs array of its owning
-<code>ProcessedMediaStream</code> at the given time relative to the output stream (or later, if it cannot be removed in time).
-If <code>time</code> is omitted, the input is removed as soon as possible and the <code>MediaInput</code> is removed from
-the destionation stream's <code>input</code> array immediately.
-After removal, the <code>MediaInput</code> object is no longer used; its attributes retain their current values
-and do not change unless explicitly set. All method calls are ignored. Additional calls to <code>remove</code> with an earlier time
-can advance the removal time, but once removal is scheduled it cannot be stopped or delayed.
-
-<h3 id="worker-processing">4.4 Worker Processing</h3>
-
-<p>A <code>ProcessedMediaStream</code> with a worker computes its output by dispatching a sequence of <code>onprocessmedia</code> callbacks to the worker, passing each a <code>ProcessMediaEvent</code> parameter. A <code>ProcessMediaEvent</code> provides audio sample buffers for each input stream. Each sample buffer for a given <code>ProcessMediaEvent</code> has the same duration, so the inputs presented to the worker are always in sync. (Inputs may be added or removed between <code>ProcessMediaEvent</code>s, however.) The sequence of buffers provided for an input stream is the audio data to be played by that input stream. The user-agent will precompute data for the input streams as necessary.
-
-<p>For example, if a Worker computes the output sample for time T as a function of the [T - 1s, T + 1s] interval of an input stream, then initially the Worker would simply refuse to output anything until it has received at least 1s of input stream data, forcing the user-agent to precompute the input stream at least 1s ahead of the current time. (Note that large Worker latencies will increase the latency of changes to the media graph.)
-
-<p class="note">Note that <code>Worker</code>s do not have access to most DOM API objects. In particular, <code>Worker</code>s have no direct access to <code>MediaStream</code>s.
-
-<p class="note">Note that a <code>ProcessedMediaStream</code>'s worker cannot be a
-<code>SharedWorker</code>. This ensures that the worker can run in the same process as the page in multiprocess browsers, so media streams can be confined to a single process.
-
-<p class="todo">Currently <code>ProcessMediaEvent</code> does not offer access to video data. This should be added later.
-
-<pre><code>partial interface DedicatedWorkerGlobalScope {
- attribute Function onprocessmedia;
-};</pre></code>
-
-<p>The <code>onprocessmedia</code> attribute is the function to be called whenever stream data needs to be processed.
-A <code>ProcessMediaEvent</code> is passed as the single parameter to each call to the <code>onprocessmedia</code> callback.
-For a given <code>ProcessedMediaStream</code>, the same <code>ProcessMediaEvent</code> is passed in every call to the
-<code>onprocessmedia</code> callback. This allows the callback function to maintain per-stream state.
-
-<pre><code>interface ProcessMediaEvent : Event {
- readonly attribute double inputTime;
-
- readonly attribute any params;
- readonly attribute double paramsStartTime;
-
- readonly attribute MediaInputBuffer inputs[];
- readonly attribute long audioSampleRate;
- readonly attribute short audioChannels;
- reaodnly attribute long audioLength;
-
- void writeAudio(in Float32Array data);
-
- void finish();
-};</pre></code>
-
-<p>The <code>inputTime</code> attribute returns the duration of the input that has been consumed by the
-<code>ProcessedMediaStream</code> for this worker.
-
-<p>The <code>params</code> attribute provides a structured clone of the parameters object set by
-<code>ProcessedMediaStream.setParams</code>. The same object is returned in each event, except when the object has
-been changed by <code>setParams</code> between events. <p>The <code>paramsStartTime</code> attribute returns the first time (measured in duration of input consumed for this stream) that this <code>params</code> object was set.
-
-<p class="note">Note that the parameters objects are constant over the duration of the inputs presented in the
-event. Frequent changes to parameters will reduce the length of the input buffers that can be presented to
-the worker.
-
-<p><code>inputs</code> provides access to <code>MediaStreamBuffers</code> for each active input stream
-(in the same order as those streams appear in the <code>ProcessedMediaStream.inputs</code> array).
-
-<p><code>audioSampleRate</code> and <code>audioChannels</code> represent the format of the input and
-output audio sample buffers. <code>audioSampleRate</code> is the number of samples per second.
-<code>audioChannels</code> is the number of channels; the channel mapping is as defined in the Vorbis specification.
-These values are constant for a given <code>ProcessedMediaStream</code>. When the <code>ProcessedMediaStream</code>
-was constructed using the Worker constructor, these values are the values passed as parameters there. When the
-<code>ProcessedMediaStream</code> was constructed via <code>MediaStream.createProcessor</code>, the values are
-chosen to match the first active input stream (or 44.1KHz, 2 channels if there is no active input stream).
-
-<p><code>audioLength</code> is the duration of the input(s) multiplied by the sample rate. If there are no inputs,
-the user-agent will choose a value representing the suggested amount of audio that the worker should produce.
-
-<p><code>writeAudio(data)</code> writes audio data to the stream output.
-The output has a single audio track. If there is an active input with an audio track, then the metadata for the output audio track is set to the metadata for the audio track of the last active input that has an audio track, otherwise the output audio track's <code>kind</code> is "main" and the other metadata attriutes are the empty string. The data for the output audio track is the concatenation of the
-inputs to each <code>writeAudio</code> call before the event handler returns. The data buffer is laid out
-with the channels non-interleaved, as for the input buffers (see below). The length of <code>data</code> must be
-a multiple of <code>audioChannels</code>; if not, then only the sample values up to the largest multiple of <code>audioChannels</code> less than the data length are used.
-
-<p>It is permitted to write less audio than the duration of the inputs (including none). This indicates latency in the filter. Normally the user-agent will dispatch another event to provide
-more input until the worker starts producing output. It is also permitted to write more audio than the duration of the inputs, for example if there are no inputs.
-Filters with latency should respond to an event with no inputs by writing out some of their buffered data; the user-agent
-is draining them.
-
-<p class="note">A synthesizer with no inputs can output as much data as it wants; the UA will buffer data and fire events as necessary. Filters that misbehave, e.g. by always writing zero-length buffers, will cause the stream to block due to an underrun.
-
-<p>If <code>writeAudio</code> is not called during the event handler, then the input audio buffers are added together and written to the output.
-
-<p>If <code>writeAudio</code> is called outside the event handler, the call is ignored.
-
-<p>Calling <code>finish()</code> puts the stream into the finished state (once any previously buffered output has been consumed). The event callback will never be called again. <code>finish()</code> can be called at any time, inside or outside the event handler.
-
-<p>The output video track is computed as if there was no worker (see above).
-
-<p class="todo">This will change when we add video processing.
-
-<pre><code>interface MediaInputBuffer {
- readonly attribute any params;
- readonly attribute double paramsStartTime;
-
- readonly attribute Float32Array audioSamples;
-};</pre></code>
-
-<p>The <code>params</code> attribute provides a structured clone of the parameters object set by
-<code>MediaInput.setParams</code>. The same object is returned in each event, except when the object has
-been changed by <code>setParams</code> between events. <p>The <code>paramsStartTime</code> attribute returns the first time (measured in duration of input consumed for this stream) that this <code>params</code> object was set.
-
-<p><code>audioSamples</code> gives access to the audio samples for each input stream. The array length will be <code>event.audioLength</code> multiplied by <code>event.audioChannels</code>. The samples are floats ranging from -1 to 1, laid out non-interleaved, i.e. consecutive segments of <code>audioLength</code> samples each. The durations of the input buffers for the input streams will be equal. The <code>audioSamples</code> object will be a fresh object in each event. For inputs with no audio track, <code>audioSamples</code> will be all zeroes.
-
-<h2 id="built-in-processing-engine">5. Built-In Processing Engines</h2>
-
-<h3 id="default-processing-engine">5.1. Default Processing Engine</h2>
-
-<p>A <code>ProcessedMediaStream</code> with the default processing engine produces output as follows:
-<ul>
-<li>If no active input has an audio track, the output has no audio track. Otherwise, the output has a single
-audio track whose metadata (<code>id</code>, <code>kind</code>, <code>label</code>, and <code>language</code>)
-is equal to that of the audio track for the last active input that has an audio track. The output audio track
-is produced by adding the samples of the audio tracks of the active inputs together.
-<li>If no active input has a video track, the output has no video track. Otherwise, the output has a single
-video track whose metadata (<code>id</code>, <code>kind</code>, <code>label</code>, and <code>language</code>)
-is equal to that of the video track for the last active input that has a video track. The output video track
-is produced by compositing together all the video frames from the video tracks of the active inputs, with the video
-frames from higher-numbered inputs on top of the video frames from lower-numbered inputs; each
-video frame is letterboxed to the size of the video frame for the last active input that has a video track.
-<p class="note">This means if the last input's video track is opaque, the video output is simply the video track of the last input.
-</ul>
-
-<h3 id="lastinput-processing-engine">5.2. "LastInput" Processing Engine</h2>
-
-<p>A <code>ProcessedMediaStream</code> with the "LastInput" processing engine simply produces the last active input stream as output. If there are no active input streams, it produces the same output as the default processing engine.
-
-<h2 id="media-graph-considerations">6. Media Graph Considerations</h2>
-
-<h3 id="cycles">6.1. Cycles</h3>
-
-<p>While a <code>ProcessedMediaStream</code> has itself as a direct or indirect input stream (considering only active inputs), it is blocked.
-
-<h3 id="blocking">6.2. Blocking</h2>
-
-<p>At each moment, every stream should not be blocked except as explicitly required by this specification.
-
-<h2 id="canvas-recording">7. Canvas Recording</h2>
-
-<p>To enable video synthesis and some easy kinds of video effects we can record the contents of a canvas:
-
-<pre><code>partial interface HTMLCanvasElement {
- readonly attribute MediaStream stream;
-};</pre></code>
-
-<p>The <code>stream</code> attribute is a stream containing a video track with the "live" contents of the canvas as video frames whose size is the size of the canvas, and no audio track. It always returns the same stream for a given element.
-
-<h2 id="implementation-considerations">8. Implementation Considerations</h2>
-
-<p class="todo">Here will be some non-normative implementation suggestions.
-
-<h2 id="examples">9. Examples</h2>
-
-<p class="todo">Add Worker scripts for these examples.
-
-<ol>
-<li>Play video with processing effect applied to the audio track
-
-<pre><code><video src="foo.webm" id="v" controls></video>
-<audio id="out" autoplay></audio>
-<script>
-document.getElementById("out").src =
- document.getElementById("v").captureStream().createWorkerProcessor(new Worker("effect.js"));
-</script></pre></code>
-
-<li>Play video with processing effects mixing in out-of-band audio tracks (in sync)
-
-<pre><code><video src="foo.webm" id="v"></video>
-<audio src="back.webm" id="back"></audio>
-<audio id="out" autoplay></audio>
-<script>
- var mixer = document.getElementById("v").captureStream().createWorkerProcessor(new Worker("audio-ducking.js"));
- mixer.addInput(document.getElementById("back").captureStream());
- document.getElementById("out").src = mixer;
- function startPlaying() {
- document.getElementById("v").play();
- document.getElementById("back").play();
- }
- // MediaController is a more convenient API because it ties together control of the elements,
- // but using streams is more flexible (e.g. they can be seeked to different offsets).
-</script></pre></code>
-
-<li>Capture microphone input and stream it out to a peer with a processing effect applied to the audio
-
-<pre><code><script>
- navigator.getUserMedia('audio', gotAudio);
- function gotAudio(stream) {
- peerConnection.addStream(stream.createWorkerProcessor(new Worker("effect.js")));
- }
-</script></pre></code>
-
-<li>Capture microphone input and visualize it as it is being streamed out to a peer and recorded
-
-<pre><code><canvas id="c"></canvas>
-<script>
- navigator.getUserMedia('audio', gotAudio);
- var streamRecorder;
- function gotAudio(stream) {
- var worker = new Worker("visualizer.js");
- var processed = stream.createWorkerProcessor(worker);
- worker.onmessage = function(event) {
- drawSpectrumToCanvas(event.data, document.getElementById("c"));
- }
- streamRecorder = processed.record();
- peerConnection.addStream(processed);
- }
-</script></pre></code>
-
-<li>Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record
-
-<pre><code><canvas id="c"></canvas>
-<audio src="back.webm" id="back"></audio>
-<script>
- navigator.getUserMedia('audio', gotAudio);
- var streamRecorder;
- function gotAudio(stream) {
- var worker = new Worker("visualizer.js");
- var processed = stream.createWorkerProcessor(worker);
- worker.onmessage = function(event) {
- drawSpectrumToCanvas(event.data, document.getElementById("c"));
- }
- var mixer = processed.createProcessor();
- mixer.addInput(document.getElementById("back").captureStream());
- streamRecorder = mixer.record();
- peerConnection.addStream(mixer);
- }
-</script></pre></code>
-
-<li>Receive audio streams from peers, mix them with spatialization effects, and play
-
-<pre><code><script>
- var worker = new Worker("spatializer.js");
- var spatialized = stream.createWorkerProcessor(worker);
- peerConnection.onaddstream = function (event) {
- spatialized.addInput(event.stream).params = {x:..., y:..., z:...};
- };
- (new Audio(spatialized)).play();
-</script></pre></code>
-
-<li>Seamlessly chain from the end of one input stream to another
-
-<p class="note">This method requires that you know each stream's duration, which is a bit suboptimal.
-To get around that we'd need new API, perhaps a new kind of ProcessedMediaStream that plays streams in
-serial.
-
-<pre><code><audio src="in1.webm" id="in1" preload></audio>
-<audio src="in2.webm" id="in2"></audio>
-<script>
- var in1 = document.getElementById("in1");
- in1.onloadedmetadata = function() {
- var mixer = in1.captureStreamUntilEnded().createProcessor();
- var in2 = document.getElementById("in2");
- mixer.addInput(in2.captureStreamUntilEnded(), in1.duration);
- (new Audio(mixer)).play();
- in1.play();
- }
-</script></pre></code>
-
-<li>Seamlessly switch from one input stream to another, e.g. to implement adaptive streaming
-
-<p class="note">There are two ways to implement seamless switching: seek the second resource to before the
-current time and then run the decoder faster than real-time to catch up to the first resource's play point, or
-seek the second resource to after the current time and enable it when the first resource catches up to the seek
-point. The first is more robust if the seek takes unexpectedly long, but the second is less demanding on the
-decoder. Only the second method is currently implementable with this API (since by design there is no way to drive MediaStreams faster than real-time). If we want to support the first method as well, the right way would be to
-add API to media elements to let them seek to synchronize with a given MediaStream.
-
-<pre><code><audio src="in1.webm" id="in1" preload></audio>
-<audio src="in2.webm" id="in2"></audio>
-<audio id="out" autoplay></audio>
-<script>
- var stream1 = document.getElementById("in1").captureStream();
- var mixer = stream1.createProcessor("LastInput");
- document.getElementById("out").src = mixer;
- function switchStreams() {
- var in2 = document.getElementById("in2");
- in2.currentTime = in1.currentTime + 10; // arbitrary, but we should be able to complete the seek within this time
- mixer.addInput(in2.captureStream(), mixer.currentTime + 10);
- in2.play();
- // in2 will be blocked until the input port is enabled
- in2.onplaying = function() { mixer.inputs[0].remove(); };
- }
-</script></pre></code>
-
-<li>Synthesize samples from JS data
-
-<pre><code><audio id="out" autoplay></audio>
-<script>
- document.getElementById("out").src =
- new ProcessedMediaStream(new Worker("synthesizer.js"));
-</script></pre></code>
-
-<li>Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking
-
-<pre><code><script>
- var effectsMixer = ...;
- function playSound(src) {
- var audio = new Audio(src);
- audio.oncanplaythrough = function() {
- var stream = audio.captureStreamUntilEnded();
- var port = effectsMixer.addInput(stream);
- port.blockOutput = false;
- audio.play();
- }
- }
-</script></pre></code>
-
-<li>Trigger a sound sample to be played through the effects graph in five seconds
-
-<pre><code><script>
- var effectsMixer = ...;
- var audio = new Audio(...);
- function triggerSound() {
- var sound = audio.clone();
- var stream = sound.captureStreamUntilEnded();
- sound.play();
- effectsMixer.addInput(stream, effectsMixer.currentTime + 5);
- }
-</script></pre></code>
-
-<li>Capture video from a camera and analyze it (e.g. face recognition)
-
-<pre><code><script>
- navigator.getUserMedia('video', gotVideo);
- function gotVideo(stream) {
- stream.createWorkerProcessor(new Worker("face-recognizer.js"));
- }
-</script></pre></code>
-
-<li>Capture video, record it to a file and upload the file (e.g. Youtube)
-
-<pre><code><script>
- navigator.getUserMedia('video', gotVideo);
- var streamRecorder;
- function gotVideo(stream) {
- streamRecorder = stream.record();
- }
- function stopRecording() {
- streamRecorder.getRecordedData(gotData);
- }
- function gotData(blob) {
- var x = new XMLHttpRequest();
- x.open('POST', 'uploadMessage');
- x.send(blob);
- }
-</script></pre></code>
-
-<li>Capture video from a canvas, record it to a file then upload
-
-<pre><code><canvas width="640" height="480" id="c"></canvas>
-<script>
- var canvas = document.getElementById("c");
- var streamRecorder = canvas.stream.record();
- function stopRecording() {
- streamRecorder.getRecordedData(gotData);
- }
- function gotData(blob) {
- var x = new XMLHttpRequest();
- x.open('POST', 'uploadMessage');
- x.send(blob);
- }
- var frame = 0;
- function updateCanvas() {
- var ctx = canvas.getContext("2d");
- ctx.clearRect(0, 0, 640, 480);
- ctx.fillText("Frame " + frame, 0, 200);
- ++frame;
- }
- setInterval(updateCanvas, 30);
-</script></pre></code>
-</ol>
-
-<h1>Related Work</h1>
-
-<ul>
-<li><a href="https://wiki.mozilla.org/RTCStreamAPI">W3C-RTC charter</a> (Harald et. al.)
-<li><a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html">WhatWG proposal (Ian Hickson et. al.)</a>
-<li><a href="http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html">Chrome audio API</a>
-<li><a href="https://wiki.mozilla.org/Audio_Data_API">Mozilla audio API</a>
-<li><a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#mediacontroller">WhatWG MediaController API</a>
-</body>
-</html>
--- a/StreamProcessing/main.css Fri Oct 28 18:27:39 2011 +1300
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,128 +0,0 @@
-
-/* Style for a Working Group Editors' Draft */
-
-/*
- Copyright 1997-2003 W3C (MIT, ERCIM, Keio). All Rights Reserved.
- The following software licensing rules apply:
- http://www.w3.org/Consortium/Legal/copyright-software */
-
-/* $Id: base.css,v 1.25 2006/04/18 08:42:53 bbos Exp $ */
-
-body {
- padding: 2em 1em 2em 70px;
- margin: 0;
- font-family: sans-serif;
- color: black;
- background: white;
- background-position: top left;
- background-attachment: fixed;
- background-repeat: no-repeat;
-}
-:link { color: #00C; background: transparent }
-:visited { color: #609; background: transparent }
-a:active { color: #C00; background: transparent }
-
-a:link img, a:visited img { border-style: none } /* no border on img links */
-
-a img { color: white; } /* trick to hide the border in Netscape 4 */
-@media all { /* hide the next rule from Netscape 4 */
- a img { color: inherit; } /* undo the color change above */
-}
-
-th, td { /* ns 4 */
- font-family: sans-serif;
-}
-
-h1, h2, h3, h4, h5, h6 { text-align: left }
-/* background should be transparent, but WebTV has a bug */
-h1, h2, h3 { color: #005A9C; background: white }
-h1 { font: 170% sans-serif }
-h2 { font: 140% sans-serif }
-h3 { font: 120% sans-serif }
-h4 { font: bold 100% sans-serif }
-h5 { font: italic 100% sans-serif }
-h6 { font: small-caps 100% sans-serif }
-
-.hide { display: none }
-
-div.head { margin-bottom: 1em }
-div.head h1 { margin-top: 2em; clear: both }
-div.head table { margin-left: 2em; margin-top: 2em }
-
-p.copyright { font-size: small }
-p.copyright small { font-size: small }
-
-@media screen { /* hide from IE3 */
-a[href]:hover { background: #ffa }
-}
-
-pre { margin-left: 2em }
-dt, dd { margin-top: 0; margin-bottom: 0 } /* opera 3.50 */
-dt { font-weight: bold }
-
-pre, code {
- font-family: monospace;
- overflow: auto;
- margin: 0;
-}
-pre > code {
- display: block;
- padding: 1em;
- border: 1px solid black;
- background: #ddd;
- margin: 0.5em 2em;
- margin-bottom: 1em;
- font-size: 120%;
-}
-code var { color: #f44; }
-
-.note {
- margin: 1em;
- padding: 1em;
- background: yellow;
-}
-
-.XXX {
- margin: 1em;
- padding: 1em;
- background: pink;
-}
-.XXX:before {
- content: "XXX "
-}
-
-.todo {
- margin: 1em;
- padding: 1em;
- background: cyan;
-}
-.todo:before {
- content: "TODO: "
-}
-
-.example {
- margin: 1em;
- padding: 1em;
- background: lime;
-}
-
-ul.toc, ol.toc {
- list-style: disc; /* Mac NS has problem with 'none' */
- list-style: none;
-}
-
-@media aural {
- h1, h2, h3 { stress: 20; richness: 90 }
- .hide { speak: none }
- p.copyright { volume: x-soft; speech-rate: x-fast }
- dt { pause-before: 20% }
- pre { speak-punctuation: code }
-}
-
-body {
-/* background-image: url(http://www.w3.org/StyleSheets/TR/logo-ED); */
-}
-
-#toc li {
- list-style-type:none;
-}
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/streams/StreamProcessing.html Thu Nov 03 11:10:05 2011 +1300
@@ -0,0 +1,778 @@
+<!DOCTYPE HTML>
+<html>
+<head>
+<title>MediaStream Processing API</title>
+<link rel="stylesheet" href="main.css">
+</head>
+<body>
+
+<div class="head">
+ <h1>MediaStream Processing API</h1>
+ <h2>Draft Proposal</h2>
+ <dl><dt>Editor:</dt><dd>Robert O'Callahan, Mozilla Corporation <robert@ocallahan.org></dd>
+</div>
+
+<h2>Status of this Document</h2>
+<p>This document is a draft specification proposal with no official status. Send comments to the <a href="mailto:public-audio@w3.org">W3C audio mailing list</a>, or <a href="mailto:robert@ocallahan.org">Robert O'Callahan</a>. It is inappropriate to cite this document except as a work in progress.
+
+<h2>Abstract</h2>
+
+<p>A number of existing or proposed features for the Web platform deal with continuous real-time media:
+<ul>
+<li>HTML media elements
+<li>Synchronization of multiple HTML media elements (e.g. proposed HTML MediaController)
+<li>Capture and recording of local audio and video input (e.g. proposed HTML Streams)
+<li>Peer-to-peer streaming of audio and video streams (e.g. proposed WebRTC and HTML Streams)
+<li>Advanced audio APIs that allow complex mixing and effects processing (e.g. Mozilla's AudioData, Chrome's AudioNode)
+</ul>
+Many use-cases require these features to work together. This proposal makes HTML Streams the foundation for integrated Web media processing by creating a mixing and effects processing API for HTML Streams.
+
+<h2>Table of Contents</h2>
+
+<ol id="toc">
+ <li><a href="#introduction">1. Introduction</a>
+ <ol>
+ <li><a href="#scenarios">1.1. Scenarios</a>
+ </ol>
+ <li><a href="#mediastreams">2. MediaStreams</a>
+ <ol>
+ <li><a href="#scenarios">2.1. The Semantics Of MediaStreams</a>
+ <li><a href="#media-formats">2.2. Media Formats</a>
+ <li><a href="#mediastream-extensions">2.3. MediaStream Extensions</a>
+ </ol>
+ <li><a href="#media-elements">3. Media Elements</a>
+ <ol>
+ <li><a href="#media-element-extensions">3.1. Media Element Extensions</a>
+ <li><a href="#audio-element-extensions">3.2. Audio Element Extensions</a>
+ </ol>
+ <li><a href="#stream-mixing-and-processing">4. Stream Mixing And Processing</a>
+ <ol>
+ <li><a href="#time-varying-attributes">4.1. Time-varying Attributes</a>
+ <li><a href="#processedmediastream">4.2. ProcessedMediaStream</a>
+ <li><a href="#mediainput">4.3. MediaInput</a>
+ <li><a href="#worker-processing">4.4. Worker Processing</a>
+ </ol>
+ <li><a href="#built-in-processing-engines">5. Built-In Processing Engines</a>
+ <ol>
+ <li><a href="#default-processing-engine">5.1. Default Processing Engine</a>
+ <li><a href="#lastinput-processing-engine">5.2. "LastInput" Processing Engine</a>
+ </ol>
+ <li><a href="#media-graph-considerations">6. Media Graph Considerations</a>
+ <li><a href="#canvas-recording">7. Canvas Recording</a>
+ <li><a href="#implementation-considerations">8. Implementation Considerations</a>
+ <li><a href="#examples">9. Examples</a>
+</ol>
+
+<h2 id="introduction">1. Introduction</h2>
+
+<p>The ideas here build on <a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html">Ian Hickson's proposal for HTML Streams</a>, adding features partly inspired by <a href="https://wiki.mozilla.org/Audio_Data_API"> the Mozilla audio API</a> and <a href="http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html">the Chrome audio API</a>. Unlike previous audio API proposals, the API presented here integrates with proposed API for media capture from local devices, integrates with proposed API for peer-to-peer media streaming, handles audio and video in a unified framework, incorporates Worker-based Javascript audio processing, and specifies synchronization across multiple media sources and effects. The API presented here does not include a library of "native" effects; those should be added as new "named effects" (described below), perhaps as a "level 2" spec.
+
+<p>The work here is nascent. Until a prototype implementation exists, this proposal is likely to be incomplete and possibly not even implementable.
+
+<h3 id="scenarios">1.1. Scenarios</h3>
+
+<p>These are concrete usage scenarios that have helped guide the design of the API. They are higher-level than use-cases.
+
+<ol>
+<li>Play video with processing effect applied to the audio track (e.g. high-pass filter)
+<li>Play video with processing effects mixing in out-of-band audio tracks (in sync) (e.g. mixing in an audio commentary with audio ducking)
+<li>Capture microphone input and stream it out to a peer with a processing effect applied to the audio (e.g. XBox 360 chat with voice distortion)
+<li>Capture microphone input and visualize it as it is being streamed out to a peer and recorded (e.g. Internet radio broadcast)
+<li>Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record (e.g. Internet radio broadcast)
+<li>Receive audio streams from peers, mix them with spatialization effects, and play (e.g. live chat with spatial feature)
+<li>Seamlessly chain from the end of one input stream to another (e.g. playlists, audio/video editing)
+<li>Seamlessly switch from one input stream to another (e.g. adaptive streaming)
+<li>Synthesize samples from JS data (e.g. game emulators or MIDI synthesizer)
+<li>Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking (e.g. game sound effects)
+<li>Trigger a sound sample to be played through the effects graph at a given time (e.g. game sound effects)
+<li>Capture video from a camera and analyze it (e.g. face recognition)
+<li>Capture video and audio, record it to a file and upload the file (e.g. Youtube upload)
+<li>Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast", or composite multiple video sources with effects into a single canvas then record)
+</ol>
+
+<h2 id="mediastreams">2. MediaStreams</h2>
+
+<h3 id="the-semantics-of-mediastreams">2.1. The Semantics Of MediaStreams</h3>
+
+<p>The description of MediaStreams here extends and must remain compatible with
+<a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html#stream-api">HTML MediaStreams</a>.
+
+<p>Each MediaStream DOM object has an underlying media stream. The underlying media streams form a graph;
+some streams (represented by ProcessedMediaStream DOM objects) can take other streams as inputs and compute an output
+stream.
+
+<p>To avoid interruptions due to script execution, script execution can overlap with media stream processing;
+media streams continue to play and change state during script execution. However, to simplify the DOM programming model,
+we limit the interaction of MediaStream DOM objects with their underlying media streams. Specifically:
+<ul>
+<li>Changes to the MediaStream and MediaInput DOM objects are batched together between <em>stable states</em> (as defined by the HTML spec, while script is not running), and propagate as an atomic change to the media stream graph to take effect
+after some delay.
+This ensures that incomplete changes to the media stream graph during script execution do not result in transient glitches in the output, and that scripted graph changes do not interfere with concurrent media processing.
+<div class="example">Thus the following script would never cause an interruption in audio output, since no stable state occurs between the two volume changes:
+<pre><code> stream.inputs[0].volume = 0;
+ if (needToPlay()) {
+ stream.inputs[0].volume = 1.0;
+ }</code></pre>
+</div>
+
+<p class="todo">Specify exactly which attributes (and methods) are subject to this regime, possibly extending to
+attributes and methods already defined in HTML for media elements etc.
+<li>State changes from the media stream graph are only propagated back to the MediaStream DOM objects during a <em>stable state</em>. This ensures that between stable states (usually, during the execution of a script event handler), MediaStream DOM APIs always reflect a consistent snapshot of the state of the media stream graph.
+</ul>
+In this spec, references to <code>MediaStream</code>s and <code>MediaInput</code>s refer to the DOM-visible state, and references to <em>media streams</em> and <em>input ports</em> refer to the underlying real-time media stream graph.
+
+<p>A stream is an abstraction of a time-varying video and/or audio signal. At a given point in time, a media stream can be <em>blocked</em>, that is, not playing for some reason. All non-blocked streams play at the same constant rate --- real time. Streams cannot be seeked or played at a rate other than real time. For convenience we define a stream's "current time" as the duration it has played since it was created, but (unless blocking is involved) streams do not have independent timelines; they are synchronized.
+
+<p>At the implementation level, and when processing media data with Workers, we assume that each stream has a buffered window of media data available, containing the sample it is currently playing (or that it will play the next time it unblocks). This buffer defines the stream's contents into the future.
+
+<p>A stream can be in the <em>finished</em> state. A finished stream is always blocked and can
+never leave the finished state --- it will never produce any more content.
+
+<p>A stream that has no consumers must block. Stream consumers defined in this specification are media elements and <code>ProcessedMediaStream</code>s (see below). This avoids situations where streams that aren't connected
+to anything and could be garbage-collected have to be kept running because their stopping could possibly be observed (e.g. because a Worker is being used to generate data for the stream and the Worker can observe whether the <code>ProcessMediaEvent</code> is being sent). A muted audio element can be used as a dummy sink if necessary.
+
+<div class="note">
+<p>We do not allow streams to have independent timelines (e.g. no adjustable playback
+rate or seeking within an arbitrary stream), because that can lead to a single stream being
+consumed at multiple different "current times" simultaneously, which requires either unbounded buffering
+or multiple internal decoders and buffers for a single stream. It seems simpler and more
+predictable for performance to require authors to create multiple streams (if necessary) and change
+the playback rate in the original stream sources to handle such situations.
+<p>For example, consider this hard case:
+<ul>
+<li>Three media element input streams: http://slow, http://fast, and http://fast2
+<li>http://slow is mixed with http://fast
+<li>http://fast is mixed with http://fast2
+</ul>
+Does the http://fast stream have to provide data at two different offsets? This spec's answer is no.
+This leads us to the conclusion that if a stream feeds into a blocked mixer, then it itself must be
+blocked. Since obviously a mixer with a blocked input must also be blocked, the entire graph of
+connected streams block as a unit. This means that the mixing of http://fast and http://fast2 will
+be blocked by delays in http://slow in the above scenario.
+<p>Authors can avoid this by explicitly splitting streams that may need to progress at
+different rates --- in the above case, by using two separate media elements each loading
+http://fast. The HTML spec encourages implementations to share cached media data between
+media elements loading the same URI.
+</div>
+
+<p>A media stream contains video and audio tracks. Tracks can start and end at any time. Each track
+contains a stream of audio or video data.
+
+<h3 id="media-formats">2.2 Media Formats</h3>
+
+<p>This spec mostly treats the formats used for stream audio and video data
+as an implementation detail. In particular, whether stream buffers are compressed or uncompressed, what compression
+formats might be used, or what uncompressed formats might be used (e.g. audio sample rates, channels, and sample
+representation)
+are not specified, and are not directly observable. An implementation might even support changing formats over
+time within a single stream. Media data is implicitly resampled as necessary, e.g. when mixing streams with different formats. Non-normative suggestions for resampling algorithms will be provided in section 7.
+
+<p>Built-in audio processing filters guarantee that if all the audio inputs constantly have the same uncompressed format
+(same audio sample rate and channel configuration), the audio output will have the same format and there will be no unnecessary resampling.
+
+<p>When samples are exposed to a Worker for processing, the user-agent chooses a fixed uncompressed audio format (sample rate and channel configuration) for its inputs and outputs; see section 4.4.
+
+<p class="todo">However, suggested resampling algorithms will be provided in an appendix.
+
+<h3 id="mediastream-extensions">2.3 MediaStream Extensions</h3>
+
+<pre><code>partial interface MediaStream {
+ readonly attribute double currentTime;
+
+ ProcessedMediaStream createProcessor(optional in DOMString namedEffect);
+ ProcessedMediaStream createWorkerProcessor(in Worker worker);
+};</code></pre>
+
+<p>The <code>currentTime</code> attribute returns the amount of time that this <code>MediaStream</code> has played since it was created.
+
+<p>The <code>createProcessor(namedEffect)</code> method returns a new <code>ProcessedMediaStream</code> with this <code>MediaStream</code> as its sole input.
+The <code>ProcessedMediaStream</code> is configured with a built-in processing engine named by <code>namedEffect</code>,
+or the default processing engine if <code>namedEffect</code> is omitted. If <code>namedEffect</code> is not supported
+by this user-agent, <code>createProcessor</code> returns null. User-agents adding nonstandard named effects should use
+vendor prefixing, e.g. "MozUnderwaterBubbles". The stream's <code>autofinish</code> attribute
+is set to true.
+
+<p>The <code>createWorkerProcessor(worker)</code> method returns a new <code>ProcessedMediaStream</code> with this <code>MediaStream</code> as its sole input.
+The stream is configured with <code>worker</code> as its processing engine.
+The stream's <code>autofinish</code> flag is set to true.
+
+<p class="todo">Add event handlers or callback functions for all finished and blocking state changes?
+
+<h2 id="media-elements">3. Media Elements</h2>
+
+<h3 id="media-element-extensions">3.1 Media Element Extensions</h3>
+
+<p>We extend HTML media elements to produce and consume streams. When an HTML media element
+produces a stream, it acts as a resource loader and control mechanism; the stream consists of whatever the
+media element is currently playing. When a media element consumes a stream, it acts a playback
+mechanism for the stream.
+
+<pre><code>partial interface HTMLMediaElement {
+ readonly attribute MediaStream stream;
+
+ MediaStream captureStream();
+ MediaStream captureStreamUntilEnded();
+ attribute boolean captureAudio;
+
+ attribute any src;
+};</pre></code>
+
+<p>The <code>stream</code> attribute returns a stream which always plays whatever the element is playing. The stream is blocked while the media element is not playing. It is never finished, even when playback ends
+(since the element might load a new resource or seek within the current resource). The <code>stream</code> attribute for a given element always returns
+the same stream. When the stream changes to blocked, we fire the <code>waiting</code> event for the media element,
+and when it changes to unblocked we fire the <code>playing</code> event for the media element.
+
+<p class="XXX">Currently the HTML media element spec says that <code>playing</code> would fire on an element
+that is able to play except that a downstream <code>MediaController</code> is blocked. This is incompatible
+with the above. I think that part of the HTML media spec should be changed so that only elements that are actually
+going to play fire <code>playing</code>.
+
+<p>The <code>captureStream()</code> method returns a new <code>MediaStream</code> that plays the same audio and video as <code>stream</code>.
+<code>captureStream()</code> sets the <code>captureAudio</code> attribute to true.
+
+<p>The <code>captureStreamUntilEnded()</code> method returns a new <code>MediaStream</code> that plays the same audio and video as <code>stream</code>, until the element next reaches
+the "ended playback" state, at which point this stream will enter the finished state.
+<code>captureStreamUntilEnded()</code> sets the <code>captureAudio</code> attribute to true.
+
+<p>While the <code>captureAudio</code> attribute is true, the element does not produce direct audio output.
+Audio output is still sent to <code>stream</code> and the resource stream. This attribute is NOT reflected into the DOM. It
+is initially false.
+
+<p>The <code>src</code> attribute is extended to allow it to be set to a <code>MediaStream</code>. The element
+will play the contents of the given stream. This is treated as a live stream; seeking and <code>playbackRate</code>
+are not supported.
+
+<p>The <code>URL.createObjectURL(stream)</code> method defined for HTML MediaStreams can create a URL to be
+used as a source for a media element. Setting a media element to use this source URL is equivalent to setting
+the media element <code>src</code> to the given stream.
+
+<h3 id="audio-element-extensions">3.2 Audio Element Extensions</h3>
+
+<p>We add an <code>Audio</code> constructor taking a <code>MediaStream</code> as a parameter.
+This sets the initial <code>src</code> to the stream.
+
+<pre><code>[NamedConstructor=Audio(MediaStream src)]
+partial interface HTMLAudioElement {
+}</code></pre>
+
+<h2 id="stream-mixing-and-processing">4. Stream Mixing And Processing</h2>
+
+<h3 id="time-varying-attributes">4.1 Time-varying Attributes</h3>
+
+<p>To enable precise control over the timing of attribute changes, many attributes can be set using a
+"timed setter" method taking a <code>startTime</code> parameter. The user-agent will attempt to make the change take
+effect when the subject stream's "current time" is exactly the given <code>startTime</code> --- certainly no earlier, but possibly later if the change request is processed after the stream's current time has reached <code>startTime</code>. <code>startTime</code> is optional; if ommitted, the change takes effect as soon as possible.
+
+<p>Using a timed setter method never changes the observed attribute value immediately. Setter method changes always take effect after the next <em>stable state</em>, as described in section 2.1. Setting the attribute value changes the observed attribute value immediately, but the change to the underlying media stream will still not take effect until after
+the next stable state.
+
+<p>Multiple pending changes to an attribute are allowed. Calling the setter method with
+<code>startTime</code> T sets the value of the attribute for all times T' >= T to the desired value (wiping out the effects of previous calls to the setter method with a time greater than or equal to <code>startTime</code>). Therefore
+by calling the setter method multiple times with increasing <code>startTime</code>, a series of change requests
+can be built up. Setting the attribute directly sets the value of the attribute for all future times, wiping
+out any pending setter method requests.
+
+<h3 id="processedmediastream">4.2 ProcessedMediaStream</h3>
+
+<p>A <code>ProcessedMediaStream</code> combines zero or more input streams and applies some processing to
+combine them into a single output stream.
+
+<pre><code>[Constructor(),
+ Constructor(in Worker worker, in optional long audioSampleRate, in optional short audioChannels)]
+interface ProcessedMediaStream : MediaStream {
+ readonly attribute MediaInput[] inputs;
+ MediaInput addInput(in MediaStream input, in optional double outputStartTime, in optional double inputStartTime);
+
+ attribute any params;
+ void setParams(in any params, in optional double startTime);
+
+ readonly attribute boolean autofinish;
+};</pre></code>
+
+<p>The constructors create a new <code>ProcessedMediaStream</code> with no inputs.
+The second constructor creates a new <code>ProcessedMediaStream</code> with a Worker
+processing engine, setting the
+audio sample rate to <code>audioSampleRate</code> and setting the number of
+audio channels to <code>audioChannels</code> (defaulting to 2). These parameters control the audio sample
+format used by the Worker (see below). Both constructors initialize <code>autofinish</code> to false.
+
+<p class="todo">Specify valid values for <code>audioChannels</code> and <code>audioSampleRate</code>.
+
+<p>The <code>inputs</code> attribute returns an array of <code>MediaInput</code>s, one for
+each stream currently configured as an input to the <code>ProcessedMediaStream</code>. (A stream can be used as multiple inputs to the same <code>ProcessedMediaStream</code>.) It is
+initially empty if constructed via the <code>ProcessedMediaStream()</code> constructor, or
+contains a single element if constructed via <code>MediaStream.createProcessor</code>.
+
+<p>The <code>addInput(input, outputStartTime, inputStartTime)</code> method adds a new <code>MediaInput</code> to the end of the
+<code>inputs</code> array, whose input stream is <code>input</code>. The <code>outputStartTime</code>
+and <code>inputStartTime</code> attributes control when an input port is enabled and helps
+synchronize inputs with outputs. The input port is enabled when the input stream's current time is
+<code>inputStartTime</code> and the output stream's current time is <code>outputStartTime</code>.
+
+<p>More precisely, when the <code>addInput</code> call takes effect (see section 2.1),
+the user-agent runs the following steps:
+<ol>
+<li>If the <code>outputStartTime</code> was omitted, set it to the output stream's current time.
+<li>If the <code>inputStartTime</code> was omitted, set it to the input stream's current time.
+<li>Compute the <em>deadline miss delay</em>: the maximum of
+ <ul>
+ <li>The input stream's curent time minus <code>inputStartTime</code>
+ <li>The output stream's current time minus <code>outputStartTime</code>
+ </ul>
+<li>If the deadline miss delay is greater than zero, add it to the <code>inputStartTime</code> and <code>outputStartTime</code>. (This would be a good place for user-agents to emit a developer-accessible warning.)
+<li>While the input stream's current time is less than <code>inputStartTime</code>, or the output stream's current time is less than <code>outputStartTime</code>:
+ <ul>
+ <li>Whenever the input stream's current time is equal to <code>inputStartTime</code>, block the input stream.
+ <li>Whenever the output stream's current time is equal to <code>outputStartTime</code>, block the output stream.
+ </ul>
+<li>Enable the input port.
+</ol>
+
+<div class="note">It is easy to (mis)use start times to cause deadlocks in media processing, i.e. a
+graph configuration that will continually block until it is modified. For example,
+<pre><code> var p = inputStream.createProcessor();
+ p.addInput(inputStream, 5);</pre></code>
+In this example, <code>inputStream</code> is used as an input to <code>p</code> twice. <code>inputStream</code> must block until <code>p</code> has played 5s of output, but also <code>p</code> cannot play anything until <code>inputStream</code> unblocks. It seems hard to design an API that's hard to deadlock; even creating a cycle will cause deadlock.</div>
+
+<p>A <code>MediaInput</code> represents an input port. An input port is <em>active</em> while it is enabled (see below) and the input stream is not finished.
+
+<p>The <code>params</code> attribute and the <code>setParams(params, startTime)</code> timed setter method set the paramters for this stream. On setting, a <em>structured clone</em> of this object is made. The clone is sent to
+the worker (if there is one) during media processing. On getting, a fresh clone is returned.
+
+<p>When an input stream finishes, at the next stable state any <code>MediaInput</code>s for that
+stream are automatically removed.
+
+<p>When the <code>autofinish</code> attribute is true, then when all stream inputs are finished (including if there are no inputs), the stream will automatically enter the finished state and never produce any more output (even if new inputs are attached).
+
+<h3 id="mediainput">4.3 MediaInput</h3>
+
+<p>A <code>MediaInput</code> object controls how an input stream contributes to the combined stream.
+
+<pre><code>interface MediaInput {
+ readonly attribute MediaStream stream;
+
+ attribute double volume;
+ void setVolume(in double volume, in optional double startTime, in optional double fadeTime);
+
+ attribute any params;
+ void setParams(in any params, in optional double startTime);
+
+ attribute boolean blockInput;
+ attribute boolean blockOutput;
+
+ void remove(in optional double time);
+};</pre></code>
+
+<p>The <code>stream</code> attribute returns the <code>MediaStream</code> connected to this input.
+The input stream is treated as having at most one audio and/or video track; all enabled audio tracks are mixed
+together and the rest are dropped, and all video tracks other than the selected video track are dropped.
+
+<p class="todo">Add additional API to select particular tracks.
+
+<p>The <code>volume</code> volume attribute and the <code>setVolume</code> timed setter method
+control the input volume; the input stream's audio is multiplied by this volume before
+being processed. The <code>setVolume</code> method takes an additional <code>fadeTime</code> parameter; when greater
+than zero, the volume is changed gradually from the value just before <code>startTime</code> to
+the new value over the given fade time. The transition function is chosen so that if one stream changes from V1 to V2
+and another stream changes from V2 to V1 over the same interval, the sum of the volumes at each point in
+time is V1 + V2. This attribute is initially 1.0.
+
+<p class="todo">Specify the exact transition function. Tim says "w=cos((pi/2)*t)^2 for t=0...1".
+
+<p>The <code>params</code> attribute and the <code>setParams(params, startTime)</code> timed setter method set the paramters for this input. On setting, a <em>structured clone</em> of this object is made. The clone is sent to
+the worker (if there is one) during media processing. On getting, a fresh clone is returned.
+
+<p>For the timed setter methods of <code>MediaInput</code>, the subject stream is the output stream, so changes take effect when the output stream's current time is equal to <code>startTime</code>.
+
+<p>The <code>blockInput</code> and <code>blockOutput</code> attributes control
+how the blocking status of the input stream is related to the blocking status of the output stream.
+When <code>blockOutput</code> is true and the input port is active, if the input stream is blocked then the output stream must be blocked. While an active input is blocked and the output is not blocked, the input is treated as having no tracks. When <code>blockInput</code> is true and the input port is active, if the output is blocked,
+then the input stream must be blocked. When false, while the output is blocked and an active input is not, the input will simply be discarded. These attributes are initially true.
+
+<p class="XXX">Need to look again at these. It's not clear we have use cases for both attributes, and I haven't implemented them yet and they could be hard to implement.
+
+<p>The <code>remove(time)</code> method removes this <code>MediaInput</code> from the inputs array of its owning
+<code>ProcessedMediaStream</code> at the given time relative to the output stream (or later, if it cannot be removed in time).
+If <code>time</code> is omitted, the input is removed as soon as possible and the <code>MediaInput</code> is removed from
+the destionation stream's <code>input</code> array immediately.
+After removal, the <code>MediaInput</code> object is no longer used; its attributes retain their current values
+and do not change unless explicitly set. All method calls are ignored. Additional calls to <code>remove</code> with an earlier time
+can advance the removal time, but once removal is scheduled it cannot be stopped or delayed.
+
+<h3 id="worker-processing">4.4 Worker Processing</h3>
+
+<p>A <code>ProcessedMediaStream</code> with a worker computes its output by dispatching a sequence of <code>onprocessmedia</code> callbacks to the worker, passing each a <code>ProcessMediaEvent</code> parameter. A <code>ProcessMediaEvent</code> provides audio sample buffers for each input stream. Each sample buffer for a given <code>ProcessMediaEvent</code> has the same duration, so the inputs presented to the worker are always in sync. (Inputs may be added or removed between <code>ProcessMediaEvent</code>s, however.) The sequence of buffers provided for an input stream is the audio data to be played by that input stream. The user-agent will precompute data for the input streams as necessary.
+
+<p>For example, if a Worker computes the output sample for time T as a function of the [T - 1s, T + 1s] interval of an input stream, then initially the Worker would simply refuse to output anything until it has received at least 1s of input stream data, forcing the user-agent to precompute the input stream at least 1s ahead of the current time. (Note that large Worker latencies will increase the latency of changes to the media graph.)
+
+<p class="note">Note that <code>Worker</code>s do not have access to most DOM API objects. In particular, <code>Worker</code>s have no direct access to <code>MediaStream</code>s.
+
+<p class="note">Note that a <code>ProcessedMediaStream</code>'s worker cannot be a
+<code>SharedWorker</code>. This ensures that the worker can run in the same process as the page in multiprocess browsers, so media streams can be confined to a single process.
+
+<p class="todo">Currently <code>ProcessMediaEvent</code> does not offer access to video data. This should be added later.
+
+<pre><code>partial interface DedicatedWorkerGlobalScope {
+ attribute Function onprocessmedia;
+};</pre></code>
+
+<p>The <code>onprocessmedia</code> attribute is the function to be called whenever stream data needs to be processed.
+A <code>ProcessMediaEvent</code> is passed as the single parameter to each call to the <code>onprocessmedia</code> callback.
+For a given <code>ProcessedMediaStream</code>, the same <code>ProcessMediaEvent</code> is passed in every call to the
+<code>onprocessmedia</code> callback. This allows the callback function to maintain per-stream state.
+
+<pre><code>interface ProcessMediaEvent : Event {
+ readonly attribute double inputTime;
+
+ readonly attribute any params;
+ readonly attribute double paramsStartTime;
+
+ readonly attribute MediaInputBuffer inputs[];
+ readonly attribute long audioSampleRate;
+ readonly attribute short audioChannels;
+ reaodnly attribute long audioLength;
+
+ void writeAudio(in Float32Array data);
+
+ void finish();
+};</pre></code>
+
+<p>The <code>inputTime</code> attribute returns the duration of the input that has been consumed by the
+<code>ProcessedMediaStream</code> for this worker.
+
+<p>The <code>params</code> attribute provides a structured clone of the parameters object set by
+<code>ProcessedMediaStream.setParams</code>. The same object is returned in each event, except when the object has
+been changed by <code>setParams</code> between events. <p>The <code>paramsStartTime</code> attribute returns the first time (measured in duration of input consumed for this stream) that this <code>params</code> object was set.
+
+<p class="note">Note that the parameters objects are constant over the duration of the inputs presented in the
+event. Frequent changes to parameters will reduce the length of the input buffers that can be presented to
+the worker.
+
+<p><code>inputs</code> provides access to <code>MediaStreamBuffers</code> for each active input stream
+(in the same order as those streams appear in the <code>ProcessedMediaStream.inputs</code> array).
+
+<p><code>audioSampleRate</code> and <code>audioChannels</code> represent the format of the input and
+output audio sample buffers. <code>audioSampleRate</code> is the number of samples per second.
+<code>audioChannels</code> is the number of channels; the channel mapping is as defined in the Vorbis specification.
+These values are constant for a given <code>ProcessedMediaStream</code>. When the <code>ProcessedMediaStream</code>
+was constructed using the Worker constructor, these values are the values passed as parameters there. When the
+<code>ProcessedMediaStream</code> was constructed via <code>MediaStream.createProcessor</code>, the values are
+chosen to match the first active input stream (or 44.1KHz, 2 channels if there is no active input stream).
+
+<p><code>audioLength</code> is the duration of the input(s) multiplied by the sample rate. If there are no inputs,
+the user-agent will choose a value representing the suggested amount of audio that the worker should produce.
+
+<p><code>writeAudio(data)</code> writes audio data to the stream output.
+The output has a single audio track. If there is an active input with an audio track, then the metadata for the output audio track is set to the metadata for the audio track of the last active input that has an audio track, otherwise the output audio track's <code>kind</code> is "main" and the other metadata attriutes are the empty string. The data for the output audio track is the concatenation of the
+inputs to each <code>writeAudio</code> call before the event handler returns. The data buffer is laid out
+with the channels non-interleaved, as for the input buffers (see below). The length of <code>data</code> must be
+a multiple of <code>audioChannels</code>; if not, then only the sample values up to the largest multiple of <code>audioChannels</code> less than the data length are used.
+
+<p>It is permitted to write less audio than the duration of the inputs (including none). This indicates latency in the filter. Normally the user-agent will dispatch another event to provide
+more input until the worker starts producing output. It is also permitted to write more audio than the duration of the inputs, for example if there are no inputs.
+Filters with latency should respond to an event with no inputs by writing out some of their buffered data; the user-agent
+is draining them.
+
+<p class="note">A synthesizer with no inputs can output as much data as it wants; the UA will buffer data and fire events as necessary. Filters that misbehave, e.g. by always writing zero-length buffers, will cause the stream to block due to an underrun.
+
+<p>If <code>writeAudio</code> is not called during the event handler, then the input audio buffers are added together and written to the output.
+
+<p>If <code>writeAudio</code> is called outside the event handler, the call is ignored.
+
+<p>Calling <code>finish()</code> puts the stream into the finished state (once any previously buffered output has been consumed). The event callback will never be called again. <code>finish()</code> can be called at any time, inside or outside the event handler.
+
+<p>The output video track is computed as if there was no worker (see above).
+
+<p class="todo">This will change when we add video processing.
+
+<pre><code>interface MediaInputBuffer {
+ readonly attribute any params;
+ readonly attribute double paramsStartTime;
+
+ readonly attribute Float32Array audioSamples;
+};</pre></code>
+
+<p>The <code>params</code> attribute provides a structured clone of the parameters object set by
+<code>MediaInput.setParams</code>. The same object is returned in each event, except when the object has
+been changed by <code>setParams</code> between events. <p>The <code>paramsStartTime</code> attribute returns the first time (measured in duration of input consumed for this stream) that this <code>params</code> object was set.
+
+<p><code>audioSamples</code> gives access to the audio samples for each input stream. The array length will be <code>event.audioLength</code> multiplied by <code>event.audioChannels</code>. The samples are floats ranging from -1 to 1, laid out non-interleaved, i.e. consecutive segments of <code>audioLength</code> samples each. The durations of the input buffers for the input streams will be equal. The <code>audioSamples</code> object will be a fresh object in each event. For inputs with no audio track, <code>audioSamples</code> will be all zeroes.
+
+<h2 id="built-in-processing-engine">5. Built-In Processing Engines</h2>
+
+<h3 id="default-processing-engine">5.1. Default Processing Engine</h2>
+
+<p>A <code>ProcessedMediaStream</code> with the default processing engine produces output as follows:
+<ul>
+<li>If no active input has an audio track, the output has no audio track. Otherwise, the output has a single
+audio track whose metadata (<code>id</code>, <code>kind</code>, <code>label</code>, and <code>language</code>)
+is equal to that of the audio track for the last active input that has an audio track. The output audio track
+is produced by adding the samples of the audio tracks of the active inputs together.
+<li>If no active input has a video track, the output has no video track. Otherwise, the output has a single
+video track whose metadata (<code>id</code>, <code>kind</code>, <code>label</code>, and <code>language</code>)
+is equal to that of the video track for the last active input that has a video track. The output video track
+is produced by compositing together all the video frames from the video tracks of the active inputs, with the video
+frames from higher-numbered inputs on top of the video frames from lower-numbered inputs; each
+video frame is letterboxed to the size of the video frame for the last active input that has a video track.
+<p class="note">This means if the last input's video track is opaque, the video output is simply the video track of the last input.
+</ul>
+
+<h3 id="lastinput-processing-engine">5.2. "LastInput" Processing Engine</h2>
+
+<p>A <code>ProcessedMediaStream</code> with the "LastInput" processing engine simply produces the last active input stream as output. If there are no active input streams, it produces the same output as the default processing engine.
+
+<h2 id="media-graph-considerations">6. Media Graph Considerations</h2>
+
+<h3 id="cycles">6.1. Cycles</h3>
+
+<p>While a <code>ProcessedMediaStream</code> has itself as a direct or indirect input stream (considering only active inputs), it is blocked.
+
+<h3 id="blocking">6.2. Blocking</h2>
+
+<p>At each moment, every stream should not be blocked except as explicitly required by this specification.
+
+<h2 id="canvas-recording">7. Canvas Recording</h2>
+
+<p>To enable video synthesis and some easy kinds of video effects we can record the contents of a canvas:
+
+<pre><code>partial interface HTMLCanvasElement {
+ readonly attribute MediaStream stream;
+};</pre></code>
+
+<p>The <code>stream</code> attribute is a stream containing a video track with the "live" contents of the canvas as video frames whose size is the size of the canvas, and no audio track. It always returns the same stream for a given element.
+
+<h2 id="implementation-considerations">8. Implementation Considerations</h2>
+
+<p class="todo">Here will be some non-normative implementation suggestions.
+
+<h2 id="examples">9. Examples</h2>
+
+<p class="todo">Add Worker scripts for these examples.
+
+<ol>
+<li>Play video with processing effect applied to the audio track
+
+<pre><code><video src="foo.webm" id="v" controls></video>
+<audio id="out" autoplay></audio>
+<script>
+document.getElementById("out").src =
+ document.getElementById("v").captureStream().createWorkerProcessor(new Worker("effect.js"));
+</script></pre></code>
+
+<li>Play video with processing effects mixing in out-of-band audio tracks (in sync)
+
+<pre><code><video src="foo.webm" id="v"></video>
+<audio src="back.webm" id="back"></audio>
+<audio id="out" autoplay></audio>
+<script>
+ var mixer = document.getElementById("v").captureStream().createWorkerProcessor(new Worker("audio-ducking.js"));
+ mixer.addInput(document.getElementById("back").captureStream());
+ document.getElementById("out").src = mixer;
+ function startPlaying() {
+ document.getElementById("v").play();
+ document.getElementById("back").play();
+ }
+ // MediaController is a more convenient API because it ties together control of the elements,
+ // but using streams is more flexible (e.g. they can be seeked to different offsets).
+</script></pre></code>
+
+<li>Capture microphone input and stream it out to a peer with a processing effect applied to the audio
+
+<pre><code><script>
+ navigator.getUserMedia('audio', gotAudio);
+ function gotAudio(stream) {
+ peerConnection.addStream(stream.createWorkerProcessor(new Worker("effect.js")));
+ }
+</script></pre></code>
+
+<li>Capture microphone input and visualize it as it is being streamed out to a peer and recorded
+
+<pre><code><canvas id="c"></canvas>
+<script>
+ navigator.getUserMedia('audio', gotAudio);
+ var streamRecorder;
+ function gotAudio(stream) {
+ var worker = new Worker("visualizer.js");
+ var processed = stream.createWorkerProcessor(worker);
+ worker.onmessage = function(event) {
+ drawSpectrumToCanvas(event.data, document.getElementById("c"));
+ }
+ streamRecorder = processed.record();
+ peerConnection.addStream(processed);
+ }
+</script></pre></code>
+
+<li>Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record
+
+<pre><code><canvas id="c"></canvas>
+<audio src="back.webm" id="back"></audio>
+<script>
+ navigator.getUserMedia('audio', gotAudio);
+ var streamRecorder;
+ function gotAudio(stream) {
+ var worker = new Worker("visualizer.js");
+ var processed = stream.createWorkerProcessor(worker);
+ worker.onmessage = function(event) {
+ drawSpectrumToCanvas(event.data, document.getElementById("c"));
+ }
+ var mixer = processed.createProcessor();
+ mixer.addInput(document.getElementById("back").captureStream());
+ streamRecorder = mixer.record();
+ peerConnection.addStream(mixer);
+ }
+</script></pre></code>
+
+<li>Receive audio streams from peers, mix them with spatialization effects, and play
+
+<pre><code><script>
+ var worker = new Worker("spatializer.js");
+ var spatialized = stream.createWorkerProcessor(worker);
+ peerConnection.onaddstream = function (event) {
+ spatialized.addInput(event.stream).params = {x:..., y:..., z:...};
+ };
+ (new Audio(spatialized)).play();
+</script></pre></code>
+
+<li>Seamlessly chain from the end of one input stream to another
+
+<p class="note">This method requires that you know each stream's duration, which is a bit suboptimal.
+To get around that we'd need new API, perhaps a new kind of ProcessedMediaStream that plays streams in
+serial.
+
+<pre><code><audio src="in1.webm" id="in1" preload></audio>
+<audio src="in2.webm" id="in2"></audio>
+<script>
+ var in1 = document.getElementById("in1");
+ in1.onloadedmetadata = function() {
+ var mixer = in1.captureStreamUntilEnded().createProcessor();
+ var in2 = document.getElementById("in2");
+ mixer.addInput(in2.captureStreamUntilEnded(), in1.duration);
+ (new Audio(mixer)).play();
+ in1.play();
+ }
+</script></pre></code>
+
+<li>Seamlessly switch from one input stream to another, e.g. to implement adaptive streaming
+
+<p class="note">There are two ways to implement seamless switching: seek the second resource to before the
+current time and then run the decoder faster than real-time to catch up to the first resource's play point, or
+seek the second resource to after the current time and enable it when the first resource catches up to the seek
+point. The first is more robust if the seek takes unexpectedly long, but the second is less demanding on the
+decoder. Only the second method is currently implementable with this API (since by design there is no way to drive MediaStreams faster than real-time). If we want to support the first method as well, the right way would be to
+add API to media elements to let them seek to synchronize with a given MediaStream.
+
+<pre><code><audio src="in1.webm" id="in1" preload></audio>
+<audio src="in2.webm" id="in2"></audio>
+<audio id="out" autoplay></audio>
+<script>
+ var stream1 = document.getElementById("in1").captureStream();
+ var mixer = stream1.createProcessor("LastInput");
+ document.getElementById("out").src = mixer;
+ function switchStreams() {
+ var in2 = document.getElementById("in2");
+ in2.currentTime = in1.currentTime + 10; // arbitrary, but we should be able to complete the seek within this time
+ mixer.addInput(in2.captureStream(), mixer.currentTime + 10);
+ in2.play();
+ // in2 will be blocked until the input port is enabled
+ in2.onplaying = function() { mixer.inputs[0].remove(); };
+ }
+</script></pre></code>
+
+<li>Synthesize samples from JS data
+
+<pre><code><audio id="out" autoplay></audio>
+<script>
+ document.getElementById("out").src =
+ new ProcessedMediaStream(new Worker("synthesizer.js"));
+</script></pre></code>
+
+<li>Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking
+
+<pre><code><script>
+ var effectsMixer = ...;
+ function playSound(src) {
+ var audio = new Audio(src);
+ audio.oncanplaythrough = function() {
+ var stream = audio.captureStreamUntilEnded();
+ var port = effectsMixer.addInput(stream);
+ port.blockOutput = false;
+ audio.play();
+ }
+ }
+</script></pre></code>
+
+<li>Trigger a sound sample to be played through the effects graph in five seconds
+
+<pre><code><script>
+ var effectsMixer = ...;
+ var audio = new Audio(...);
+ function triggerSound() {
+ var sound = audio.clone();
+ var stream = sound.captureStreamUntilEnded();
+ sound.play();
+ effectsMixer.addInput(stream, effectsMixer.currentTime + 5);
+ }
+</script></pre></code>
+
+<li>Capture video from a camera and analyze it (e.g. face recognition)
+
+<pre><code><script>
+ navigator.getUserMedia('video', gotVideo);
+ function gotVideo(stream) {
+ stream.createWorkerProcessor(new Worker("face-recognizer.js"));
+ }
+</script></pre></code>
+
+<li>Capture video, record it to a file and upload the file (e.g. Youtube)
+
+<pre><code><script>
+ navigator.getUserMedia('video', gotVideo);
+ var streamRecorder;
+ function gotVideo(stream) {
+ streamRecorder = stream.record();
+ }
+ function stopRecording() {
+ streamRecorder.getRecordedData(gotData);
+ }
+ function gotData(blob) {
+ var x = new XMLHttpRequest();
+ x.open('POST', 'uploadMessage');
+ x.send(blob);
+ }
+</script></pre></code>
+
+<li>Capture video from a canvas, record it to a file then upload
+
+<pre><code><canvas width="640" height="480" id="c"></canvas>
+<script>
+ var canvas = document.getElementById("c");
+ var streamRecorder = canvas.stream.record();
+ function stopRecording() {
+ streamRecorder.getRecordedData(gotData);
+ }
+ function gotData(blob) {
+ var x = new XMLHttpRequest();
+ x.open('POST', 'uploadMessage');
+ x.send(blob);
+ }
+ var frame = 0;
+ function updateCanvas() {
+ var ctx = canvas.getContext("2d");
+ ctx.clearRect(0, 0, 640, 480);
+ ctx.fillText("Frame " + frame, 0, 200);
+ ++frame;
+ }
+ setInterval(updateCanvas, 30);
+</script></pre></code>
+</ol>
+
+<h1>Related Work</h1>
+
+<ul>
+<li><a href="https://wiki.mozilla.org/RTCStreamAPI">W3C-RTC charter</a> (Harald et. al.)
+<li><a href="http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html">WhatWG proposal (Ian Hickson et. al.)</a>
+<li><a href="http://chromium.googlecode.com/svn/trunk/samples/audio/specification/specification.html">Chrome audio API</a>
+<li><a href="https://wiki.mozilla.org/Audio_Data_API">Mozilla audio API</a>
+<li><a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#mediacontroller">WhatWG MediaController API</a>
+</body>
+</html>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/streams/main.css Thu Nov 03 11:10:05 2011 +1300
@@ -0,0 +1,128 @@
+
+/* Style for a Working Group Editors' Draft */
+
+/*
+ Copyright 1997-2003 W3C (MIT, ERCIM, Keio). All Rights Reserved.
+ The following software licensing rules apply:
+ http://www.w3.org/Consortium/Legal/copyright-software */
+
+/* $Id: base.css,v 1.25 2006/04/18 08:42:53 bbos Exp $ */
+
+body {
+ padding: 2em 1em 2em 70px;
+ margin: 0;
+ font-family: sans-serif;
+ color: black;
+ background: white;
+ background-position: top left;
+ background-attachment: fixed;
+ background-repeat: no-repeat;
+}
+:link { color: #00C; background: transparent }
+:visited { color: #609; background: transparent }
+a:active { color: #C00; background: transparent }
+
+a:link img, a:visited img { border-style: none } /* no border on img links */
+
+a img { color: white; } /* trick to hide the border in Netscape 4 */
+@media all { /* hide the next rule from Netscape 4 */
+ a img { color: inherit; } /* undo the color change above */
+}
+
+th, td { /* ns 4 */
+ font-family: sans-serif;
+}
+
+h1, h2, h3, h4, h5, h6 { text-align: left }
+/* background should be transparent, but WebTV has a bug */
+h1, h2, h3 { color: #005A9C; background: white }
+h1 { font: 170% sans-serif }
+h2 { font: 140% sans-serif }
+h3 { font: 120% sans-serif }
+h4 { font: bold 100% sans-serif }
+h5 { font: italic 100% sans-serif }
+h6 { font: small-caps 100% sans-serif }
+
+.hide { display: none }
+
+div.head { margin-bottom: 1em }
+div.head h1 { margin-top: 2em; clear: both }
+div.head table { margin-left: 2em; margin-top: 2em }
+
+p.copyright { font-size: small }
+p.copyright small { font-size: small }
+
+@media screen { /* hide from IE3 */
+a[href]:hover { background: #ffa }
+}
+
+pre { margin-left: 2em }
+dt, dd { margin-top: 0; margin-bottom: 0 } /* opera 3.50 */
+dt { font-weight: bold }
+
+pre, code {
+ font-family: monospace;
+ overflow: auto;
+ margin: 0;
+}
+pre > code {
+ display: block;
+ padding: 1em;
+ border: 1px solid black;
+ background: #ddd;
+ margin: 0.5em 2em;
+ margin-bottom: 1em;
+ font-size: 120%;
+}
+code var { color: #f44; }
+
+.note {
+ margin: 1em;
+ padding: 1em;
+ background: yellow;
+}
+
+.XXX {
+ margin: 1em;
+ padding: 1em;
+ background: pink;
+}
+.XXX:before {
+ content: "XXX "
+}
+
+.todo {
+ margin: 1em;
+ padding: 1em;
+ background: cyan;
+}
+.todo:before {
+ content: "TODO: "
+}
+
+.example {
+ margin: 1em;
+ padding: 1em;
+ background: lime;
+}
+
+ul.toc, ol.toc {
+ list-style: disc; /* Mac NS has problem with 'none' */
+ list-style: none;
+}
+
+@media aural {
+ h1, h2, h3 { stress: 20; richness: 90 }
+ .hide { speak: none }
+ p.copyright { volume: x-soft; speech-rate: x-fast }
+ dt { pause-before: 20% }
+ pre { speak-punctuation: code }
+}
+
+body {
+/* background-image: url(http://www.w3.org/StyleSheets/TR/logo-ED); */
+}
+
+#toc li {
+ list-style-type:none;
+}