Proposal: Media Capture and Streams Settings API v5

1.1 Rationale

Provide a predictable home for developers to find and modify device settings

A previous proposal went out of its way to strongly associate LocalMediaStream objects with devices. This seemed like a good design because local device configuration is always on the local media stream. This made for a stable, dependable API surface for all local media stream instances (no guesswork).

Prevent track-list mutations

A previous proposal also removed the track lists on local media streams (resulting in some dramatic inheritance changes). Mutable tracks lists on LocalMediaStream objects seemed like the wrong design considering the current thinking that a getUserMedia request would only ever produce a LocalMediaStream with at most one audio or video track.

Some feedback even suggested re-considering the "at most one video/audio track per request to getUserMedia".

While thinking about these goals and the feedback, I began to consider a few things:

Device-centric tracks

With tracks supplemented with device-characteristics (duck-typing), the LocalMediaStream's stop() API was a convenience feature for stopping all tracks backed by a device on the LocalMediaStream object. With device- centric tracks a stop() API should be present on the tracks themselves.

Mutable track lists

Mutable track lists were not a desirable feature while I was locked into considering the LocalMediaStream as strongly associated with device-control. What is actually necessary, is that there is a something immutable associated with devices--that "something" doesn't necessarily need to be a LocalMediaStream or any MediaStream-like object at all! Once I unlocked that line of thinking, I began to experiment with the notion of a device list which then ultimately brought back a use-case for having mutable track lists for MediaStream objects. (It did not bring back a need for LocalMediaStream objects themselves though.)

Work flow for access to additional device streams

It is now understood that to request additional streams for different devices (e.g., the second camera on a dual-camera mobile phone), one must invoke getUserMedia a second time. In my prior proposal, this would result in a separate LocalMediaStream instance. At this point there are two LocalMediaStream objects each with their own devices. While this was nice for consistency of process, it was a challenge in terms of use of the objects with a MediaStream consumer like the Video tag.

To illustrate this challenge, consider how my prior proposal required a re-hookup of the MediaStream to a video tag consumer:

1. First request to getUserMedia
2. LocalMediaStream (1) obtained from success callback
3. createObjectURL and preview in a video tag
4. Second call to getUserMedia
5. LocalMediaStream (2) obtained from success callback
6. createObjectURL and preview in same video tag

Note that this process has to bind a completely new LocalMediaStream to the video tag a second time (if re-using the same video tag) only because the second LocalMediaStream object was different than the first.

It is much more efficient for developer code to simply add/remove tracks to a MediaStream that are relevant, without needing to change the consumer of the MediaStream.

Usage of getUserMedia for permission rather than for additional device access

The getUserMedia method is the gateway for permission to media. This proposal does not suggest changing that concept. It does suggest, however, that more information can be made available for discovery of additional devices within the approved "category" or "type" of media, and provide a way to obtain those additional devices without needing to go through the "permissions" route (i.e., getUserMedia).

Importance of restricting control to LocalMediaStream

Upon reflection of the feedback around the prior proposal, the relative importance of restricting control of the devices associated with tracks on the LocalMediaStream to only the LocalMediaStream did not seem as vital, insofar as the device-level access via the track is not directly available through a PeerConnection to a remote browser.

2.1 Updating MediaStreamTrack

This section defines MediaStreamTrack in order to add the new "new" state and associated event handlers. The definition is otherwise identical to the current definition except that the defined constants are replaced by strings (using an enumerated type).

interface MediaStreamTrack : EventTarget {
             attribute DOMString           id;
    readonly attribute DOMString           kind;
    readonly attribute DOMString           label;
             attribute boolean             enabled;
    readonly attribute TrackReadyStateEnum readyState;
             attribute EventHandler        onstart;
             attribute EventHandler        onmute;
             attribute EventHandler        onunmute;
             attribute EventHandler        onended;
};

enum TrackReadyStateEnum {
    "new",
    "live",
    "muted",
    "ended"
};

2.2 Creating Derived Tracks

MediaStreamTrack objects cannot be instantiated directly. To create an instance of a MediaStreamTrack, one of its derived track types may be instantiated directly. These derived types are defined in this section. Each of these track types has general IDL attributes specific to all tracks of the given type as well as a mechanism to obtain the device object that is providing the source for this track.

It's important to note that the camera's green light doesn't come on when a new track is created; nor does the user get prompted to enable the camera/microphone. Those actions only happen after the developer has requested that a media stream containing "new" tracks be bound to a source via getUserMedia. Until that point tracks are inert.

[Constructor]
interface VideoStreamTrack : MediaStreamTrack {
    readonly attribute VideoFacingEnum    facing;
    readonly attribute VideoStreamSource? source;
};

[Constructor]
interface AudioStreamTrack : MediaStreamTrack {
    readonly attribute unsigned long      level;
    readonly attribute AudioStreamSource? source;
};

3.1 Local Video and Audio Sources

VideoStreamSource and AudioStreamSource objects are created by the user agent to represent a camera or microphone device/source for which the source attributes can be inspected and/or changed. At the moment these are limited to local cameras, local microphones, and peer connection sources, but additional sources can be defined later (such a local file system sources for images or audio files).

interface VideoStreamSource : EventTarget {
    readonly attribute unsigned long          width;
    readonly attribute unsigned long          height;
    readonly attribute float                  frameRate;
    readonly attribute VideoRotationEnum      rotation;
    readonly attribute VideoMirrorEnum        mirror;
    readonly attribute float                  zoom;
    readonly attribute VideoFocusModeEnum     focusMode;
    readonly attribute VideoFillLightModeEnum fillLightMode;
    void                 stop ();
    static unsigned long getNumDevices ();
};

enum VideoFacingEnum {
    "unknown",
    "user",
    "environment"
};

enum VideoRotationEnum {
    "0",
    "90",
    "180",
    "270"
};

enum VideoMirrorEnum {
    "none",
    "horizontal",
    "vertical"
};

enum VideoFocusModeEnum {
    "notavailable",
    "auto",
    "manual"
};

enum VideoFillLightModeEnum {
    "notavailable",
    "auto",
    "off",
    "flash",
    "on"
};

interface AudioStreamSource : EventTarget {
    readonly attribute unsigned long gain;
    void                 stop ();
    static unsigned long getNumDevices ();
};

3.2 Camera sources with "high-resolution picture" modes

The PictureStreamSource derived interface is created by the user agent if the camera source providing the VideoStreamSource supports an optional "high-resolution picture mode" with picture settings that are separate from those of its basic video source (which is usually considered its preview mode).

The PictureStreamSource object presents a set of capabilities and controls for taking high-resolution pictures. The unique settings of this object are only applied at the time when the takePicture API is invoked.

interface PictureStreamSource : VideoStreamSource {
    readonly attribute unsigned long photoWidth;
    readonly attribute unsigned long photoHeight;
    void takePicture ();
             attribute EventHandler  onpicture;
             attribute EventHandler  onpictureerror;
};

[Constructor(DOMString type, optional BlobEventInit blobInitDict)]
interface BlobEvent : Event {
    readonly attribute Blob data;
};

dictionary BlobEventInit : EventInit {
    Blob data;
};

3.3 Remote Media Sources

When MediaStreams are transmitted over the network by way of a peer connection, the tracks that are created on the remote side of the MediaStream will have a remote media source attached as the track's source. This source object allows remote-consumers of the MediaStream's tracks to request specific changes to the tracks. These change requests will be serviced by the RTCPeerConnection source object which is streaming the media over the network.

interface VideoStreamRemoteSource : EventTarget {
    readonly attribute unsigned long width;
    readonly attribute unsigned long height;
    readonly attribute float         frameRate;
    readonly attribute float         bitRate;
};

interface AudioStreamRemoteSource : EventTarget {
    readonly attribute float bitRate;
};

3.4 Other Settings (out-of-scope in this proposal)

The following settings have been proposed, but are not included in this version to keep the initial set of settings scoped to those that:

1. cannot be easily computed in post-processing
2. are not redundant with other settings
3. are settings found in nearly all devices (common)
4. can be easily tested for conformance

Each setting also includes a brief explanatory rationale for why it's not included:

1. horizontalAspectRatio - easily calculated based on width/height in the dimension values
2. verticalAspectRatio - see horizontalAspectRatio explanation
3. orientation - can be easily calculated based on the width/height values and the current rotation
4. aperatureSize - while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
5. shutterSpeed - see aperatureSize explanation
6. denoise - may require specification of the algorithm processing or related image processing filter required to implement.
7. effects - sounds like a v2 or independent feature (depending on the effect).
8. faceDetection - sounds like a v2 feature. Can also be done using post-processing techniques (though perhaps not as fast...)
9. antiShake - sounds like a v2 feature.
10. geoTagging - this can be independently associated with a recorded picture/video/audio clip using the Geolocation API. Automatically hooking up Geolocation to Media Capture sounds like an exercise for v2 given the possible complications.
11. highDynamicRange - not sure how this can be specified, or if this is just a v2 feature.
12. skintoneEnhancement - not a particularly common setting.
13. shutterSound - Can be accomplished by syncing custom audio playback via the <audio> tag if desired. By default, there will be no sound issued.
14. redEyeReduction - photo-specific setting. (Could be considered if photo-specific settings are introduced.)
15. meteringMode - photo-specific setting. (Could be considered if photo-specific settings are introduced.)
16. iso - photo-specific setting. while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
17. sceneMode - while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
18. antiFlicker - not a particularly common setting.
19. zeroShutterLag - this seems more like a hope than a setting. I'd rather just have implementations make the shutter snap as quickly as possible after takePicture, rather than requiring an opt-in/opt-out for this setting.

The following settings may be included by working group decision:

1. exposure
2. exposureCompensation (is this the same as exposure?)
3. autoExposureMode
4. brightness
5. contrast
6. saturation
7. sharpness
8. evShift
9. whiteBalance

4.1 Expectations around changing settings

Browsers provide a media pipeline from sources to sinks. In a browser, sinks are the <img>, <video> and <audio> tags. Traditional sources include streamed content, files and web resources. The media produced by these sources typically does not change over time - these sources can be considered to be static.

The sinks that display these sources to the user (the actual tags themselves) have a variety of controls for manipulating the source content. For example, an <img> tag scales down a huge source image of 1600x1200 pixels to fit in a rectangle defined with width="400" and height="300".

The getUserMedia API adds dynamic sources such as microphones and cameras - the characteristics of these sources can change in response to application needs. These sources can be considered to be dynamic in nature. A <video> element that displays media from a dynamic source can either perform scaling or it can feed back information along the media pipeline and have the source produce content more suitable for display.

This proposal assumes that MediaStream sinks (such as <video>, <audio>, and even RTCPeerConnection) will continue to have mechanisms to further transform the source stream beyond that which the settings described in this proposal offer. (The sink transformation options, including those of RTCPeerConnection are outside the scope of this proposal.)

The act of changing a setting on a stream's source will, by definition, affect all down-level sinks that are using that source. Many sinks may be able to take these changes in stride, such as the <video> element or RTCPeerConnection. Others like the Recorder API may fail as a result of a source change.

The RTCPeerConnection is an interesting object because it acts simultaneously as both a sink and a source for over-the-network streams. As a sink, it has source transformational capabilities (e.g., lowering bit-rates, scaling-up or down resolutions, adjusting frame-rates), and as a source it may have its own settings changed by a track source that it provides (in this proposal, such sources are the VideoStreamRemoteSource and AudioStreamRemoteSource objects).

To illustrate how changes to a given source impact various sinks, consider the following example. This example only uses width and height, but the same principles apply to any of the settings exposed in this proposal. In the first figure a home client has obtained a video source from its local video camera. The source device's width and height are 800 pixels by 600 pixels, respectively. Three MediaStream objects on the home client contain tracks that use this same source. The three media streams are connected to three different sinks, a <video> element (A), another <video> element (B), and a peer connection (C). The peer connection is streaming the source video to an away client. On the away client there are two media streams with tracks that use the peer connection as a source. These two media streams are connected to two <video> element sinks (Y and Z).

Note that in the current state, all of the sinks on the home client must apply a transformation to the original source's dimensions. A is scaling the video up (resulting in loss of quality), B is scaling the video down, and C is also scaling the video up slightly for sending over the network. On the away client, sink Y is scaling the video way down, while sink Z is not applying any scaling.

Using the settings APIs defined in the next section, the home client's video source is changed to a higher resolution (1920 by 1200 pixels).

Note that the source change immediately effects all of the sinks on home client, but does not impact any of the sinks (or sources) on the away client. With the increase in the home client source video's dimensions, sink A no longer has to perform any scaling, while sink B must scale down even further than before. Sink C (the peer connection) must now scale down the video in order to keep the transmission constant to the away client.

While not shown, an equally valid settings change request could be made of the away client video source (the peer connection on the away client's side). This would not only impact sink Y and Z in the same manner as before, but would also cause re-negotiation with the peer connection on the home client in order to alter the transformation that it is applying to the home client's video source. Such a change would not change anything related to sink A or B or the home client's video source.

It is fairly obvious that changes to a given source will impact sink consumers. However, in some situations changes to a given sink may also be cause for implementations to adjust the characteristics of a source's stream. This is illustrated in the following figures. In the first figure below, the home client's video source is sending a video stream sized at 1920 by 1200 pixels. The video source is also unconstrained, such that the exact source dimensions are flexible as far as the application is concerned. Two MediaStream objects contain tracks that use this same source, and those MediaStreams are connected to two different <video> element sinks A and B. Sink A has been sized to width="1920" and height="1200" and is displaying the sources video without any transformations. Sink B has been sized smaller and as a result, is scaling the video down to fit its rectangle of 320 pixels across by 200 pixels down.

When the application changes sink A to a smaller dimension (from 1920 to 1024 pixels wide and from 1200 to 768 pixels tall), the browser's media pipeline may recognize that none of its sinks require the higher source resolution, and needless work is being done both on the part of the source and on sink A. In such a case and without any other constraints forcing the source to continue producing the higher resolution video, the media pipeline may change the source resolution:

In the above figure, the home client's video source resolution was changed to the max(sinkA, sinkB) in order to optimize playback. While not shown above, the same behavior could apply to peer connections and other sinks.

4.2 StreamSourceSettings mix-in interface

VideoStreamSource implements StreamSourceSettings;

AudioStreamSource implements StreamSourceSettings;

VideoStreamRemoteSource implements StreamSourceSettings;

AudioStreamRemoteSource implements StreamSourceSettings;

[NoInterfaceObject]
interface StreamSourceSettings {
    (MediaSettingsRange or MediaSettingsList) getRange (DOMString settingName);
    any                                       get (DOMString settingName);
    void                                      set (MediaTrackConstraint setting, optional boolean isMandatory = false);
};

dictionary MediaSettingsRange {
    any max;
    any min;
    any initial;
};

dictionary MediaSettingsList {
    sequence<any> values;
    any           initial;
};

4.3 Tracking the result of constraint application

AudioStreamSource implements MediaSettingsEventHandlers;

VideoStreamSource implements MediaSettingsEventHandlers;

AudioStreamRemoteSource implements MediaSettingsEventHandlers;

VideoStreamRemoteSource implements MediaSettingsEventHandlers;

[NoInterfaceObject]
interface MediaSettingsEventHandlers {
             attribute EventHandler onsettingserror;
             attribute EventHandler onsettingschanged;
};

[Constructor(DOMString type, optional MediaSettingsEventInit eventInitDict)]
interface MediaSettingsEvent : Event {
    readonly attribute DOMString[] settings;
};

dictionary MediaSettingsEventInit : EventInit {
    sequence<DOMString> settings;
};

5.1 Video Constraints

The following constraints are applicable to video devices

dictionary VideoConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint)                     width;
    (unsigned long or MinMaxULongSubConstraint)                     height;
    (unsigned long or MinMaxULongSubConstraint)                     photoWidth;
    (unsigned long or MinMaxULongSubConstraint)                     photoHeight;
    VideoRotationEnum      rotation;
    VideoMirrorEnum        mirror;
    (float or MinMaxFloatSubConstraint)                     zoom;
    VideoFocusModeEnum     focusMode;
    VideoFillLightModeEnum fillLightMode;
    (float or MinMaxFloatSubConstraint)                     frameRate;
    (float or MinMaxFloatSubConstraint)                     bitRate;
};

5.2 Audio Constraints

The following constraints are applicable to audio devices

dictionary AudioConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint) gain;
};

5.3 Common sub-constraint structures

dictionary MinMaxULongSubConstraint {
    unsigned long max;
    unsigned long min;
};

dictionary MinMaxFloatSubConstraint {
    float max;
    float min;
};

6.1 Getting access to a video and/or audio device (if available)

var audioTrack = (AudioStreamSource.getNumDevices() > 0) ? new AudioStreamTrack() : null;
if (audioTrack)
   audioTrack.onstart = mediaStarted;
var videoTrack = (VideoStreamSource.getNumDevices() > 0) ? new VideoStreamTrack() : null;
if (videoTrack)
   videoTrack.onstart = mediaStarted;
var MS = new MediaStream();
MS.addTrack(audioTrack);
MS.addTrack(videoTrack);
navigator.getUserMedia(MS);

function mediaStarted() {
   // One of the video/audio devices started.
}

6.2 Previewing the local video/audio in HTML5 video tag -- scenario is unchanged

function mediaStarted() {
   // objectURL technique
   document.querySelector("video").src = URL.createObjectURL(MS, { autoRevoke: true }); // autoRevoke is the default
   // direct-assign technique
   document.querySelector("video").srcObject = MS; // Proposed API at this time
}

6.3 Applying resolution constraints

function mediaStarted() {
   var videoDevice = videoTrack.source;
   var maxWidth = videoDevice.getRange("width").max;
   var maxHeight = videoDevice.getRange("height").max;
   // Check for 1080p+ support
   if ((maxWidth >= 1920) && (maxHeight >= 1080)) {
      // See if I need to change the current settings...
      if ((videoDevice.width < 1920) && (videoDevice.height < 1080)) {
         videoDevice.set({ width: maxWidth}, true);
         videoDevice.set({ height: maxHeight}, true);
         videoDevice.onsettingserror = failureToComply;
      }
   }
   else
      failureToComply();
}

function failureToComply(e) {
   if (e)
      console.error("Devices failed to change " + e.settings); // 'width' and/or 'height'
   else
      console.error("Device doesn't support at least 1080p");
}

6.4 Changing zoom in response to user input:

function mediaStarted() {
   setupRange( videoTrack.source );
}

function setupRange(videoDevice) {
   var zoomCaps = videoDevice.getRange("zoom");
   // Check to see if the device supports zooming...
   if (zoomCaps.min != zoomCaps.max) {
      // Set HTML5 range control to min/max values of zoom
      var zoomControl = document.querySelector("input[type=range]");
      zoomControl.min = zoomCaps.min;
      zoomControl.max = zoomCaps.max;
      zoomControl.value = videoDevice.zoom;
      zoomControl.onchange = applySettingChanges;
   }
}

function applySettingChanges(e) {
   videoTrack.source.set({ zoom: parseFloat(e.target.value)}, true);
}

6.5 Adding the local media tracks into a new media stream:

function mediaStarted() {
   return new MediaStream( [ videoTrack, audioTrack ]);
}

6.6 Take a picture, show the picture in an image tag:

function mediaStarted() {
   var videoDevice = videoTrack.source;
   // Check if this device supports a picture mode...
   if (videoDevice.takePicture) {
       videoDevice.onpicture = showPicture;
       // Turn on flash only for the snapshot...if available
       if (videoDevice.fillLightMode != "notavailable")
          videoDevice.set({ fillLightMode: "flash"}, true);
       else
          console.info("Flash not available");
       videoDevice.takePicture();
   }
}

function showPicture(e) {
   var img = document.querySelector("img");
   img.src = URL.createObjectURL(e.data);
}

6.7 Show a newly available device

This scenario is not currently possible with this proposal.

6.8 Show all available video devices:

var totalVideoDevices = VideoStreamSource.getNumDevices();
var videoTracksList = [];
for (var i = 0; i < totalVideoDevices; i++) {
   var mediaStream = new MediaStream( new VideoStreamTrack() );
   // Create a video element and add it to the UI
   var videoTag = document.createElement('video');
   videoTag.srcObject = mediaStream;
   document.body.appendChild(videoTag);
   // Request to have the track connected to a source device (queue these up in the for-loop)
   navigator.getUserMedia(mediaStream);
}