Proposal: Settings API Version 4

1.1 Rationale

Provide a predictable home for developers to find and modify device settings

A previous proposal went out of its way to strongly associate LocalMediaStream objects with devices. This seemed like a good design because local device configuration is always on the local media stream. This made for a stable, dependable API surface for all local media stream instances (no guesswork).

Prevent track-list mutations

A previous proposal also removed the track lists on local media streams (resulting in some dramatic inheritance changes). Mutable tracks lists on LocalMediaStream objects seemed like the wrong design considering the current thinking that a getUserMedia request would only ever produce a LocalMediaStream with at most one audio or video track.

Some feedback even suggested re-considering the "at most one video/audio track per request to getUserMedia".

While thinking about these goals and the feedback, I began to consider a few things:

Device-centric tracks

With tracks supplemented with device-characteristics (duck-typing), the LocalMediaStream's stop() API was a convenience feature for stopping all tracks backed by a device on the LocalMediaStream object. With device- centric tracks a stop() API should be present on the tracks themselves.

Mutable track lists

Mutable track lists were not a desirable feature while I was locked into considering the LocalMediaStream as strongly associated with device-control. What is actually necessary, is that there is a something immutable associated with devices--that "something" doesn't necessarily need to be a LocalMediaStream or any MediaStream-like object at all! Once I unlocked that line of thinking, I began to experiment with the notion of a device list which then ultimately brought back a use-case for having mutable track lists for MediaStream objects. (It did not bring back a need for LocalMediaStream objects themselves though.)

Workflow for access to additional device streams

It is now understood that to request additional streams for different devices (e.g., the second camera on a dual-camera mobile phone), one must invoke getUserMedia a second time. In my prior proposal, this would result in a separate LocalMediaStream instance. At this point there are two LocalMediaStream objects each with their own devices. While this was nice for consistency of process, it was a challenge in terms of use of the objects with a MediaStream consumer like the Video tag.

To illustrate this challenge, consider how my prior proposal required a re-hookup of the MediaStream to a video tag consumer:

1. First request to getUserMedia
2. LocalMediaStream (1) obtained from success callback
3. createObjectURL and preview in a video tag
4. Second call to getUserMedia
5. LocalMediaStream (2) obtained from success callback
6. createObjectURL and preview in same video tag

Note that this process has to bind a completely new LocalMediaStream to the video tag a second time (if re-using the same video tag) only because the second LocalMediaStream object was different than the first.

It is much more efficient for developer code to simply add/remove tracks to a MediaStream that are relevant, without needing to change the consumer of the MediaStream.

Usage of getUserMedia for permission rather than for additional device access

The getUserMedia method is the gateway for permission to media. This proposal does not suggest changing that concept. It does suggest, however, that more information can be made available for discovery of additional devices within the approved "category" or "type" of media, and provide a way to obtain those additional devices without needing to go through the "permissions" route (i.e., getUserMedia).

Importance of restricting control to LocalMediaStream

Upon reflection of the feedback around the prior proposal, the relative importance of restricting control of the devices associated with tracks on the LocalMediaStream to only the LocalMediaStream did not seem as vital, insofar as the device-level access via the track is not directly available through a PeerConnection to a remote browser.

2.1 Local and remote video tracks

MediaStreamTrack objects that are of kind "video" and that are located in a MediaStream's videoTracks list will be instances of a VideoStreamTrack. The VideoStreamTrack provides basic (read-only) properties pertinent to all sources of video.

I'm intentionally keeping this interface as sparse as possible. Features about the video that can be calculated like aspect ratio are not provided.

interface VideoStreamTrack : MediaStreamTrack {
    readonly attribute unsigned long width;
    readonly attribute unsigned long height;
};

2.2 Local and remote audio tracks

MediaStreamTrack objects that are of kind "audio" and that are located in a MediaStream's audioTracks list will be instances of an AudioStreamTrack. The AudioStreamTrack provides basic (read-only) properties pertinent to all sources of audio.

interface AudioStreamTrack : MediaStreamTrack {
};

2.3 Camera device tracks

VideoDeviceTracks are created by the user agent to represent a camera device that provides local video.

interface VideoDeviceTrack : VideoStreamTrack {
    readonly attribute PictureDeviceTrack? pictureTrack;
    readonly attribute VideoFacingEnum     facing;
    void stop ();
};

enum VideoFacingEnum {
    "unknown",
    "user",
    "environment"
};

2.4 Cameras with "high-resolution picture" modes

The PictureDeviceTrack interface is created by the user agent if the camera device providing the VideoDeviceTrack supports an optional "high-resolution picture mode" with picture settings different (better) from those of its basic video constraints.

This track is initially available from a VideoDeviceTrack via the pictureTrack property. This track type is not present in the video device list (MediaDeviceList). Likewise, it cannot be stopped directly, and its VideoStreamTrack inherited attributes reflect the values of its "owning" VideoDeviceTrack.

The PictureDeviceTrack is essentially a specialized VideoStreamTrack (this track type is of kind "video"). It may be explicitly added to a videoTracks list (MediaStreamTrackList) in order to output its track video to a <video> tag, but its preview video stream reflects the owning VideoDeviceTrack's settings, rather than the settings directly available on this object. Rather the settings of this object are only applied at the time when the takePicture API is invoked.

interface PictureDeviceTrack : VideoStreamTrack {
    void takePicture ();
             attribute EventHandler onpicture;
};

[Constructor(DOMString type, optional PictureEventInit eventInitDict)]
interface PictureEvent : Event {
    readonly attribute Blob data;
};

2.5 Microphone device tracks

AudioDeviceTracks are created by the user agent to represent a microphone device that provides local audio.

interface AudioDeviceTrack : AudioStreamTrack {
    readonly attribute unsigned long level;
    void stop ();
};

3.1 Grouping setting features

Settings are organized into two groups: value ranges (a continuum of values) and enumerated values. Value ranges include a min and max value, while enumerated values are provided in an array with an associated length. Both groups of settings include an "initial" value, which is the value that is expected to be the device's default value when it is acquired.

The key to changing settings in either setting group is the request() API. This is the mechanism for asynchronously requesting that the device change the value of the setting that the setting group is applicable to. The mechanics for applying setting change requests follows exactly the model used when applying constraints at getUserMedia invocation. Each team a request() is made, the user agent begins building up an [internally-represented] constraint structure which is associated with the device making the request (and only that device). For example, if a "width" setting change request is made, the user agent creates a constraint structure equivalent to the following getUserMedia constraint (except that this constraint only applies to the specific device--not all video devices):

{ video: { optional: [ { width: value } ] } }

If this is the only request during this script-execution task, then when control returns to the user agent, this constraint will be committed (i.e., like an indexedDB transaction) and the constraint application logic will evaluate the request making changes to the current device if applicable.

If there is another request during the same script-execution task, it is appended to the optional list. Since order is important in the optional constraints list, the first requested setting has priority over the next.

The request() API also has a flag used to signal to the UA that the requested setting change should be mandatory. In this case, the constraint is added to the mandatory set, and replaces an existing setting in that set if the names collide (last setting wins). My expectation is that if a mandatory constraint cannot be satisfied, then the UA must end that stream as a result of the failure.

Unlike constraints built using dictionaries for getUserMedia, the constraint structures produced by calls to the request() API will always be individual proposed values, rather than min/max ranges. This is because min/max information is already available within the relevant settings, and can be included in calculations before making the call to request(). Therefore, I didn't feel it was necessary to clutter the API surface with that feature.

MediaSettingsRange objects should be used when the setting can generally actually assume a value along a continuum of values. This specification should indicate what the range of values must be for each setting. Given that implementations of various hardware may not exactly map to the same range, an implementation should make a reasonable attempt to translate and scale the hardware's setting onto the mapping provided by this specification. If this is not possible due to a hardware setting supporting (for example) fewer levels of granularity, then the implementation should make the device settings min value reflect the min value reported in this specification, and the same for the max value. Then for values in between the min and max, the implementation may round to the nearest supported value and report that value in the setting.

MediaSettingsList objects should order their enumerated values from minimum to maximum where it makes sense, or in the order defined by the enumerated type where applicable.

interface MediaSettingsRange {
    readonly attribute any max;
    readonly attribute any min;
    readonly attribute any initial;
    void request (any value, optional boolean mandatory);
};

interface MediaSettingsList {
    readonly attribute unsigned long length;
    getter any item (unsigned long index);
    readonly attribute any           initial;
    void       request (any value, optional boolean mandatory);
};

3.2 Basic settings for pictures and video devices

Settings (read/writable) are defined as separate properties from their read-only counterparts. This allows for a variety of benefits:

• Some read-only settings can be surfaced on super-interfaces (like VideoStreamTrack) to benefit non-device centric tracks.
• Some read-only settings have no corresponding read/write version (for example "facing" cannot be changed).
• Some read/write settings have no (or do not really require) a similar read-only value (for example, dimensions is already reported separately as width/height).
• Simple to access read-only settings are more convenient (versus accessing an object to then get the property).

VideoDeviceTrack implements PictureAndVideoSettings;

PictureDeviceTrack implements PictureAndVideoSettings;

[NoInterfaceObject]
interface PictureAndVideoSettings {
    readonly attribute MediaSettingsList      dimensions;
    readonly attribute unsigned long          rotation;
    readonly attribute MediaSettingsList?     rotations;
    readonly attribute float                  zoom;
    readonly attribute MediaSettingsRange?    zooms;
    readonly attribute VideoFocusModeEnum     focusMode;
    readonly attribute MediaSettingsList?     focusModes;
    readonly attribute VideoFillLightModeEnum fillLightMode;
    readonly attribute MediaSettingsList?     fillLightModes;
};

dictionary VideoDimensionDict {
    unsigned long width;
    unsigned long height;
};

enum VideoFocusModeEnum {
    "notavailable",
    "auto",
    "manual"
};

enum VideoFillLightModeEnum {
    "notavailable",
    "auto",
    "off",
    "on"
};

3.3 Expanded settings for video devices

partial interface VideoDeviceTrack {
    readonly attribute float               framesPerSecond;
    readonly attribute MediaSettingsRange? framesPerSeconds;
};

3.4 Settings for audio devices

partial interface AudioDeviceTrack {
    readonly attribute MediaSettingsRange? levels;
};

3.5 Tracking the result of constraint application

AudioDeviceTrack implements MediaConstraintResultEventHandlers;

VideoDeviceTrack implements MediaConstraintResultEventHandlers;

PictureDeviceTrack implements MediaConstraintResultEventHandlers;

MediaDeviceList implements MediaConstraintResultEventHandlers;

[NoInterfaceObject]
interface MediaConstraintResultEventHandlers {
             attribute EventHandler onconstrainterror;
             attribute EventHandler onconstraintsuccess;
};

[Constructor(DOMString type, optional ConstraintErrorEventInit eventInitDict)]
interface ConstraintErrorEvent : Event {
    readonly attribute DOMString[] optionalConstraints;
    readonly attribute DOMString[] mandatoryConstraints;
};

dictionary ConstraintErrorEventInit : EventInit {
    sequence<'DOMString> optionalConstraints;
    sequence<'DOMString> mandatoryConstraints;
};

4.1 MediaDeviceList interface

interface MediaDeviceList : EventTarget {
    readonly attribute unsigned long length;
    getter any item (unsigned long index);
    readonly attribute unsigned long totalEnabled;
    void       select (MediaTrackConstraints constraints);
             attribute EventHandler  ondeviceadded;
             attribute EventHandler  ondeviceremoved;
};

4.2 DeviceEvent interface

[Constructor(DOMString type, optional DeviceEventInit eventInitDict)]
interface DeviceEvent : Event {
    readonly attribute MediaStreamTrack device;
};

4.3 DeviceEventInit dictionary

dictionary DeviceEventInit : EventInit {
    MediaStreamTrack device;
};

4.4 DeviceListAccess mix-in interface

AudioDeviceTrack implements DeviceListAccess;

VideoDeviceTrack implements DeviceListAccess;

[NoInterfaceObject]
interface DeviceListAccess {
    readonly attribute MediaDeviceList devices;
    readonly attribute unsigned long?  deviceIndex;
};

5.1 Video Constraints

The following constraints are applicable to video devices

dictionary VideoConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint)                     width;
    (unsigned long or MinMaxULongSubConstraint)                     height;
    (float or MinMaxFloatSubConstraint)                     horizontalAspectRatio;
    (float or MinMaxFloatSubConstraint)                     verticalAspectRatio;
    unsigned long          rotation;
    (float or MinMaxFloatSubConstraint)                     zoom;
    VideoFocusModeEnum     focusMode;
    VideoFillLightModeEnum fillLightMode;
    (float or MinMaxFloatSubConstraint)                     framesPerSecond;
};

5.2 Audio Constraints

The following constraints are applicable to audio devices

dictionary AudioConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint) level;
};

5.3 Common sub-constraint structures

dictionary MinMaxULongSubConstraint {
    unsigned long max;
    unsigned long min;
};

dictionary MinMaxFloatSubConstraint {
    float max;
    float min;
};

enum VideoOrientationDict {
    "landscape",
    "portrait"
};

6.1 Getting access to a video and/or audio device (if available) -- scenario unchanged

navigator.getUserMedia({audio: true, video: true}, gotMedia, failedToGetMedia);

function gotMedia(mediastream) {
   // The recieved mediastream is using its initial settings (it's clean)
}

6.2 Previewing the local video/audio in HTML5 video tag -- scenario is unchanged

function gotMedia(mediastream) {
   // objectURL technique
   document.querySelector("video").src = URL.createObjectURL(mediastream, { autoRevoke: true }); // autoRevoke is the default
   // direct-assign technique
   document.querySelector("video").srcObject = mediastream; // Proposed API at this time
}

6.3 Applying resolution constraints

function gotMedia(mediastream) {
   var videoDevice = mediastream.videoTracks[0];
   var maxDimensions = videoDevice.dimensions[videoDevice.dimensions.length - 1];
   // Check for 1080p+ support
   if ((maxDimensions.width >= 1920) && (maxDimensions.height >= 1080)) {
      // See if I need to change the current settings...
      if ((videoDevice.width < 1920) && (videoDevice.height < 1080)) {
         videoDevice.dimensions.request(maxDimensions, true);
         videoDevice.onconstrainterror = failureToComply;
      }
   }
   else
      failureToComply();
}

function failureToComply(e) {
   if (e)
      console.error("Device failed to change " + e.mandatoryConstraints[0]); // 'dimension'
   else
      console.error("Device doesn't support at least 1080p");
}

6.4 Changing zoom in response to user input:

function gotMedia(mediastream) {
   setupRange( mediastream.videoTracks[0] );
}

function setupRange(videoDevice) {
   // Check to see if the device supports zooming...
   if (videoDevice.zooms) {
      // Set HTML5 range control to min/max values of zoom
      var zoomControl = document.querySelector("input[type=range]");
      zoomControl.min = videoDevice.zooms.min;
      zoomControl.max = videoDevice.zooms.max;
      zoomControl.value = videoDevice.zoom;
      zoomControl.zoomController = videoDevice.zooms; // Store the setting
      zoomControl.onchange = applySettingChanges;
   }
}

function applySettingChanges(e) {
   e.target.zoomController.request(parseFloat(e.target.value), true);
}

6.5 Adding the local media tracks into a new media stream -- scenario is unchanged

function gotMedia(mediastream) {
   return new MediaStream( [ mediastream.videoTracks[0], mediastream.audioTracks[0] ]);
}

6.6 Take a picture, show the picture in an image tag

function gotMedia(mediastream) {
   var videoDevice = mediastream.videoTracks[0];
   // Check if this device supports a picture mode...
   var pictureDevice = videoDevice.pictureTrack;
   if (pictureDevice) {
       pictureDevice.onpicture = showPicture;
       // Turn on flash only for the snapshot...if available
       if (pictureDevice.fillLightModes) { // If there's an object here, then the flash is supported
          pictureDevice.fillLightModes.request("on", true);
       }
       else
          console.info("Flash not available");
       pictureDevice.takePicture();
   }
}

function showPicture(e) {
   var img = document.querySelector("img");
   img.src = URL.createObjectURL(e.data);
}

6.7 Show a newly available device

function gotMedia(mediastream) {
   mediastream.videoTracks[0].devices.addEventListener("deviceadded", enableAndShowNewDevice);
}

function enableAndShowNewDevice(e) {
   // Show the new video device as soon as it's available
   // New device is muted when it first becomes available
   e.device.enabled = true;
   var mStream = new MediaStream(e.device);
   document.querySelector("video").srcObject = mStream; // Using the proposed direct-assignment API
}

6.8 Show all available video devices

function gotMedia(mediastream) {
   var deviceList = mediastream.videoTracks[0].devices;
   for (var i = 0; i < deviceList.length; i++) {
      var videoDevice = deviceList[i];
      videoDevice.enabled = true;
      // Create a video element and add it to the UI
      var videoTag = document.createElement('video');
      videoTag.srcObject = new MediaStream([videoDevice]);
      document.body.appendChild(videoTag);
   }
}