W3C

Proposal: Settings API Version 4

Updated: 05 October 2012 (previous revision)

This is an out-of-date proposal. Click here for a newer version (v5)

Author:
Travis Leithead, Microsoft

Abstract

This proposal describes additions and suggested changes to the Media Capture and Streams specification in order to support the goal of device settings retrieval and modification. This proposal incorporates feedback from three prior proposals with the same goal [v3] [v2] [v1].

Table of Contents

1. Remove LocalMediaStream interface

In this proposal, the derived LocalMediaStream interface is removed. Rather than returning a LocalMediaStream instance in the NavigatorUserMediaSuccessCallback, a vanilla MediaStream object is returned. The primary difference is in the tracks contained in that MediaStream object.

1.1 Rationale

The LocalMediaStream object currently extends MediaStream by adding a single method "stop()". In my prior proposals, this object was radically altered in order to facilitate several goals:
Provide a predictable home for developers to find and modify device settings
A previous proposal went out of its way to strongly associate LocalMediaStream objects with devices. This seemed like a good design because local device configuration is always on the local media stream. This made for a stable, dependable API surface for all local media stream instances (no guesswork).
Prevent track-list mutations
A previous proposal also removed the track lists on local media streams (resulting in some dramatic inheritance changes). Mutable tracks lists on LocalMediaStream objects seemed like the wrong design considering the current thinking that a getUserMedia request would only ever produce a LocalMediaStream with at most one audio or video track.

Some feedback even suggested re-considering the "at most one video/audio track per request to getUserMedia".

While thinking about these goals and the feedback, I began to consider a few things:

Device-centric tracks
With tracks supplemented with device-characteristics (duck-typing), the LocalMediaStream's stop() API was a convenience feature for stopping all tracks backed by a device on the LocalMediaStream object. With device- centric tracks a stop() API should be present on the tracks themselves.
Mutable track lists
Mutable track lists were not a desirable feature while I was locked into considering the LocalMediaStream as strongly associated with device-control. What is actually necessary, is that there is a something immutable associated with devices--that "something" doesn't necessarily need to be a LocalMediaStream or any MediaStream-like object at all! Once I unlocked that line of thinking, I began to experiment with the notion of a device list which then ultimately brought back a use-case for having mutable track lists for MediaStream objects. (It did not bring back a need for LocalMediaStream objects themselves though.)
Workflow for access to additional device streams
It is now understood that to request additional streams for different devices (e.g., the second camera on a dual-camera mobile phone), one must invoke getUserMedia a second time. In my prior proposal, this would result in a separate LocalMediaStream instance. At this point there are two LocalMediaStream objects each with their own devices. While this was nice for consistency of process, it was a challenge in terms of use of the objects with a MediaStream consumer like the Video tag.

To illustrate this challenge, consider how my prior proposal required a re-hookup of the MediaStream to a video tag consumer:

  1. First request to getUserMedia
  2. LocalMediaStream (1) obtained from success callback
  3. createObjectURL and preview in a video tag
  4. Second call to getUserMedia
  5. LocalMediaStream (2) obtained from success callback
  6. createObjectURL and preview in same video tag

Note that this process has to bind a completely new LocalMediaStream to the video tag a second time (if re-using the same video tag) only because the second LocalMediaStream object was different than the first.

It is much more efficient for developer code to simply add/remove tracks to a MediaStream that are relevant, without needing to change the consumer of the MediaStream.

Usage of getUserMedia for permission rather than for additional device access
The getUserMedia method is the gateway for permission to media. This proposal does not suggest changing that concept. It does suggest, however, that more information can be made available for discovery of additional devices within the approved "category" or "type" of media, and provide a way to obtain those additional devices without needing to go through the "permissions" route (i.e., getUserMedia).
Importance of restricting control to LocalMediaStream
Upon reflection of the feedback around the prior proposal, the relative importance of restricting control of the devices associated with tracks on the LocalMediaStream to only the LocalMediaStream did not seem as vital, insofar as the device-level access via the track is not directly available through a PeerConnection to a remote browser.

2. New MediaStreamTrack (derived) types

This proposal consolidates settings directly into the tracks that are provided by devices. However, in order to do this efficiently and in a future-extensible manner, the highly-generic MediaStreamTrack is now extended for specific characteristics of the devices it embodies, resulting in a hierarchy:

2.1 Local and remote video tracks

MediaStreamTrack objects that are of kind "video" and that are located in a MediaStream's videoTracks list will be instances of a VideoStreamTrack. The VideoStreamTrack provides basic (read-only) properties pertinent to all sources of video.

Note

There is no takePicture API on a VideoStreamTrack because a simple frame-grab can be accomplished using a combination of a <video> and <canvas> APIs (takePicture is intended for use with a camera's high-resolution picture mode, not for arbitrary video frame capture).

I'm intentionally keeping this interface as sparse as possible. Features about the video that can be calculated like aspect ratio are not provided.

2.1.1 VideoStreamTrack interface

interface VideoStreamTrack : MediaStreamTrack {
    readonly attribute unsigned long width;
    readonly attribute unsigned long height;
};
Attributes
width of type unsigned long, readonly
The "natural" width (in pixels) of the video flowing through the track. In the case of a VideoDeviceTrack, this value represents the current setting of the camera's sensor (still in terms of number of pixels). This value is independent of the camera's rotation (if the camera's rotation setting is changed, it does not impact this value). For example, consider a camera setting with width of 1024 pixels and height of 768 pixels. If the camera's rotation setting is changed by 90 degrees, the width is still reported as 1024 pixels. However, a <video> element used to preview this track would report a width of 768 pixels (the effective width with rotation factored in).
height of type unsigned long, readonly
The "natural" height (in pixels) of the video in this track. See the "width" attribute for additional info.

2.2 Local and remote audio tracks

MediaStreamTrack objects that are of kind "audio" and that are located in a MediaStream's audioTracks list will be instances of an AudioStreamTrack. The AudioStreamTrack provides basic (read-only) properties pertinent to all sources of audio.

2.2.1 AudioStreamTrack interface

interface AudioStreamTrack : MediaStreamTrack {
};

2.3 Camera device tracks

VideoDeviceTracks are created by the user agent to represent a camera device that provides local video.

2.3.1 VideoDeviceTrack interface

interface VideoDeviceTrack : VideoStreamTrack {
    readonly attribute PictureDeviceTrack? pictureTrack;
    readonly attribute VideoFacingEnum     facing;
    void stop ();
};
Attributes
pictureTrack of type PictureDeviceTrack, readonly, nullable
If the device providing this VideoDeviceTrack supports a "high-resolution picture mode", this attribute will be reference to a PictureDeviceTrack object. Otherwise, this attribute will be null.
facing of type VideoFacingEnum, readonly
From the user's perspective, this attribute describes whether this camera is pointed toward the user ("user") or away from the user ("environment"). If this information cannot be reliably obtained, for example from a USB external camera, the value "unknown" is returned.
Methods
stop
Causes this track to enter the ENDED state. Same behavior of the old LocalMediaStream's stop API, but only affects this device track.
No parameters.
Return type: void

2.3.2 VideoFacingEnum enumeration

enum VideoFacingEnum {
    "unknown",
    "user",
    "environment"
};
Enumeration description
unknownThe relative directionality of the camera cannot be determined by the user agent based on the hardware.
userThe camera is facing toward the user (a self-view camera).
environmentThe camera is facing away from the user (viewing the environment).

2.4 Cameras with "high-resolution picture" modes

The PictureDeviceTrack interface is created by the user agent if the camera device providing the VideoDeviceTrack supports an optional "high-resolution picture mode" with picture settings different (better) from those of its basic video constraints.

This track is initially available from a VideoDeviceTrack via the pictureTrack property. This track type is not present in the video device list (MediaDeviceList). Likewise, it cannot be stopped directly, and its VideoStreamTrack inherited attributes reflect the values of its "owning" VideoDeviceTrack.

The PictureDeviceTrack is essentially a specialized VideoStreamTrack (this track type is of kind "video"). It may be explicitly added to a videoTracks list (MediaStreamTrackList) in order to output its track video to a <video> tag, but its preview video stream reflects the owning VideoDeviceTrack's settings, rather than the settings directly available on this object. Rather the settings of this object are only applied at the time when the takePicture API is invoked.

2.4.1 PictureDeviceTrack interface

interface PictureDeviceTrack : VideoStreamTrack {
    void takePicture ();
             attribute EventHandler onpicture;
};
Attributes
onpicture of type EventHandler
Register/unregister for "picture" events. The handler should expect to get a PictureEvent object as its first parameter.
Issue 2

Is an "error" event necessary here too?

Methods
takePicture
Temporarily mutes the owning VideoDeviceTrack's stream, then asynchronously switches the camera into "high resolution picture mode", applies the PictureDeviceTrack settings (a snapshot from the time the takePicture API was called, and records/encodes an image using a user-agent determined format into a Blob object. Finally, queues a task to fire a "picture" event with the resulting Blob instance.
Issue 1

Could consider providing a hint or setting for the desired picture format.

No parameters.
Return type: void
Note

In the previous proposal, the PictureEvent returned a Canvas ImageData object, however it makes sense to return a compressed format (PNG/JPEG), especially given that picture snapshots will be very high resolution, and ImageData objects are essentially raw images.

2.4.2 PictureEvent interface

[Constructor(DOMString type, optional PictureEventInit eventInitDict)]
interface PictureEvent : Event {
    readonly attribute Blob data;
};
Attributes
data of type Blob, readonly
Returns a Blob object whose type attribute indicates the encoding of the picture data. An implementation must return a Blob in a format that is capable of being viewed in an HTML <img> tag.

2.5 Microphone device tracks

AudioDeviceTracks are created by the user agent to represent a microphone device that provides local audio.

2.5.1 AudioDeviceTrack interface

interface AudioDeviceTrack : AudioStreamTrack {
    readonly attribute unsigned long level;
    void stop ();
};
Attributes
level of type unsigned long, readonly
The sensitivity of the microphone. This value must be a whole number between 0 and 100 inclusive. When a MediaStreamTrack is muted, the level attribute must return 0. A value of 100 means the microphone is configured for maximum gain.
Methods
stop
Causes this track to enter the ENDED state. Same behavior of the old LocalMediaStream's stop API, but only for this device track.
No parameters.
Return type: void

3. Settings Retrieval/Application

As noted in prior proposals, camera/microphone settings must be applied asynchronously to ensure that web applications can remain responsive for all device types that may not respond quickly to setting changes.

My prior proposals used a monolithic dictionary of settings for inspection and application. This proposal takes a different approach, considering the feedback for more-direct access to settings, expected patterns for settings adjustment (which is generally one setting at at time as initiated by a web application UI), difficulties in understanding what values were read-only vs. writable, and the current already-defined constraint application engine.

3.1 Grouping setting features

Settings are organized into two groups: value ranges (a continuum of values) and enumerated values. Value ranges include a min and max value, while enumerated values are provided in an array with an associated length. Both groups of settings include an "initial" value, which is the value that is expected to be the device's default value when it is acquired.

The key to changing settings in either setting group is the request() API. This is the mechanism for asynchronously requesting that the device change the value of the setting that the setting group is applicable to. The mechanics for applying setting change requests follows exactly the model used when applying constraints at getUserMedia invocation. Each team a request() is made, the user agent begins building up an [internally-represented] constraint structure which is associated with the device making the request (and only that device). For example, if a "width" setting change request is made, the user agent creates a constraint structure equivalent to the following getUserMedia constraint (except that this constraint only applies to the specific device--not all video devices):

{ video: { optional: [ { width: value } ] } }

If this is the only request during this script-execution task, then when control returns to the user agent, this constraint will be committed (i.e., like an indexedDB transaction) and the constraint application logic will evaluate the request making changes to the current device if applicable.

If there is another request during the same script-execution task, it is appended to the optional list. Since order is important in the optional constraints list, the first requested setting has priority over the next.

The request() API also has a flag used to signal to the UA that the requested setting change should be mandatory. In this case, the constraint is added to the mandatory set, and replaces an existing setting in that set if the names collide (last setting wins). My expectation is that if a mandatory constraint cannot be satisfied, then the UA must end that stream as a result of the failure.

Unlike constraints built using dictionaries for getUserMedia, the constraint structures produced by calls to the request() API will always be individual proposed values, rather than min/max ranges. This is because min/max information is already available within the relevant settings, and can be included in calculations before making the call to request(). Therefore, I didn't feel it was necessary to clutter the API surface with that feature.

MediaSettingsRange objects should be used when the setting can generally actually assume a value along a continuum of values. This specification should indicate what the range of values must be for each setting. Given that implementations of various hardware may not exactly map to the same range, an implementation should make a reasonable attempt to translate and scale the hardware's setting onto the mapping provided by this specification. If this is not possible due to a hardware setting supporting (for example) fewer levels of granularity, then the implementation should make the device settings min value reflect the min value reported in this specification, and the same for the max value. Then for values in between the min and max, the implementation may round to the nearest supported value and report that value in the setting.

Note

For example, if the setting is fluxCapacitance, and has a specified range from -10 (min) to 10 (max) in this specification, but the implementation's fluxCapacitance hardware setting only supports values of "off" "medium" and "full", then -10 should be mapped to "off", 10 should map to "full", and 0 should map to "medium". A request to change the value to 3 should be rounded down to the closest supported setting (0).

MediaSettingsList objects should order their enumerated values from minimum to maximum where it makes sense, or in the order defined by the enumerated type where applicable.

3.1.1 MediaSettingsRange interface

interface MediaSettingsRange {
    readonly attribute any max;
    readonly attribute any min;
    readonly attribute any initial;
    void request (any value, optional boolean mandatory);
};
Attributes
max of type any, readonly
The maximum value of this setting.

The type of this value is specific to the setting. Each setting will describe a specific type. That type must be returned for this attribute.

min of type any, readonly
The minimum value of this setting.

The type of this value is specific to the setting. Each setting will describe a specific type. That type must be returned for this attribute.

initial of type any, readonly
The initial value of this setting. When the object associated with this setting is first made available to the application, the current value of the setting should be set to the initial value. For example, in a browsing scenario, if one web site changes this setting and a subsequent web site gets access to this same setting, the setting should have been reset back to its initial value.

The type of this value is specific to the setting. Each setting will describe a specific type. That type must be returned for this attribute.

Methods
request
Creates an internal constraint based on the setting name with the provided value, adds that constraint into the pending constraint structure (to the MediaTrackConstraint array by default or replaces an entry in the MediaTrackConstraintSet if the mandatory flag is set) and queues a task (if not already queued) to process the pending constraint structure at the conclusion of this task.

The mandatory parameter defaults to false.

The value parameter type of this method is specific to the setting. Each setting will describe a specific type. That type must be provided for this paramter. If the type does align, then the implementation should throw a TypeError exception.

ParameterTypeNullableOptionalDescription
valueany
mandatoryboolean
Return type: void

3.1.2 MediaSettingsList interface

interface MediaSettingsList {
    readonly attribute unsigned long length;
    getter any item (unsigned long index);
    readonly attribute any           initial;
    void       request (any value, optional boolean mandatory);
};
Attributes
length of type unsigned long, readonly
The length of the enumerated values that this setting may assume.
initial of type any, readonly
The initial value of this setting. When the object associated with this setting is first made available to the application, the current value of the setting should be set to the initial value. For example, in a browsing scenario, if one web site changes this setting and a subsequent web site gets access to this same setting, the setting should have been reset back to its initial value.

The type of this value is specific to the setting. Each setting will describe a specific type. That type must be returned for this attribute.

Methods
item
Retrieves the value of the indexed enumerated item of this setting. Items should be sorted from min (at index 0) to max where applicable, or in the order listed in the enumerated type otherwise.

The type of this value is specific to the setting. Each setting will describe a specific type. That type must be returned for this attribute.

ParameterTypeNullableOptionalDescription
indexunsigned long
Return type: getter any
request
Creates an internal constraint based on the setting name with the provided value, adds that constraint into the pending constraint structure (to the MediaTrackConstraint array by default or replaces an entry in the MediaTrackConstraintSet if the mandatory flag is set) and queues a task (if not already queued) to process the pending constraint structure at the conclusion of this task.

The mandatory parameter defaults to false.

The value parameter type of this method is specific to the setting. Each setting will describe a specific type. That type must be provided for this paramter. If the type does align, then the implementation should throw a TypeError exception.

ParameterTypeNullableOptionalDescription
valueany
mandatoryboolean
Return type: void

3.2 Basic settings for pictures and video devices

Settings (read/writable) are defined as separate properties from their read-only counterparts. This allows for a variety of benefits:

Note

These are pluralized for compactness and easy identification as a "setting". The more verbose "widthSettings", "horizontalAspectRatioSettings", "orientationSettings", etc., were considered (and may still be considered).

Note
The following settings have been proposed, but are not included in this version to keep the initial set of settings scoped to those that:
  1. cannot be easily computed in post-processing
  2. are not redundant with other settings
  3. are settings found in nearly all devices (common)
  4. can be easily tested for conformance
Each setting also includes a brief explanatory rationale for why it's not included:
  1. width - I've used "dimension" for the setting instead, since resolutions of the camera are nearly always in step-wise pairs of width/height combinations. These are thus an enumerated type rather than a range continuum of possible width/height (independent) pairs.
  2. height - see width explanation
  3. horizontalAspectRatio - easily calculated based on width/height in the dimension values
  4. verticalAspectRatio - see horizontalAspectRatio explanation
  5. orientation - can be easily calculated based on the width/height values and the current rotation
  6. aperatureSize - while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
  7. shutterSpeed - see aperatureSize explanation
  8. denoise - may require specification of the algorithm processing or related image processing filter required to implement.
  9. effects - sounds like a v2 or independent feature (depending on the effect).
  10. faceDetection - sounds like a v2 feature. Can also be done using post-processing techniques (though perhaps not as fast...)
  11. antiShake - sounds like a v2 feature.
  12. geoTagging - this can be independently associated with a recorded picture/video/audio clip using the Geolocation API. Automatically hooking up Geolocation to Media Capture sounds like an exercise for v2 given the possible complications.
  13. highDynamicRange - not sure how this can be specified, or if this is just a v2 feature.
  14. skintoneEnhancement - not a particularly common setting.
  15. shutterSound - Can be accomplished by syncing custom audio playback via the <audio> tag if desired. By default, there will be no sound issued.
  16. redEyeReduction - photo-specific setting. (Could be considered if photo-specific settings are introduced.)
  17. meteringMode - photo-specific setting. (Could be considered if photo-specific settings are introduced.)
  18. iso - photo-specific setting. while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
  19. sceneMode - while more common on digital cameras, not particularly common on webcams (major use-case for this feature)
  20. antiFlicker - not a particularly common setting.
  21. zeroShutterLag - this seems more like a hope than a setting. I'd rather just have implementations make the shutter snap as quickly as possible after takePicture, rather than requiring an opt-in/opt-out for this setting.
The following settings are up for debate in my opinion:
  1. exposure
  2. exposureCompensation (is this the same as exposure?)
  3. autoExposureMode
  4. brightness
  5. contrast
  6. saturation
  7. sharpness
  8. evShift
  9. whiteBalance

Some of the above settings are available as constraints, and so are included in the proposed set of constraints in the last section.

3.2.1 PictureAndVideoSettings mix-in interface

VideoDeviceTrack implements PictureAndVideoSettings;
PictureDeviceTrack implements PictureAndVideoSettings;
[NoInterfaceObject]
interface PictureAndVideoSettings {
    readonly attribute MediaSettingsList      dimensions;
    readonly attribute unsigned long          rotation;
    readonly attribute MediaSettingsList?     rotations;
    readonly attribute float                  zoom;
    readonly attribute MediaSettingsRange?    zooms;
    readonly attribute VideoFocusModeEnum     focusMode;
    readonly attribute MediaSettingsList?     focusModes;
    readonly attribute VideoFillLightModeEnum fillLightMode;
    readonly attribute MediaSettingsList?     fillLightModes;
};
Attributes
dimensions of type MediaSettingsList, readonly
The MediaSettingsList reports values of type VideoDimensionDict. The width/height reported are of the camera's sensor, not reflecting a particular orientation. VideoDimensionDict items in the MediaSettingsList list must be primarily sorted by width (smallest to largest), and then sub-sorted by height (smallest to largest).
rotation of type unsigned long, readonly
The current rotation value in use by the camera. If not supported, the property returns the value 0.
rotations of type MediaSettingsList, readonly, nullable
The MediaSettingsList reports values of type unsigned long. (0-90-180-270) degrees).
Issue 3

Rotation makes me think I could set this to 45 degrees or some such. Maybe there's a better setting name for this. I only want to support right-angles.

zoom of type float, readonly
The current zoom scale value in use by the camera. If the zooms property is not available (not supported), then this property will always return 1.0.
Issue 4

In the case that a camera device supports both optical and digital zoom, does it make sense to have just one property? I expect this to be the "digitalZoom" version, which is more common on devices.

zooms of type MediaSettingsRange, readonly, nullable
The MediaSettingsRange reports values of type float. (initial value is 1. The float value is a scale factor, for example 0.5 is zoomed out by double, while 2.0 is zoomed in by double. Requests should be rounded to the nearest supporting zoom factor by the implementation (when zoom is supported).
focusMode of type VideoFocusModeEnum, readonly
The camera's current focusMode state.
focusModes of type MediaSettingsList, readonly, nullable
The MediaSettingsList reports values of type VideoFocusModeEnum (less the "notavailable" value). The initial value is "auto".
fillLightMode of type VideoFillLightModeEnum, readonly
The camera's current fill light mode.
Note

fillLight seemed more appropriate a term to use for both cameras and photo settings.

fillLightModes of type MediaSettingsList, readonly, nullable
The MediaSettingsList reports values of type VideoFillLightModeEnum (less the "notavailable" value). The initial value is "auto".

3.2.2 VideoDimensionDict dictionary

dictionary VideoDimensionDict {
    unsigned long width;
    unsigned long height;
};
Dictionary VideoDimensionDict Members
width of type unsigned long
A supported camera width (long axis).
height of type unsigned long
A supported camera height (short axis).
Note

The following enums had many more values in the prior proposal, but in the interest of testing, I've scoped the initial list to those that seem most easily testable.

3.2.3 VideoFocusModeEnum enumeration

enum VideoFocusModeEnum {
    "notavailable",
    "auto",
    "manual"
};
Enumeration description
notavailableThis camera does not have an option to change focus modes.
autoThe camera auto-focuses.
manualThe camera must be manually focused.

3.2.4 VideoFillLightModeEnum enumeration

enum VideoFillLightModeEnum {
    "notavailable",
    "auto",
    "off",
    "on"
};
Enumeration description
notavailableThis camera does not have an option to change fill light modes (e.g., the camera does not have a flash).
autoThe camera's fill light will be enabled when required (typically low light conditions). Otherwise it will be off.
offThe camera's fill light will not be used.
onThe camera's fill light will be turned on until this setting is changed again, or the underlying track object has ended.

3.3 Expanded settings for video devices

3.3.1 VideoDeviceTrack partial interface

partial interface VideoDeviceTrack {
    readonly attribute float               framesPerSecond;
    readonly attribute MediaSettingsRange? framesPerSeconds;
};
Attributes
framesPerSecond of type float, readonly
The camera's currently configured (estimated) framesPerSecond.
framesPerSeconds of type MediaSettingsRange, readonly, nullable
The MediaSettingRange reports values of type float.
Issue 5

I wonder if this should just be a MediaSettingsList with the common values of 15, 30, and 60. Are there really any other values coming from hardware?

3.4 Settings for audio devices

Note

My previous proposal included a "bassTone" and "trebleTone" setting value, but on reflection, those settings are more relevant to playback than to microphone device settings. Those settings have been removed.

3.4.1 AudioDeviceTrack partial interface

partial interface AudioDeviceTrack {
    readonly attribute MediaSettingsRange? levels;
};
Attributes
levels of type MediaSettingsRange, readonly, nullable
The MediaSettingRange reports values of type unsigned long.

3.5 Tracking the result of constraint application

3.5.1 MediaConstraintResultEventHandlers mix-in interface

AudioDeviceTrack implements MediaConstraintResultEventHandlers;
VideoDeviceTrack implements MediaConstraintResultEventHandlers;
PictureDeviceTrack implements MediaConstraintResultEventHandlers;
MediaDeviceList implements MediaConstraintResultEventHandlers;
[NoInterfaceObject]
interface MediaConstraintResultEventHandlers {
             attribute EventHandler onconstrainterror;
             attribute EventHandler onconstraintsuccess;
};
Attributes
onconstrainterror of type EventHandler
Register/unregister for "constrainterror" events. The handler should expect to get a ConstraintErrorEvent object as its first parameter. The event is fired asynchronously after [potentially many] settings change requests are made but resulted in one or more failures to meet those constraints. The ConstraintErrorEvent reports the name of the settings that could not be applied.
onconstraintsuccess of type EventHandler
Register/unregister for "constraintsuccess" events. The handler should expect to get a DeviceEvent object as its first parameter. The event is fired asynchronously after the [potentially many] settings change requests are made and applied successfully. Note, if any one setting change fails, then the "constrainterror" event fires instead. The DeviceEvent will fire on the track making the settings request (with the device attribute referring to the same object), with the exception of the MediaDeviceList (see the MediaDeviceList's select() API).

3.5.2 ConstraintErrorEvent interface

[Constructor(DOMString type, optional ConstraintErrorEventInit eventInitDict)]
interface ConstraintErrorEvent : Event {
    readonly attribute DOMString[] optionalConstraints;
    readonly attribute DOMString[] mandatoryConstraints;
};
Attributes
optionalConstraints of type array of DOMString, readonly
A list of optional constraints that failed or succeeded (depending on the event type).
mandatoryConstraints of type array of DOMString, readonly

3.5.3 ConstraintErrorEventInit dictionary

dictionary ConstraintErrorEventInit : EventInit {
    sequence<'DOMString> optionalConstraints;
    sequence<'DOMString> mandatoryConstraints;
};
Dictionary ConstraintErrorEventInit Members
optionalConstraints of type sequence<'DOMString>
List of optional constraints to populate into the ConstraintErrorEvent object's optionalConstraints readonly attribute.
mandatoryConstraints of type sequence<'DOMString>
List of mandatory constraints to populate into the ConstraintErrorEvent object's mandatoryConstraints readonly attribute.

4. Device Lists

One common problem with all my previous proposals, and with the existing model for using getUserMedia to request access to additional devices, is the problem of discovery of multiple devices. As I understand it, the existing recommendation relies on "guessing" by making a second (or third, etc.) request to getUserMedia for access to additional devices. This model has two primary advantages:

First, it ensures privacy by making sure that each device request could be approved by the user. I say "could" because there is no current requirement that the user agent be involved, especially when re-requesting a device type that was already approved, for example, a second "video" device. I surmise that a request for a different class of device ("audio", when exclusive "video" was previously approved), would be cause for an implementation to ask the user for approval.

Second, it ensure privacy by not leaking any information about additional devices until the code has successfully requested a device.

Unfortunately, this model does not provide a means for discovery of additional devices. Such a discovery mechanism could be trivially added to this proposal in the form of a device-specific "totalDevices" property, but there's an opportunity for considering a solution that both streamlines the usability of multiple devices while maintaining the privacy benefits of the current model.

The device list is such a proposal. The device list offers the following benefits:

A device list is merely a list of all AudioDeviceTrack or VideoDeviceTrack objects that are available to the application. Device lists are device-type specific, so there is one device list for all AudioDeviceTrack objects and one device list for all VideoDeviceTrack objects. There is only one instance of each of these lists at any time, and the lists are LIVE (meaning the user agent keeps them up-to-date at all times). Device track objects are added to the list as soon as they are available to the application (e.g., as soon as they are plugged-in) A device track object in the device list will have a readyState set to either LIVE or MUTED). Device tracks are removed from the list when they are unplugged, or otherwise disassociated with their device source such that their readyState changes to ENDED.

Every non-ended device track object will belong to a device list. Of course, the same device track object may also belong to zero or more MediaStreamTrackList objects. The device list provides the one-stop list for all devices of that type regardless of which MediaStream's (if any) the device track objects also belong to.

4.1 MediaDeviceList interface

interface MediaDeviceList : EventTarget {
    readonly attribute unsigned long length;
    getter any item (unsigned long index);
    readonly attribute unsigned long totalEnabled;
    void       select (MediaTrackConstraints constraints);
             attribute EventHandler  ondeviceadded;
             attribute EventHandler  ondeviceremoved;
};

4.1.1 Attributes

length of type unsigned long, readonly
The number of devices in this list (including those that are MUTED and LIVE.
totalEnabled of type unsigned long, readonly
The number of devices in this list that whose readyState is in the LIVE state.
ondeviceadded of type EventHandler
Register/unregister for "deviceadded" events. The handler should expect to get a DeviceEvent object as its first parameter. The event is fired whenever a new video device (or audio device, depending on the device list) becomes available for use. This can happen when a new device is plugged in, for example. Previously ended device tracks are not re-used, and if the user agent is able to re-purpose a physical device for use in the application, it fires the "deviceadded" event providing a new device track object (in its default initial state).
ondeviceremoved of type EventHandler
Register/unregister for "deviceremoved" events. The handler should expect to get a DeviceEvent object as its first parameter. The event is fired whenever an existing video device (or audio device, depending on the device list) moves into the ENDED state. Note that before dispatching this event, the device in question is removed from the device list.

4.1.2 Methods

item
Retrieves a device object (an AudioDeviceTrack if this is the audio devices list or a VideoDeviceTrack if this is the video devices list).
ParameterTypeNullableOptionalDescription
indexunsigned long
Return type: getter any
select
Test a set of optional and/or mandatory constraints across the set of devices in the device list, with the goal of selecting a single device. This will queue a task to fire either a "constrainterror" or "constraintsuccess" event depending on the result. The "constraintsuccess" event includes the selected device on the DeviceEvent object's device attribute.

No devices (or their settings) are modified by this API. This API only tests the provided constraints against all the device's capabilities and reports a matching device via the "constraintsuccess" event, or no matches via "constrainterror" event.

ParameterTypeNullableOptionalDescription
constraintsMediaTrackConstraints
Return type: void

4.2 DeviceEvent interface

[Constructor(DOMString type, optional DeviceEventInit eventInitDict)]
interface DeviceEvent : Event {
    readonly attribute MediaStreamTrack device;
};

4.2.1 Attributes

device of type MediaStreamTrack, readonly
Contains a reference to the relevant device.

The actual object referenced by the device attribute will be a derived device track object such as an AudioDeviceTrack, VideoDeviceTrack or PictureDeviceTrack.

4.3 DeviceEventInit dictionary

dictionary DeviceEventInit : EventInit {
    MediaStreamTrack device;
};

4.3.1 Dictionary DeviceEventInit Members

device of type MediaStreamTrack
Video, Audio, or Picture device track used to initialize the "device" property on the DeviceEvent.

Device lists are only accessible from an existing device track object. In other words, the device list itself can only be accessed from one of the devices contained within it (this is an inside-to-outside reference). To help orient the traversal of the list, each device track object includes a (dynamically updated) device index property. If a given device track transitions to the ENDED state, then it will not belong to the device list any longer and its device index property becomes invalid (null); however, the device list itself will still be accessible from that object.

4.4 DeviceListAccess mix-in interface

AudioDeviceTrack implements DeviceListAccess;
VideoDeviceTrack implements DeviceListAccess;
[NoInterfaceObject]
interface DeviceListAccess {
    readonly attribute MediaDeviceList devices;
    readonly attribute unsigned long?  deviceIndex;
};

4.4.1 Attributes

devices of type MediaDeviceList, readonly
A reference to the device list for the associated video (or audio) device.
deviceIndex of type unsigned long, readonly, nullable
The current index of this device in the device list. This value can be dynamically changed when other devices are added to (or removed from) the device list. If this device is removed from the device list (because it enters the ENDED state), then the deviceIndex property returns null to signal that this device is not in the device list any longer.

5. Constraints for navigator.getUserMedia/MediaDeviceList.select

This proposal defines several constraints for use with video and audio devices.

These constraints are applied against the device's range or set of enumerated possible settings, but do not result in a setting change on the device. To change actual settings, use the request() API on each setting.

5.1 Video Constraints

The following constraints are applicable to video devices

5.1.1 VideoConstraints dictionary

dictionary VideoConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint)                     width;
    (unsigned long or MinMaxULongSubConstraint)                     height;
    (float or MinMaxFloatSubConstraint)                     horizontalAspectRatio;
    (float or MinMaxFloatSubConstraint)                     verticalAspectRatio;
    unsigned long          rotation;
    (float or MinMaxFloatSubConstraint)                     zoom;
    VideoFocusModeEnum     focusMode;
    VideoFillLightModeEnum fillLightMode;
    (float or MinMaxFloatSubConstraint)                     framesPerSecond;
};
Dictionary VideoConstraints Members
width of type unsigned longMinMaxULongSubConstraint
A device that supports the desired width or width range.
height of type unsigned longMinMaxULongSubConstraint
A device that supports the desired height or height range.
horizontalAspectRatio of type floatMinMaxFloatSubConstraint
A device that supports the desired horizontal aspect ratio (width/height)
verticalAspectRatio of type floatMinMaxFloatSubConstraint
A device that supports the desired vertical aspect ratio (height/width)
rotation of type unsigned long
A device that supports the desired rotation.
zoom of type floatMinMaxFloatSubConstraint
A device that supports the desired zoom setting.
focusMode of type VideoFocusModeEnum
A device that supports the desired focus mode.
fillLightMode of type VideoFillLightModeEnum
A device that supports the desired fill light (flash) mode.
framesPerSecond of type floatMinMaxFloatSubConstraint
A device that supports the desired frames per second.

5.2 Audio Constraints

The following constraints are applicable to audio devices

5.2.1 AudioConstraints dictionary

dictionary AudioConstraints : MediaTrackConstraintSet {
    (unsigned long or MinMaxULongSubConstraint) level;
};
Dictionary AudioConstraints Members
level of type unsigned longMinMaxULongSubConstraint
A device that supports the desired level or level range.

5.3 Common sub-constraint structures

5.3.1 MinMaxULongSubConstraint dictionary

dictionary MinMaxULongSubConstraint {
    unsigned long max;
    unsigned long min;
};
Dictionary MinMaxULongSubConstraint Members
max of type unsigned long
unsigned long min
min of type unsigned long

5.3.2 MinMaxFloatSubConstraint dictionary

dictionary MinMaxFloatSubConstraint {
    float max;
    float min;
};
Dictionary MinMaxFloatSubConstraint Members
max of type float
float min
min of type float

5.3.3 VideoOrientationEnum enumeration

enum VideoOrientationDict {
    "landscape",
    "portrait"
};
Enumeration description
landscapeThe long axis of the "effective" video (width/height + rotation) is landscape.
portraitThe long axis of the "effective" video (width/height + rotation) is portrait.

6. Example usage scenarios

As provided in the 3rd version of this proposal, the following JavaScript examples demonstrate how the Settings APIs defined in this proposal could be used.

6.1 Getting access to a video and/or audio device (if available) -- scenario unchanged

navigator.getUserMedia({audio: true, video: true}, gotMedia, failedToGetMedia);

function gotMedia(mediastream) {
   // The recieved mediastream is using its initial settings (it's clean)
}

6.2 Previewing the local video/audio in HTML5 video tag -- scenario is unchanged

function gotMedia(mediastream) {
   // objectURL technique
   document.querySelector("video").src = URL.createObjectURL(mediastream, { autoRevoke: true }); // autoRevoke is the default
   // direct-assign technique
   document.querySelector("video").srcObject = mediastream; // Proposed API at this time
}

6.3 Applying resolution constraints

function gotMedia(mediastream) {
   var videoDevice = mediastream.videoTracks[0];
   var maxDimensions = videoDevice.dimensions[videoDevice.dimensions.length - 1];
   // Check for 1080p+ support
   if ((maxDimensions.width >= 1920) && (maxDimensions.height >= 1080)) {
      // See if I need to change the current settings...
      if ((videoDevice.width < 1920) && (videoDevice.height < 1080)) {
         videoDevice.dimensions.request(maxDimensions, true);
         videoDevice.onconstrainterror = failureToComply;
      }
   }
   else
      failureToComply();
}

function failureToComply(e) {
   if (e)
      console.error("Device failed to change " + e.mandatoryConstraints[0]); // 'dimension'
   else
      console.error("Device doesn't support at least 1080p");
}

6.4 Changing zoom in response to user input:

function gotMedia(mediastream) {
   setupRange( mediastream.videoTracks[0] );
}

function setupRange(videoDevice) {
   // Check to see if the device supports zooming...
   if (videoDevice.zooms) {
      // Set HTML5 range control to min/max values of zoom
      var zoomControl = document.querySelector("input[type=range]");
      zoomControl.min = videoDevice.zooms.min;
      zoomControl.max = videoDevice.zooms.max;
      zoomControl.value = videoDevice.zoom;
      zoomControl.zoomController = videoDevice.zooms; // Store the setting
      zoomControl.onchange = applySettingChanges;
   }
}

function applySettingChanges(e) {
   e.target.zoomController.request(parseFloat(e.target.value), true);
}

6.5 Adding the local media tracks into a new media stream -- scenario is unchanged

function gotMedia(mediastream) {
   return new MediaStream( [ mediastream.videoTracks[0], mediastream.audioTracks[0] ]);
}

6.6 Take a picture, show the picture in an image tag

function gotMedia(mediastream) {
   var videoDevice = mediastream.videoTracks[0];
   // Check if this device supports a picture mode...
   var pictureDevice = videoDevice.pictureTrack;
   if (pictureDevice) {
       pictureDevice.onpicture = showPicture;
       // Turn on flash only for the snapshot...if available
       if (pictureDevice.fillLightModes) { // If there's an object here, then the flash is supported
          pictureDevice.fillLightModes.request("on", true);
       }
       else
          console.info("Flash not available");
       pictureDevice.takePicture();
   }
}

function showPicture(e) {
   var img = document.querySelector("img");
   img.src = URL.createObjectURL(e.data);
}

6.7 Show a newly available device

Note

A newly available device occurs whenever an existing device that was being used by another application (with exclusive access) is relinquished and becomes available for this application to use. Of course, plugging-in a new device also causes a device to become available.

function gotMedia(mediastream) {
   mediastream.videoTracks[0].devices.addEventListener("deviceadded", enableAndShowNewDevice);
}

function enableAndShowNewDevice(e) {
   // Show the new video device as soon as it's available
   // New device is muted when it first becomes available
   e.device.enabled = true;
   var mStream = new MediaStream(e.device);
   document.querySelector("video").srcObject = mStream; // Using the proposed direct-assignment API
}

6.8 Show all available video devices

function gotMedia(mediastream) {
   var deviceList = mediastream.videoTracks[0].devices;
   for (var i = 0; i < deviceList.length; i++) {
      var videoDevice = deviceList[i];
      videoDevice.enabled = true;
      // Create a video element and add it to the UI
      var videoTag = document.createElement('video');
      videoTag.srcObject = new MediaStream([videoDevice]);
      document.body.appendChild(videoTag);
   }
}