Copyright © 2013 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This specification extends HTMLMediaElement to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
The working groups maintains a list of all bug reports that the editors have not yet tried to address. This draft highlights some of the pending issues that are still to be discussed in the working group. No decision has been taken on the outcome of these issues including whether they are valid.
Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways. Vendors interested in implementing this specification before it eventually reaches the Candidate Recommendation stage should join the mailing list mentioned below and take part in the discussions.
This document was published by the HTML Working Group as a First Public Working Draft. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to public-html-media@w3.org (subscribe, archives). All feedback is welcome.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
This specification allows JavaScript to dynamically construct media streams for <audio> and <video>. It defines objects that allow JavaScript to pass media segments to an HTMLMediaElement. A buffering model is also included to describe how the user agent should act when different media segments are appended at different times. Byte stream specifications for WebM, ISO Base Media File Format, and MPEG-2 Transport Streams are given to specify the expected format of byte streams used with these extensions.
This specification was designed with the following goals in mind:
A sequence of bytes that contain all of the initialization information required to decode a sequence of media segments. This includes codec initialization data, Track ID mappings for multiplexed segments, and timestamp offsets (e.g. edit lists).
The byte stream format specifications contain format specific examples.
A sequence of bytes that contain packetized & timestamped media data for a portion of the presentation timeline. Media segments are always associated with the most recently appended initialization segment.
The byte stream format specifications contain format specific examples.
A buffer that holds initialization data and coded frames that will be decoded and rendered. This buffer may not exist in actual implementations, but it is intended to represent media data that will be decoded no matter what media segments are appended to update the SourceBuffer
. This distinction is important when considering appends that happen close to the current playback position. See Track Buffer to Decoder Buffer transfer for details.
A position in a media segment where decoding and continuous playback can begin without relying on any previous data in the segment. For video this tends to be the location of I-frames. In the case of audio, most audio frames can be treated as a random access point. Since video tracks tend to have a more sparse distribution of random access points, the location of these points are usually considered the random access points for multiplexed streams.
The presentation start time is the earliest time point in the presentation and specifies the initial playback position and earliest possible position. All presentations created using this specification have a presentation start time of 0.
A MediaSource object URL is a unique Blob URI created by createObjectURL()
. It is used to attach a MediaSource
object to an HTMLMediaElement.
These URLs are the same as what the File API specification calls a Blob URI, except that anything in the definition of that feature that refers to File and Blob objects is hereby extended to also apply to MediaSource
objects.
A Track ID is a byte stream format specific identifier that marks sections of the byte stream as being part of a specific track. The Track ID in a track description identifies which sections of a media segment belong to that track.
A byte stream format specific structure that provides the Track ID, codec configuration, and other metadata for a single track. Each track description inside a single initialization segment must have a unique Track ID.
A unit of compressed media data that has a presentation timestamp and decode timestamp. The presentation timestamp indicates when the frame should be rendered. The decode timestamp indicates when the frame needs to be decoded. If frames can be decoded out of order, then the decode timestamp must be present in the bytestream. If frames cannot be decoded out of order and a decode timestamp is not present in the bytestream, then the decode timestamp is equal to the presentation timestamp.
SourceBuffer
object is the MediaSource
object that created it.appendArrayBuffer()
or appendStream()
calls on a SourceBuffer
without any intervening abort()
calls. The
media segments in an append sequence must be adjacent and monotonically increasing in time without any gaps. An
abort()
call starts a new append sequence which allows media segments to be appended in non-monotonically
increasing order.The subsections below outline the buffering model for this specification. It describes the various rules and behaviors associated with appending
data to an individual SourceBuffer
. At the highest level, the web application creates SourceBuffer
objects and appends sequences of
initialization segments and media segments to update their state. The media element pulls media data out of the
MediaSource
object, plays it, and fires events just like it would if a normal URL was passed to the src
attribute.
The web application is expected to monitor media element events to determine when it needs to append more media segments.
There are several ways that media segments can overlap segments in the SourceBuffer
. Behavior for the different overlap situations are described below. If more than one overlap applies, then the start overlap must be resolved first, followed by any complete overlaps, and finally the end overlap. If a segment contains multiple tracks then the overlap is resolved independently for each track.
Bug 19673 - Seamless audio signal transitions at splice points
Bug 19784 - timestampOffset with multiplexed Media Segments
The figure above shows how the SourceBuffer
is updated when a new media segment completely overlaps a segment in the buffer. In this case, the new segment completely replaces the old segment.
The figure above shows how the SourceBuffer
is updated when the beginning of a new media segment overlaps a segment in the buffer. In this case, the new segment replaces all the old media data in the overlapping region. Since media segments are constrained to starting with random access points, this provides a seamless transition between segments.
When an audio frame in the SourceBuffer
overlaps with the start of the new media segment special behavior is required. At a minimum implementations must support dropping the old audio frame that overlaps the start of the new segment and insert silence for the small gap that is created. Higher quality implementations may support crossfading or crosslapping between the overlapping audio frames. No matter which strategy is implemented, no gaps are created in the ranges reported by buffered
and playback must never stall at the overlap.
The figure above shows how the SourceBuffer
is updated when the end of a new media segment overlaps the beginning of a segment in the buffer. In this case, the SourceBuffer
tries to keep as much of the old segment as possible. The amount saved depends on where the closest random access point, in the old segment, is to the end of the new segment. In the case of audio, if the gap is smaller than the size of an audio frame, then the SourceBuffer
may render silence for this gap. This gap must not be reflect in buffered
. The entire new segment must be added to the SourceBuffer
, but it is up to the implementation to determine how much of the old segment data is retained.
An implementation may keep old segment data before the end of the new segment to avoid creating a gap if it wishes. Doing this though can significantly increase implementation complexity and could cause delays at the splice point.
The web application can use buffered
to determine how much of the old segment was preserved.
The figure above shows how the SourceBuffer
is updated when the new media segment is in the middle of the old segment. This condition is handled by first resolving the start overlap and then resolving the end overlap.
The track buffer represents the media that the web application would like the media element to play. The decoder buffer contains the data that will actually get decoded and rendered. In most cases the decoder buffer will simply contain a subset of the track buffer near the current playback position. These two buffers start to diverge when media segments that overlap or are very close to the current playback position are appended. Depending on the contents of the new media segment it may not be possible to switch to the new data immediately because there isn't a random access point close enough to the current playback position. The quality of the implementation determines how much data is considered "in the decoder buffer." It should transfer data to the decoder buffer as late as possible whilst maintaining seamless playback. Some implementations may be able to instantiate multiple decoders or decode the new data significantly faster than real-time to achieve a seamless splice immediately. Other implementations may delay until the next random access point before switching to the newly appended data. Notice that this difference in behavior is only observable when appending close to the current playback position. The decoder buffer represents a media subsegment, like a group of pictures or something with similar decode dependencies, that the media element commits to playing. This commitment may be influenced by a variety of things like limited decoding resources, hardware decode buffers, a jitter buffer, or the desire to limit implementation complexity.
Here is an example to help clarify the role of the decoder buffer. Say the current playback position has a timestamp of 8 and the media element pulled frames with timestamp 9 & 10 into the decoder buffer. The web application then appends a higher quality media segment that starts with a random access point at timestamp 9. The track buffer will get updated with the higher quality data, but the media element won't be able to switch to this higher quality data until the next random access point at timestamp 20. This is because a frame for timestamp 9 is already in the decoder buffer. The decoder buffer represents the "point of no return." for decoding. If a seek occurs the media element may choose to use the higher quality data since a seek might imply flushing the decoder buffer and the user expects a break in playback.
The MediaSource object represents a source of media data for an HTMLMediaElement. It keeps track of the readyState
for this source as well as a list of SourceBuffer
objects that can be used to add media data to the presentation. MediaSource objects are created by the web application and then attached to an HTMLMediaElement. The application uses the SourceBuffer
objects in sourceBuffers
to add media data to this source. The HTMLMediaElement fetches this media data from the MediaSource
object when it is needed during playback.
enum ReadyState {
"closed",
"open",
"ended"
};
Enumeration description | |
---|---|
closed | Indicates the source is not currently attached to a media element. |
open |
The source has been opened by a media element and is ready for data to be appended to the SourceBuffer objects in sourceBuffers .
|
ended |
The source is still attached to a media element, but endOfStream() has been called.
|
enum EndOfStreamError {
"network",
"decode"
};
Enumeration description | |
---|---|
network |
Terminates playback and signals that a network error has occured. Note If the JavaScript fetching media data encounters a network error it should use this status code to terminate playback. |
decode |
Terminates playback and signals that a decoding error has occured. Note If the JavaScript code fetching media data has problems parsing the data it should use this status code to terminate playback. |
[Constructor]
interface MediaSource : EventTarget {
readonly attribute SourceBufferList
sourceBuffers;
readonly attribute SourceBufferList
activeSourceBuffers;
readonly attribute ReadyState
readyState;
attribute unrestricted double duration;
SourceBuffer
addSourceBuffer (DOMString type);
void removeSourceBuffer (SourceBuffer
sourceBuffer);
SourceBuffer
? getSourceBuffer (VideoTrack videoTrack);
SourceBuffer
? getSourceBuffer (AudioTrack audioTrack);
SourceBuffer
? getSourceBuffer (TextTrack textTrack);
void endOfStream (optional EndOfStreamError
error);
void setTrackInfo (VideoTrack track, DOMString kind, DOMString language);
void setTrackInfo (AudioTrack track, DOMString kind, DOMString language);
void setTrackInfo (TextTrack track, DOMString kind, DOMString language);
static bool isTypeSupported (DOMString type);
};
activeSourceBuffers
of type SourceBufferList
, readonlyContains the subset of sourceBuffers
that are providing the
selected video track, the
enabled audio tracks, and the
"showing" or "hidden" text tracks.
The Changes to selected/enabled track state section describes how this attribute gets updated.
duration
of type unrestricted doubleAllows the web application to set the presentation duration. The duration is initially set to NaN when the MediaSource
object is created.
On getting, run the following steps:
readyState
attribute is "closed"
then return NaN and abort these steps.On setting, run the following steps:
INVALID_ACCESS_ERR
exception and abort these steps.readyState
attribute is not "open"
then throw an INVALID_STATE_ERR
exception and abort these steps.appendArrayBuffer()
, appendStream()
and endOfStream()
can update the duration under certain circumstances.
readyState
of type ReadyState
, readonlyIndicates the current state of the MediaSource
object. When the MediaSource
is created readyState
must be set to "closed"
.
sourceBuffers
of type SourceBufferList
, readonlySourceBuffer
objects associated with this MediaSource
. When readyState
equals "closed"
this list will be empty. Once readyState
transitions to "open"
SourceBuffer objects can be added to this list by using addSourceBuffer()
.
addSourceBuffer
Adds a new SourceBuffer
to sourceBuffers
.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
type | DOMString | ✘ | ✘ |
SourceBuffer
When this method is invoked, the user agent must run the following steps:
INVALID_ACCESS_ERR
exception and abort these steps.SourceBuffer
objects in sourceBuffers
, then throw a NOT_SUPPORTED_ERR
exception and abort these steps.QUOTA_EXCEEDED_ERR
exception and abort these steps.readyState
attribute is not in the "open"
state then throw an INVALID_STATE_ERR
exception and abort these steps.SourceBuffer
object and associated resources.sourceBuffers
and queue a task to fire a simple event named addsourcebuffer
at sourceBuffers
.endOfStream
Signals the end of the stream.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
error |
| ✘ | ✔ |
void
When this method is invoked, the user agent must run the following steps:
readyState
attribute is not in the "open"
state then throw an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute value to "ended"
.sourceended
at the MediaSource
.SourceBuffer
objects in sourceBuffers
.This allows the duration to properly reflect the end of the appended media segments. For example, if the duration was explicitly set to 10 seconds and only media segments for 0 to 5 seconds were appended before endOfStream() was called, then the duration will get updated to 5 seconds.
appendArrayBuffer()
and appendStream()
has been played."network"
HTMLMediaElement.readyState
attribute equals HAVE_NOTHING
HTMLMediaElement.readyState
attribute is greater than HAVE_NOTHING
"decode"
HTMLMediaElement.readyState
attribute equals HAVE_NOTHING
HTMLMediaElement.readyState
attribute is greater than HAVE_NOTHING
INVALID_ACCESS_ERR
exception.getSourceBuffer
Gets the SourceBuffer
object that created a specific VideoTrack
.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
videoTrack | VideoTrack | ✘ | ✘ |
SourceBuffer
, nullableWhen this method is invoked, the user agent must run the following steps:
SourceBuffer
object in sourceBuffers
then return null.SourceBuffer
object in sourceBuffers
that created videoTrack.
getSourceBuffer
Gets the SourceBuffer
object that created a specific AudioTrack
.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
audioTrack | AudioTrack | ✘ | ✘ |
SourceBuffer
, nullableWhen this method is invoked, the user agent must run the following steps:
SourceBuffer
object in sourceBuffers
then return null.SourceBuffer
object in sourceBuffers
that created audioTrack.
getSourceBuffer
Gets the SourceBuffer
object that created a specific TextTrack
.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
textTrack | TextTrack | ✘ | ✘ |
SourceBuffer
, nullableWhen this method is invoked, the user agent must run the following steps:
SourceBuffer
object in sourceBuffers
then return null.SourceBuffer
object in sourceBuffers
that created textTrack.
isTypeSupported
, staticCheck to see whether the MediaSource
is capable of creating SourceBuffer
objects for the the specified MIME type.
If true is returned from this method, it only indicates that the MediaSource
implementation is capable of creating SourceBuffer
objects for the specified MIME type. An addSourceBuffer()
call may still fail if sufficient resources are not available to support the addition of a new SourceBuffer
.
This method returning true implies that HTMLMediaElement.canPlayType() will return "maybe" or "probably" since it does not make sense for a MediaSource
to support a type the HTMLMediaElement knows it cannot play.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
type | DOMString | ✘ | ✘ |
bool
When this method is invoked, the user agent must run the following steps:
removeSourceBuffer
Removes a SourceBuffer
from sourceBuffers
.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
sourceBuffer |
| ✘ | ✘ |
void
When this method is invoked, the user agent must run the following steps:
INVALID_ACCESS_ERR
exception and abort these steps.sourceBuffers
then throw a NOT_FOUND_ERR
exception and abort these steps.audioTracks
, videoTracks
, and textTracks
for all tracks associated with sourceBuffer and queue a task to fire a simple event named change at the modified lists.activeSourceBuffers
, then remove it from activeSourceBuffers
and queue a task to fire a simple event named removesourcebuffer
at activeSourceBuffers
.sourceBuffers
and queue a task to fire a simple event named removesourcebuffer
at sourceBuffers
.setTrackInfo
Set the kind
and language
of the VideoTrack
track.
This method would be unnecessary if the kind
and language
attributes of VideoTrack
were not read-only.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
track | VideoTrack | ✘ | ✘ | |
kind | DOMString | ✘ | ✘ | |
language | DOMString | ✘ | ✘ |
void
setTrackInfo
Set the kind
and language
of the AudioTrack
track.
This method would be unnecessary if the kind
and language
attributes of AudioTrack
were not read-only.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
track | AudioTrack | ✘ | ✘ | |
kind | DOMString | ✘ | ✘ | |
language | DOMString | ✘ | ✘ |
void
setTrackInfo
Set the kind
and language
of the TextTrack
track.
This method would be unnecessary if the kind
and language
attributes of TextTrack
were not read-only.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
track | TextTrack | ✘ | ✘ | |
kind | DOMString | ✘ | ✘ | |
language | DOMString | ✘ | ✘ |
void
Event name | Interface | Dispatched when... |
---|---|---|
sourceopen |
Event |
readyState transitions from "closed" to "open" or from "ended" to "open" . |
sourceended |
Event |
readyState transitions from "open" to "ended" . |
sourceclose |
Event |
readyState transitions from "open" to "closed" or "ended" to "closed" . |
A MediaSource
object can be attached to a media element by assigning a MediaSource object URL to the media element src
attribute or the src attribute of a <source> inside a media element. A MediaSource object URL is created by passing a MediaSource object to createObjectURL()
.
If the resource fetch algorithm absolute URL matches the MediaSource object URL, run the following steps right before the "Perform a potentially CORS-enabled fetch" step in the resource fetch algorithm.
readyState
is NOT set to "closed"
readyState
attribute to "open"
.sourceopen
at the MediaSource
.appendArrayBuffer()
and appendStream()
.The following steps are run in any case where the media element is going to transition to NETWORK_EMPTY and queue a task to fire a simple event named emptied at the media element. These steps should be run right before the transition.
readyState
attribute to "closed"
.duration
attribute to NaN.SourceBuffer
objects from activeSourceBuffers
.removesourcebuffer
at activeSourceBuffers
.SourceBuffer
objects from sourceBuffers
.removesourcebuffer
at sourceBuffers
.sourceclose
at the MediaSource
.Run the following steps as part of the "Wait until the user agent has established whether or not the media data for the new playback position is available, and, if it is, until it has decoded enough data to play back that position" step of the seek algorithm:
SourceBuffer
object in activeSourceBuffers
.
activeSourceBuffers
is missing media segments for the new playback position
HTMLMediaElement.readyState
attribute to HAVE_METADATA
.appendArrayBuffer()
or appendStream()
.
The web application can use buffered
to determine what the media element needs to resume playback.
The following steps are periodically run during playback to make sure that all of the SourceBuffer
objects in activeSourceBuffers
have enough data to ensure uninterrupted playback. Appending new segments and changes to activeSourceBuffers
also cause these steps to run because they affect the conditions that trigger state transitions.
The web application can monitor changes in HTMLMediaElement.readyState
to drive media segment appending.
Bug 18592 - How much is "enough data to ensure uninterrupted playback"
buffered
for all objects in activeSourceBuffers
do not contain TimeRanges
for the current playback position:HTMLMediaElement.readyState
attribute to HAVE_METADATA
.HAVE_METADATA
, then queue a task to fire a simple event named loadedmetadata
at the media element.buffered
for all objects in activeSourceBuffers
contain TimeRanges
that include the current playback position and enough data to ensure uninterrupted playback:HTMLMediaElement.readyState
attribute to HAVE_ENOUGH_DATA
.canplaythrough
at the media element.HAVE_CURRENT_DATA
.buffered
for at least one object in activeSourceBuffers
contains a TimeRange
that includes the current playback position but not enough data to ensure uninterrupted playback:HTMLMediaElement.readyState
attribute to HAVE_FUTURE_DATA
.HTMLMediaElement.readyState
was less than HAVE_FUTURE_DATA
, then queue a task to fire a simple event named canplay
at the media element.HAVE_CURRENT_DATA
.buffered
for at least one object in activeSourceBuffers
contains a TimeRange
that ends at the current playback position and does not have a range covering the time immediately after the current position:HTMLMediaElement.readyState
attribute to HAVE_CURRENT_DATA
.HAVE_CURRENT_DATA
, then queue a task to fire a simple event named loadeddata
at the media element.During playback activeSourceBuffers
needs to be updated if the selected video track, the enabled audio tracks, or a text track mode changes. When one or more of these changes occur the following steps need to be followed.
SourceBuffer
associated with the previously selected video track is not associated with any other enabled tracks, run the following steps:
SourceBuffer
from activeSourceBuffers
.removesourcebuffer
at activeSourceBuffers
SourceBuffer
associated with the newly selected video track is not already in activeSourceBuffers
, run the following steps:
SourceBuffer
to activeSourceBuffers
.addsourcebuffer
at activeSourceBuffers
SourceBuffer
associated with this track is not associated with any other enabled or selected track, then run the following steps:SourceBuffer
associated with the audio track from activeSourceBuffers
removesourcebuffer
at activeSourceBuffers
SourceBuffer
associated with this track is not already in activeSourceBuffers
, then run the following steps:
SourceBuffer
associated with the audio track to activeSourceBuffers
addsourcebuffer
at activeSourceBuffers
SourceBuffer
associated with this track is not associated with any other enabled or selected track, then run the following steps:SourceBuffer
associated with the text track from activeSourceBuffers
removesourcebuffer
at activeSourceBuffers
SourceBuffer
associated with this track is not already in activeSourceBuffers
, then run the following steps:
SourceBuffer
associated with the text track to activeSourceBuffers
addsourcebuffer
at activeSourceBuffers
Follow these steps when duration
needs to change to a new duration.
duration
is equal to new duration, then abort these steps.duration
.
duration
to new duration.remove(new duration, old duration)
on all objects in sourceBuffers
.
media controller duration
to new duration and run the HTMLMediaElement duration change algorithm.Bug 20327 - Continuous splice flag
interface SourceBuffer : EventTarget {
readonly attribute boolean appending;
readonly attribute TimeRanges buffered;
attribute double timestampOffset;
void appendArrayBuffer (ArrayBuffer data);
void appendStream (Stream stream, optional unsigned long long maxSize);
void abort ();
void remove (double start, double end);
};
appending
of type boolean, readonlyIndicates whether an appendArrayBuffer()
or appendStream()
operation is still being processed.
buffered
of type TimeRanges, readonlyIndicates what TimeRanges
are buffered in the SourceBuffer
.
When the attribute is read the following steps must occur:
sourceBuffers
attribute of the parent media source then throw an INVALID_STATE_ERR
exception and abort these steps.timestampOffset
of type doubleControls the offset applied to timestamps inside subsequent media segments that are appended to this SourceBuffer
. The timestampOffset
is initially set to 0 which indicates that no offset is being applied.
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
sourceBuffers
attribute of the parent media source, then throw an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute of the parent media source is not in the "open"
state, then throw an INVALID_STATE_ERR
exception and abort these steps.INVALID_STATE_ERR
and abort these steps.Bug 19676 - timestampOffset accuracy
abort
Aborts the current segment and resets the segment parser.
void
When this method is invoked, the user agent must run the following steps:
sourceBuffers
attribute of the parent media source then throw an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute of the parent media source is not in the "open"
state then throw an INVALID_STATE_ERR
exception and abort these steps.appending
attribute equals true, then run the following steps:
appending
attribute to false.abort
at this SourceBuffer
object.appendend
at this SourceBuffer
object.appendArrayBuffer
Appends the segment data in an ArrayBuffer to the source buffer.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
data | ArrayBuffer | ✘ | ✘ |
void
When this method is invoked, the user agent must run the following steps:
INVALID_ACCESS_ERR
exception and abort these steps.sourceBuffers
attribute of the parent media source then throw an INVALID_STATE_ERR
exception and abort these steps.appending
attribute equals true, then throw an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute of the parent media source is in the "closed"
state then throw an INVALID_STATE_ERR
exception and abort these steps.If the readyState
attribute of the parent media source is in the "ended"
state then run the following steps:
readyState
attribute of the parent media source to "open"
sourceopen
at the parent media source .If the buffer full flag equals true, then throw a QUOTA_EXCEEDED_ERR
exception and abort these step.
The web application must use remove()
to free up space in the SourceBuffer
.
appending
attribute to true.appendstart
at this SourceBuffer
object.appendStream
Appends segment data to the source buffer from a Stream.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
stream | Stream | ✘ | ✘ | |
maxSize | unsigned long long | ✘ | ✔ |
void
When this method is invoked, the user agent must run the following steps:
INVALID_ACCESS_ERR
exception and abort these steps.sourceBuffers
attribute of the parent media source then throw
an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute of the parent media source is in the "closed"
state then throw
an INVALID_STATE_ERR
exception and abort these steps.appending
attribute equals true, then throw an INVALID_STATE_ERR
exception and abort these
steps.If the readyState
attribute of the parent media source is in the "ended"
state then run
the following steps:
readyState
attribute of the parent media source to "open"
sourceopen
at the parent media source .If the buffer full flag equals true, then throw a QUOTA_EXCEEDED_ERR
exception and abort these step.
The web application must use remove()
to free up space in the SourceBuffer
.
appending
attribute to true.appendstart
at this SourceBuffer
object.remove
Removes media for a specific time range.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
start | double | ✘ | ✘ | |
end | double | ✘ | ✘ |
void
When this method is invoked, the user agent must run the following steps:
duration
, then throw an INVALID_ACCESS_ERR
exception and abort these steps.INVALID_ACCESS_ERR
exception and abort these steps.sourceBuffers
attribute of the parent media source then throw an INVALID_STATE_ERR
exception and abort these steps.readyState
attribute of the parent media source is not in the "open"
state then throw an INVALID_STATE_ERR
exception and abort these steps.For each track buffer in this source buffer, run the following steps:
duration
If this track buffer has a random access point timestamp that is greater than or equal to end, then update remove end timestamp to that timestamp.
Random access point timestamps can be different across tracks because the dependencies between coded frames within a track are usually different than the dependencies in another track.
If this object is in activeSourceBuffers
, the current playback position is greater than or equal to start and less than the remove end timestamp, and HTMLMediaElement.readyState
is greater than HAVE_METADATA
, then set the HTMLMediaElement.readyState
attribute to HAVE_METADATA
and stall playback.
This transition occurs because media data for the current position has been removed. Playback cannot progress until media for the current playback position is appended or the selected/enabled tracks change.
A track buffer stores the track descriptions and coded frames for an individual
track. The track buffer is updated as initialization segments and media segments are appended to the
SourceBuffer
.
Each track buffer has a last decode timestamp variable that stores the decode timestamp of the last coded frame appended in the current append sequence. The variable is initially unset to indicate that no coded frames have been appended yet.
Event name | Interface | Dispatched when... |
---|---|---|
appendstart |
Event |
appending transitions from false to true. |
appendend |
Event |
appending transitions from true to false. |
error |
Event |
An error occurred during the append. |
abort |
Event |
The append was aborted by an abort() call. |
All SourceBuffer objects have an internal append state variable that keeps track of the high-level segment parsing state. It is initially set to WAITING_FOR_SEGMENT and can transition to the following states as data is appended.
Append state name | Description |
---|---|
WAITING_FOR_SEGMENT | Waiting for the start of an initialization segment or media segment to be appended. |
PARSING_INIT_SEGMENT | Currently parsing an initialization segment. |
PARSING_MEDIA_SEGMENT | Currently parsing a media segment. |
The input buffer is a byte buffer that is used to hold unparsed bytes across appendArrayBuffer()
and appendStream()
calls. The buffer is empty when the SourceBuffer object is created.
The buffer full flag keeps track of whether appendArrayBuffer()
or
appendStream()
is allowed to accept more bytes. It is set to false when the SourceBuffer object is created and gets updated
as data is appended and removed.
When this algorithm is invoked, run the following steps:
If the append state equals WAITING_FOR_SEGMENT, then run the following steps:
If the append state equals PARSING_INIT_SEGMENT, then run the following steps:
If the append state equals PARSING_MEDIA_SEGMENT, then run the following steps:
If the input buffer does not contain a complete media segment header yet, then jump to the need more data step below.
Implementations may choose to implement this state as an incremental parser so that it is not necessary to have the entire media segment before running the coded frame processing algorithm.
SourceBuffer
is full and cannot accept more media data, then set the buffer full flag to true.Set append state to WAITING_FOR_SEGMENT.
Incremental parsers should only do this transition after the entire media segment has been received.
appending
attribute to false.appendend
at this SourceBuffer
object.When the parser state needs to be reset, run the following steps:
When an error occurs during an append, run the following steps:
appending
attribute to false.error
at this SourceBuffer
object.
Need a way to convey error information.
appendend
at this SourceBuffer
object.When a Stream is passed to appendStream()
, the following steps are run to transfer data from the
Stream to the SourceBuffer
. This algorithm is initialized with the stream and maxSize parameters
from the appendStream()
call.
If the buffer full flag equals true, then run the append error algorithm and abort this algorithm.
The web application must use remove()
to free up space in the SourceBuffer
.
appending
attribute to false.appendend
at this SourceBuffer
object.The following steps are run when the segment parser loop successfully parses a complete initialization segment:
Each SourceBuffer object has an internal first initialization segment flag that tracks whether the first initialization segment has been appended. This flag is set to false when the SourceBuffer is created and updated by the algorithm below.
duration
attribute if it currently equals NaN:
endOfStream("decode")
and abort these steps.endOfStream("decode")
and abort these steps.
If the first initialization segment flag is false, then run the following steps:
For each audio track in the initialization segment, run following steps:
AudioTrack
object.id
property on new audio track.
If audioTracks
.length
equals 0, then run
the following steps:
enabled
property on new audio track to true.audioTracks
.For each video track in the initialization segment, run following steps:
VideoTrack
object.id
property on new video track.
If videoTracks
.length
equals 0, then run
the following steps:
selected
property on new video track to true.videoTracks
.For each text track in the initialization segment, run following steps:
TextTrack
object with its properties populated with the appropriate
information from the initialization segment.mode
property on new text track equals "showing"
or
"hidden"
, then set active track flag to true.
textTracks
.SourceBuffer
to activeSourceBuffers
.addsourcebuffer
at activeSourceBuffers
If the HTMLMediaElement.readyState
attribute is HAVE_NOTHING
, then run the following steps:
sourceBuffers
have first initialization segment flag set to false, then abort
these steps.HTMLMediaElement.readyState
attribute to HAVE_METADATA
.loadedmetadata
at the media element.HTMLMediaElement.readyState
attribute is greater than
HAVE_CURRENT_DATA
, then set the HTMLMediaElement.readyState
attribute to HAVE_METADATA
.
When complete coded frames have been parsed by the segment parser loop then the following steps are run:
For each coded frame in the media segment run the following steps:
If timestampOffset
is not 0, then run the following steps:
timestampOffset
to the presentation timestamp.timestampOffset
to the decode timestamp.endOfStream("decode")
, and abort these steps.endOfStream("decode")
and abort these steps.If the HTMLMediaElement.readyState
attribute is HAVE_METADATA
and the new coded frames cause all objects in activeSourceBuffers
to have media data for the current playback position, then run the following steps:
HTMLMediaElement.readyState
attribute to HAVE_CURRENT_DATA
.HAVE_CURRENT_DATA
, then queue a task to fire a simple event named loadeddata
at the media element.If the HTMLMediaElement.readyState
attribute is HAVE_CURRENT_DATA
and the new coded frames cause all objects in activeSourceBuffers
to have media data beyond the current playback position, then run the following steps:
HTMLMediaElement.readyState
attribute to HAVE_FUTURE_DATA
.canplay
at the media element.If the HTMLMediaElement.readyState
attribute is HAVE_FUTURE_DATA
and the new coded frames cause all objects in activeSourceBuffers
to have enough data to start playback, then run the following steps:
HTMLMediaElement.readyState
attribute to HAVE_ENOUGH_DATA
.canplaythrough
at the media element.duration
, then run the duration change algorithm with new duration set to the maximum of the current duration and the highest end timestamp reported by HTMLMediaElement.buffered
.SourceBufferList is a simple container object for SourceBuffer
objects. It provides read-only array access and fires events when the list is modified.
interface SourceBufferList : EventTarget {
readonly attribute unsigned long length;
getter SourceBuffer (unsigned long index);
};
length
of type unsigned long, readonlyIndicates the number of SourceBuffer
objects in the list.
SourceBuffer
Allows the SourceBuffer objects in the list to be accessed with an array operator (i.e. []).
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
index | unsigned long | ✘ | ✘ |
getter
When this method is invoked, the user agent must run the following steps:
length
attribute then return undefined and abort these steps.SourceBuffer
object in the list.Event name | Interface | Dispatched when... |
---|---|---|
addsourcebuffer |
Event |
When a SourceBuffer is added to the list. |
removesourcebuffer |
Event |
When a SourceBuffer is removed from the list. |
partial interface URL {
static DOMString createObjectURL (MediaSource
mediaSource);
};
createObjectURL
, staticCreates URLs for MediaSource
objects.
This algorithm is intended to mirror the behavior of the File API createObjectURL() method with autoRevoke set to true.
Parameter | Type | Nullable | Optional | Description |
---|---|---|---|---|
mediaSource |
| ✘ | ✘ |
DOMString
When this method is invoked, the user agent must run the following steps:
This section specifies what existing attributes on the HTMLMediaElement should return when a MediaSource
is attached to the element.
The HTMLMediaElement.seekable attribute returns a new static normalized TimeRanges object created based on the following steps:
duration
equals NaNTimeRanges
object.duration
equals positive InfinityHTMLMediaElement.buffered
attribute.duration
.The HTMLMediaElement.buffered
attribute returns a new static normalized TimeRanges object created based on the following steps:
buffered
for each SourceBuffer
object in activeSourceBuffers
.If readyState
is "ended"
, then run the following steps:
Bug 18615 - Define how SourceBuffer.buffered maps to HTMLMediaElement.buffered
Bug 18400 - Define and document timestamp heuristics
The bytes provided through appendArrayBuffer()
and appendStream()
for a SourceBuffer
form a logical byte stream. The format of this byte stream depends on the media container format in use and is defined in a byte stream format specification. Byte stream format specifications based on WebM , the ISO Base Media File Format, and MPEG-2 Transport Streams are provided below. These format specifications are intended to be the authoritative source for how data from these containers is formatted and passed to a SourceBuffer
. If a MediaSource
implementation claims to support any of these container formats, then it must implement the corresponding byte stream format specification described below.
This section provides general requirements for all byte stream formats:
The number and type of tracks must be consistent.
For example, if the first initialization segment has 2 audio tracks and 1 video track, then all initialization segments that follow it in the byte stream must describe 2 audio tracks and 1 video track.
Track IDs do not need to be the same across initialization segments if the segment describes only one track of each type.
For example, if an initialization segment describes a single audio track and a single video track, the internal Track IDs do not need to be the same.
Codecs changes are not allowed.
For example, a byte stream that starts with an initialization segment that specifies a single AAC track and later contains an initialization segment that specifies a single AMR-WB track is not allowed. Support for multiple codecs is handled with multiple SourceBuffer
objects.
Video frame size changes are allowed and must be supported seamlessly.
This will cause the <video> display region to change size if the web application does not use CSS or HTML attributes (width/height) to constrain the element size.
Audio channel count changes are allowed, but they may not be seamless and could trigger downmixing.
This is a quality of implementation issue because changing the channel count may require reinitializing the audio device, resamplers, and channel mixers which tends to be audible.
buffered
.
This is intended to simplify switching between audio streams where the frame boundaries don't always line up across encodings (e.g. Vorbis).
For example, if I1 is associated with M1, M2, M3 then the above must hold for all the combinations I1+M1, I1+M2, I1+M1+M2, I1+M2+M3, etc.
Byte stream specifications must at a minimum define constraints which ensure that the above requirements hold. Additional constraints may be defined, for example to simplify implementation.
This section defines segment formats for implementations that choose to support WebM.
A WebM initialization segment must contain a subset of the elements at the start of a typical WebM file.
The following rules apply to WebM initialization segments:
A WebM media segment is a single Cluster element.
The following rules apply to WebM media segments:
A SimpleBlock element with its Keyframe flag set signals the location of a random access point for that track. Media segments containing multiple tracks are only considered a random access point if the first SimpleBlock for each track has its Keyframe flag set. The order of the multiplexed blocks must conform to the WebM Muxer Guidelines.
This section defines segment formats for implementations that choose to support the ISO Base Media File Format ISO/IEC 14496-12 (ISO BMFF).
Bug 18933 - Segment byte boundaries are not defined
An ISO BMFF initialization segment must contain a single Movie Header Box (moov). The tracks in the Movie Header Box must not contain any samples (i.e. the entry_count in the stts, stsc and stco boxes must be set to zero). A Movie Extends (mvex) box must be contained in the Movie Header Box to indicate that Movie Fragments are to be expected.
The initialization segment may contain Edit Boxes (edts) which provide a mapping of composition times for each track to the global presentation time.
An ISO BMFF media segment must contain a single Movie Fragment Box (moof) followed by one or more Media Data Boxes (mdat).
The following rules apply to ISO BMFF media segments:
A random access point as defined in this specification corresponds to a Stream Access Point of type 1 or 2 as defined in Annex I of ISO/IEC 14496-12.
This section defines segment formats for implementations that choose to support MPEG-2 Transport Streams (MPEG-2 TS) specified in ISO/IEC 13818-1.
MPEG-2 TS media and initialization segments must conform to the MPEG-2 TS Adaptive Profile (ISO/IEC 13818-1:2012 Amd. 2).
The following rules must apply to all MPEG-2 TS segments:
An MPEG-2 TS initialization segment must contain a single PAT and a single PMT. Other SI, such as CAT, that are invariant for all subsequent media segments, may be present.
The following rules apply to all MPEG-2 TS media segments:
A random access point as defined in this specification corresponds to Elementary Stream Random Access Point as defined in ISO/IEC 13818-1.
Timestamp rollovers and discontinuities must be handled by the UA. The UA's MPEG-2 TS implementation must maintain an internal offset
variable, MPEG2TS_timestampOffset, to keep track of the offset that needs to be applied to timestamps
that have rolled over or are part of a discontinuity. MPEG2TS_timestampOffset is initially set to 0 when the SourceBuffer
is
created. This offset must be applied to the timestamps as part of the conversion process from MPEG-2 TS packets
into coded frames for the coded frame processing algorithm. This results in the coded frame timestamps
for a packet being computed by the following equations:
Coded Frame Presentation Timestamp = (MPEG-2 TS presentation timestamp) + MPEG2TS_timestampOffset Coded Frame Decode Timestamp = (MPEG-2 TS decode timestamp) + MPEG2TS_timestampOffset
MPEG2TS_timestampOffset is updated in the following ways:
abort()
is called, MPEG2TS_timestampOffset must be set to 0.timestampOffset
is successfully set, MPEG2TS_timestampOffset must be set to 0.Example use of the Media Source Extensions
<script> function onSourceOpen(videoTag, e) { var mediaSource = e.target; var sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vorbis,vp8"'); videoTag.addEventListener('seeking', onSeeking.bind(videoTag, mediaSource)); videoTag.addEventListener('progress', onProgress.bind(videoTag, mediaSource)); var initSegment = GetInitializationSegment(); if (initSegment == null) { // Error fetching the initialization segment. Signal end of stream with an error. mediaSource.endOfStream("network"); return; } // Append the initialization segment. var firstAppendHandler = function(e) { var sourceBuffer = e.target; sourceBuffer.removeEventListener('appendend', firstAppendHandler); // Append some initial media data. appendNextMediaSegment(mediaSource); }; sourceBuffer.addEventListener('appendend', firstAppendHandler); sourceBuffer.appendArrayBuffer(initSegment); } function appendNextMediaSegment(mediaSource) { if (mediaSource.readyState == "ended") return; // If we have run out of stream data, then signal end of stream. if (!HaveMoreMediaSegments()) { mediaSource.endOfStream(); return; } // Make sure the previous append is not still pending. if (mediaSource.sourceBuffers[0].appending) return; var mediaSegment = GetNextMediaSegment(); if (!mediaSegment) { // Error fetching the next media segment. mediaSource.endOfStream("network"); return; } mediaSource.sourceBuffers[0].appendArrayBuffer(mediaSegment); } function onSeeking(mediaSource, e) { var video = e.target; // Abort current segment append. mediaSource.sourceBuffers[0].abort(); // Notify the media segment loading code to start fetching data at the // new playback position. SeekToMediaSegmentAt(video.currentTime); // Append a media segment from the new playback position. appendNextMediaSegment(mediaSource); } function onProgress(mediaSource, e) { appendNextMediaSegment(mediaSource); } </script> <video id="v" autoplay> </video> <script> var video = document.getElementById('v'); var mediaSource = new MediaSource(); mediaSource.addEventListener('sourceopen', onSourceOpen.bind(this, video)); video.src = window.URL.createObjectURL(mediaSource); </script>
Version | Comment |
---|---|
04 January 2013 |
|
14 December 2012 | Pubrules, Link Checker, and Markup Validation fixes. |
13 December 2012 |
|
08 December 2012 |
|
06 December 2012 |
|
28 November 2012 |
|
09 November 2012 | Converted document to ReSpec. |
18 October 2012 | Refactored SourceBuffer.append() & added SourceBuffer.remove(). |
8 October 2012 |
|
1 October 2012 | Fixed various addsourcebuffer & removesourcebuffer bugs and allow append() in ended state. |
13 September 2012 | Updated endOfStream() behavior to change based on the value of HTMLMediaElement.readyState. |
24 August 2012 |
|
22 August 2012 |
|
17 August 2012 | Minor editorial fixes. |
09 August 2012 | Change presentation start time to always be 0 instead of using format specific rules about the first media segment appended. |
30 July 2012 | Added SourceBuffer.timestampOffset and MediaSource.duration. |
17 July 2012 | Replaced SourceBufferList.remove() with MediaSource.removeSourceBuffer(). |
02 July 2012 | Converted to the object-oriented API |
26 June 2012 | Converted to Editor's draft. |
0.5 | Minor updates before proposing to W3C HTML-WG. |
0.4 | Major revision. Adding source IDs, defining buffer model, and clarifying byte stream formats. |
0.3 | Minor text updates. |
0.2 | Updates to reflect initial WebKit implementation. |
0.1 | Initial Proposal |