The core primitives of the Streams API is now being developped at WHATWG GitHub repository. Please join us to finalize the core primitives. Once it's done, it's planned to be ported to here and extended to meet requirements specific to browser environment.

This document is not complete. It is subject to major changes and, while early experimentations are encouraged, it is therefore not intended for implementation.

To check recent changes and rationale for them, please visit Mercurial history. Check open bugs at Bugzilla using this link. If you wish to submit a bug, please use this link. All comments and bug reports are welcome.

This specification provides an API for representing a stream of data in web applications, as well as programmatically reading and writing it. This includes:

This API is designed to be used in conjunction with other APIs and elements on the web platform, notably:

Introduction

Web applications should have the ability to acquire, manipulate, and pass data in a wide variety of forms, including as a sequence of data made available over time. This specification defines the basic representation for streams of data, and programmatic ways to read and write streams of data and errors raised on those operations.

The WritableStream interface defines a general protocol for data consuming APIs to communicate with data producing code. In these cases, the data consuming API, such as a decoder, provides a WritableStream for other applications to write to, enabling the decoder to begin decoding data as it becomes available. The data is written to the internal data sink inside the data consuming API using:

Actual transfer of data to the data sink may happen either synchronously or asynchronously. WritableStream hides the details of actual communication with the data sink while allowing for an efficient transfer of data.

The ReadableStream interface defines a general protocol for data producing APIs to communicate with data consuming code. This interface represents the potential for an infinite amount of data which are obtained over time and read once. Data consuming code reads data from the internal data source inside the data producing API using:

Actual transfer of data from the data source may happen either synchronously or asynchronously. The ReadableStream hides the details of actual communication with the data source while allowing for an efficient transfer of data.

With the combination of the following features, this interface suite responds to various simple and complex needs of data stream handling.

The ByteStream is a simple implementation of both the WritableStream and ReadableStream interface. A ByteStream has a single queue of bytes which works as a data sink for the WritableStream interface and as a data source for the ReadableStream interface.

Read and write operations on these interfaces are implemented using Promise. When an operation completes, the Promise returned by a method will be fulfilled, and then the fulfill callback set to the Promise will handle the result. Error conditions that may arise during an operation will be handled by the reject callback set to the Promise.

The base stream classes are defined as JavaScript primitives. Extensions to it for use in browsers are defined separately.

Examples below will be illustrative.

The example below demonstrates how to read one object from a ReadableStream and process it.

readableStream.read().then(
  function (result) {
    processData(result.data);
  },
  function (error) {
    processError(error);
  });
		

The example below demonstrates how to read binary data from a ReadableStream representing binary data and process it.

// Determine the size of the next binary chunk.
var nextChunkSize = parse(header);
readableStream.readBytesAs = "arraybuffer";
readableStream.readBytes(nextChunkSize).then(
  function (result) {
    processResult(result.data);
  },
  function (error) {
    handleError(error);
  });
		

The example below demonstrates how to read text data from a ReadableStream representing binary data and process it.

// Determine the size of the chunk containing text data.
var textChunkSize = parse(header);
readableStream.readBytesAs = "text";
readableStream.readEncoding = "UTF-8";
readableStream.readBytes(textChunkSize).then(
  function (result) {
    processData(result.data);
  },
  function (error) {
    handleError(error);
  });
		

The example below demonstrates how to roughly adjust the amount of data to fetch from the underlying data source and buffer in a ReadableStream.

// Tell stream that we're ready to consume 1024 bytes.
stream.pullAmount = 1024;
		

The example below demonstrates how to read bytes from a ReadableStream until an EOF is encountered.

// Read data from the ReadableStream repeatedly
function readUntilEof() {
  readableStream.read().then(
    function (result) {
      processData(result.data);

      if (!result.eof) {
        readUntilEof();
      }
    },
    function (error) {
      handleError(error);
    });
}

readUntilEof();
		

The example below demonstrates how to obtain a ReadableStream from XMLHttpRequest to begin playing a large video in readystate LOADING. The example takes the ReadableStream from a producer, XMLHttpRequest, and gives it to a consumer, the video tag.

function handler() {
  if(this.readyState == this.LOADING) {
    var readableStream = this.response;
    var readableStreamURL = URL.createObjectURL(readableStream);
    document.getElementById("myVideoTag").src = readableStreamURL;
  }
}

var client = new XMLHttpRequest();
client.onreadystatechange = handler;
client.setRequestHeader('customHeader', 'value');
client.setRequestHeader('customHeader2', 'value2');
client.open("GET", "myvideo.h264");
client.responseType = "stream";
client.send();

If the consumer implements the WritableStream interface, we can use pipe() to transfer data to it directly.

function handler() {
  if(this.readyState == this.LOADING) {
    var readableStream = this.response;
    readableStream.pipe(destinationStream).then(handleSuccess, handleFailure);
  }
}

var client = new XMLHttpRequest();
client.onreadystatechange = handler;
client.open("GET", "/video");
client.responseType = "stream";
client.send();
		

The example below demonstrates how to use write() to load a ReadableStream into the audio tag, whose data could be processed and built dynamically at read time. ByteStream in the example is a class which implements both WritableStream and ReadableStream. It accepts bytes generated by the code via WritableStream methods, and the written data will be consumed by the audio element via the object URL which is a part of ReadableStream's functionality.

var stream = new ByteStream("audio/mp3");

function generateAndWriteData() {
  // Do work to create more data to place into the stream.
  var data = generateMusic();

  // If we have no more data to process and place in the stream, we close the stream.
  if (data == null){
    stream.writeClose();
  } else{
    // Wait until write() completes.
    stream.write(data).then(
      function () {
        generateAndWriteData();
      },
      function (error) {
        handleError(error);
      });
  }
}

var streamURL = URL.createObjectURL(stream);
document.getElementById('audioTag').src = streamURL;

generateAndWriteData();
		

A producer can also do work only when pulled by using awaitSpaceAvailable(). This method informs the data producing code of that the data consuming API is ready to consume data. This is useful when high-performance is necessary.

function poll() {
  stream.awaitSpaceAvailable().then(
    function (pulledAmount) {
      stream.write(generateRandomBytes(pulledAmount));
      poll();
    }
  );
}

poll();
		

The example below demonstrates how the timing to pull new data is determined. When data of cost X is loaded and passed to the Promise, the ReadableStream marks X bytes of pullAmount as used. The next read() call automatically clears the mask to request the data source to produce X more data. If precise flow control is needed, you update pullAmount before the next read(). In the example, the producer API is requested to produce 16 bytes, and then in the fulfill callback, pullAmount is set to the remaining number of bytes not to let the API produce more data.

stream.pullAmount = 16;
function read() {
  stream.read().then(
    function (result) {
      process(result);
      stream.pullAmount -= result.size;
      if (stream.pullAmount == 0) {
        // Done
      } else if (stream.eof) {
        // Done
      } else {
        read();
      }
    }
  );
}
		

Terminology

In this section, when it is required to run steps with task queuing if necessary, run the following additional steps before the steps listed in the algorithm:

  1. If in the event loop
    Run the rest of the algorithm
    Otherwise
    Queue a task which runs the rest of the algorithm

WritableStream

The WritableStream interface defines a protocol for APIs which consume a data stream. The protocol includes:

  1. How to receive data stream
  2. How to notify the writer (data producing code) of completion of writing
  3. How to notify the writer of how much data can be accepted currently
By returning a Promise and delaying fulfillment of it, the WritableStream realizes flow control.

The actual data consumer behind the WritableStream is called a data sink and is identified by dataSink. A data sink consumes stream of data and notifies the WritableStream of the cost of data the data sink can newly accept. For each data sink, it must be defined how to calculate cost of each object which the data sink can consume.

A WritableStream has a one-to-one mapping with the associated data sink, and has defined semantics for interacting with the data sink internally.

WritableStream Interface

// Generic properties
Promise<unsigned long long> write()

This method writes the specified data to dataSink.

write() returns a Promise to make it easy for byte stream producing code to react to backpressure.

This method must run the steps below:

  1. If sinkGone is set, return a Promise rejected with sinkGoneReason
  2. Let latchedEncoding be the current value of writeEncoding
  3. If costOverride argument is specified, let amountToWrite be costOverride argument. Otherwise, let amountToWrite be the cost of data
  4. Let writePromise be a newly-created Promise
  5. Let pendingWrite be a newly-created PendingWriteDescriptor
  6. Set pendingWrite.promise to writePromise
  7. Push pendingWrite to pendingWriteQueue
  8. Set spaceAvailable to spaceAvailable - amountToWrite
  9. Run the steps below called write operation possibly asynchronously:
    If data is a DOMString
    Write data to dataSink together with latchedEncoding as encoding parameter. Interpretation of latchedEncoding is up to dataSink.
    Otherwise
    Write data to dataSink.
  10. Return writePromise

WritableStream interface doesn't guarantee that data argument is cloned. Modifications made on data argument after write() method call may affect the data written to dataSink.

any data
Data to write.
optional [Clamp] unsigned long long costOverride
Overrides cost of data.
Promise<unsigned long long> awaitSpaceAvailable()

The returned Promise will be fulfilled with the data cost which dataSink can accept when dataSink becomes able to accept data with any non-zero amount of cost.

This method must run the steps below:

  1. If sinkGone is set, return a Promise rejected with sinkGoneReason
  2. If awaitSpacePromise is not null, return awaitSpacePromise and terminate these steps
  3. Let newPromise be a newly-created Promise
  4. Set awaitSpacePromise to newPromise
  5. Check space
  6. Return newPromise

Promise<undefined> writeClose()

This method tells the WritableStream that no more data will be written to it.

Once this method has been called on a WritableStream, no further method calls can be made on the WritableStream.

This method must run the steps below:

  1. Set writeClosePromise be a newly-created Promise
  2. Run the steps below called close operation possibly asynchronously:
    1. Send the write EOF signal to dataSink
  3. Return writeClosePromise

Promise<undefined> writeAbort(optional any reason)

This method tells the WritableStream that no more data will be written to it with indication of error. The details of the error will be given by reason argument. The interpretation of reason is up to dataSink.

Once this method has been called on a WritableStream, no further method calls can be made on the WritableStream.

This method must run the steps below:

  1. Set writeAbortPromise to a newly-created Promise
  2. Run the steps below possibly asynchronously:
    1. Send the write abort signal to dataSink with reason
  3. Return writeAbortPromise

// Binary data specific properties
attribute DOMString writeEncoding

A DOMString that represents the label of an encoding [[!EncodingDetermination]] to be passed to dataSink together with data passed to write() method. It will be used by dataSink for encoding determination. This attribute returns the label last set, or the empty DOMString if never set.

This parameter is not designed to be specified as an argument of write() since it's not likely to be changed frequently.

Data Sink Model

The data sink is the underlying mechanism which a WritableStream interacts with and stores its data in.

A data sink to which the WritableStream interface writes bytes can be anything which:

Interacting with the Data Sink

This section defines the internal mechanisms for how the WritableStream interacts with dataSink. This section specifies both the internal properties as well as the algorithms.

A WritableStream has an associated integer variable spaceAvailable which holds the cost of data that dataSink requested but not yet written to it. This variable is initialized to 0 on construction.

A WritableStream has an associated Promise awaitSpacePromise which is used by the awaitSpaceAvailable() method. This variable is initialized to null on construction.

To abort awaitSpace, run the steps below:

  1. If awaitSpacePromise is null, terminate these steps
  2. Let detachedWaitPromise be awaitSpacePromise
  3. Set awaitSpacePromise to null
  4. Reject detachedWaitPromise with an "AbortError"

A struct type PendingWriteDescriptor has the following members:

A WritableStream has an associated queue pendingWriteQueue which holds PendingWriteDescriptors. This queue is initialized to an empty queue on construction.

To check space, run the steps below:

  1. If pendingWriteQueue is empty and spaceAvailable is not 0 and awaitSpacePromise is not null, run the steps below:
    1. Let detachedPromise be awaitSpacePromise
    2. Set awaitSpacePromise to null
    3. Fulfill detachedPromise with spaceAvailable

An associated Promise writeClosePromise. This variable is initialized to null on construction.

An associated Promise writeAbortPromise. This variable is initialized to null on construction.

An associated boolean sinkGone. This variable is initialized to false on construction.

An associated object sinkGoneReason. This variable is initialized to null on construction.

When dataSink notified the WritableStream of acknowledgement of data writing with writtenAmount which is the cost of the written data, run the steps below with task queuing if necessary:

  1. Pop the head element from pendingWriteQueue, and let pendingWrite be the element
  2. If pendingWrite.promise is not null, fulfill it with writtenAmount

When dataSink notifies of that data sink is gone with an object reason which represents the reason, run the steps below with task queuing if necessary:

  1. Set sinkGone
  2. Set sinkGoneReason to reason
  3. Abort awaitSpace

When dataSink notifies of failure of data writing with an object reason which represents the detail of the failure, run the steps below with task queuing if necessary:

  1. Pop the head element from pendingWriteQueue, and let pendingWrite be the element
  2. If pendingWrite.promise is not null, reject it with reason

When dataSink notifies of acknowledgement of the write EOF signal (called "done" notification), fulfill writeClosePromise with undefined

When dataSink notifies of failure of the write EOF signal with reason (called "error" notification), reject writeClosePromise with reason

When dataSink notifies of acknowledgement of the write abort signal, fulfill writeAbortPromise with undefined

When dataSink notifies of failure of the write abort signal with reason, reject writeAbortPromise with reason

When dataSink requests amountNewlyRequested more data (called "done and want more" notification), run the steps below with task queuing if necessary:

  1. Set spaceAvailable to spaceAvailable + amountNewlyRequested
  2. Check space

ReadableStream

The ReadableStream interface defines a protocol for APIs which produce a data stream. This protocol includes:

  1. How to receive read requests from the reader (data consuming code) and pass output data to the reader
  2. How to transfer data in bulk to another WritableStream (by using the pipe() method)
  3. How to mirror data to multiple destinations (by using the fork())
By returning a Promise and delaying fulfillment of it, the ReadableStream realizes asynchronous data consumption.

Interfaces introduced in this section provide simple and convenient ways to consume data. They connect arbitrary data stream producer and data consuming code using Promises and method invocations.

A ReadableStream has a one-to-one mapping with an associated data source, and has defined semantics for interacting with the data source internally. A data source, however, can have a one-to-many relationship with ReadableStreams, in cases where fork() is used.

ReadableStream Interface

// Generic properties
attribute unsigned long long pullAmount

This attribute provides the ReadableStream with a hint of how much data the reader can consume currently.

Each implementation of the ReadableStream must initialize the value of pullAmount on construction.

When this attribute is set, the user agent must request data production.

This flow control functionality is provided as a separated attribute rather than as an argument of the read() method based on the assumption that this value is not changed frequently.
It's possible that the cost of data received by the read() method is more than the pullAmount value.
Promise<StreamReadResult> read()

This method reads data from the ReadableStream. The speed of reading can be roughly adjusted by using pullAmount. pullAmount doesn't necessarily limit the size of data being read by this method.

This method must run the steps below:

  1. If pendingRead is not null, return a Promise rejected with an "InvalidStateError"
  2. Let readPromise be a newly-created Promise
  3. Set pendingRead to a newly-created PendingReadDescriptor
  4. Set pendingRead.promise to readPromise
  5. Set pendingRead.remaining to undefined
  6. Set pendingRead.destination to null
  7. Set pendingRead.bytesAs to undefined
  8. Set pendingRead.encoding to undefined
  9. Set amountBeingReturned to 0
  10. Request data production possibly asynchronously
  11. Return readPromise

How new data is automatically fetched

This method is useful if
  1. You don't care in what size and as how many fragments the data will be received
  2. You want to limit the number of bytes read for flow control
Promise<StreamReadResult> pipe()

This method bulk transfers data from the ReadableStream to another WritableStream.

Fulfillment of the returned Promise doesn't necessarily mean that the data transferred to destination has been successfully read from it.
TODO: Rewrite this to update amountBeingReturned as numBytesAcknowledged of PendingWriteDescriptors are updated

This method must run the steps below:

  1. If pendingRead is not null, return a Promise rejected with an "InvalidStateError"
  2. Set pipePromise be a newly-created Promise
  3. Set pendingRead to a newly-created PendingReadDescriptor
  4. Set pendingRead.promise to pipePromise
  5. Set pendingRead.remaining to undefined
  6. Set pendingRead.destination to destination argument
  7. Set pendingRead.bytesAs to undefined
  8. Set pendingRead.encoding to undefined
  9. Set totalAmountTransferred to 0
  10. Wait for destination possibly asynchronously
  11. Return pipePromise

WritableStream destination
Destination WritableStream.
ReadableStream fork()

This method creates a new ReadableStream which refers to the same dataSource as the current ReadableStream. Data sources are range reference counted so that a range in a data source is freed only when all the ReadableStreams sharing the data source finish consuming it.

This method must run the steps below:

  1. Let branch be a new ReadableStream which refers to dataSource and has the same amountRequested value, readDataBuffer contents and eofReached value as the current ReadableStream.
  2. If amountRequested is not 0, up to amountRequested bytes arriving in the future must be forwarded to branch.
  3. Return branch.

Promise<undefined> readAbort(optional any reason)

This method tells the ReadableStream that no more data will be read from it optionally with indication of error. The details of the error will be given by reason argument. The interpretation of reason is up to dataSource.

Once this method has been called on a ReadableStream, no further method calls can be made on the ReadableStream.

This method must run the steps below:

  1. Set abortPromise to a newly-created Promise
  2. Send the read abort with reason to dataSource possibly asynchronously
  3. Return abortPromise

// Binary data specific properties
attribute StreamReadType readBytesAs

Specifies what data type will be returned by readBytes() method calls. This attribute can be set to any of the supported types in StreamReadType.

The default value for this attribute is "as-is".

Values other than "as-is" works only when the data to be read from the ReadableStream represents binary data.

attribute DOMString readEncoding

Specifies a DOMString that represents the label of an encoding [[!EncodingDetermination]]. If set, it will be used as part of the encoding determination used when processing a readBytes() method call.

This attribute works only when the data to be read from the ReadableStream represents binary data.

This parameter is not designed to be specified as an argument of the reading methods since it's not likely to be changed frequently.
Promise<StreamReadResult> readBytes()

This method reads binary data from the ReadableStream. The returned Promise will be fulfilled when any of the following conditions is met:

  • The total size of produced bytes becomes equal to the size argument
  • The read EOF signal is received from dataSource
The cost of data returned by this method can be less than size when data cannot be fully converted into the type specified by readBytesAs or the read EOF signal is received.

This method must run the steps below:

  1. If pendingRead is not null, return a Promise rejected with an "InvalidStateError"
  2. Let readPromise be a newly-created Promise
  3. Set pendingRead to a newly-created PendingReadDescriptor
  4. Set pendingRead.promise to readPromise
  5. Set pendingRead.remaining to size argument
  6. Set pendingRead.destination to null
  7. Set pendingRead.bytesAs to the current value of readBytesAs
  8. Set pendingRead.encoding to the current value of readEncoding
  9. If size argument is not undefined, set readBytesPullAmount to size
  10. Set amountBeingReturned to 0
  11. Request data production possibly asynchronously
  12. Return readPromise

optional [Clamp] unsigned long long size
Upper limit of total cost of data to be read
This method is useful if
  1. You don't want to get notified of new data unless bytes of the specified number become available.
  2. You don't want to get your data fragmented into multiple parts, thus avoiding the need to buffer and combine chunks yourself.
Promise<StreamReadResult> pipeBytes()

This method bulk transfers bytes from the ReadableStream to another WritableStream.

TODO: Rewrite this to update amountBeingReturned as numBytesAcknowledged of PendingWriteDescriptors are updated

This method must run the steps below:

  1. If pendingRead is not null, return a Promise rejected with an "InvalidStateError"
  2. Set pipePromise be a newly-created Promise
  3. Set pendingRead to a newly-created PendingReadDescriptor
  4. Set pendingRead.promise to pipePromise
  5. Set pendingRead.remaining to the size argument
  6. Set pendingRead.destination to destination argument
  7. Set pendingRead.bytesAs to undefined
  8. Set pendingRead.encoding to undefined
  9. Set totalAmountTransferred to 0
  10. Wait for destination possibly asynchronously
  11. Return pipePromise

WritableStream destination
Destination WritableStream.
optional [Clamp] unsigned long long size
Number of bytes to transfer.

Data Source Model

A data source produces data stream to be consumed via the ReadableStream interface instances. A ReadableStream retrieves data from an associated data source. The data source model is not directly surfaced in the API, and is described here to provide details on how internal operations such as fork() can be handled.

A data source can be anything that:

To allow multiple consumers, we use a wrapper data source wrapper for range reference counting on produced bytes. A data source wrapper has the following functionalities:

When fork() is called, a new reader is registered to the original ReadableStream's data source and the new ReadableStream uses read() method to fetch data from it.

Interacting with the Data Source

This interface is designed to work even if operations on dataSource are asynchronous. If dataSource is synchronously accessible, some of the algorithms can be simplified.

A ReadableStream has an associated data source referred by dataSource from which the ReadableStream retrieves data. The data source model is explained in this section. A data source can be shared by multiple ReadableStream instances when fork() is used. A ReadableStream is registered with a data source on construction and given a reader ID. The dataSource of a ReadableStream refers to its data source, and readerId of a ReadableStream refers to its reader ID.

A ReadableStream has the following internal mechanisms:

An associated integer variable readBytesPullAmount which temporarily overrides pullAmount if necessary for a readBytes() method call. This variable is initialized to 0 on construction.

An associated integer variable pipePullAmount which temporarily overrides pullAmount if necessary for a pipe() method call. This variable is initialized to 0 on construction.

An associated integer variable amountRequested which holds the cost of data being retrieved from the dataSource. This variable is initialized to 0 on construction.

An associated integer variable amountBeingReturned which holds the cost of data consumed on the last read operation. This variable is initialized to 0 on construction.

amountBeingReturned delays replenishment of pullAmount until the next read()/pipe() operation is made.

An associated queue readDataBuffer which holds data retrieved from dataSource. This queue is initialized to an empty queue on construction.

Read data buffer related variables

An associated binary data queue splicedBinaryBuffer which holds binary data spliced from readDataBuffer. THis queue is initialized to an empty queue on construction.

A struct type PendingReadDescriptor has the following members:

A WritableStream has an associated PendingReadDescriptor pendingRead. This variable is initialized to null on construction.

An associated Promise abortPromise. This variable is initialized to null on construction.

An associated flag eofReached which indicates that the read EOF is received from the dataSource.

An associated object sourceErrorDetail which is initialized to null. Once the read EOF including indication of an error is received from the dataSource, this variable is set to the error detail object.

An associated integer variable totalAmountTransferred which holds the total cost of data transferred for the current pipe() method call. This variable is initialized to 0 on construction.

To request data production, run the steps below:

  1. Let readableAmount be sum of the total cost in readDataBuffer and the total number of bytes in splicedBinaryBuffer
  2. Let amountToNewlyRequest be max(max(pullAmount, readBytesPullAmount, pipePullAmount) - (amountRequested + readableAmount + amountBeingReturned), 0)
  3. Set amountRequested to amountRequested + amountToNewlyRequest
  4. Send a request to the dataSource to produce amountToNewlyRequest more data. This step is called pull operation.

To splice binary data, repeat the steps below while readDataBuffer is not empty:

  1. If pendingRead.size is 0, return from this algorithm
  2. Let head be the first element in readDataBuffer
  3. If head is not an object representing binary data
    1. Let readPromise be pendingRead.promise
    2. Set pendingRead to null
    3. Reject readPromise with an "InvalidStateError"
    4. Return from this algorithm with failure
    If pendingRead.size is undefined
    1. Pop head from readDataBuffer
    2. Push head to splicedBinaryBuffer
    Implementations may choose to pop only part of bytes readable if it helps reducing memory copy.
    If the number of bytes in head is not greater than pendingRead.size
    1. Pop head from readDataBuffer
    2. Push head to splicedBinaryBuffer
    3. Set pendingRead.size to pendingRead.size - (the number of bytes in head)
    Otherwise
    1. Split head into two elements first and last where the number of bytes in first is pendingRead.size
    2. Replace head in readDataBuffer with last
    3. Push first to splicedBinaryBuffer
    4. Set pendingRead.size to 0

To output data, run the steps below:

  1. Let result be a newly-created StreamReadResult
  2. If pendingRead.bytesAs is undefined
    1. Pop one element from readDataBuffer, and set result.data to the element
    2. Let amountConsumed be cost of the element
    Otherwise
    1. Splice binary data
    2. If the last step failed, terminate these steps
    3. If pendingRead.size is not undefined and eofReached is not set and the number of bytes in splicedBinaryBuffer is less than pendingRead.size, terminate these steps
    4. Switch on pendingRead.bytesAs:
      "arraybuffer"
      1. Set result.data to an object of the type specified by pendingRead.bytesAs that represents binary data stored in splicedBinaryBuffer
      2. Let amountConsumed be the number of bytes in splicedBinaryBuffer
      "text"
      1. Set result.data to a DOMString that is the result of decoding as many bytes in splicedBinaryBuffer as possible using pendingRead.encoding
      2. Let unusedBytes be the bytes in splicedBinaryBuffer which were not used in the last step, and let amountConsumed be the number of bytes in splicedBinaryBuffer used in the last step
      3. If eofReached is set and unusedBytes is not empty, run the steps below:
        1. Let readPromise be pendingRead.promise
        2. Set pendingRead to null
        3. Reject readPromise with an "EncodingError"
      4. Prepend unusedBytes to readDataBuffer
      "none"
      1. Set result.data to undefined
      2. Let amountConsumed be the number of bytes in splicedBinaryBuffer
    5. Clear splicedBinaryBuffer
  3. Set result.size to amountConsumed
  4. Set result.eof to true if readDataBuffer is empty and eofReached is set
  5. Set result.error to sourceErrorDetail
  6. Set amountBeingReturned to amountConsumed
  7. Set readBytesPullAmount to 0
  8. Let readPromise to pendingRead.promise
  9. Set pendingRead to null
  10. Fulfill readPromise with result

To wait for destination, run the steps below:

  1. Let onFulfilled be a callback which runs the steps below:
    1. Let space be the first argument
    2. If pendingRead.size is specified
      Set pipePullAmount to min(pendingRead.size, space)
      If pullAmount is set to 0, pipe() transfers data in pull style. I.e. only data of the same cost as one writable to destination are retrieved from the data source.
      Setting pullAmount to a small value doesn't have an effect of throttling the pace of pipe().
      Otherwise
      Set pipePullAmount to space
    3. Set amountBeingReturned to 0
    4. Request data production
  2. Let onRejected be a callback which runs the steps below:
    1. Set pipePullAmount to 0
    2. Let pipePromise be pendingRead.promise
    3. Set pendingRead to null
    4. Reject pipePromise with the first argument
  3. Run the algorithm of awaitSpaceAvailable() method on destination, and let promise be the returned Promise
  4. Transform promise of destination with onFulfilled and onRejected

To transfer data, run the steps below:

  1. Let onFulfilled be a callback which runs the steps below:
    1. Let writtenAmounts be the first argument
    2. Set totalAmountTransferred to totalAmountTransferred + (the sum of elements in writtenAmounts)
    3. If eofReached is set or pendingRead.size is 0
      1. Set pipePullAmount to 0
      2. Let pipePromise be pendingRead.promise
      3. Set pendingRead to null
      4. Let result be a newly-created StreamReadResult
      5. Set result.data to undefined
      6. Set result.eof to true if eofReached
      7. Set result.error to sourceErrorDetail
      8. Set result.amountConsumed to totalAmountTransferred
      9. Set totalAmountTransferred to 0
      10. Fulfill pipePromise with result
        At this point, amountBeingReturned is not yet reset
      Need to revisit. Finish when write to the destination completes?
      Otherwise
      Wait for destination
  2. Let onRejected be a callback which runs the steps below:
    1. Set pipePullAmount to 0
    2. Let pipePromise be pendingRead.promise
    3. Set pendingRead to null
    4. Reject pipePromise with the first argument
  3. If pendingRead.size is undefined
    1. Let writePromises be an array of Promises.
    2. Repeat the steps below while readDataBuffer is not empty:
      1. Pop one element from readDataBuffer, and let head be the element
      2. Set amountBeingReturned to amountBeingReturned + (cost of head)
      3. Run the algorithm of write() method with head on pendingRead.destination
      4. Push the Promise returned by the last step to writePromises
    3. Let allPromise be waiting for all of writePromises
    4. Transform allPromise with onFulfilled and undefined
    Otherwise
    1. Splice binary data
    2. If the last step failed, terminate these steps
    3. Repeat the steps below while splicedBinaryBuffer is not empty:
      1. Pop one element from splicedBinaryBuffer, and let head be the element
      2. Run the algorithm of write() method with head to pendingRead.destination

To process received data and signal, run the steps below:

  1. If pendingRead is null, terminate these steps
  2. If pendingRead.destination is null
    Output data
    Otherwise
    Transfer data

A dataSource delivers produced data to the associated ReadableStream by running the steps below called push operation with task queuing if necessary:

  1. Let receivedData be the produced data
  2. Let receivedAmount be the size of receivedData
  3. Set amountRequested to amountRequested - receivedAmount
  4. Push receivedData to readDataBuffer
  5. Process received data and signal

When pullAmount is increased, new data will be fetched

A dataSource sends the read EOF signal to the associated ReadableStream by running the steps below called close operation with task queuing if necessary:

  1. Set eofReached and sourceErrorDetail
  2. Process received data and signal

When the read abort signal is acknowledged by dataSource, fulfill abortPromise with undefined

StreamReadResult Interface

This interface represents the result of methods on ReadableStream.

readonly attribute boolean eof
Set if the operation resulted in the read EOF
readonly attribute any data
The contents read (and converted into an object of the specified type if needed)
readonly attribute unsigned long long amountConsumed
The cost of the data read
readonly attribute any error
Once any error occurred, set to the error detail object on all StreamReadResults returned after the error

StreamReadType enum

Data can be read as various data types from ReadableStream. This enum defines DOMString values to specify the data types.

arraybuffer
readBytes() attempts to convert produced data into an ArrayBuffer, and then returns it
text
readBytes() attempts to convert produced data into a DOMString, and then returns it
none
readBytes() returns nothing. Useful for seeking data by skipping some amount of data. User agents may implement some optimization for "none" type read for example omitting internal data transfer.

ByteStream

This section introduces a simple implementation of the WritableStream and ReadableStream named ByteStream, as well as accompanying interfaces required as part of the ByteStream implementation. This includes a constructor to build a ByteStream object and the type attribute.

ByteStream Interface

This interface represents a sequence of bytes which can be read only once over time and to which we can push data.

A ByteStream has an associated bufferedDataQueue. The buffer can be stored in memory or backed by slower devices such as a disk.

The ByteStream inherits the WritableStream interface. Its dataSink is bufferedDataQueue wrapped with a data source wrapper.

The ByteStream inherits the ReadableStream interface. Its dataSource is also bufferedDataQueue.

bufferedDataQueue just forwards retrieval requests coming from the data source wrapper to the WritableStream as a notification of the number of newly acceptable bytes with the same amount.

Constructor()
Constructs a ByteStream and sets the type to the specified value.
in unsigned long long pullAmount
Specifies the initial value of pullAmount.
in optional DOMString type
Specifies the MIME type [[!RFC2046]] of the ByteStream.

URIs for Stream

To reference a ByteStream, the same URI used for Blobs and Files in 6.7. A URI for Blob and File reference of the File API specification should be used. [[!FILE-API]] The definitions of Origin, Lifetime, Referencing, and Dereferencing of a Blob should be applied to a ByteStream.

Creating and Revoking a Stream URI

A Stream URI is a Blob URI that is referencing a ReadableStream. These URIs are created and revoked using methods exposed on the URL object, as defined in 6.7.5. Creating and Revoking a Blob URI of the File API specification. [[!FILE-API]]

URL.createObjectURL and URL.revokeObjectURL should both be extended as follows:

static DOMString? createObjectURL()
ReadableStream stream
The ReadableStream for which a Blob URI will be created
DOMString type

The media type of this stream. The media type is returned as an ASCII-encoded string in lower case representing the media type, expressed as an RFC2046 MIME type [[!RFC2046]]. A string is a valid media type if it matches the media-type token defined in section 3.7 "Media Types" of RFC 2616 [[!HTTP11]].

The extension onto createObjectURL should have the following steps added.

Returns a unique Blob URL each time it is called on a ReadableStream.

This method must run the steps below:

If stream is non-null and in scope of the global object's URL property from which this static method is called
Run the steps below:
  1. If pendingRead of stream is not null, return null
  2. Set pendingRead of stream to a newly-created PendingReadDescriptor
  3. Return a unique Blob URI that can be used to dereference stream
  4. Add an entry to the Blob URL Store for this Blob URL
Otherwise
Return null

static DOMString? createFor()
ReadableStream stream
DOMString type

The extension onto createFor should have the following steps added.

Returns a unique Blob URL each time it is called on a ReadableStream. Blob URLs created with this method are said to be auto-revoking since user-agents are responsible for the revocation of Blob URLs created with this method, subject to the lifetime stipulation for Blob URLs.

This method must run the steps below:

If stream is non-null and in scope of the global object's URL property from which this static method is called
Run the steps below:
  1. If pendingRead of stream is not null, return null
  2. Set pendingRead of stream to a newly-created PendingReadDescriptor
  3. Return a unique Blob URI that can be used to dereference stream
  4. Add an entry to the Blob URL Store for this Blob URL
  5. Add an entry to the Revocation List for this Blob URL
Otherwise
Return null

static void revokeObjectURL(in DOMString url)

The extension onto revokeObjectURL should have the following steps added.

If url is a Blob URI which refers to a ReadableStream and it's in the same origin of the global object’s URL property on which this static method was calld
Return a 404 response code when the URL is dereferenced
Otherwise
Do nothing. A message on their error console may be displayed.

Stream Consumers and Producers

Byte streams can be both produced and consumed by various APIs. APIs which create streams are identified as producers, and ones which read and act on a byte stream are known as consumers. This section identifies some of the APIs where Streams may be produced and consumed.

The list of producers and consumers below is not an exhaustive list. It is placed here as informative for the time being.

Stream Consumers

This section outlines APIs which can consume a byte stream

Stream Producers

This section outlines APIs which can produce a byte stream

Security Considerations

A ReadableStream should have the same security considerations as a Blob. This is outlined in 6.8. Security Considerations of the File API specification. [[!FILE-API]] Because a ReadableStream uses a Blob URI, cross origin requests on a ReadableStream will not be supported.

Extension of XMLHttpRequest

This specification proposes an extension to XMLHttpRequest [[!XMLHTTPREQUEST2]] to add support for ByteStream. This section is temporary and is meant to provide a recommendation for how ByteStream should be incorporated into XMLHttpRequest. This will extend XMLHttpRequest to allow for receiving and uploading of a ByteStream. One such scenario is providing access to data during readyState 3 (LOADING). The sections below document in detail what extensions must be done to XMLHttpRequest to support ByteStream.

Addition of stream response entity body

The section named "Response Entity Body" in XMLHttpRequest specification [[!XMLHTTPREQUEST2]] should have the following additions:

The stream response entity body is either a ByteStream representing the response entity body or null. If the stream response entity body is null, let it be the return value of the following algorithm:

  1. If the response entity body is null, return an empty ByteStream object.
  2. Return a ByteStream object representing the response entity body.

Addition of "stream" responseType

A new value for the responseType attribute "stream" should be introduced.

In the IDL list in the section named "Interface XMLHttpRequest" in XMLHttpRequest specification [[!XMLHTTPREQUEST2]], the definition of XMLHttpRequestResponseType enum should now read:

enum XMLHttpRequestResponseType {
  "",
  "arraybuffer",
  "blob",
  "stream",
  "document",
  "json",
  "text"
}

Modification on the response attribute

The algorithm of the response attribute should be modified to handle the new responseType value "stream". A Stream is binary data obtained sequentially over time. Given this, a ByteStream should be accessible in readyState 3 (LOADING).

The section named "The response attribute" in XMLHttpRequest specification [[!XMLHTTPREQUEST2]] should now read:

The response attribute must return the result of running these steps:

If responseType is the empty string or "text"
The same as the original XMLHttpRequest specification.
If responseType is "stream"
  1. If the state is not LOADING or DONE, return null.
  2. If the error flag is set, return null.
  3. Return the stream response entity body.
Otherwise
The same as the original XMLHttpRequest specification.

send()

The switch in otherwise case of step 4 of The section named "The send() method" in XMLHttpRequest specification [[!XMLHTTPREQUEST2]] should have the following additions:

ReadableStream

If the object's type attribute is not the empty string let mime type be its value.

Set pendingRead of the ReadableStream to a newly-created PendingReadDescriptor.

Let the request entity body be the raw data represented by data.

Terminology

ArrayBuffer and ArrayBufferView are defined in Typed Array specification.

Blob is defined in File API specification.

Promise is defined in Promise Objects specification. Shorthand phrases to refer operations on Promises are borrowed from Writing Promise-Using Specifications.

DOMString is defined in DOM Level 3 Core specification.

XMLHttpRequest is defined in [[!XMLHTTPREQUEST2]].

AbortError is defined in DOM4 specification. InvalidStateError is defined in DOM4 specification. SyntaxError is defined in DOM4 specification. EncodingError is defined in DOM4 specification.

Requirements and Use Cases

The ByteStream type allows for completion of several end-to-end experiences. This section covers what the requirements are for this API, and illustrates some use cases.

Acknowledgements

Thanks to Eliot Graff for editorial assistance. Special thanks to the W3C. The editor would like to thank Adrian Bateman, Anne van Kesteren, Austin William Wright, Aymeric Vitte, Domenic Denicola, Elliott Sprehn, Francois-Xavier Kowalski, Harris Syed, Isaac Schlueter, Jonas Sicking, Kenneth Russell, Kinuko Yasuda, Lindsay Verola, Michael Davidson, Rob Manson, Taiju Tsuiki, Yusuke Suzuki, Yutaka Hirano, for their contributions to this specification.