integrating some of the suggestions by Werther Azevedo for the game scenario - generative and multi-track music requirements
authorOlivier Thereaux <Olivier.Thereaux@bbc.co.uk>
Thu, 06 Sep 2012 16:25:40 +0100
changeset 158c5ecdca5be3f
parent 157 bf0e920450e6
child 159 c51f23cdcd61
integrating some of the suggestions by Werther Azevedo for the game scenario - generative and multi-track music requirements
reqs/Overview.html
     1.1 --- a/reqs/Overview.html	Wed Sep 05 15:13:25 2012 +0300
     1.2 +++ b/reqs/Overview.html	Thu Sep 06 16:25:40 2012 +0100
     1.3 @@ -147,15 +147,17 @@
     1.4        <section>
     1.5        <h3>3D game with music and convincing sound effects</h3>
     1.6        <p>A commuter is playing a 3D first-person adventure game on their mobile device. The game is built entirely using open web technologies, and includes rich, convincing sound piped through the commuter's stereo headphones.</p>
     1.7 -      <p>As soon as the game starts, a musical background starts, loops seamlessly, and transitions smoothly from one music track to another as the player enters a house.</p>
     1.8 +      <p>As soon as the game starts, a musical background starts, loops seamlessly, and transitions smoothly from one music track to another as the player enters a house. Some of the music is generated live, and reacts to the state of the game: tempo, time signature, note properties and envelopes change depending on the the health level of the characters and their actions.</p>
     1.9        <p>While walking in a corridor, the player can hear the muffled sound of a ticking grandfather's clock. Following the direction of the sound and entering a large hall, the sound of the clock becomes clear, reverberating in the large hall. At any time, the sound of the clock spatialized in real-time based on the position of the player's character in the room (relative to the clock) and the current camera angle.</p>
    1.10 -      <p>As the soundscape changes, bringing a more somber, scary atmosphere to the scene, the player equips a firearm. Suddenly,  a giant snake springs from behind a corner, its hissing becoming a little louder as the snake turns its head towards the player. The weapon fires at the touch of a key, and the player can hear the sound of bullets in near-perfect synchronization with the firing, as well as the sound of bullets ricocheting against walls. The sounds are played immediately after the player presses the key, but the action and video frame rate can remain smooth even when a lot of sounds (bullets being fired, echoing and ricocheting, sound of the impacts, etc) are played at the same time. The snake is now dead, and many flies gather around it, and around the player's character, buzzing and zooming in the virtual space of the room.</p>
    1.11 +      <p>The soundscape changes, bringing a more somber, scary atmosphere to the scene: the once full orchestral underscore is slowly reduced, instrument by instrument, to a lonely and echoing cello. The player equips a firearm. Suddenly, a giant snake springs from behind a corner, its hissing becoming a little louder as the snake turns its head towards the player. The weapon fires at the touch of a key, and the player can hear the sound of bullets in near-perfect synchronization with the firing, as well as the sound of bullets ricocheting against walls. The sounds are played immediately after the player presses the key, but the action and video frame rate can remain smooth even when a lot of sounds (bullets being fired, echoing and ricocheting, sound of the impacts, etc) are played at the same time. The snake is now dead, and many flies gather around it, and around the player's character, buzzing and zooming in the virtual space of the room.</p>
    1.12  
    1.13        <h4>Notes and Implementation Considerations</h4>
    1.14        
    1.15        <ol>
    1.16          <li><p>Developing the soundscape for a game as the one described above can benefit from a <em>modular, node based approach</em> to audio processing. In our scenario, some of the processing needs to happen for a number of sources at the same time (e.g room effects) while others (e.g mixing and spatialization) need to happen on a per-source basis. A graph-based API makes it very easy to envision, construct and control the necessary processing architecture, in ways that would be possible with other kinds of APIs, but more difficult to implement. The fundamental <code>AudioNode</code> construct in the Web Audio API supports this approach.</p></li>
    1.17          <li><p>While a single looping music background can be created today with the <a href="http://www.w3.org/TR/html5/the-audio-element.html#the-audio-element" title="4.8.7 The audio element &#8212; HTML5">HTML5 &lt;audio&gt; element</a>, the ability to transition smoothly from one musical background to another requires additional capabilities that are found in the Web Audio API including <em>sample-accurate playback scheduling</em> and <em>automated cross-fading of multiple sources</em>. Related API features include <code>AudioBufferSourceNode.start()</code> and <code>AudioParam.setValueAtTime()</code>.</p></li>
    1.18 +        <li><p>The musical background of the game not only involves seamless looping and transitioning of full tracks, but also the automated creation of generative music from basic building blocks or algorithms (“Some of the music is generated live, and reacts to the state of the game”), as well as the creation and evolution of a musical score from multiple instrument tracks (“the once full orchestral underscore is slowly reduced, instrument by instrument”). Related requirements for such features are developed in details within the <a href="#online-music-production-tool">Online music production tool</a> and <a href="#music-creation-environment-with-sampled-instruments">Music Creation Environment with Sampled Instruments</a> scenarios.</p>
    1.19 +          
    1.20          <li><p>The scenario illustrates many aspects of the creation of a credible soundscape. The game character is evolving in a virtual three-dimensional environment and the soundscape is at all times spatialized: a <em>panning model</em> can be used to spatialize sound sources in the game (<code>AudioPanningNode</code>); <em>obstruction / occlusion</em> modeling is used to muffle the sound of the clock going through walls, and the sound of flies buzzing around would need <em>Doppler Shift</em> simulation to sound believable (also supported by <code>AudioPanningNode</code>).  The listener's position is part of this 3D model as well (<code>AudioListener</code>).</p></li>
    1.21          <li><p>As the soundscape changes from small room to large hall, the game benefits from the <em>simulation of acoustic spaces</em>, possibly through the use of a <em>convolution engine</em> for high quality room effects as supported by <code>ConvolverNode</code> in the Web Audio API.</p></li> 
    1.22          <li><p>Many sounds in the scenario are triggered by events in the game, and would need to be played with low latency. The sound of the bullets as they are fired and ricochet against the walls, in particular, illustrate a requirement for <em>basic polyphony</em> and <em>high-performance playback and processing of many sounds</em>. These are supported by the general ability of the Web Audio API to include many sound-generating nodes with independent scheduling and high-throughput native algorithms.</p></li>