The NA3 Audio Server

Function: int move_sound(server, id,heading, velocity, time);CLIENT *server;sound_id id;position heading;float velocity;float time;Move the sound source identified by id on the machine indicated by server in the given heading at the given velocity for the given period of time. For spatial audio, the heading is a polar vector of the (azimuth, elevation) form. For panned audio, the heading is a slope of the form depth/width. The velocity is given in meters per second. Time is given in seconds. Doppler shift is also computed during the movement. If put_sound or move_sound are called during this movement, the request(s) will be enqueued until the movement completes.Function: int sync_movement(server, id,???)CLIENT *server;sound_id id;??????;Provide a synchronization callback when the movement queue for the sound source identified by id on the machine indicated by server has been emptied. This allows clients to synchronize with the motion of sound sources. Additional parameters will be needed but have not been determined.Function: int change_volume(server, id,volume);CLIENT *server;sound_id id;int volume;Changes the volume of the sound identified by id on the machine indicated by server to volume.Function: int change_bend(server, id,bend);CLIENT *server;sound_id id;int bend;Changes the bend of the sound identified by id on the machine indicated by server to bend.Function: int change_muffle(server, id,muffle);CLIENT *server;sound_id id;int muffle;Changes the muffling of the sound identified by id on the machine indicated by server to muffle.Function: int change_distort(server, id,distort);CLIENT *server;sound_id id;int distort;Changes the distortion of the sound identified by id on the machine indicated by server to distort.Function: int query_sounds(server, list);CLIENT *server;play_list list;Returns play information about the sounds currently playing and paused on the machine indicated by server. The play information is contained in list, which is a pointer to a structure:struct play_list_item {play_info info;play_list next;};where play_info is this structure:struct play_info {char *filename;sound_id id;int looped;float volume;float bend;float muffle;float bend;float distort;position direction;float distance;int paused;};Function: int query_environment(server,environ);CLIENT *server;space *environ;Returns play information about the listening environment on the machine indicated by server. The play information is contained in environ, which is a pointer to a space structure.CLIENT *server;char *path;float rate;float volume;float muffle;float distort;position direction;float distance;int looped;int full_spatial; Plays the audio file named in path on the machine indicated by server. If the filename in path is relative (i.e.,“machine_gun1.au” instead of “/home/packages/sounds/ new-sound-effects/machine_gun1.au”), the server will look in default directories (currently “/usr/lib/sounds” and “/usr/ local/lib/sounds”) for the files instead of the directory in which the server was run. The rate parameter determines playback speed; e.g, if rate is 1.5, the audio file will be played 1.5 times as fast as normal. The volume and muffle parameters control the relative volume and muffling effects for that sound.volume is given in decibels, where 0dB is normal volume level;muffle is in the range -1.0 to 1.0 where 0 does not muffle or thin, positive numbers muffle (low-pass filter), and negative numbers thin (high-pass fil-ter). The distortion parameter controls logarithmic com-pression and exponential expansion. This parameter can be any real number. A value of zero has no effect, positive val-ues compress, and negative values expand. The direction parameter is a union that can specify either an azimuth and angle (in degrees) for spatial audio or a panning value for simple stereo panning. For spatial audio, direction is relative to the origin of the audio environment’s coordinate system. For stereo panning, negative values denote positions to the left of center and positive values denote positions to the right. The distance value is given in meters. The looped parameter causes the sound to start over from the beginning when finished; when set to 1, the file is played looped, when set to 0 the file is played normally. The full_spatial flag is use to select between full HRTF-based spatialization and low-cost stereo panning. The return value is of type soun-d_id, and can be used to refer to this instance of the sound in other functions; if this number is negative, it indicates an error.Function: int define_environment(server,environ);CLIENT *server;space environ;This function defines the listening environment in which the played sounds are produced on the machine server. The environment is specified in a space data structure which is defined astypedef struct {float t60;float depth;float warmth;} space;The t60 parameter is reverberation time—the number of seconds needed for reverberation to fall to an inaudible level (-60dB) after the sound stops. The depth parameterspecifies the longest dimension of the listening space in meters. Listening spaces are assumed to be rectangular prisms of fixed proportions, chosen to give pleasant sounding reverberation. The warmth parameter can be any real number, with positive values giving a “warm, boomy”sort of sound and negative values giving a “tinny, metallic”sort of sound. When the server is first opened, the default environment will be define as anechoic (or “dry”), with t60 = 0 sec. (A typical concert hall has t60 = 2 sec. A typical office room has t60 = 0.5 sec.) The function returns NULL on success, or returns a negative-valued error on failure. This error will usually be due to out of range parameters or a shortage of system resources.Function: int stop_sound(server, id);CLIENT *server;sound_id id;Stops the sound identified by id on the machine indicated by server.Function: int pause_sound(server, id);CLIENT *server;sound_id id;Pauses the sound identified by id on the machine indicated by server.Function: int resume_sound(server, id);CLIENT *server;sound_id id;Unpauses the sound identified by id on the machine indicated by server.Function: int sync_audio(server, id,index, ???);CLIENT *server;sound_id id;float index;??????;Provide a synchronization callback when the offset index is reached in the sound file identified by id on the machine indicated by server. The index is given is seconds. If the index is greater than the duration of the file, synchronization is at the end of the file. A negative index will also force synchronization at the end of the file. If the given index has been passed in the playing of the file, synchronization is immediate and a special positive-valued error code is returned. Additional parameters may be needed.Function: int put_sound(server, id,direction, distance);CLIENT *server;sound_id id;position direction;float distance;Changes the location of the sound source in the listening space.APPENDIX - NA3 DESIGN DOCUMENTCurrent State of the Netaudio ServerVersion 2 of the netaudio server (NA2) has been complete since about 3/93. NA2 allows multiple clients to connect to the audio server and play sounds, with all clients having simultaneous access to the audio output hardware. Simulta-neous sounds are mixed for output. NA2 provides the user with the ability to start, pause, continue, stop, and query the status of sounds while they are playing. NA2 also provides a small set of processing effects which can be applied to indi-vidual sounds as they are played. These effects include vol-ume control, tone control, and pitch/speed shifting. Starting late in 1994, a series of changes was applied to NA2 to produce a new version of the server.Overview of the ModificationsThere were three major modifications to the netaudio server, resulting in a new server to be called netaudio version 3 (NA3).The first modification is a differentiation of the sound source and the listening environment. This differentiation involved changes in the signal processing path and resulted in some minor changes in the programmer’s interface to the netaudio server.The second modification was to divide the netaudio server into a number of user-level threads (provided by Solaris 2.x) to allow for the effective utilization of multiprocessor work-stations. The multithreading of netaudio did not result in any changes in the programmer’s interface to the server.The third modification is for netaudio to operate with an internal sample rate of 32kHz (instead of the current 8kHz) and use floating point for internal representation of samples (instead of the current uLaw/short format). The original 8kHz/uLaw format can be selected at compile-time.The Source/Environment ModelAn important aspect of the architecture of NA3 is the divi-sion of audio effects into three groups: properties of the source, effects of the relationship between the listener and the source, and effects of the environment. Effects of the source would include source volume, pitch shifting, muf-fling, thinning, and distortion (so-called “filtears” effects). The relationship between the source and listener determines effects such as spatialization, atmospheric dispersion, dis-tance-related attenuation, and Doppler shift. Dense reverber-ation is largely an environmental effect.Although this division of effort does not precisely model the physics of a listening environment, it is a close approxima-tion which renders a much functional parallelism and pro-vides a framework for describing listening conditions with manageable amounts of specification data.MultithreadingVersion 2 of the netaudio server uses a single process, a set of signals, and a select structure. While this architecture gives good performance on a single-processor machine, it is not suitable for utilizing multiprocessor machines.The new NA3 server uses five types of threads: a control thread, source threads, effects threads, environmental thread(s), and an output thread.The control thread initializes the server’s devices and data structures and receive and process RPC calls from clients.Each active audio source has a source thread which reads the audio from a file or device, correct sample rate and format (if needed) and apply the “source effects” outlined previously.Each active source also has an effect thread which applies audio effects due to the relationship between the source an the listener (listed previously).Each listening environment has a thread to compute environ-mental effects (namely dense reverberation). W e are cur-rently providing for only one listening environment.The output thread sends processed audio to /dev/audio for presentation to the listener.Sample Rate and Sample FormatA sample rate of 32kHz has been chosen for NA3 because this rate is sufficiently high to allow for true HRTF-based spatial audio but sufficiently low to allow the spatial audio to be generated in real time by a SPARCstation 10.Our benchmarks have shown that SPARCstations compute audio functions most quickly using double precision floating point values. However, double precision samples can also take up too much memory. As a general rule, audio process-ing functions inside NA3 will store samples in memory as float and perform calculations with double (the valid range for an audio sample will be -1…1). The use of floating point also eases concerns over accuracy and dynamic rangein the design of the signal processing functions.Function definitionsThe following functions provide the programmer’s interface to the netaudio server.AudioServer *open_audio_server(host);char *host;This function opens a connection to the audio server named in host. If host is NULL, it will look at the AUDI-OSERVER environment variable for a host name; if that is NULL, it will look in the DISPLAY environment variable to get a host name.Function: sound_id play_sound(server,path, rate, volume,muffle, distort,direction, distance,looped);pass filtering in the Doppler shift units, resulting in lower quality audio at a lower computational cost. While this method works well for general purpose computers, it is of limited usefulness in DSP arrays.Advanced Acoustic ModelingAdvanced modeling occurs at two levels. The high level algorithms analyze the listening environment and produce a set of signal processing parameters, such as filter coefficients and delay line lengths. These algorithms are in many ways similar to graphical renders. This rendering would corre-spond to the layer 2 software of NA3. The lower level algo-rithms perform signal processing to actually apply the parameters to the sound stream. This processing would cor-respond to the layer 3 software.The rendering algorithms will greatly increase the computa-tional cost of the layer 2 software If these algorithms are to be expected to run in real time on current generation proces-sors, layer 2 must be divided into multiple threads.The cost of the signal processing pipeline in layer 3 will also increase, and most of this increase will be in the second stage. To provide adequate performance on current or near-term processors, this second stage for each source must be implemented with multiple threads feeding their outputs into a single mixer.SUMMARYWhile NA3 has a good fundamental design, time limitations have prevented us from implementing many of the features we had planned. We sincerely hope that someone will pick continue this work in the future. Even in its present state, however, NA3 has the potential to deliver complex auditory environments for a variety of applications. At this time, a demonstration of the NA3 server is available via FTP (File Transfer Protocol) or WWW (W orld Wide Web) at file:// /pub/audiotools/NA3demo.tar.Z. Please read the README.html file in the same directory before copying the demo.ACKNOWLEDGEMENTSWe would like to thank NASA Marshall Space Flight Center for the initial funding of the Mercator project. W e would also like to thank Sun Microsystems Labs, Inc. for further fund-ing of spatial audio research. The author would also like to thank Beth Mynatt, Mark Lee, Jouke Verlinden, Phillip Seaver, and Tom Rodriguez for the parts they played in the development of the various NetAudio servers. REFERENCES1.Burgess, D.A. (1992) Techniques for low cost spatialaudio, Proceedings of the Fifth Annual Symposium onUser Interface Software and Technology (UIST ‘92),ACM, New York, 1992, pp. 53-59.2.Burgess, D.A., Verlinden, J.C., (1193) An Architecturefor Spatial Audio Servers, GVU Technical Report GIT-GVU-93-34.3.Mynatt, E.D., Edwards, W.K., (1992) Mapping GUIs toauditory interfaces, Proceedings of the Fifth AnnualSymposium on User Interface Software and Technology (UIST ‘92), ACM, New York, 1992, pp. 61-70.Automatic Source Format and Rate ConversionThe first stage of the signal processing path can be easily modified to accept either mu-Law or linear formats and con-vert them transparently to the floating-point format used internally by NA3. In the present implementation, the source format must be fixed at compile time. Transparent sample rate conversion can also be supported. Since the present implementation uses a 32kHz internal rate, 8kHz, 16kHz, 24kHz, and 48kHz can be converted cheaply and accurately in real time. However, accurate conversion of 22.05kHz and 44.1kHz may be considerably more expensive because 441 and 320 have no common multiples other than 1. POSSIBLE LONG-TERM ENHANCEMENTSThe architecture of NA3 has been planned to allow a variety of major modifications to support sophisticated auditory dis-plays. Some useful possible modifications are described here.Multiple EnvironmentsSuppose you are in a virtual space consisting of a lattice of rooms, as in the inside of a building. To model the acoustics of this space well, your audio renderer must support multiple simultaneous environments. NA3 can be extended in this manner by adding a new stage to the layer 3 signal process-ing pipeline between the existing stages 3 and 4. This new stage, the meta-environment stage, would mix the inputs from the various environments (or rooms) while applying muffling and attenuation based on the characteristics of thespace between the listener and each environment (or room). Figure 3. A Simple Field Pattern. This is the cartioid pattern associated with many general-purpose microphones. This pattern could be described as having a single lobe with a width of 120 degrees. The circle marks the half-power radius which is used to define the edges of the lobe. More complex fields can be described with multiple lobes, possibly overlap-ping, with different patterns at different frequencies to describe directionally dependent timbre putational costs can be greatly reduced by bypassing or simplifying stage 2 for sources outside of the listener’s envi-ronment. The new stage can then generate rough stereo pan-ning cues to suggest the general directions of the other environments.Hardware IndependenceAn important feature in the design of NA3 is that the layer 3 signal processing is portable. Our current implementation is written in C for a shared-memory MIMD computer system. However, the same processing network can be built using, for example, a network of dedicated DSP devices. To actu-ally support this sort of portability requires a formalization of the protocol between layer 2 and 3 which would not be dependent on shared memory. We can do this by adding a set of function calls in layer 2 to modify the play list in standard-ized ways, thus hiding the present shared-memory coupling between the layers and allowing it to be replaced with other types of communication channels.While the general architecture of layer 3 would remain the same, the synchronization and buffering mechanisms would almost certainly change.Resource ManagementIn the present design, as audio sources are added to the envi-ronment, the computational load increases linearly until it outgrows the available processing power. At this point, NA3 will fail to deliver samples to the output in real time, result-ing in gaps in the output stream. What we would rather have is a system which monitors its demands for resources and degrades gracefully as the environment outgrows available computing power.Three techniques are suggested for implementing graceful degradation, and these can be used in any combination. All of these techniques focus on the reduction of work in stage 2 of the signal processing pipeline, because this is where most of the computing power is spent.The first method is based on the clustering of sources. If, from the listener’s perspective, two audio sources are sepa-rated by a sufficiently small angle, they can be combined into a single source between stages 1 and 2. This clustering of the sources eliminates the need for a stage 2 processing thread. As the demand for more computing power grows, the thresh-old angle for clustering can be expanded to allow more opportunities for elimination of stage 2 processing.The second method is based on a fixed number of stage 2 processing channels, each of fixed quality, as would be the case in an array of DSP devices. In this method we use some metric based on distance, source volume, and a programmer-assigned priority. When the audio sources outnumber the stage 2 processing channels, we can use this metric to decide which sources get processed.The third method is to modify the individual processing channels by eliminating features or using cheaper models of effects. For example, to cut costs we could turn off the low-cessing pipeline in layer 3. In fact, the file descriptor need not refer to an actual disk file. Layer 2 could, in principle, open an audio input device, a pipe, or even a network socket to get its file descriptor. The network socket is of particular interest, because it would allow a remote client to feed live audio to the server.To support client-server audio transfers, we have designed a set of mechanisms in layers 1 and 2 to create network sock-ets between a client and a source thread in the layer 3 soft-ware of the server. Once such a connection is open, the client can send live audio to the server simply by writing samples to the new socket through a function call in layer 1. This function can also implement error checking and data com-pression transparently to the programmer.When choosing a network protocol and transmission format for audio, the designer should bear in mind that timely deliv-ery of audio is at least as important as error-free delivery. Late delivery of audio packets results in gaps in the audio stream which are no more desirable than glitches due to erro-neous data. Late delivery of audio can also result in a loss in synchronization between sound events and other media in a mixed-media presentation. If there is a need for enhanced reliability, it is suggested that the designer use robust com-pression and error-correcting codes rather than error-free network protocols based on retransmission.The support of network audio feeds will also require modifi-cations to the signal processing network in layer 3. If a first stage thread were to wait indefinitely on a packet, the entire signal processing network would eventually block waiting for audio from that thread. Because audio packets from a network connection may not arrive in a timely manner, the first stage of the signal processing pipeline must be able to time-out when reading audio sources. This stage 1 thread can then generate a period of silence to replace the late audio packet, thus allowing the server to process other sources in a timely manner. Sequence numbers or time-tags may also be desirable to recognize out-of-order packets and to provide a means to recognize and discard packets which have previ-ously timed-out.SecurityIn the current implementation of NA3, any client can control any existing sound event, including those created by other clients. Furthermore, any client can connect to the server and begin to insert new events into the environment. Some secu-rity mechanism is needed to control access both to the envi-ronment and to individual sound events, although it is not clear what this mechanism should be. In any case, whatever security mechanism is used must be implemented in layer 2, because this is the layer which controls client access to the server.The simplest sort of security would be store a client ID in the playlist entry for each sound event. Layer 2 software can then refuse to allow a client to modify sound events which it did not initiate. Additional security classes can easily be added for super-clients who can modify any events and for unprotected events which can be modified by any client. Access to the environment can be controlled by a mechanism similar to xhost in the X Window System.Silence ProcessingThe audio stream from a live source may contain long peri-ods on silence. Furthermore, unreliable network connections may suffer from numerous packet time-out, producing more long silent periods. It is wasteful to expend CPU time pro-cessing silence. To reduce such waste, a new type of token can be introduced into the synchronization queues which would signify a buffer of silence. Silence detection for the source can be preformed in stage 1 of the signal processing pipeline. Other pipeline layers can be modified to recognize the silence token and take appropriate cost-saving actions. Better Acoustic ModelingDistance modeling can be enhanced by adding a LPF to stage 2 of the signal processing pipeline. This LPF can be used to model atmospheric dispersion associated with the distance of a source.Doppler shift can also be added to stage 2. The introduction of Doppler shift presents two problems, however. The first is that Doppler shift computation can result in aliasing noise if not implemented with the proper low-pass filters, and these filters can be computationally expensive. The second prob-lem is that Doppler shift causes the data rates in and out of stage 2 of the pipeline to differ, requiring more complex buffering software in this stage.An extra dimension of interactivity can be added by model-ing the field patterns of sound sources in stage 2. Natural sound sources generally vary in volume and timbre when heard from different angles. The same is true of sound sen-sors, like microphones. The function relating volume and timbre to orientation is called the field pattern of the source. (See figure 3.) The field pattern can be modeled easily with a low-order digital filter by changing filter coefficients as a function of source orientation. Following the conventions of antenna design in electrical engineering, the source’s field pattern can be specified as a set of lobes in different fre-quency bands. A lobe is a local maximum in the field pattern function, and is specified as having a particular direction, strength, and width.Listener Status and Motion TrackingIf this server is to be used to provide audio to an immersive virtual environment, an additional data structure should be introduced into the server to store the listener’s position and orientation in the listening space. The issue of how to inte-grate motion tracker inputs into the server has not been resolved. The tracking system will likely be a shared resource, so we cannot mange it directly from the audio server. If the tracker has a server of it’s own, NA3 could be directed to connect to that server. Otherwise, the client will be forced to provide NA3 with tracker data at regular inter-vals. In either case, a timely transport mechanism should be used for tracker data.all of the sounds in the environment. A fourth stage is used to perform any required sample rate or format conversion prior to output and controls communication with the output device.The signal processing network is implemented with one thread for each pipeline stage for each source. Because the third and fourth stages of an environment are shared among all sources, the total number of signal processing threads for N sources is 2N+2.To reduce computational requirements, stage 1 and 2 threads exit when their sound events complete and block when these events are paused. Threads in stages 3 and 4 sleep when there are no active events, although they will continue to process silence if all events are paused.These threads communicate and synchronize via blocking queues. Reads from these queues block when the queue is empty. Such a queue also has a maximum length and writes block on full queues. Such a queue is implemented with an array and a pair of condition variables. T o reduce synchroni-zation and sample indexing overhead, the threads process audio on a buffer-by-buffer basis. The values passed through the blocking queues are tokens which can identify audio buffers or indicate the pause or death of an audio event. The choices of buffer size and queue length are important for the performance of the server. Because processing within each buffer is hidden from the server (for reasons of effi-ciency), the buffer size determines the accuracy of temporal synchronization within server. Larger buffers result in less accurate estimates of time in the audio stream. Smaller buff-ers provide lower processing latency and better temporal synchronization at the expense of greater synchronization overhead. The selection of maximum queue length has a similar effect. Longer queues allow the server greater toler-ance for irregular CPU scheduling at the expense of greater processing latency and decreased temporal synchronization accuracy.Presently, stage 1 of the signal processing network supports 16-bit linear PCM and 8-bit mu-Law encoded sources, although input format selection must be made at compile time. Sample rate conversion is not supported. A first-order FIR filter is provided for muffling/thinning effects.Stage 2 supports stereo panning based on attenuation and a delay line. Stereo panning can easily be replaced with full HRTF-based spatial sound (Burgess, 1992) if computing resources are available. We expect that a multiprocessor SPARC Viking compute server can provide 1.3 3D sources for each 50MHz CPU. Stage 2 also performs attenuation based on the distance of a source from the listener.Stage 3 supports dense reverberation. Reverb is generated with a cascade of allpass filters. A LPF is included inside one of the allpass loops to model the warmth of a listening space, although this access feature is not presently supported by the higher layers.POSSIBLE NEAR-TERM ENHANCEMENTSWe have many more ideas for NA3 than time and resources permit us to execute. There are several enhancements for NA3 which could be implemented without significantly changing its present architecture and synchronization mech-anisms.Output Delay ControlA significant problem with the current version of NA3 is that the audio output may lag as much as several seconds behind real time. This is due to uncontrolled buffering which occurs inside the Sun audio device driver. Previously, this problem was solved by writing zero-length buffers for /dev/audio. However, this solution was based on an undocumented fea-ture of the SunOS device driver, and does not work in newer Solaris systems.A possible solution is to use a wait loop which periodically polls /dev/audio for the number of samples played. This sample count is then compared to a count of samples written to /dev/audio to compute the number of buffered samples. The polling loop cycles until the number of buffered samples is sufficiently small. The problem with this scheme is that so many cycles are wasted on busywaiting, and the thr_yield function does not reduce this waste. Therefore, some other waiting mechanism must be built. This new mechanism should probably be based on SIGALRM, a standard Unix signal which is controlled by the system clock.Client Synchronization with the Audio StreamAlthough the programmer’s interface provides adequate means to control and query sound events, it is presently lack-ing in any good means to synchronize the programmer’s application with a particular point in an audio stream. To provide this facility, the RPC interface between layers 1and 2 must be extended to support callbacks. A callback is a cli-ent-side routine which is registered with the RPC service so that it can be activated by the server. Such callbacks could be used directly by the programmer to attach actions to syn-chronization events. The callbacks could also be used within the layer 1 software to provide calls which block until syn-chronization conditions are met, thus hiding the callback mechanism from programmers who do not need to use it directly.Lacking callback mechanisms, a stopgap synchronization scheme can be implemented by busywaiting on event query requests. Although this is poor software engineering, in prac-tice, the client will spend most of its time blocked on net-work I/O instead of actually spinning. This scheme could be implemented entirely inside the layer 1 software, with no modification to the server.Live Audio Sources and Network Audio Feeds Although the present implementation of NA3 supports only disk-based audio sources, the server can be extended to sup-port other source types by modifying layer 2. When a new sound event is requested, layer 2 opens the file and then places the file descriptor into the playlist. This file descriptor is then used by the source threads in stage 1 of the signal pro-。