Remote Data Access and Streaming Techniques

Efficient remote data access and streaming techniques are core technologies that affect file formats with advanced data layout, dump routines for simulations, compression techniques, stream processing algorithms, and specialized algorithms. Proper deployment of these techniques achieves important capabilities ranging from remote data access and visualization with high resiliency and low latency to pipelined computations that don't use any intermediated storage and realize super-linear speedups. We are providing the ability to integrate computation and data movement to benefit our stakeholders who run computations and analysis on different, often geographically distributed platforms. We build on top of our prototypical implementation where we demonstrated at SC04 real-time visualization of output from a large-scale simulation executed on BlueGene/L. Our deployment will be compatible with most data management workflows including the infrastructure provided by the SDM center and integrated in visualization tools such as VisIt, and EnSight. Our efficient compression libraries and optimize dump routines will provide efficient I/O for simulations first and for the subsequent data access and analysis stages. We are also providing a number of progressive streaming modules starting from, volume rendering, isosurface computation, slicing, simplification of surface and volumetric meshes, and many others specifically requested by the applications.