With an increase in the generation and use of multi-dimensional datasets – in the spatial, scientific, medical, and engineering industries, among others – it has been decided to extend the SQL database language with support for massive multi-dimensional arrays.
Initiated by Prof. Peter Baumann from Jacobs University, work has commenced on the forthcoming standard: SQL/MDA.
At last week’s meeting in Beijing all national bodies participating in the SQL working group of ISO unanimously agreed on the importance of arrays in SQL. Following thorough assessment of all options available, the group has accepted the proposal of Baumann for further elaboration. The new standard will be named ISO 9075 SQL/MDA, for ‘Multi-Dimensional Arrays.’
SQL has had huge success in running any-size databases in the fields of business and administration. However, the ‘big data’ of the spatial and other sciences are structured differently. Instead of simple tables, they often consist of multi-dimensional ‘data cubes’. Specifically, in the spatial sciences, this can encompass 1-dimensional sensor data, 2-D satellite imagery, 3-D x/y/t image time-series as well as x/y/z geophysical voxel data, and 4-D x/y/z/t weather data. Outside of the spatial sciences, in the medical sciences, there is laserscan microscopy and brain scans. And, when it comes to astrophysics, data can grow as large as simulations of the whole universe.
But SQL is not able to find, filter, and process such multi-dimensional arrays, and, consequently, arrays are largely maintained outside databases. Recognising this shortcoming, Peter Baumann, Professor of Computer Science at Jacobs University Bremen, and his group have long been researching on ways to extend SQL appropriately. The rasdaman system, which the group established, has now coined a new technology: Array Databases.
In a recent technology demonstration, more than 1,000 computers collaborated in a cloud to jointly compute the result of a single database query. This ‘distributed query processing’ means a massive speed increase, and research challenges on multi-Petabyte data cubes can be answered that were previously unsolvable.
International data centres are already using this tool to allow scientists to gain unanticipated insights into their spatio-temporal data cubes, and rasdaman installations can be found at NASA, ESA, British Geological Survey, Plymouth Marine Laboratory, Deutscher Wetterdienst, and many more.
