The release of the new version of TileDB 2.0 was recently announced in which integration is added to work with different cloud services, ability to use different algorithms, improvements with the different storage engines and other things.
For those who are unfamiliar with TileDB, they should know that this is database designed to help data science teams make discoveries faster by giving them a more powerful way to store, update, analyze and share large sets of diverse data.
About TileDB
TileDB consists of a new multidimensional array data format, a fast, embeddable, open source C ++ storage engine with data science tool integrations and a cloud service for easy serverless calculation and data management.
TileDB is optimized to store matrices and data used in multidimensional scientific calculations, such as various systems for processing genetic information, spatial and financial data, that is, systems that operate with dispersed or continuously filled multidimensional matrices.
TileDB offers a standalone and embedded C ++ library which ships with API in C, C ++, Python, R, Java and Go and you have direct access to the TileDB arrays.
The library is integrated with Spark, Dask, PrestoDB, MariaDB, Arrow, and geospatial libraries such as PDAL, GDAL, and Rasterio. TileDB pushes as much compute as possible to storagesuch as SQL engine filter conditions and Dask and Spark data frame calculations.
Alongside the database is TileDB Cloud, a pay-as-you-go service that you can use to share TileDB arrays in the cloud with other users and perform serverless calculations on them.
Of the key features of TileDB the following stand out:
- Effective methods for storing sparse arrays, the data of which does not follow continuously, the array is filled with chunks, and most of the elements remain empty or take the same value.
- Ability to access data in key value format or sets of columns (DataFrame);
- Support for integration with AWS S3, Google Cloud Storage, and Azure Blob Storage.
- TileDB efficiently supports data versioning natively embedded in its format and storage engine.
- It has a variety of optimizations around parallel I / O in cloud object stores and multi-threaded calculations (such as classification, compression, etc.).
- Ability to use different data compression and encryption algorithms.
- Support for checksum integrity.
- It works in multithreaded mode with input / output parallelization.
- Support for versioning of stored data, even for retrieving state at a certain point in the past or for atomic updates of large integer sets.
- Ability to link metadata.
- Data grouping support.
- Integration modules to be used as a low-level storage engine in Spark, Dask, MariaDB, GDAL, PDAL, Rasterio, gVCF and PrestoDB.
- C ++ API binding libraries for Python, R, Java, and Go languages.
The project code is written in C ++ and distributed under the MIT license and is compatible with Linux, macOS, and Windows.
About version 2.0
The 2.0 version stands out for its compatibility with the «DataFrame» concept, which allows you to store data in the form of columns of values arbitrary length, bound to specific attributes and that the redesigned API for R.
Storage is also optimized for processing sparse matrices heterogeneous in size (different types of data can be stored in cells and it is possible to merge different types of columns, for example, in which the name, time and price are stored).
Added support for columns with string data, as well as modules were added for integration with Google Cloud Storage and Azure Blob Storage.
Finally if you want to know more about this new version, pYou can check the release note at the following link.
Y to learn more about its installation, implementation and documentation, you can do it in the following link