Hey, the basic idea is to have a rust native and export to Python thru PyO3. This will allow a boost in performance and use in a Data Engineering scenario instead of using parquet also it might included inside a Apache Iceberg catalog. Withing that scenario integration with ML tools like Pytorch or DE tools likes Spark will be seamless.
Il giorno ven 5 gen 2024 alle ore 13:45 liuyong <[email protected]> ha scritto: > Hi, has the tsfile submodule planned to provide the native interface for > prevalent analytical frameworks or programming languages? Especially, the > Python community. The Python community consists of many packages powerful > on time series data analysis, such as the sktime for Machine Learning and > PyTorch for Deep Learning, which highly relies on training on a huge amount > of time series data. > > > As we develop the native AI engine for IoTDB (AINode). We find that the > tsfile, which takes advantage of the fast reading and dense compression of > tsfile organization, may further benefit the incorporation of the DB and AI > System with authority management and free from numerous and costly data > transfers compared with loading from the local file system. And there are > still some open problems as loading the data for training deep models in > iterations and inference of the data as the stream, which pose novel > challenges to the operator supported by the native interface. > > > > Perhaps a good starting point would be to implement a PyTorch-like > dataloader, where the data source could support the tsfile. > > > > -- Life is a chess game - Anonymous.
