Hi everyone! We are currently writing a blog post ( https://github.com/huggingface/blog/pull/1283) about the synergies between the Hugging Face `datasets` library and Apache Arrow, and how to use the Compute API to analyze HF datasets out-of-core. This will soon be published on the HF blog: https://huggingface.co/blog.
We thought it might be cool to cross-post (not necessarily in its exact same form) on the Arrow blog, if that's something that you'd be interested in. Look forward to hearing what you think! Best, Chris
