I couldn't get S3 select to work with a limit (it's not designed for that),
I have used clickhouse with S3 select with Arrow support (for snowflake)
and it works great.
On Sat, 13 Feb 2021, 5:24 pm Rémi Dettai, wrote:
> Thank you Daniel for taking the time to go through the slides!
>
> S3 select
Thank you Daniel for taking the time to go through the slides!
S3 select is an interesting beast, but I think the benefit we could draw
from it in this usecase is pretty limited:
- for now Buzz focuses on Parquet data, which already allows efficient
projection capabilities (it uses HTTP Range requ
Hi Andy,
Is Ballista or the Rust meeting still looking for sponsors? Please send me
some details as I'm still in the trying to get C level buy in, but I think
it's moving along :)
I'd like to do something similar to TwoSigmas BeakerX or Flint at my work,
while especially focusing on low level op
Thanks for sharing the slides Rémi! That looks really cool.
One question I have after this, do you plan to use S3 Select (
https://aws.amazon.com/blogs/aws/s3-glacier-select/)?Seems it would fit
your architecture nicely and I think shouldn't be too hard to create the
query from the filters/project
Thanks for the notes Andy. Here is the slide deck I presented, for further
reference:
https://docs.google.com/presentation/d/1uZ5PbazC1zCX24k0Hh-UItddIh9BRvD5GL7NUDgc9eQ/edit?usp=sharing
If anyone wants to see how it works in practice and does not have an AWS
account to try it out, feel free to re
Attendees
-
Andy Grove
-
Benjamin Blodgett
-
Marc Prud’Hommeaux
-
Mike Seddon
-
Jorge Leitao
-
Andrew Lamb
-
Fernando Herrera
-
Neville Dipale
-
Remi Dettai
(Please let me know if I have misspelled anyone’s names)
Topics Discussed
A quick reminder that the bi-weekly Arrow Rust sync call starts about an
hour from now. Everyone is welcome.
https://meet.google.com/ctp-yujs-aee
Thanks,
Andy.
