> Which connectors would be commonly used when reading in Arrow format?
Filesystem?
Currently, yes. The better way is it can be combined used with
different connector,
but I have not figured out how to integrate the Arrow format
deserializer with the
`DecodingFormat` or `DeserializationSchema`
Which connectors would be commonly used when reading in Arrow format?
Filesystem?
On Wed, Apr 12, 2023 at 4:27 AM Jacky Lau wrote:
> Hi
>I also think arrow format will be useful when reading/writing with
> message queue.
>Arrow defines a language-independent columnar memory format for
Hi
I also think arrow format will be useful when reading/writing with
message queue.
Arrow defines a language-independent columnar memory format for flat and
hierarchical data, organized for efficient analytic operations on modern
hardware like CPUs and GPUs. The Arrow memory format also
Hi all,
Thanks for your input.
@Ran > However, as mentioned in the issue you listed, it may take a lot of work
and the community's consideration for integrating Arrow.
To clarify, this proposal solely aims to introduce flink-arrow as a new format,
similar to flink-csv and flink-protobuf. It
Hi all,
How do Flink formats relate to or interact with Paimon (formerly
Flink-Table-Store)? If the Flink format interface is used there, then it
may be useful to consider Arrow along with other columnar formats.
Separately, from previous experience, I've seen the Arrow format be useful
as an
Hi,
To be honest, I haven't seen that much demand for supporting the Arrow
format directly in Flink as a flink-format. I'm wondering if there's really
much benefit for the Flink project to add another file format, over
properly supporting the format that we already have in the project.
Best
It is a good point that flink integrates apache arrow as a format.
Arrow can take advantage of SIMD-specific or vectorized optimizations,
which should be of great benefit to batch tasks.
However, as mentioned in the issue you listed, it may take a lot of work
and the community's consideration for
Hi guys
I'm opening this thread to discuss supporting the Apache Arrow format
in Flink.
Arrow is a language-independent columnar memory format that has become
widely used in different systems, and It can also serve as an
inter-exchange format between other systems.
So, using it directly