I think it’s better to provide lower-level APIs. Though high-level APIs are
preferred by many users, but lower-level APIs are still needed to enhance the
expressiveness and ease the optimization.
Flink APIs now provide good abstraction for many operations, but higher
abstraction always means p
I do actually think that all levels of abstraction have their value.
If you wish, we have (top to bottom):
(1) SQL
(2) Table API with simplified Stream/Table duality
(3) DataStream API / windows
(4) DataStream API with custom windows and triggers
(5) Custom Operators
The Data Scientist may n
I agree with Vasia that for data scientist it's likely easier to learn the
high-level api. I like the material from
http://dataartisans.github.io/flink-training/ but all of them focus on the
high level api.
Maybe we could have a guide (blog post, lecture, whatever) on how to get
into Flink as a St
Jamie,
I think you raise a valid concern but I would hesitate to accept the suggestion
that the low-level API be promoted to app developers.
Higher-level abstractions tend to be more constrained and more optimized,
whereas lower-level abstractions tend to be more powerful, be more laborious to
Hi Jamie,
thanks for sharing your thoughts on this! You're raising some interesting
points.
Whether users find the lower-level primitives more intuitive depends on
their background I believe. From what I've seen, if users are coming from
the S4/Storm world and are used to the "compositional" way
You lost me at lattice, Aljoscha ;)
I do think something like the more powerful N-way FlatMap w/ Timers
Aljoscha is describing here would probably solve most of the problem.
Often Flink's higher level primitives work well for people and that's
great. It's just that I also spend a fair amount of t
Hi,
I'm also not sure whether we should start teaching Flink by demonstrating
the low-level APIs. According to my experience, people new to Flink should
first learn a very basic set of primitive operations. Usually this is map,
flatmap, join, windows, etc. The semantics of these operations is well
Hi All,
I also thought about this recently. A good think would be to add a good
user facing operator that behaves more or less like an enhanced FlatMap
with multiple inputs, multiple outputs, state access and keyed timers. I'm
a bit hesitant, though, since users rarely think about the implications
It really depends on the skill level of the developer. Using low-level
API requires to think about many details (eg. state handling etc.) that
could be done wrong.
As Flink gets a broader community, more people will use it who might not
have the required skill level to deal with low-level API. For
Hi Jamie,
I agree that it is often much easier to work on the lower level APIs if you
know what you are doing.
I think it would be nice to have very clean abstractions on that level so
we could teach this to the users first but currently I thinm its not easy
enough to be good starting point.
The
Hey all,
I've noticed a few times now when trying to help users implement particular
things in the Flink API that it can be complicated to map what they know
they are trying to do onto higher-level Flink concepts such as windowing or
Connect/CoFlatMap/ValueState, etc.
At some point it just become
11 matches
Mail list logo
