Hi Jacky Lau,
1) yes I saw that
2) I saw sources like IntegerSource which are bounded and which extend
RichParallelSourceFunction. This is why I mentioned it.
3) True, there is an hadoop ES connector in Beam but it is more of a
side connector, the main one is ElasticsearchIO here:
hi Jacky Lau :
I agree with jark's point of view, the use of es is not just to read data,
more use is to group query, aggregate these.
Best,
Forward
Jacky Lau 于2020年6月5日周五 下午2:47写道：
> hi Etienne Chauchot:
> you can read here https://www.jianshu.com/p/d32e17dab90c, which is
> chinese.But you
hi Etienne Chauchot:
you can read here https://www.jianshu.com/p/d32e17dab90c, which is
chinese.But you can konw that slice api has poor performance in es-hadoop
project .
And i found that es-hadoop has removed this and disable sliced scrolls by
default. you can see below, which i found in the
Thanks for Jacky and Yangze.
1) we will reorganize your design and move it to the flip
2) we will support lookup and DynamicSource. and current DynamicSource's
optimize, such as supportLimitPushDown/supportLimitPushdown doesn't have
achieved. so we will do it after it have done
Thanks for your
hi Etienne Chauchot:
thanks for your discussion.
for 1) we do not supprt es unbouded source currently
for 2) RichParallelSourceFunction is used for streaming ,InputFormat is for
batch
for 3) i downloaded beam just now. and the beam es connector is also using
es-hadoop. i have read the code of
Thanks Jacky for starting this discussion.
The requirement of ES source has been proposed in the community many
times. +1 for the feature from my side.
Here are my thoughts:
1. streaming source
As we only support bounded source for JDBC and HBase, so I think it's fine
to have a bounded ES
Hi,
I made the Elasticsearch connector of Apache Beam and I was thinking
about doing the same for Flink when I came by this discussion. I have
some comments regarding the design doc:
1. Streaming source:
ES has data streams features but only for time series data; the aim of
this source is
Hi, Jackey.
Thanks for driving this discussion. I think this proposal should be a
FLIP[1] since it impacts the public interface. However, as we have
only some preliminary discussions atm, a design draft would be ok. But
it would be better to organize your document according to [2].
I've two
Hi Robert Metzger:
Thanks for your response. could you please read this docs.
https://www.yuque.com/jackylau-sc7w6/bve18l/14a2ad5b7f86998433de83dd0f8ec067
. Any Is it any problem here? we are worried about
we do not think throughly. thanks.
--
Sent from:
Hi Jacky,
thanks a lot for starting the discussion. I have no objections to adding
support for reading data from ElasticSearch as well, as long as we clearly
state the performance and correctness implications / guarantees in the
Flink documentation.
On Tue, Jun 2, 2020 at 9:52 AM Jacky Lau
Hi all!
We have started some preliminary work on the flink elasticsearch integration
(es connector for es version7) at hikvision research institute.
It seems that the integration should think throughly. And we want to
contribute our code for the conmunity.
So I think I should open a discussion
11 matches
Mail list logo
