Re: getting an error when configuring state backend to hdfs

Thanks ! My apology for my late response. all good advices I did put the flink-hadoop-fs jar in /lib as Chesnay suggested. and from the IDE simply use file:// as Yun suggested On Mon, Dec 24, 2018 at 6:32 AM Yun Tang wrote: > Hi Avi > > For application running in your IDE, please set the checkp

Re: [ANNOUNCE] Apache Flink 1.6.3 released

Thanks a lot for being our release manager Gordon. Great job! And also a big thanks to the community for making this release possible. Cheers, Till On Mon, Dec 24, 2018 at 2:11 AM vino yang wrote: > Thanks for being the release manager Gordon. And glad to see Flink 1.6.3 > released. > > Best, >

Re: HA with HDFS question

Hi Steve, I think your question is specific to HDFS HA setup. Flink HA addresses failover issues only for job manager and job meta state. The storage layer for savepoints/checkpoints and its failover are responsibility of HDFS deployment. Flink uses HDFS as external system, available over locatio

Re: Connecting to kafka with tls

Hi Avi, have you tried to setup FlinkKafkaConsumer/FlinkKafkaProducer source/sink [1] with Kafka TLS config properties [2]? Best, Andrey [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html

Re: Running Flink on Yarn

Hi, I suppose you apply windowing to a keyed stream or SQL time-windowed join? Globally windowed streams are non-parallel and processed/stored in one slot. In case of keyed stream, total range of key values is distributed among slots. Each slot processes/stores only a subrange of keys. Window st

Re: Running Flink on Yarn

Thanks for the quick response Andrey. I'm doing a SQL time-windowed join on non-keyed stream. So all the thread in various task slot in the same TM will share this state. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Iterations and back pressure problem

We've tried using iterations feature and in case of significant load the job sometimes stalls and stops processing events due to high back pressure both in tasks that produces records for iteration and all the other inputs to this task. It looks like a back pressure loop the task can't handle all t

Re: Running Flink on Yarn

If you mean time-windowed join documented here [1]. I think it implicitly uses keyed stream [2] where the key is the field in equi-join predicate. The window state is also keyed [3] in this case. I also cc Timo and Piotr, they might add more to this topic. [1] https://ci.apache.org/projects/flink/

Re: Running Flink on Yarn

I am using time-windowed join only. Here's a sample query - SELECT a1.order_id, a2.order.restaurant_id FROM awz_s3_stream1 a1 INNER JOIN awz_s3_stream2 a2 ON CAST(a1.order_id AS VARCHAR) = a2.order_id AND a1.to_state = 'PLACED' AND a1.proctime BETWEEN a2.proctime - INTERVAL '2' HOUR AND a2.proct

Re: Iterations and back pressure problem

Hi Sergey, It seems to be a known issue. Community will hopefully work on this but I do not see more updates since the last answer to the similar question [1], see also [2] and [3]. Best, Andrey [1] http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3CBFD8C506-5B41-47D8-B735-488D03

Re: Running Flink on Yarn

I think the data buffered for join will be distributed among threads by order_id (a1 and a2 will be internally keyed). Each thread will have non-shared window state (for 2 hours) per certain order_id's. Slots will share some common JVM resources mentioned in docs, also access to state DB but not th

Re: Iterations and back pressure problem

Hi Sergey, As Andrey noted, it’s a known issue with (currently) no good solution. I talk a bit about how we worked around it on slide 26 of my Flink Forward talk

getting Timeout expired while fetching topic metadata

Hi all, very new to flink so my apology if it seems trivial. We deployed flink on gcloud I am trying to connect to kafka but keep getting this error: *org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata* this how my properties look like val consumerPropert

Re: getting Timeout expired while fetching topic metadata

Hi Avi, Can you try to add this properties props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); Thanks, Miki On Mon, Dec 24, 2018 at 8:19 PM Avi Levi wrote: > Hi all, > very new to flink so my apology if it seems trivial. > We deployed flink on gcloud > I am trying to connect to ka

Re: getting Timeout expired while fetching topic metadata

Thanks Miki, we had a bug in the certs. however the property you suggested gave as better logs so it really cleared things. THANKS !!! On Mon, Dec 24, 2018 at 8:45 PM miki haiat wrote: > Hi Avi, > Can you try to add this properties > > props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SS

Re: [ANNOUNCE] Apache Flink 1.6.3 released

Thanks a lot for being our release manager Gordon. Thanks a lot for made this release possible! Cheers, Jincheng Tzu-Li (Gordon) Tai 于2018年12月23日周日 下午9:35写道： > Hi, > > The Apache Flink community is very happy to announce the release of > Apache Flink 1.6.3, which is the third bugfix release for

Re: [ANNOUNCE] Apache Flink 1.7.1 released

Thanks for being the release manager Chesnay! Thanks a lot for made this release possible! Thanks, Jincheng Chesnay Schepler 于2018年12月23日周日 上午3:34写道： > The Apache Flink community is very happy to announce the release of > Apache Flink 1.7.1, which is the first bugfix release for the Apache > Fl