Re: How to understand create watermark for Kafka partitions

Hi Alex, >> But why also say created watermark for each Kafka topic partitions ? IMO, the official documentation has explained the reason. Just copied here: When using Apache Kafka as a data source, each Kafka par

Re: Join a datastream with tables stored in Hive

Hi Krzysztof, What you raised also interested us a lot to achieve in Flink. Unfortunately, there is no in place solution in Table/SQL API yet, but you have 2 options which are both close to this thus need some modifications. 1. The first one is use temporal table function [1]. It needs you to wri

Re: Join a datastream with tables stored in Hive

Sorry I forgot to paste the reference url. Best, Kurt [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table-function [2] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table

Re: [DISCUSS] have separate Flink distributions with built-in Hive dependencies

cc user ML in case anyone want to chime in On Fri, Dec 13, 2019 at 00:44 Bowen Li wrote: > Hi all, > > I want to propose to have a couple separate Flink distributions with Hive > dependencies on specific Hive versions (2.3.4 and 1.2.1). The distributions > will be provided to users on Flink down

Re: [DISCUSS] have separate Flink distributions with built-in Hive dependencies

+1, this is definitely necessary for better user experience. Setting up environment is always painful for many big data tools. Bowen Li 于2019年12月13日周五 下午5:02写道： > cc user ML in case anyone want to chime in > > On Fri, Dec 13, 2019 at 00:44 Bowen Li wrote: > >> Hi all, >> >> I want to propose

Re: State Processor API: StateMigrationException for keyed state

Sorry I posted it but I guess it got dropped when the message was formatted. Here’s another attempt: 2019-12-12 12:50:08 java.io.IOException: Failed to restore state backend at org.apache.flink.state.api.input.KeyedStateInputFormat.getStreamOperatorStateContext(KeyedStateInputFormat.java:231

Re: Flink 'Job Cluster' mode Ui Access

Thank you for the logs. Flink can indeed find the WebUI files in the distribution, which is a bit odd. Since there are no static files serve in this case, the StaticFileServerHandler is never set up in the first place (hence why we didn't find any log statements). What I also found in the lo

Re: [ANNOUNCE] Apache Flink 1.8.3 released

Thanks a lot Hequn for being our release manager and to the community for making this release happen :-) Cheers, Till On Thu, Dec 12, 2019 at 9:02 AM Zhu Zhu wrote: > Thanks Hequn for driving the release and everyone who makes this release > possible! > > Thanks, > Zhu Zhu > > Wei Zhong 于2019年

TypeInformation problem

I was refactoring some Flink code to use IndexedSeq rather than Array. When I compiled the code I had failures that required according to the URL below the following to be inserted /* * Type information (see https://stackoverflow.com/questions/37920023/could-not-find-implicit-value-for-evidence

Flink slot utilization

Hi I'm testing Flink to do stream processing, in my use case there are multiples pipelines processing messages from multiple Kafka sources. I have some questions regarding the jobs and slots. 1) When I deploy a new job, it takes a job slot in the TM, the job never ends (I think it doesn't end bec

[ANNOUNCE] Zhu Zhu becomes a Flink committer

Hi everyone, I'm very happy to announce that Zhu Zhu accepted the offer of the Flink PMC to become a committer of the Flink project. Zhu Zhu has been an active community member for more than a year now. Zhu Zhu played an essential role in the scheduler refactoring, helped implementing fine graine

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

Congrats Zhu Zhu and welcome on board! Best, Fabian Am Fr., 13. Dez. 2019 um 17:51 Uhr schrieb Till Rohrmann < trohrm...@apache.org>: > Hi everyone, > > I'm very happy to announce that Zhu Zhu accepted the offer of the Flink PMC > to become a committer of the Flink project. > > Zhu Zhu has been

Re: Sample Code for querying Flink's default metrics

Additionally, when an old job completes and I run a new job on the Flink Yarn session mode cluster, when I query for metrics before they become available for the new job, I sometimes get the last metrics for the old job instead. This happens even if I wait for the TaskManager to be released by Flin

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

Congratulations! :) > On 13 Dec 2019, at 18:05, Fabian Hueske wrote: > > Congrats Zhu Zhu and welcome on board! > > Best, Fabian > > Am Fr., 13. Dez. 2019 um 17:51 Uhr schrieb Till Rohrmann < > trohrm...@apache.org>: > >> Hi everyone, >> >> I'm very happy to announce that Zhu Zhu accepted th

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

Congratulations!:) On Fri, Dec 13, 2019 at 9:45 AM Piotr Nowojski wrote: > Congratulations! :) > > > On 13 Dec 2019, at 18:05, Fabian Hueske wrote: > > > > Congrats Zhu Zhu and welcome on board! > > > > Best, Fabian > > > > Am Fr., 13. Dez. 2019 um 17:51 Uhr schrieb Till Rohrmann < > > trohrm..

Re: Join a datastream with tables stored in Hive

Very interesting, Kurt! Yes, I also imagined it's rather a very common case. In my company we currently have 3 clients wanting this functionality. I also just realized this slight difference between Temporal Join and Temporal Table Function Join, that there are actually two methods:) Regarding opt

Questions about taskmanager.memory.off-heap and taskmanager.memory.preallocate

Hi Community, I have a question about the taskmanager.memory.preallocate config in the doc https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-preallocate

Jobmanager not properly fenced when killed by YARN RM

Hi, Recently I've seen a situation when a JobManager received a stop signal from YARN RM but failed to exit and got in the restart loop, and keeps failing because the TaskManager containers are disconnected (killed by RM as well) before finally exited when hit the limit of the restart policy. This