Re: FLINK-13497 / "Could not create file for checking if truncate works" / HDFS

Hi >From the given stack trace, maybe you could solve the "replication problem" first, File /okd-dev/3fe6b069-43bf-4d86-9762-4f501c9db16e could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation, and maybe

Re: add() method of AggregateFunction not called even though new watermark is emitted

Hi, Thx for the replies - Congxian & Dawdi. Watermarks are advancing.Not sure how to check every new generated watermark is reaching end of the window I did check the Flink UI for the currentInputWatermark and it is increasing monotonically. Narrowed down the problem to not calling the

Re: Challenges Deploying Flink With Savepoints On Kubernetes

Thanks you till. We will try to shift to latest flink version. Regards Bhaskar On Mon, Oct 14, 2019 at 7:43 PM Till Rohrmann wrote: > Hi Vijay, > > Flink usually writes first the checkpoint data to disk and then writes the > pointer to the files to ZooKeeper. Hence, if you see a ZooKeeper

group.id更改，通过savepoint启动的Flink任务，Kafka consumer是否仍然可以获取到保存在状态中的start position？

Hi，all根据文档，如果从checkpoint或者savepoint中恢复任务，则Kafka Consumer会使用状态中的start position。 https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html#kafka-consumers-start-position-configuration 请问通过savepoint恢复的任务，如果group.id更改，Kafka consumer是否仍然可以获取到保存在状态中的start

Re: [flink sql] table in subquery join temporal table raise java.lang.NullPointerException

Thanks for reporting this issue. I've pulled in Jark and Kurt who might help you with this problem. Cheers, Till On Sat, Oct 12, 2019 at 5:42 PM hzp wrote: > Hi all, > > I'm using flink sql to join a temporal table in a subquery, but it raises > java.lang.NullPointerException when execute. > >

Re: Challenges Deploying Flink With Savepoints On Kubernetes

Hi Vijay, Flink usually writes first the checkpoint data to disk and then writes the pointer to the files to ZooKeeper. Hence, if you see a ZooKeeper entry, then the files should be there. I assume that there is no other process accessing and potentially removing files from the checkpoint

FLINK-13497 / "Could not create file for checking if truncate works" / HDFS

Hello,    We recently upgraded our product from Flink 1.7.2 to Flink 1.9, and we experience repeated failing jobs with   java.lang.RuntimeException: Could not create file for checking if truncate works. You can disable support for truncate() completely via BucketingSink.setUseTruncate(false).

Re: Batch Job in a Flink 1.9 Standalone Cluster

Thanks for the insight Roman, and also for the GC tips. There are 2 reasons why I wanted to see this memory released. First as a way to just confirm my understanding of Flink memory segment handling. Second is that I run a single standalone cluster that runs both streaming and batch jobs, and

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Thank you all for the encouraging feedback! So far the reaction to add this to Flink was exclusively positive, which is really great to see! To make this happen, here would be the next steps: (1) As per the bylaws, a contribution like that would need a PMC vote, because it is a commitment to

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

+1 to add Stateful Function to FLINK core repository. Best, tison. Becket Qin 于2019年10月14日周一 下午4:16写道： > +1 to adding Stateful Function to Flink. It is a very useful addition to > the Flink ecosystem. > > Given this is essentially a new top-level / first-citizen API of Flink, it > seems

Re: 【flink sql】table in subquery join temporal table raise java.lang.NullPointerException

Hi, Thanks for reply. They are the same issue. > 在 2019年10月14日，下午5:17，Benoît Paris 写道： > > https://issues.apache.org/jira/browse/FLINK-14200 >

Re: 【flink sql】table in subquery join temporal table raise java.lang.NullPointerException

This seems to be related to: https://issues.apache.org/jira/browse/FLINK-14200 (Temporal Table Function Joins do not work on Tables (only TableSources) on the query side) On Sat, Oct 12, 2019 at 5:56 PM hzp wrote: > Hi all, > > I'm using flink sql to join a temporal table in a subquery, but

kafka-json row_time register

my data come from kafka and json formated。 I register source table with row_time using timestamps_from_field (python api with flink-190). As api described，timestamps_from_field can handle long and timestamp. but json_schema do not contain long and in original data there are no timestamp data-type

Re: StreamingFileSink

As for StreamingFileSink and compressed output, see the StreamingFileSink.forBulkFormat and BulkWriter.Factory classes. Simple example (using apache commons-io and commons-compress): val writer = new BulkWriter.Factory[String] { override def create(out: FSDataOutputStream): BulkWriter[String]

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

+1 to adding Stateful Function to Flink. It is a very useful addition to the Flink ecosystem. Given this is essentially a new top-level / first-citizen API of Flink, it seems better to have it the Flink core repo. This will also avoid letting this important new API to be blocked on potential

Re: Batch Job in a Flink 1.9 Standalone Cluster

Forced GC does not mean that JVM will even try to release the freed memory back to the operating system. This highly depends on the JVM and garbage collector used for your Flink setup, but most probably it's the jvm8 with the ParallelGC collector. ParallelGC is known to be not that aggressive

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi all, Big +1 for contributing Stateful Functions to Flink and as for the main question at hand, I would vote for putting it in the main repository. I understand that this can couple the release cadence of Flink and Stateful Functions although I think the pros of having a "you break it, you fix

Re: add() method of AggregateFunction not called even though new watermark is emitted

Hi Vijay, Could you check if the Watermark for the aggregate operator advances? You should be able to check that in the Flink WebUI. Could it be that the Watermark does not advance for all of the upstream operators? The watermark for a particular operator is a minimum of watermarks received from

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Hi Stephan, Big +1 for adding stateful functions to Apache Flink! The use cases unlocked with this feature are very interesting and promising. Regarding to whether to place it into Flink core repository, personally I perfer to put it in the main repository. This feature introduces a new set of

FLINK WEEKLY 2019/41

FLINK WEEKLY 2019/41 很高兴和大家分享上周 FLINK 社区的发展。上周 Stephan Ewen 在 Flink Forward Berlin 上宣布了基于 FLINK 的通用计算库 Stateful Function ，使用 Stateful Function 可以将 FLINK 的应用场景扩展到现有的几乎所有数据系统上。具体的邮件链接参考本次 WEEKLY 最后的社区发展部分 用户问题 Flink 1.8 版本如何进行