[Statefun] Unable to locate the launcher jar

Hello! I was trying to run the python greeter example from the statefun example o my own flink cluster (where I use to deploy regular flink jobs) and I follow all setup steps listed here

Re: Migrating createTemporaryView to new Table api.

Hi! To implement the renaming of fields with the new API, try this: tableEnv.createTemporaryView( "AgentStream", inputStream, Schema.newBuilder() .columnByExpression("useragent", "f0") .columnByExpression("expectedDeviceClass", "f1")

Re: How to refresh topics to ingest with KafkaSource?

Hi! I suppose you want to read from different topics every now and then? Does the topic-pattern option [1] in Table API Kafka connector meet your needs? [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/kafka/#topic-pattern Preston Price 于2021年10月14日周四 上午1:34写道：

Re: pyflink keyed stream checkpoint error

Hi guys, I'm still running into this problem. I checked the logs, and there is no evidence that the python process crashed. I checked the process IDs and they are still active after the error. No `killed process` messages in /var/log/messages. I don't think it's necessarily related to checkpointin

Re: How to deserialize Avro enum type in Flink SQL?

> > Can you provide a minimal reproducer (without confluent schema registry) > with a valid input? > Please download and unzip the attached file. - src/main/avro/MyProtocol.avdl - MyRecord, MyEntry, and the MyEnumType is defined - "mvn generate-sources" will auto-generate Java clas

Issue with Flink UI for Flink 1.14.0

Hello, I just started testing Flink 1.14.0 and noticed some weird behavior. This is for a Flink cluster with zookeeper for HA and two job managers (one leader, one backup). The UI on the leader works fine. The UI on the other job manager does not load any job-specific data. Same applies to the

How to refresh topics to ingest with KafkaSource?

The KafkaSource, and KafkaSourceBuilder appear to prevent users from providing their own KafkaSubscriber. Am I overlooking something? In my case I have an external system that controls which topics we should be ingesting, and it can change over time. I need to add, and remove topics as we refresh

Re: Replacing S3 Client in Hadoop plugin

Hi, Could you elaborate on why you would like to replace the S3 client? Best regards, Martijn On Wed, 13 Oct 2021 at 17:18, Tamir Sagi wrote: > I found the dependency > > > org.apache.hadoop > hadoop-aws > 3.3.1 > > > apparently its possible, there is a method > setAmazonS3Clien

Re: Replacing S3 Client in Hadoop plugin

I found the dependency org.apache.hadoop hadoop-aws 3.3.1 apparently its possible, there is a method setAmazonS3Client I think I found the solution. Thanks. Tamir. From: Tamir Sagi Sent: Wednesday, October 13, 2021 5:44 PM To: user@flink.apache

Migrating createTemporaryView to new Table api.

To test a Flink Table UDF I wrote a while ago I created this code to test it: (Full link: https://github.com/nielsbasjes/yauaa/blob/v6.0/udfs/flink-table/src/test/java/nl/basjes/parse/useragent/flink/table/TestTableFunction.java#L80 ) // The base execution environment StreamExecutionEnvironment

Replacing S3 Client in Hadoop plugin

Hey community. I would like to know if there is any way to replace the S3 client in Hadoop plugin[1] to a custom client(AmazonS3). I did notice that Hadoop plugin supports replacing the implementation of S3AFileSystem using "fs.s3a.impl" (in flink-conf.yaml it will be "s3.impl") but not the cli

RE: FlinkJobNotFoundException

Hi Matthias, Do you have any update here? Thank you, Doug From: Gusick, Doug S [Engineering] Sent: Thursday, October 7, 2021 9:03 AM To: Hailu, Andreas [Engineering] ; Matthias Pohl Cc: user@flink.apache.org; Erai, Rahul [Engineering] Subject: RE: FlinkJobNotFoundException Hi Matthias, I j

Re: Flink fault tolerance guarantees

Hi Yuval, If the pipeline fails before the next checkpoint all the records in the buffer should be replayed beginning from the last taken checkpoint. The replay usually starts from the source and reading records again from the external system. The assumption is always that after a successful ch

Re: Flink fault tolerance guarantees

Hi Yuval, If your pipeline can implement an exactly-once delivery guarantee depends on your pipeline. Usually Flink’s fault tolerance mechanism is built around periodically snapshots of intermediate states called checkpoints. As long as checkpointing is enabled and all the operators you are usi

Flink fault tolerance guarantees

Hi, Assume I have a source, stateful operator and a sink operator: Source -> Batch data and upload -> Push message to sink -> Sink runs insert / merges into a data warehouse. I am wondering, what would happen in case the data is uploaded from the stateful operator, and has not yet reached the si

Re: How to deserialize Avro enum type in Flink SQL?

First of all, are you sure the input data is correct? From the stacktrace it seems to me the issue might be that the input data is invalid. Looking at the code of AvroToRowDataConverters, It sounds like STRING should work with avro enums. Can you provide a minimal reproducer (without confluent sch

Re: I/O reactor status: STOPPED after moving to elasticsearch7 connector

Hi Oran, can you check your ES logs / metrics? Most issues we see with the ES sink are around incorrect batching and/or overloaded clusters. Could it be your ES write queue is building up? On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster wrote: > Flink version 1.13.1 > ES Version 7.12.0 > Flink dep