Apache Flink + Apache arrow flight

Hi Have anyone done any POC around using arrow or arrow flight with flink ? Is there any use case available around these 2 APIs being used ? I am thinking to use arrow as default data format , so the actions and transformations would be fast where data shuffling is required . Any insights or pointe

Re: Flink STOP with savepoint

Hi Diwakar, The client log doesn't contain much useful info except that the operation timed out. You could try: 1. Check the JM log to see if there is any relevant info. 2. Increase the client timeout to see if that helps. Best, Zhanghao Chen From: Diwakar

Re: Query regarding ClassCastException

Hi Team, Just a follow up mail regarding the query. Any leads would be appreciated. Thanks and Regards Mahima Agarwal On Mon, Mar 21, 2022 at 2:56 PM Mahima Agarwal wrote: > Hi Team, > > I am getting a ClassCastException in my flink job's business logic when > using the below code to clone

Flink STOP with savepoint

Hello Everyone, I'm running Flink 1.11 as EMR 6.1 as a Yarn application. I'm trying to use STOP command to capture savepoint and restart job from the same savepoint during redeployment. flink stop -p $JOB_RUNNING -yid $YARN_APP_ID Problem : job completes savepoint on Flink UI but it throw the f

Re: DBT-flink profile?

Hi Georg, May I have a double confirmation for integrating with dbt, are you currenty want to use it for batch jobs or streaming jobs? Best, Yun Gao -- Sender:Georg Heiler Date:2022/03/25 01:27:26 Recipient:user Theme:DBT-flink p

Re: Using Amazon EC2 Spot instances with Flink

https://aws.amazon.com/blogs/compute/optimizing-apache-flink-on-amazon-eks-using-amazon-ec2-spot-instances/ Sharing this link FWIW. Jeremy From: David Anderson Date: Thursday, March 24, 2022 at 1:32 PM To: "Vasileva, Valeriia" Cc: "user@flink.apache.org" Subject: RE: [EXTERNAL] Using Amazon E

Re: Using Amazon EC2 Spot instances with Flink

I remember a Flink Forward talk several years ago where the speaker shared how they were running on spot instances. They were catching the notification that the instance was being shutdown, taking a savepoint, and relaunching. They were also proactively monitoring spot instance prices around the wo

DBT-flink profile?

Hi, is anyone working on a DBT Flink plugin/profile? https://docs.getdbt.com/reference/profiles.yml hosts many other databases - and I think this kind of support would be really beneficial for the SQL part of Flink. Best, Georg

Flink SQL AVG with mandatory type casting

Hi, I observe strange behavior in Flink SQL: For an input stream: CREATE TABLE input_stream ( duration int, rating int ) WITH ( 'connector' = 'kafka', 'topic' = 't', 'scan.startup.mode' = 'earliest-offset', 'format' = 'avro-confluent', 'avro-confluent.schema-registry.

Re: Flink kafka consumer disconnection, application processing stays behind

Hi Qingsheng, thank you a lot for you response. The message I see from the consumer before the log exception I provided previously is this: "locationInformation": "org.apache.kafka.clients.NetworkClient.handleTimedOutRequests(NetworkClient.java:778)", "logger": "org.apache.kafka.clients.Networ

Re: Python UDF Gauge Metrics not working & error log on Vectorized UDF

Hi Dian, Thanks for the confirmation. Regards, Jesry Pada tanggal Kam, 24 Mar 2022 pukul 19.32 Dian Fu menulis: > Hi Jesry, > > Regarding the gauge metrics, I have verified that it's a bug introduced > when bumping the Beam version. I have opened a ticket [1] to track it. > > Regarding the wa

Re: Python UDF Gauge Metrics not working & error log on Vectorized UDF

Hi Jesry, Regarding the gauge metrics, I have verified that it's a bug introduced when bumping the Beam version. I have opened a ticket [1] to track it. Regarding the warning message, I think you could just ignore it. Per my understanding, it's saying that it's using a deprecated API of pyarrow.

Re: Flink OOM issue

Hi, Phoebe Which version of Flink are you using? This OOM shows insufficient meta space. As the error message says, you can increase 'taskmanager.memory.jvm-metaspace.size’(default value is 256m) and check if this error persists > 2022年3月23日 上午3:37，Phoebe Kwok 写道： > > java.lang.OutOfMemoryErr

Re: Using Amazon EC2 Spot instances with Flink

HI Vasileva, Do you want to run Flink programs on EC2? If yes, it can be deploy in standalone mode, reference this doc[1] [1]https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ > 2022年3月23日 上午6:33，Vasileva, Valeriia > 写道： > > Hello,

SQL Client Kafka (UPSERT?) Sink for confluent-avro

Hi, how can I get Flinks SQL client to nicely sink some data to either the regular kafka or the kafka-upsert connector? I have a table/ topic with dummy data: CREATE TABLE metrics_brand_stream ( `event_time` TIMESTAMP(3) METADATA FROM 'timestamp', WATERMARK FOR event_time AS event_time -

Re: flink SQL client with kafka confluent avro binaries setup

Hi Georg, Thanks for the feedback. What made you think that Flink required any additional dependencies except for the ones that are mentioned to download? Best regards, Martijn On Thu, 24 Mar 2022 at 10:12, Georg Heiler wrote: > Hi, > > the solution is the following: > > wget > https://repo1.

Re: flink SQL client with kafka confluent avro binaries setup

Hi, the solution is the following: wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.14.4/flink-sql-connector-kafka_2.12-1.14.4.jar -P lib/ wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/1.14.4/flink-sql-avro-confluent-re

Re: Flink kafka consumer disconnection, application processing stays behind

Hi Isidoros, I’m not sure in which kind of way the timeout and the high back pressure are related, but I think we can try to resolve the request timeout issue first. You can take a look at the request log on Kafka broker and see if the request was received by broker, and how long it takes for b