Re: Unit testing PyFlink SQL project

2022-04-23 Thread Dian Fu
Hi John, I have written an example on how to write unit tests of Flink functionalities with PyFlink in [1]. Hope it is helpful for you. Feel free to let me know if there are any problems. Regards, Dian [1] https://github.com/dianfu/pyflink-faq/tree/main/testing On Sun, Apr 24, 2022 at 9:25 AM D

Fwd: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing

2022-04-23 Thread Yi Tang
-- Forwarded message - From: Yi Tang Date: Sun, Apr 24, 2022 at 11:29 AM Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing To: Hi team; I'm trying to write to aliyun oss via FileSystem connector. The job master always hangs on finishing. Looks

Re: JobManager doesn't bring up new TaskManager during failure recovery

2022-04-23 Thread Zheng, Chenyu
Thank you so much, Yang! It looks very likely that this issue is causing the bug I met. I’ll upgrade my Flink version and test if I can reproduce that bug. BRs, Chenyu From: Yang Wang Date: Sunday, April 24, 2022 at 10:20 AM To: "Zheng, Chenyu" Cc: "user@flink.apache.org" , "user...@flink.apa

Re: JobManager doesn't bring up new TaskManager during failure recovery

2022-04-23 Thread Yang Wang
After more debugging, I think this issue is same as FLINK-24315[1], which is fixed in 1.13.3. [1]. https://issues.apache.org/jira/browse/FLINK-24315 Best, Yang Zheng, Chenyu 于2022年4月22日周五 18:27写道: > I created a JIRA ticket https://issues.apache.org/jira/browse/FLINK-27350 > to track this issue

Re: KAFKA SINK ERROR IN Flink-(version 1.14.4)

2022-04-23 Thread Dian Fu
Hi Harshit, Could you try to update the following line `ds = ds.map(lambda x: ','.join([str(value) for value in x]))` as following: `ds = ds.map(lambda x: ','.join([str(value) for value in x]), output_type=Types.STRING())` The reason is that if the output type is not specified, it will be seriali

HTTP REST API as Ingress/Egress

2022-04-23 Thread Himanshu Sareen
Team, Does flink-statefun support HTTP REST as Ingress ( like Kafka and kinesis ) I'm looking for a fault tolerant solution where an external API can invoke stateful function , access state and return response. We are using python sdk for statefun application Regards, Himanshu

Re: Unit testing PyFlink SQL project

2022-04-23 Thread Dian Fu
Hi John, >> I don't know how to fix this. I've tried adding `flink-table-planner` and `flink-table-planner-blink` dependencies with `test-jar` to my dummy pom.xml, but it still fails. What's the failure after doing this? The flink-table-planner*-tests.jar should be available in maven repository[1]

Unit testing PyFlink SQL project

2022-04-23 Thread John Tipper
Hi all, Is there an example of a self-contained repository showing how to perform SQL unit testing of PyFlink (specifically 1.13.x if possible)? I have cross-posted the question to Stack Overflow here: https://stackoverflow.com/questions/71983434/is-there-an-example-of-pyflink-sql-unit-testing

Re: Savepoint and cancel questions

2022-04-23 Thread Dan Hill
Hi Hangxiang. Thanks! 1. Ah, okay. It makes more sense considering FAILED. 2. Oh cool. I'm migrating to v1.14.4 now. 3. Yes, this is great! On Fri, Apr 22, 2022 at 8:05 PM Hangxiang Yu wrote: > Hi, Dan > 1. Do you mean put the option into savepoint command? If so, I think it > will not work