Re: Apache Flink - A question about Tables API and SQL interfaces

Hi Mans, +1 for Austin's reply. I would like to add something about "allow lateness". After introduce Windowing table-valued function in Flink 1.13, User could use two SQL solution to do window aggregate. And 'allow lateness' behavior is different in these two solutions. 1. If ad

How to deal with the abnormal extraction time of the calculated column？

Hi My Friends： How to deal with the abnormal extraction time of the calculated column ？ My DDL ： CREATE TABLE mytable ( xx string, t as get_event_time_from_xx(xx), WATERMARK FOR t AS t - INTERVAL '1' SECOND ) WITH (xxx) If an exception occurs inside my function

Flink 1.13.0 reactive mode: Job stop and cannot restore from checkpoint

Hi, We run our application in Flink 1.13.0, Kubernetes standalone application cluster with reactive mode enabled. The application has stopped and cannot restore today, so we try to restore the application from checkpoint. However, the application cannot restart from checkpoint due to the following

Re: Root Exception can not be shown on Web UI in Flink 1.13.0

Hi, Gary I think it might be a bug. So would you like to open a jira for this. And could you share the exception ,which the TaskManagerLocation is null? It might be very helpful to verify the cause. Best, Guowei On Thu, May 13, 2021 at 10:36 AM Yangze Guo wrote: > Hi, it seems to be related t

Re: RabbitMQ source does not stop unless message arrives in queue

Hi Jose, hey Austin!! I know we were just recently looking at trying to consume a fixed number of messages from an RMQ source, process them and output them to an RMQ sink. As a naive first attempt at stopping the job when the target number of messaged had been processed, we put a counter state

Re: RabbitMQ source does not stop unless message arrives in queue

Hey Jose, Thanks for bringing this up – it indeed sounds like a bug. There is ongoing work to update the RMQ source to the new interface, which might address some of these issues (or should, if it is not already), tracked in FLINK-20628[1]. Would you be able to create a JIRA issue for this, or wou

Incorrect flink documentation example

Hi, I was trying out examples given here to convert Datastream to Table and one of the example gives out error. My code is as follows : /StreamExecutionEnvironment env = S

Long checkpoint duration for Kafka source operators

Hello, I have an application that reads from two Kafka sources, joins them, and produces to a Kafka sink. The application is experiencing long end to end checkpoint durations for the Kafka source operators. I'm hoping I could get some direction in how to debug this further. Here is a UI screensho

RabbitMQ source does not stop unless message arrives in queue

Hi, I am using Flink 1.12 to read from and write to a RabbitMQ cluster. Flink's RabbitMQ source has some surprising behavior when a stop-with-savepoint request is made. *Expected Behavior:* The stop-with-savepoint request stops the job with a FINISHED state. *Actual Behavior:* The stop-with-save

Re: Apache Flink - A question about Tables API and SQL interfaces

Hi Mans, There are currently no public APIs for doing so, though if you're willing to deal with some breaking changes there are some experimental config options for late events in the Table API and SQL, seen in the WIndowEmitStrategy class[1]. Best, Austin [1]: https://github.com/apache/flink/bl

Re: How does JobManager terminate dangling task manager

Hi narasimha, For each TaskManager, there are two kinds of connections to the JobManager process. - One single connection to the ResourceManager, which allows RM to monitor the slots' availability and assign them to Flink jobs. - Connections to each JobMaster that the slots of this TM are assigned

Re: How does JobManager terminate dangling task manager

Hi, In fact, not only JobManager(ResoruceManager) will kill TimeOut's TaskManager, but if TaskManager finds that it cannot connect to JobManager(ResourceManager), it will also exit by itself. You can look at the time period during which the HB timeout occurred and what happened in the log. Under no

Re: Nested match_recognize query not supported in SQL ?

By the way, views that use MATCH_RECOGNIZE don't work in Flink 1.11. [1] [1] https://issues.apache.org/jira/browse/FLINK-20077 On Thu, May 13, 2021 at 11:06 AM David Anderson wrote: > I was able to get something like this working, but only by introducing a > view: > > CREATE TEMPORARY VIEW mmm

Re: Nested match_recognize query not supported in SQL ?

I was able to get something like this working, but only by introducing a view: CREATE TEMPORARY VIEW mmm AS SELECT id FROM events MATCH_RECOGNIZE (...); SELECT * FROM event WHERE id IN (SELECT id FROM mmm); Regards, David On Tue, May 11, 2021 at 9:22 PM Tejas wrote: > Hi, > I am using flink 1

Re: Regarding Stateful Functions

Hi Austin, Thanks for your insights. We are currently following a microservice architecture for accomplishing our data processing requirements. We are planning to use Flink as our unified platform for all data processing tasks. Although most of our use cases are a suitable fit for Flink, there