[jira] [Created] (FLINK-32593) DelimitedInputFormat will cause record loss for multi-bytes delimit when a delimit is seperated to two splits

2023-07-14 Thread Zhaofu Liu (Jira)
Zhaofu Liu created FLINK-32593: -- Summary: DelimitedInputFormat will cause record loss for multi-bytes delimit when a delimit is seperated to two splits Key: FLINK-32593 URL: https://issues.apache.org/jira/browse/FLIN

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-14 Thread Piotr Nowojski
Hi All, We had a lot of off-line discussions. As a result I would suggest dropping the idea of introducing an end-to-end-latency concept, until we can properly implement it, which will require more designing and experimenting. I would suggest starting with a more manual solution, where the user ne

Re: [VOTE] Graduate the FileSink to @PublicEvolving

2023-07-14 Thread Jing Ge
Hi, After talking with Dong offline, it turns out @PublicEvolving are intended for public use and should have stable behavior. FileSink is not ready for that according to the data loss issue raised in the discussion [1]. This vote is cancelled. Best regards, Jing [1] https://lists.apache.org/th

[DISCUSS] Connectors, Formats, and even User Code should also be pluggable.

2023-07-14 Thread zhiqiang li
Hi devs, I have observed that in [1], connectors and formats are pluggable, allowing user code to be easily integrated. The advantages of having pluggable connectors are evident, as it helps avoid conflicts between different versions of jar packages. If classloader isolation is not used, shadin

Issue with flink 1.16 and hive dialect

2023-07-14 Thread ramkrishna vasudevan
Hi All, I am not sure if this was already discussed in this forum. In our set up with 1.16.0 flink we have ensured that the setup has all the necessary things for Hive catalog to work. The flink dialect works fine functionally (with some issues will come to that later). But when i follow the ste

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

2023-07-14 Thread Yunfeng Zhou
Hi Matt, 1. I tried to add back the tag serialization process back to my POC code and run the benchmark again, this time the performance improvement is roughly reduced by half. It seems that both the serialization and the judgement process have a major contribution to the overhead reduction in the

[jira] [Created] (FLINK-32592) Mixed-up job execution on concurrent job submission

2023-07-14 Thread Fabio Wanner (Jira)
Fabio Wanner created FLINK-32592: Summary: Mixed-up job execution on concurrent job submission Key: FLINK-32592 URL: https://issues.apache.org/jira/browse/FLINK-32592 Project: Flink Issue Typ

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler. Second vote.

2023-07-14 Thread Prabhu Joseph
*+1 (non-binding)* Thanks for working on this. We have seen good improvement during the cool down period with this feature. Below are details on the test results from one of our clusters: On a scale-out operation, 8 new nodes were added one by one with a gap of ~30 seconds. There were 8 restarts

[jira] [Created] (FLINK-32591) Update document of Kafka Source: Enable Dynamic Partition Discovery by Default in Kafka Source

2023-07-14 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-32591: - Summary: Update document of Kafka Source: Enable Dynamic Partition Discovery by Default in Kafka Source Key: FLINK-32591 URL: https://issues.apache.org/jira/browse/FLINK-32591

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

2023-07-14 Thread Matthias Pohl
> > I'm not aware of any decision that has already been made by the community > regarding after which 1.x minor release will we ship the 2.0 major release. > I personally don't think it's feasible to avoid 1.19, and even 1.20 > depending on the progress. Ok, thanks. That clarification helped. I h