[jira] [Created] (FLINK-33761) Snowflake as JDBC source

Boris Litvak created FLINK-33761: Summary: Snowflake as JDBC source Key: FLINK-33761 URL: https://issues.apache.org/jira/browse/FLINK-33761 Project: Flink Issue Type: Bug

Distributed cache support in async io

Hi all, I asked this question before in user mail list but got no response. Just try in this list if I could get some answer here :-). I noticed distributed cache is not supported in async io, is there any reason for that? It's understandable that state apis are not supported since the data

[jira] [Created] (FLINK-33760) Group Window agg has different result when only consuming -D records while using or not using minibatch

xuyang created FLINK-33760: -- Summary: Group Window agg has different result when only consuming -D records while using or not using minibatch Key: FLINK-33760 URL: https://issues.apache.org/jira/browse/FLINK-33760

[jira] [Created] (FLINK-33759) flink parquet writer support write nested array or map type

Cai Liuyang created FLINK-33759: --- Summary: flink parquet writer support write nested array or map type Key: FLINK-33759 URL: https://issues.apache.org/jira/browse/FLINK-33759 Project: Flink

[RESULT][VOTE] FLIP-379: Dynamic source parallelism inference for batch jobs

Dear developers, FLIP-379: Dynamic source parallelism inference for batch jobs[1] has been accepted and voted through this thread [2]. The proposal received 6 approving binding votes and there is no disapproval: - Zhu Zhu (binding) - Lijie Wang (binding) - Rui Fan (binding) - Etienne Chauchot

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

I apologize, I was a little off with my description, it's been a while since I have looked at this code but I have refreshed myself. The line I referred to earlier was correct though. This operator only processes records in a file split while the operator is idle, meaning there are no more

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

They way I understand this loop is that the ContinuiousFileReaderOperator only processes records in the background while the operator is idle, i.e. while it's not receiving any records. At the very bottom of that loop here

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Hi Rui, Sorry for the late reply. I was suggesting that perhaps we could do some testing with Kubernetes wrt configuring values for the exponential restart strategy. We've noticed that the default strategy in 1.17 caused a lot of requests to the K8s API server for unstable deployments. However,

[jira] [Created] (FLINK-33758) Implement restore tests for TemporalSort node

Jim Hughes created FLINK-33758: -- Summary: Implement restore tests for TemporalSort node Key: FLINK-33758 URL: https://issues.apache.org/jira/browse/FLINK-33758 Project: Flink Issue Type:

[jira] [Created] (FLINK-33757) Implement restore tests for Rank node

Jim Hughes created FLINK-33757: -- Summary: Implement restore tests for Rank node Key: FLINK-33757 URL: https://issues.apache.org/jira/browse/FLINK-33757 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33756) Missing record with CUMULATE/HOP windows using an optimization

Jim Hughes created FLINK-33756: -- Summary: Missing record with CUMULATE/HOP windows using an optimization Key: FLINK-33756 URL: https://issues.apache.org/jira/browse/FLINK-33756 Project: Flink

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

Thanks, Peter. Given the discussion I also agree that the consensus is to move towards the mixin interface approach (and accept its disadvantages given its advantages). +1 for the general direction of your proposed code change in https://github.com/apache/flink/pull/23876. On Tue, Dec 5, 2023 at

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

This is the loop - code reference , where it fetches all records from the split, and then only the MailboxProcessor gets control to

Re: [DISCUSS] Release flink-connector-parent v1.01

Hi Péter, My answers are inline Best Etienne Le 05/12/2023 à 05:27, Péter Váry a écrit : Hi Etienne, Which branch would you cut the release from? the parent_pom branch (consisting of a single maven pom file) I find the flink-connector-parent branches confusing. If I merge a PR to the

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

I thought for sure this was already the existing behavior with this operator. Does it not check the mailbox executor after every record read? On Tue, Dec 5, 2023 at 6:48 AM Prabhu Joseph (Jira) wrote: > Prabhu Joseph created FLINK-33753: > - > >

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

Hi Paul, I believe Xintong has answered your question. >> IIUC, in the FLIP, the main method is lost after the recovery, and only submitted jobs would be recovered. Is that right? You are right, we can't recover the execution progress of main method. So after JM crashs, only the submitted and

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

It seems to me we have a consensus to move forward with the mixin approach. I hope that everyone is aware that with the mixin interfaces we lose the opportunity of the strong type checks. This will be especially painful for generic types, where we will not have a way to ensure that the generic

[jira] [Created] (FLINK-33755) Cleanup usage of deprecated StreamExecutionEnvironment#generateSequence

Sergey Nuyanzin created FLINK-33755: --- Summary: Cleanup usage of deprecated StreamExecutionEnvironment#generateSequence Key: FLINK-33755 URL: https://issues.apache.org/jira/browse/FLINK-33755

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Thank you, Gyula, we are working on validate setting larger taskmanager.memory.jvm-overhead.fraction to ease this problem, and on the other side, we try to find a way in deployment path to ease this problem. I agree with you proposal, may be I could find sometime to make a pr for FLINK-33548

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

I understand your problem but I think you are trying to find a solution in the wrong place. Have you tried setting taskmanager.memory.jvm-overhead.fraction ? That would reserve more memory from the total process memory for non-JVM use. Gyula On Tue, Dec 5, 2023 at 1:50 PM richard.su wrote: >

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi Richard, Shouldn't the solution then be solving the glibc problem? Best regards, Martijn On Tue, Dec 5, 2023 at 1:49 PM richard.su wrote: > > Sorry, "To be clear, we need a container has memory larger than request, and > confirm this pod has Guarantee Qos." which need to be "To be clear,

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Sorry, "To be clear, we need a container has memory larger than request, and confirm this pod has Guarantee Qos." which need to be "To be clear, we need a container has memory larger than process.size, and confirm this pod has Guarantee Qos." Thanks. Richard Su > 2023年12月5日 20:47，richard.su

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi, Gyula, yes, this is a special case in our scenarios, sorry about that it's hard to understand, which we want to reserved some memory beyond the jobmanager or task manager's process.To be clear, we need a container has memory larger than request, and confirm this pod has Guarantee Qos.

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Richard, I still don't understand why the current setup doesn't work for you. According to https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/memory/mem_setup/ : The process memory config (which is what we configure) translates directly into the container request size.

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

I think the new configuration could be : "kubernetes.taskmanager.memory.amount" and "kubernetes.jobmanager.memory.amout" once we can calculate the limit-factor by the different of requests and limits. when native mode, we no longer check the process.size as default memory, but using this

Re: Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

Hi Julian, Could you please remove the duplicated "RE:" in the topic of the reply? That way we can continue this discussion to the original thread. (Apache deals with it correctly, but not all email clients/services do, e.g. GMail) Thanks, Alex On Tue, 5 Dec 2023 at 09:39, Payne, Julian wrote:

[jira] [Created] (FLINK-33754) Serialize QueryOperations into SQL

Dawid Wysakowicz created FLINK-33754: Summary: Serialize QueryOperations into SQL Key: FLINK-33754 URL: https://issues.apache.org/jira/browse/FLINK-33754 Project: Flink Issue Type:

[jira] [Created] (FLINK-33752) When Duration is greater than or equal to 1 day, the display unit is ms.

Rui Fan created FLINK-33752: --- Summary: When Duration is greater than or equal to 1 day, the display unit is ms. Key: FLINK-33752 URL: https://issues.apache.org/jira/browse/FLINK-33752 Project: Flink

[jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

Prabhu Joseph created FLINK-33753: - Summary: ContinuousFileReaderOperator consume records as mini batch Key: FLINK-33753 URL: https://issues.apache.org/jira/browse/FLINK-33753 Project: Flink

[jira] [Created] (FLINK-33751) use modules correctly when deserializing json plan

shuaiqi.guo created FLINK-33751: --- Summary: use modules correctly when deserializing json plan Key: FLINK-33751 URL: https://issues.apache.org/jira/browse/FLINK-33751 Project: Flink Issue Type:

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi, Gyula, from my opinion, this still will using flinkDeployment's resource filed to set jobManager.memory.process.size, and I have told an uncovered case that: When user wants to define a flinkdeployment with jobmanager has 1G memory resources in container field but config

[FLINK-32028] connectors/elasticsearch: error handling

Hi! We're using flink and its elasticsearch (ES) sink to process content changes at wikimedia. The connector uses ES' bulk API but is rather strict when it comes to interpreting a response: It fails if a single action of the bulk has failed. This behavior is not configurable at the moment so I

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

@Paul, Do you mean the scenario where users call `evn.execute()` multiple times in the `main()` method? I believe that is not supported currently when HA is enabled, for the exact same reason you mentioned that Flink is not aware of which jobs are executed and which are not. On the other hand,

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

This is the proposal according to FLINK-33548: spec: taskManager: resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" I honestly think this is much more intuitive and easier than using the podTemplate, which is

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi Gyula, FLINK-33548 proposes adding a new resource field to match with Kubernetes pod resource configuration. Here's my suggestion: instead of adding a new resource field, let's use a pod template for more advanced resource setup. Adding a new resource field might confuse users. This change can

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

Thanks for addressing my comments, Lijie. LGTM Best, Xintong On Tue, Dec 5, 2023 at 2:56 PM Paul Lam wrote: > Hi Lijie, > > Recovery for batch jobs is no doubt a long-awaited feature. Thanks for > the proposal! > > I’m concerned about the multi-job scenario. In session mode, users could >

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Sorry Gyula, let me explain more about the point of 2, if I avoid the override, I will got a jobmanager pod still with resources consist with “jobmanager.memory.process.size”, but a flinkdeployment with a resource larger than that. Thanks for your time. Richard Su > 2023年12月5日

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Thank you for your time, Gyula, I have more question about Flink-33548, we can have more discussion about this and make progress: 1. I agree with you about declaring resources in FlinkDeployment resource sections. But Flink Operator will override the “jobmanager.memory.process.size” and

Re: Subscribe Apache Flink development email.

Hi, aaron. If you want to subscribe the dev mail list, you need to send an e-mail to dev-subscr...@flink.apache.org . See more in [1]. Mailing list could be found here[2]. Best, Hang [1] https://flink.apache.org/what-is-flink/community/#how-to-subscribe-to-a-mailing-list [2]

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

As you can see in the jira ticket there hasn't been any progress, nobody started to work on this yet. I personally don't think it's confusing to declare resources in the FlinkDeployment resource sections. It's well documented and worked very well so far for most users. This is pretty common

RE: Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

Hey all, Thanks for this proposal, I think it makes a lot of sense. I am, curious to know what time horizon we would consider for LTS of 1.x. Customers value knowing when versions will deprecate so they can build migration into their planning and resourcing cycles, so I would be in favour of

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi, Gyula, is there had any progress in FLINK-33548? I would like to join the discussion but I haven't seen any discussion in the url. I also make flinkdeployment by flink operator, which indeed will override the process size by TaskmanagerSpec.resources or JobmanagerSpec.resources, which

Re: Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hi! Please see the discussion in https://lists.apache.org/thread/6p5tk6obmk1qxf169so498z4vk8cg969 and the ticket: https://issues.apache.org/jira/browse/FLINK-33548 We should follow the approach outlined there. If you are interested you are welcome to pick up the operator ticket. Unfortunately

Discussion: [FLINK-24150] Support to configure cpu resource request and limit in pod template

Hello everyone, I've encountered an issue while using flink kubernetes native, Despite setting resource limits in the pod template, it appears that these limits and requests are not considered during JobManager(JM) and TaskManager (TM) pod deployment. I find the a issue had opened in jira