Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

2021-07-01 Thread Sonam Mandal
Hi Stephan, Thanks for the detailed explanation! This really helps understand all this better. Appreciate your help! Regards, Sonam From: Stephan Ewen Sent: Wednesday, June 30, 2021 3:56:22 AM To: Sonam Mandal ; user@flink.apache.org Cc:

Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

2021-06-30 Thread Stephan Ewen
Hi Sonam! To answer this, let me quickly provide some background on the two ways flink deployments / job submissions work. See also here for some background: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/overview/#deployment-modes What is common in all setups is

Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

2021-06-29 Thread Sonam Mandal
Hi Matthias, Thanks for getting back to me. We are trying to build a system where users can focus on writing Flink SQL applications and we handle the full lifecycle of their Flink cluster and job. We would like to let users focus on just their SQL and UDF logic. In such an environment, we

Re: Recommended way to submit a SQL job via code without getting tied to a Flink version?

2021-06-29 Thread Matthias Pohl
Hi Sonam, what's the reason for not using the Flink SQL client? Because of the version issue? I only know that FlinkSQL's state is not backwards-compatible between major Flink versions [1]. But that seems to be unrelated to what you describe. I'm gonna add Jark and Timo to this thread. Maybe,

Recommended way to submit a SQL job via code without getting tied to a Flink version?

2021-06-22 Thread Sonam Mandal
Hello, We've written a simple tool which takes SQL statements as input and uses a StreamTableEnvironment​ to eventually submit this to the Flink cluster. We've noticed that the Flink library versions we depend on must match the Flink version running in our Kubernetes cluster for the job