Announcing public preview of Flink with Azure HDInsight on AKS

2023-10-10 Thread ramkrishna vasudevan
We are super excited to announce the public preview of Flink on Azure, as a managed platform as a service. You can read more information on this blog and reach out to me!

Re: Continuous errors with Azure ABFSS

2023-10-05 Thread ramkrishna vasudevan
Sorry for the late reply. Just in case you restart the job , is it able to safely use the checkpoint and get back to the checkpointed state? Regards Ram, On Thu, Sep 28, 2023 at 4:46 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Surendra, > > there are no exceptions in the

Re: Continuous errors with Azure ABFSS

2023-09-27 Thread ramkrishna vasudevan
Can you help with more info here? The RocksDB backend itself is in ABFS instead of local? Or you mean the checkpoint is in ABFS but local dir for RocksDB is in local storage? GetPathSTatus is done by your monitoring pages? We run Flink on ABFS so we would like to see if we can help you out.

Re: Issue with flink 1.16 and hive dialect

2023-07-16 Thread ramkrishna vasudevan
cy.kind'='metastore,success-file'); > > > [1] https://issues.apache.org/jira/browse/FLINK-27450 > [2] https://issues.apache.org/jira/browse/HIVE-21584 > [3] https://issues.apache.org/jira/browse/FLINK-32596 > > > Best regards, > Yuxia > > --

Issue with flink 1.16 and hive dialect

2023-07-14 Thread ramkrishna vasudevan
Hi All, I am not sure if this was already discussed in this forum. In our set up with 1.16.0 flink we have ensured that the setup has all the necessary things for Hive catalog to work. The flink dialect works fine functionally (with some issues will come to that later). But when i follow the

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread ramkrishna vasudevan
Congratulations !!! On Mon, Mar 27, 2023 at 2:54 PM Yu Li wrote: > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new project still

Re: CSV File Sink in Streaming Use Case

2023-03-07 Thread ramkrishna vasudevan
Hi all, One thing to note is that, the CSVBulkReader does not support the splittable property. Previously with TextInputFormat we were able to use the block size to split them, but in Streaming world this is not there. Regards Ram On Wed, Mar 8, 2023 at 7:22 AM yuxia wrote: > Hi, as the doc

Physical memory and Auto Memory settings in TM

2022-11-23 Thread ramkrishna vasudevan
Hi All, Installation of flink clusters where we govern the memory configs by specifying 'taskmanager.memory.process.size' TM memory is auto calculated. But in this calculation we generally don't check the actual physical memory available. So in case of a wrong configuration where the JVM heap

Re: Jobmanager fails to come up if the job has an issue

2022-09-26 Thread ramkrishna vasudevan
> prevent the JobManager from recovering this specific job. Keep in mind that > you have to clean up any job-related data by yourself in that case. > > I hope that helps. > Matthias > > [1] https://issues.apache.org/jira/browse/FLINK-9097 > > On Mon, Sep 26, 2022 at 12:26 P

Re: Jobmanager fails to come up if the job has an issue

2022-09-26 Thread ramkrishna vasudevan
I got some logs and stack traces from our backend storage. This is not the entire log though. Can this be useful? With these set of logs messages the job manager kept restarting. Regards Ram On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan < ramvasu.fl...@gmail.com> wrote: > Thank

Re: Jobmanager fails to come up if the job has an issue

2022-09-26 Thread ramkrishna vasudevan
as > > On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan < > ramvasu.fl...@gmail.com> wrote: > >> Hi all >> >> I have a simple job where we read for a given path in cloud storage to >> watch for new files in a given fodler. While I setup my job there

Jobmanager fails to come up if the job has an issue

2022-09-25 Thread ramkrishna vasudevan
Hi all I have a simple job where we read for a given path in cloud storage to watch for new files in a given fodler. While I setup my job there was some permission issue on the folder. The job is STREAMING job. The cluster is set in the session mode and is running on Kubernetes. The job manager