Re: Task Manager restart and RocksDB incremental checkpoints issue.

2022-11-10 Thread Yanfei Lei
Hi Vidya Sagar, Could you please share the reason for TaskManager restart? If the machine or JVM process of TaskManager crashes, the `RocksDBKeyedStateBackend` can't be disposed/closed normally, so the existing rocksdb instance directory would remain. BTW, if you use Application Mode on k8s, if

Re: Any caveats about processing abstract classes ?

2022-11-10 Thread Gen Luo
Hi Davide, I suppose it would be fine. The only difference I can figure out that may matter is the serialization. Flink uses KryoSerializer as the fallback serializer if the TypeInformation of the records is not provided, which can properly process abstract classes. This works well in most cases.

Spark Scala Contract Opportunity @USA

2022-11-10 Thread sri hari kali charan Tummala
Hi All, Is anyone looking for a spark scala contract role inside the USA? A company called Maxonic has an open spark scala contract position (100% remote) inside the USA if anyone is interested, please send your CV to kali.tumm...@gmail.com. Thanks & Regards Sri Tummala

Task Manager restart and RocksDB incremental checkpoints issue.

2022-11-10 Thread Vidya Sagar Mula
Hi, I am using RocksDB state backend for incremental checkpointing with Flink 1.11 version. Question: -- For a given Job ID, Intermediate RocksDB checkpoints are stored under the path defined with "" The files are stored with "_jobID+ radom UUID" prefixed to the location. Case : 1 -

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

2022-11-10 Thread Ryan Skraba via user
Excellent news -- welcome to the new era of easier, more timely and more feature-rich releases for everyone! Great job! Ryan On Thu, Nov 10, 2022 at 3:15 PM Leonard Xu wrote: > Thanks Chesnay and Martijn for the great work! I believe the > flink-connector-shared-utils[1] you built will help

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

2022-11-10 Thread Leonard Xu
Thanks Chesnay and Martijn for the great work! I believe the flink-connector-shared-utils[1] you built will help Flink connector developers a lot. Best, Leonard [1] https://github.com/apache/flink-connector-shared-utils > 2022年11月10日 下午9:53,Martijn Visser 写道: > > Really happy with the firs

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

2022-11-10 Thread Martijn Visser
Really happy with the first externalized connector for Flink. Thanks a lot to all of you involved! On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticsearch Connector 3.0.0. > > Apache Flink® is an

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

2022-11-10 Thread Jing Ge
Hi Etienne, Nice blog! Thanks for sharing! Best regards, Jing On Wed, Nov 9, 2022 at 5:49 PM Etienne Chauchot wrote: > Hi Yun Gao, > > FYI I just updated the article after your review: > https://echauchot.blogspot.com/2022/11/flink-howto-migrate-real-life-batch.html > > Best > > Etienne > Le

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

2022-11-10 Thread Chesnay Schepler
The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available f