Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-07 Thread Dong Lin
Hi Qingsheng, Thanks for the comment! After double checking the solution of passing recordEvaluator via env.fromSource(...), I realized that this approach does not automatically bring the feature for all sources. We would still need to update the implementation of every source in order to pass re

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread David Morávek
+1 for the separate repository under the Flink umbrella as we've already started creating more repositories with connectors, would it be possible to re-use the same build infrastructure for this one? (eg. shared set of Gradle plugins that unify the build experience)? Best, D. On Fri, Jan 7, 2022

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-07 Thread Dong Lin
Hi Martijn, Thanks for the comments! In general I agree we should avoid feature sparsity. In this particular case, connectors are a bit different than most other features in Flink. AFAIK, we plan to move connectors (including Kafka and Pulsar) out of the Flink project in the future, which means t

[jira] [Created] (FLINK-25578) Graduate Sink V1 interfaces to PublicEvolving and deprecate them

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25578: --- Summary: Graduate Sink V1 interfaces to PublicEvolving and deprecate them Key: FLINK-25578 URL: https://issues.apache.org/jira/browse/FLINK-25578 Project: Flink

[jira] [Created] (FLINK-25577) Update GCS documentation not to use no longer supported flink-shaded-hadoop artifacts

2022-01-07 Thread Jira
David Morávek created FLINK-25577: - Summary: Update GCS documentation not to use no longer supported flink-shaded-hadoop artifacts Key: FLINK-25577 URL: https://issues.apache.org/jira/browse/FLINK-25577

[jira] [Created] (FLINK-25576) Update com.h2database:h2 to 2.0.206

2022-01-07 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-25576: -- Summary: Update com.h2database:h2 to 2.0.206 Key: FLINK-25576 URL: https://issues.apache.org/jira/browse/FLINK-25576 Project: Flink Issue Type: Technical

Re: [ANNOUNCE] Apache Flink ML 2.0.0 released

2022-01-07 Thread David Morávek
Great job! <3 Thanks Dong and Yun for managing the release and big thanks to everyone who has contributed! Best, D. On Fri, Jan 7, 2022 at 2:27 PM Yun Gao wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink ML 2.0.0. > > > > Apache Flink ML provides API a

[ANNOUNCE] Apache Flink ML 2.0.0 released

2022-01-07 Thread Yun Gao
The Apache Flink community is very happy to announce the release of Apache Flink ML 2.0.0. Apache Flink ML provides API and infrastructure that simplifies implementing distributed ML algorithms, and it also provides a library of off-the-shelf ML algorithms. Please check out the release blog po

[jira] [Created] (FLINK-25575) Implement StreamGraph translation for Sink V2 interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25575: --- Summary: Implement StreamGraph translation for Sink V2 interfaces Key: FLINK-25575 URL: https://issues.apache.org/jira/browse/FLINK-25575 Project: Flink Issue

[jira] [Created] (FLINK-25574) Update Async Sink to use decomposed interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25574: --- Summary: Update Async Sink to use decomposed interfaces Key: FLINK-25574 URL: https://issues.apache.org/jira/browse/FLINK-25574 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-25573) Update Kafka Sink to use decomposed interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25573: --- Summary: Update Kafka Sink to use decomposed interfaces Key: FLINK-25573 URL: https://issues.apache.org/jira/browse/FLINK-25573 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-25572) Update File Sink to use decomposed interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25572: --- Summary: Update File Sink to use decomposed interfaces Key: FLINK-25572 URL: https://issues.apache.org/jira/browse/FLINK-25572 Project: Flink Issue Type: Sub-t

[jira] [Created] (FLINK-25571) Update Elasticsearch Sink to use decomposed interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25571: --- Summary: Update Elasticsearch Sink to use decomposed interfaces Key: FLINK-25571 URL: https://issues.apache.org/jira/browse/FLINK-25571 Project: Flink Issue Ty

[jira] [Created] (FLINK-25570) Introduce Sink V2 extension APIs

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25570: --- Summary: Introduce Sink V2 extension APIs Key: FLINK-25570 URL: https://issues.apache.org/jira/browse/FLINK-25570 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25569) Introduce decomposed Sink V2 interfaces

2022-01-07 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25569: --- Summary: Introduce decomposed Sink V2 interfaces Key: FLINK-25569 URL: https://issues.apache.org/jira/browse/FLINK-25569 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25568) Add Elasticsearch 7 Source Connector

2022-01-07 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-25568: Summary: Add Elasticsearch 7 Source Connector Key: FLINK-25568 URL: https://issues.apache.org/jira/browse/FLINK-25568 Project: Flink Issue Type: New

[RESULT][VOTE] FLIP-191: Extend unified Sink interface to support small file compaction

2022-01-07 Thread Fabian Paul
I am happy to announce that FLIP-191 [1] has been accepted by this vote [2]. There are 5 approving votes, 3 of which are binding: * Martijn Visser (non-binding) * Yun Gao (binding) * Arvid Heise (binding) * Guowei Ma (binding) * Jing Ge (non-binding) There are no disapproving votes. Thanks ever

[jira] [Created] (FLINK-25567) Casting of Multisets to Multisets

2022-01-07 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-25567: --- Summary: Casting of Multisets to Multisets Key: FLINK-25567 URL: https://issues.apache.org/jira/browse/FLINK-25567 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25566) Fail to cancel task if disk is bad for java.lang.NoClassDefFoundError

2022-01-07 Thread Liu (Jira)
Liu created FLINK-25566: --- Summary: Fail to cancel task if disk is bad for java.lang.NoClassDefFoundError Key: FLINK-25566 URL: https://issues.apache.org/jira/browse/FLINK-25566 Project: Flink Issue Ty

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread Jingsong Li
For more references on `store` and `storage`: For example, Rocksdb is a library that provides an embeddable, persistent key-value store for fast storage. [1] Apache HBase [1] is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable. [2] [1] https://github.

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread Jingsong Li
Thanks all, Combining everyone's comments, I recommend using `flink-table-store`: ## table something to do with table storage (From Till). Not only flink-table, but also for user-oriented tables. ## store vs storage - The first point I think, store is better pronounced, storage is three syllable

[jira] [Created] (FLINK-25565) Write and Read Parquet INT64 Timestamp

2022-01-07 Thread Bo Cui (Jira)
Bo Cui created FLINK-25565: -- Summary: Write and Read Parquet INT64 Timestamp Key: FLINK-25565 URL: https://issues.apache.org/jira/browse/FLINK-25565 Project: Flink Issue Type: New Feature

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread Konstantin Knauf
+1 to a separate repository assuming this repository will still be part of Apache Flink (same PMC, Committers). I am not aware we have something like "sub-projects" officially. I share Till and Timo's concerns regarding "store". On Fri, Jan 7, 2022 at 9:59 AM Till Rohrmann wrote: > +1 for the s

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread Till Rohrmann
+1 for the separate project. I would agree that flink-store is not the best name. flink-storage > flink-store but I would even more prefer a name that conveys that it has something to do with table storage. Cheers, Till On Fri, Jan 7, 2022 at 9:14 AM Timo Walther wrote: > +1 for the separate p

[DISCUSS] FLIP-209: Support to run multiple shuffle plugins in one session cluster

2022-01-07 Thread Yingjie Cao
Hi dev, I'd like to start a discussion for FLIP-209 [1] which aims to support to run multiple shuffle plugins in one session cluster. Currently, one Flink cluster can only use one shuffle service plugin configured by 'shuffle-service-factory.class'. It is not flexible enough and cannot support use

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

2022-01-07 Thread Francesco Guardiani
+1 with a separate repo and +1 with the flink-storage name On Fri, Jan 7, 2022 at 8:40 AM Jingsong Li wrote: > Hi everyone, > > Vote for create a separate sub project for FLIP-188 thread is here: > https://lists.apache.org/thread/wzzhr27cvrh6w107bn464m1m1ycfll1z > > Best, > Jingsong > > > On Fri

[jira] [Created] (FLINK-25564) TaskManagerProcessFailureStreamingRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure fails on AZP

2022-01-07 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25564: - Summary: TaskManagerProcessFailureStreamingRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure fails on AZP Key: FLINK-25564 URL: https:/

[NOTICE] Introduction of flink-table-test-utils

2022-01-07 Thread Timo Walther
Hi everyone, for the Scala-free planner effort we introduced a new `flink-table-test-utils`. It is recommended that connectors, formats, and catalogs use this module as a test dependency if table-common is not enough. We will fill this module with more test utilities that can also be used b

Re: [DISCUSS] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-07 Thread Xingbo Huang
Hi Till, I have written a more complicated PyFlink job. Compared with the previous single python udf job, there is an extra stage of converting between table and datastream. Besides, I added a python map function for the job. Because python datastream has not yet implemented Thread mode, the pytho

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

2022-01-07 Thread Timo Walther
+1 for the separate project But maybe use `flink-storage` instead of `flink-store`? I'm not a native speaker but store is defined as "A place where items may be purchased.". It almost sounds like the `flink-packages` project. Regards, Timo On 07.01.22 08:37, Jingsong Li wrote: Hi everyone,