[jira] [Created] (FLINK-16859) Introduce FileSystemTableFactory, FileSystemTableSource, FileSystemTableSink

2020-03-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-16859: Summary: Introduce FileSystemTableFactory, FileSystemTableSource, FileSystemTableSink Key: FLINK-16859 URL: https://issues.apache.org/jira/browse/FLINK-16859

[jira] [Created] (FLINK-16858) Expose partitioned by grammar

2020-03-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-16858: Summary: Expose partitioned by grammar Key: FLINK-16858 URL: https://issues.apache.org/jira/browse/FLINK-16858 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-16857) Support partition prune for PartitionableTableSource with getPartitions

2020-03-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-16857: Summary: Support partition prune for PartitionableTableSource with getPartitions Key: FLINK-16857 URL: https://issues.apache.org/jira/browse/FLINK-16857 Project:

[jira] [Created] (FLINK-16856) taskmanager.sh is broken because of invalid arguments of head

2020-03-29 Thread Yang Wang (Jira)
Yang Wang created FLINK-16856: - Summary: taskmanager.sh is broken because of invalid arguments of head Key: FLINK-16856 URL: https://issues.apache.org/jira/browse/FLINK-16856 Project: Flink

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Jingsong Li
Hi all, I'm glad to announce that the voting of FLIP-115 has passed, with 5 +1 (3 binding: Jingsong, Kurt, Jark, 2 non-binding: Benchao, Leonard) and no -1. Thanks everyone for participating! Best, Jingsong Lee On Mon, Mar 30, 2020 at 12:47 PM Jark Wu wrote: > +1 (binding) > > Best, > Jark

[jira] [Created] (FLINK-16855) Update K8s template for the standalone setup

2020-03-29 Thread Canbin Zheng (Jira)
Canbin Zheng created FLINK-16855: Summary: Update K8s template for the standalone setup Key: FLINK-16855 URL: https://issues.apache.org/jira/browse/FLINK-16855 Project: Flink Issue Type:

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Jark Wu
+1 (binding) Best, Jark On Mon, 30 Mar 2020 at 11:32, Jingsong Li wrote: > +1 (binding) > > Best, > Jingsong Lee > > On Sun, Mar 29, 2020 at 10:37 PM Benchao Li wrote: > > > +1 (non-binding) > > > > This feature will greatly enrich Flink SQL's ecology~ > > > > Leonard Xu 于2020年3月29日周日

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Tzu-Li (Gordon) Tai
Thanks. The licensing issues that Hequn found are fixed in RC4: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-Release-2-0-0-release-candidate-4-td39453.html This vote thread is hereby cancelled. Since the new RC4 only fixes licensing issues

[VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #4

2020-03-29 Thread Tzu-Li (Gordon) Tai
Hi everyone, Please review and vote on the *release candidate #4* for the version 2.0.0 of Apache Flink Stateful Functions, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Testing Guideline** You can find here [1] a doc that we can

Re: [DISCUSS] FLIP-108: Add GPU support in Flink

2020-03-29 Thread Yangze Guo
Hi, Stephan, I see your concern and I totally agree with you. The interface on RM side is now `Map getYarn/KubernetesExternalResource()`. The only valid information RM get from it is the configuration key of that external resource in Yarn/K8s. The "String/Long value" would be the same as the

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Jingsong Li
+1 (binding) Best, Jingsong Lee On Sun, Mar 29, 2020 at 10:37 PM Benchao Li wrote: > +1 (non-binding) > > This feature will greatly enrich Flink SQL's ecology~ > > Leonard Xu 于2020年3月29日周日 下午9:44写道: > > > +1 > > > > Best, > > Leonard > > > > > 在 2020年3月28日,16:25,Kurt Young 写道: > > > > > > +1

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

2020-03-29 Thread Yang Wang
Hi Konstantin, I think it is a good idea. Currently, our users also report a similar issue with resourceId of standalone cluster. When we start a standalone cluster now, the `TaskManagerRunner` always generates a uuid for the resourceId. It will be used to register to the jobmanager and not

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Hequn Cheng
Hi, @Gordon I have created the corresponding JIRAs and PRs for the license problems: - https://issues.apache.org/jira/browse/FLINK-16853 - https://issues.apache.org/jira/browse/FLINK-16854 @Igal @Gordon, Thanks a lot for the confirmation of the package name, that addresses my concerns now.

[jira] [Created] (FLINK-16854) Correct dependency versions in the NOTICE file of module statefun-ridesharing-example-simulator

2020-03-29 Thread Hequn Cheng (Jira)
Hequn Cheng created FLINK-16854: --- Summary: Correct dependency versions in the NOTICE file of module statefun-ridesharing-example-simulator Key: FLINK-16854 URL: https://issues.apache.org/jira/browse/FLINK-16854

Re: [DISCUSS] FLIP-119: Pipelined Region Scheduling

2020-03-29 Thread Yangze Guo
Hi, Zhu Zhu, Thanks for the explanation and information. I also agree that it out of the scope of this FLIP and glad to see that we are moving forward to a more extensible scheduler! Best, Yangze Guo On Mon, Mar 30, 2020 at 1:31 AM Zhu Zhu wrote: > > Thanks for the comments! > > To Xintong, >

Re: [DISCUSS] FLIP-119: Pipelined Region Scheduling

2020-03-29 Thread Xintong Song
@ZhuZhu The links work for me now. Someone might have fixed them. Never mind. Thank you~ Xintong Song On Mon, Mar 30, 2020 at 1:31 AM Zhu Zhu wrote: > Thanks for the comments! > > To Xintong, > It's a bit strange since the in page links work as expected. Would you take > another try? > >

[jira] [Created] (FLINK-16853) Update the version of protobuf-java from 3.8.0 to 3.7.1 in the NOTICE file of module statefun-flink-distribution

2020-03-29 Thread Hequn Cheng (Jira)
Hequn Cheng created FLINK-16853: --- Summary: Update the version of protobuf-java from 3.8.0 to 3.7.1 in the NOTICE file of module statefun-flink-distribution Key: FLINK-16853 URL:

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-03-29 Thread Jacky Lau
thanks Gyula Fóra. i have read it. And i think it is lark of flink catalog info, which you can see spark atlas project here https://github.com/hortonworks-spark/spark-atlas-connector -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-16852) Add metrics to the source coordinator.

2020-03-29 Thread Jiangjie Qin (Jira)
Jiangjie Qin created FLINK-16852: Summary: Add metrics to the source coordinator. Key: FLINK-16852 URL: https://issues.apache.org/jira/browse/FLINK-16852 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-16851) Add common metrics to the SourceReader base implementation.

2020-03-29 Thread Jiangjie Qin (Jira)
Jiangjie Qin created FLINK-16851: Summary: Add common metrics to the SourceReader base implementation. Key: FLINK-16851 URL: https://issues.apache.org/jira/browse/FLINK-16851 Project: Flink

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Konstantin Knauf
Hi Gordon, +1 (non-binding) * built from sources...check * build python SDK from sources...check * went through walkthrough based on local builds...check Cheers, Konstantin On Sun, Mar 29, 2020 at 6:30 PM Igal Shilman wrote: > Hi @Hequn Cheng , and @Tzu-Li (Gordon) Tai > > Indeed the

Re: [DISCUSS] FLIP-119: Pipelined Region Scheduling

2020-03-29 Thread Zhu Zhu
Thanks for the comments! To Xintong, It's a bit strange since the in page links work as expected. Would you take another try? To Till, - Regarding the idea to improve to SlotProvider interface I think it is a good idea and thanks a lot! In the current design we make slot requests for batch jobs

[jira] [Created] (FLINK-16850) Make flink-benchmark align with different releases of Flink

2020-03-29 Thread Yun Tang (Jira)
Yun Tang created FLINK-16850: Summary: Make flink-benchmark align with different releases of Flink Key: FLINK-16850 URL: https://issues.apache.org/jira/browse/FLINK-16850 Project: Flink Issue

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Igal Shilman
Hi @Hequn Cheng , and @Tzu-Li (Gordon) Tai Indeed the names are generated automatically and are following a convention. Cheers, Igal. On Sun, Mar 29, 2020 at 6:28 PM Tzu-Li (Gordon) Tai wrote: > @Hequn Cheng > On second thought I think it would not hurt to create a RC4 to fix the > version

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Tzu-Li (Gordon) Tai
@Hequn Cheng On second thought I think it would not hurt to create a RC4 to fix the version strings. Will fix those and create that now. On Mon, Mar 30, 2020 at 12:06 AM Tzu-Li (Gordon) Tai wrote: > > > On Mon, Mar 30, 2020 at 12:00 AM Tzu-Li (Gordon) Tai > wrote: > >> @Hequn Cheng >> Good

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Tzu-Li (Gordon) Tai
On Mon, Mar 30, 2020 at 12:00 AM Tzu-Li (Gordon) Tai wrote: > @Hequn Cheng > Good catches again! > > Regarding the incorrect versions: > I think technically those would not be hard blockers, since what matters > is their inclusion and licenses being acknowledged. > It would still be good to fix

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Tzu-Li (Gordon) Tai
@Hequn Cheng Good catches again! Regarding the incorrect versions: I think technically those would not be hard blockers, since what matters is their inclusion and licenses being acknowledged. It would still be good to fix those though - could you open a ticket for those? Regarding the names of

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Hequn Cheng
Hi Gordon, Thanks a lot for the new RC. I found some new blockers about licenses: - Module statefun-flink-distribution com.google.protobuf:protobuf-java:3.8.0 (The version should be 3.7.1) - Module statefun-ridesharing-example-simulator com.google.code.findbugs:jsr305:3.0.2:compile (Remove

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Benchao Li
+1 (non-binding) This feature will greatly enrich Flink SQL's ecology~ Leonard Xu 于2020年3月29日周日 下午9:44写道: > +1 > > Best, > Leonard > > > 在 2020年3月28日,16:25,Kurt Young 写道: > > > > +1 > > > > Best, > > Kurt > > > > > > On Fri, Mar 27, 2020 at 10:51 AM Jingsong Li > wrote: > > > >> Hi everyone,

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Leonard Xu
+1 Best, Leonard > 在 2020年3月28日,16:25,Kurt Young 写道: > > +1 > > Best, > Kurt > > > On Fri, Mar 27, 2020 at 10:51 AM Jingsong Li wrote: > >> Hi everyone, >> >> I'd like to start the vote of FLIP-115 [1], which introduce Filesystem >> table factory in table. This FLIP is discussed in the

[ANNOUNCE] Weekly Community Update 2020/13

2020-03-29 Thread Konstantin Knauf
Dear community, happy to share this week's Apache Flink community digest with a couple of threads around the upcoming release of Apache Flink Stateful Functions 2.0, an update on Flink 1.10.1, two FLIPs to improve Apache Flink's distributed runtime and the schedule for Flink Forward Virtual

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

2020-03-29 Thread Konstantin Knauf
Hi Yangze, Hi Till, thanks you for working on this topic. I believe it will make debugging large Apache Flink deployments much more feasible. I was wondering whether it would make sense to allow the user to specify the Resource ID in standalone setups? For example, many users still implicitly

Re: Kafka - FLink - MongoDB using Scala

2020-03-29 Thread Konstantin Knauf
cc user@f.a.o Hi Siva, I am not aware of a Flink MongoDB Connector in either Apache Flink, Apache Bahir or flink-packages.org. I assume that you are doing idempotent upserts, and hence do not require a transactional sink to achieve end-to-end exactly-once results. To build one yourself, you

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

2020-03-29 Thread Hequn Cheng
Thanks a lot for such a quick update! @Gordon Best, Hequn On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai wrote: > All blockers are resolved. > > This vote thread is cancelled. > There is a new vote thread for RC3: > >

[jira] [Created] (FLINK-16849) comment is not reasonable

2020-03-29 Thread hufeihu (Jira)
hufeihu created FLINK-16849: --- Summary: comment is not reasonable Key: FLINK-16849 URL: https://issues.apache.org/jira/browse/FLINK-16849 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-16848) Improve the performance of the RecordsWithSplitIds queue.

2020-03-29 Thread Jiangjie Qin (Jira)
Jiangjie Qin created FLINK-16848: Summary: Improve the performance of the RecordsWithSplitIds queue. Key: FLINK-16848 URL: https://issues.apache.org/jira/browse/FLINK-16848 Project: Flink

[VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

2020-03-29 Thread Tzu-Li (Gordon) Tai
Hi everyone, Please review and vote on the *release candidate #3* for the version 2.0.0 of Apache Flink Stateful Functions, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Testing Guideline** You can find here [1] a doc that we can

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

2020-03-29 Thread Tzu-Li (Gordon) Tai
All blockers are resolved. This vote thread is cancelled. There is a new vote thread for RC3: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-Release-2-0-0-release-candidate-3-td39424.html On Sat, Mar 28, 2020 at 7:00 PM Tzu-Li (Gordon) Tai