Re: [DISCUSS] FLIP-84: Improve & Refactor API of Table Module

2019-10-30 Thread Kurt Young
Thanks Terry for bringing this up. TableEnv's interface is really critical not only to users, but also for components built upon it like SQL CLI. Your proposal solved some pain points we currently have, so +1 to the proposal. I left some comments in the document. Best, Kurt On Thu, Oct 31,

Re: [DISCUSS] FLIP 69 - Flink SQL DDL Enhancement

2019-10-30 Thread Terry Wang
Hi, everyone~ Sorry so late to reply this this thread again. I am working on FLIP-84 recently to make sql api support return value which this flip depends on. I think it’s time to pick up this discussion again and there are some new updates in this flip design:

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread Yun Gao
+1 (non-binding) Very thanks for bringing this to the community! -- From:jincheng sun Send Time:2019 Oct. 31 (Thu.) 10:22 To:dev Cc:Vasiliki Kalavri Subject:Re: [VOTE] Accept Stateful Functions into Apache Flink big +1

[DISCUSS] FLIP-84: Improve & Refactor API of Table Module

2019-10-30 Thread Terry Wang
Hi everyone, TableEnvironment has provided two `Table sqlQuery(String sql)` and `void sqlUpdate(String sql)` interfaces to create a table(actually a view here) or describe an update action from one sql string. But with more use cases come up, there are some fatal shortcomings in current API

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread jincheng sun
big +1 (binding) Andrey Zagrebin 于2019年10月30日 周三23:45写道: > sorry, my +1 was non-binding, confused that it was not a committer vote but > PMC. > > On Wed, Oct 30, 2019 at 4:43 PM Chesnay Schepler > wrote: > > > +1 (binding) > > > > On 30/10/2019 15:25, Vasiliki Kalavri wrote: > > > +1 (binding)

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread tison
Thanks for your attentions! @shixiaoga...@gmail.com Yes correct. We try to avoid jobs affect one another. Also a local ClusterClient in case saves the overhead about retry before leader elected and persist JobGraph before submission in RestClusterClient as well as the net cost. @Paul Lam 1.

[jira] [Created] (FLINK-14579) enable SQL CLI to configure modules via yaml config

2019-10-30 Thread Bowen Li (Jira)
Bowen Li created FLINK-14579: Summary: enable SQL CLI to configure modules via yaml config Key: FLINK-14579 URL: https://issues.apache.org/jira/browse/FLINK-14579 Project: Flink Issue Type:

[jira] [Created] (FLINK-14580) add HiveModuleFactory, HiveModuleDescriptor, and HiveModuleValidator

2019-10-30 Thread Bowen Li (Jira)
Bowen Li created FLINK-14580: Summary: add HiveModuleFactory, HiveModuleDescriptor, and HiveModuleValidator Key: FLINK-14580 URL: https://issues.apache.org/jira/browse/FLINK-14580 Project: Flink

[jira] [Created] (FLINK-14578) load/unloadModule() should throw RuntimeException rather than checked exception

2019-10-30 Thread Bowen Li (Jira)
Bowen Li created FLINK-14578: Summary: load/unloadModule() should throw RuntimeException rather than checked exception Key: FLINK-14578 URL: https://issues.apache.org/jira/browse/FLINK-14578 Project:

[jira] [Created] (FLINK-14577) Hortonworks schema registry support for avro deserialization

2019-10-30 Thread Nikita Ryanov (Jira)
Nikita Ryanov created FLINK-14577: - Summary: Hortonworks schema registry support for avro deserialization Key: FLINK-14577 URL: https://issues.apache.org/jira/browse/FLINK-14577 Project: Flink

[jira] [Created] (FLINK-14576) Elasticsearch (v6.3.1) sink end-to-end test instable

2019-10-30 Thread Gary Yao (Jira)
Gary Yao created FLINK-14576: Summary: Elasticsearch (v6.3.1) sink end-to-end test instable Key: FLINK-14576 URL: https://issues.apache.org/jira/browse/FLINK-14576 Project: Flink Issue Type: Bug

possible backwards compatibility issue between 1.8->1.9?

2019-10-30 Thread Bekir Oguz
Hi guys, during our upgrade from 1.8.1 to 1.9.1, one of our jobs fail to start with the following exception. We deploy the job with 'allow-non-restored-state' option and from the latest checkpoint dir of the 1.8.1 version. org.apache.flink.util.StateMigrationException: The new state

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread Andrey Zagrebin
sorry, my +1 was non-binding, confused that it was not a committer vote but PMC. On Wed, Oct 30, 2019 at 4:43 PM Chesnay Schepler wrote: > +1 (binding) > > On 30/10/2019 15:25, Vasiliki Kalavri wrote: > > +1 (binding) from me. I hope this is not too late :) > > > > Thank you for this great

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread Chesnay Schepler
+1 (binding) On 30/10/2019 15:25, Vasiliki Kalavri wrote: +1 (binding) from me. I hope this is not too late :) Thank you for this great contribution! On Wed, 30 Oct 2019 at 14:45, Stephan Ewen wrote: Thank you all for voting. The voting period has passed, but only 13 PMC members have

[jira] [Created] (FLINK-14575) Wrong (parent-first) class loader during serialization while submitting jobs

2019-10-30 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-14575: --- Summary: Wrong (parent-first) class loader during serialization while submitting jobs Key: FLINK-14575 URL: https://issues.apache.org/jira/browse/FLINK-14575 Project:

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread Hequn Cheng
+1. Good catch Jingsong! Best, Hequn On Wed, Oct 30, 2019 at 7:26 PM Zhenghua Gao wrote: > +1 (non-binding) > > *Best Regards,* > *Zhenghua Gao* > > > On Wed, Oct 30, 2019 at 12:05 PM Jingsong Li > wrote: > > > Hi all: > > > > We already have the parent model of formats. we have put other > >

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread Vasiliki Kalavri
+1 (binding) from me. I hope this is not too late :) Thank you for this great contribution! On Wed, 30 Oct 2019 at 14:45, Stephan Ewen wrote: > Thank you all for voting. > > The voting period has passed, but only 13 PMC members have voted so far, > that is less than 2/3rd of the PMCs (17

Re: [DISCUSS] Introduce a location-oriented two-stage query mechanism toimprove the queryable state.

2019-10-30 Thread Yu Li
Thanks for bringing this up again Vino. Considering the current status that the community is lack of committer bandwidth on QueryableState [1] [2], I think David has made a good point/concern, that we need to collect more information about valid in-production QueryableState use cases to draw more

Re: [VOTE] Accept Stateful Functions into Apache Flink

2019-10-30 Thread Stephan Ewen
Thank you all for voting. The voting period has passed, but only 13 PMC members have voted so far, that is less than 2/3rd of the PMCs (17 members). I will take a few days to ping other members to vote, after that we will gradually lower the threshold as per the process to account for inactive

[jira] [Created] (FLINK-14574) flink-s3-fs-hadoop doesn't work with plugins mechanism

2019-10-30 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-14574: -- Summary: flink-s3-fs-hadoop doesn't work with plugins mechanism Key: FLINK-14574 URL: https://issues.apache.org/jira/browse/FLINK-14574 Project: Flink

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread Paul Lam
Hi, Thanks for starting the discussion. WRT the per-job semantic, it looks natural to me that per-job means per-job-graph, because in my understanding JobGraph is the representation of a job. Could you share some use case in which a user program should contain multiple job graphs? WRT the

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread Zhenghua Gao
+1 (non-binding) *Best Regards,* *Zhenghua Gao* On Wed, Oct 30, 2019 at 12:05 PM Jingsong Li wrote: > Hi all: > > We already have the parent model of formats. we have put other > formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, > flink-sequence-file) to flink-formats.

Re: [DISCUSS] FLIP-83: Flink End-to-end Performance Testing Framework

2019-10-30 Thread Piotr Nowojski
Hi Yu, Thanks for bringing this up. +1 for the idea and the proposal from my side. I think that the proposed Test Job List might be a bit redundant/excessive, but: - we can always adjust this later, once we have the infrastructure in place - as long as we have the computing resources and

[DISCUSS] FLIP-83: Flink End-to-end Performance Testing Framework

2019-10-30 Thread Yu Li
Hi everyone, We would like to propose FLIP-83 that adds an end-to-end performance testing framework for Flink. We discovered some potential problems through such an internal end-to-end performance testing framework before the release of 1.9.0 [1], so we'd like to contribute it to Flink community

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-30 Thread Yu Li
Congratulations Becket! Best Regards, Yu On Wed, 30 Oct 2019 at 18:01, Andrey Zagrebin wrote: > Congrats Becket! > > On Tue, Oct 29, 2019 at 6:11 PM Xuannan Su wrote: > > > Congratulations Becket! :-) > > > > Xuannan > > > > > On Oct 28, 2019, at 6:07 AM, Fabian Hueske wrote: > > > > > >

[jira] [Created] (FLINK-14573) Support time data types in Python user-defined functions

2019-10-30 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-14573: Summary: Support time data types in Python user-defined functions Key: FLINK-14573 URL: https://issues.apache.org/jira/browse/FLINK-14573 Project: Flink

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

2019-10-30 Thread Andrey Zagrebin
Congrats Becket! On Tue, Oct 29, 2019 at 6:11 PM Xuannan Su wrote: > Congratulations Becket! :-) > > Xuannan > > > On Oct 28, 2019, at 6:07 AM, Fabian Hueske wrote: > > > > Hi everyone, > > > > I'm happy to announce that Becket Qin has joined the Flink PMC. > > Let's congratulate and welcome

Re: [DISCUSS] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread Fabian Hueske
+1 from my side as well. Am Mi., 30. Okt. 2019 um 04:51 Uhr schrieb Jingsong Li < jingsongl...@gmail.com>: > Thanks jark for reminding me. > I will start a vote thread. > > On Wed, Oct 30, 2019 at 11:40 AM Jark Wu wrote: > > > Hi Jingsong, > > > > Do we need a formal vote? Or do we need to wait

[jira] [Created] (FLINK-14572) BlobsCleanupITCase failed in Travis stage core - scheduler_ng

2019-10-30 Thread Gary Yao (Jira)
Gary Yao created FLINK-14572: Summary: BlobsCleanupITCase failed in Travis stage core - scheduler_ng Key: FLINK-14572 URL: https://issues.apache.org/jira/browse/FLINK-14572 Project: Flink Issue

[jira] [Created] (FLINK-14571) flink connector for redis

2019-10-30 Thread xiaodao (Jira)
xiaodao created FLINK-14571: --- Summary: flink connector for redis Key: FLINK-14571 URL: https://issues.apache.org/jira/browse/FLINK-14571 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread SHI Xiaogang
Hi Thanks for bringing this. The design looks very nice to me in that 1. In the new per-job mode, we don't need to compile user programs in the client and can directly run user programs with user jars. That way, it's easier for resource isolation in multi-tenant platforms and is much safer. 2.

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread vino yang
+1 (non-binding) Best, Vino Dawid Wysakowicz 于2019年10月30日周三 下午4:07写道: > +1 > > Best, > > Dawid > > On 30/10/2019 04:49, Jingsong Li wrote: > > Hi all: > > > > We already have the parent model of formats. we have put other > > formats(flink-avro, flink-json, flink-parquet, flink-json,

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread Jark Wu
+1 Best, Jark On Wed, 30 Oct 2019 at 16:07, Dawid Wysakowicz wrote: > +1 > > Best, > > Dawid > > On 30/10/2019 04:49, Jingsong Li wrote: > > Hi all: > > > > We already have the parent model of formats. we have put other > > formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv,

[jira] [Created] (FLINK-14570) KeyedStateBootstrapFunction missed onTimer callback method to fire timer

2019-10-30 Thread vinoyang (Jira)
vinoyang created FLINK-14570: Summary: KeyedStateBootstrapFunction missed onTimer callback method to fire timer Key: FLINK-14570 URL: https://issues.apache.org/jira/browse/FLINK-14570 Project: Flink

[jira] [Created] (FLINK-14569) use flink to deal with batch task,when parallelism granter than 1, always 1 or 2 records loss

2019-10-30 Thread kk (Jira)
kk created FLINK-14569: -- Summary: use flink to deal with batch task,when parallelism granter than 1, always 1 or 2 records loss Key: FLINK-14569 URL: https://issues.apache.org/jira/browse/FLINK-14569 Project:

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

2019-10-30 Thread Dawid Wysakowicz
+1 Best, Dawid On 30/10/2019 04:49, Jingsong Li wrote: > Hi all: > > We already have the parent model of formats. we have put other > formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv, > flink-sequence-file) to flink-formats. flink-orc is a format too. So we can > move it to

[jira] [Created] (FLINK-14568) Add methods for interacting with temporary functions

2019-10-30 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-14568: Summary: Add methods for interacting with temporary functions Key: FLINK-14568 URL: https://issues.apache.org/jira/browse/FLINK-14568 Project: Flink

Re: Builds Notification Update

2019-10-30 Thread Xiyuan Wang
Hi, As discussed before, Flink ARM builds and test now is running as cron job in OpenLab CI. The test result can be sent to this mail list once a day from "i...@openlabtesting.org". Can you give ` i...@openlabtesting.org` the right to send e-mail to bui...@flink.apache.org? It can help

[DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread tison
(CC user list because I think users may have ideas on how per-job mode should look like) Hi all, In the discussion about Flink on k8s[1] we encounter a problem that opinions diverge in how so-called per-job mode works. This thread is aimed at stating a dedicated discussion about per-job semantic

[jira] [Created] (FLINK-14567) Aggregate query with more than two group fields can't be write into HBase sink

2019-10-30 Thread Jark Wu (Jira)
Jark Wu created FLINK-14567: --- Summary: Aggregate query with more than two group fields can't be write into HBase sink Key: FLINK-14567 URL: https://issues.apache.org/jira/browse/FLINK-14567 Project: Flink

[jira] [Created] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used

2019-10-30 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14566: --- Summary: Enable ResourceSpec to get/set whether managed memory is used Key: FLINK-14566 URL: https://issues.apache.org/jira/browse/FLINK-14566 Project: Flink Issue