[jira] [Created] (FLINK-14849) Can not submit job when use hive connector

2019-11-18 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14849: Summary: Can not submit job when use hive connector Key: FLINK-14849 URL: https://issues.apache.org/jira/browse/FLINK-14849 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-14848) BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part

2019-11-18 Thread Zhenghua Gao (Jira)
Zhenghua Gao created FLINK-14848: Summary: BaseRowSerializer.toBinaryRow wrongly process null for objects with variable-length part Key: FLINK-14848 URL: https://issues.apache.org/jira/browse/FLINK-14848

[jira] [Created] (FLINK-14847) Support retrieving Hive PK constraints

2019-11-18 Thread Rui Li (Jira)
Rui Li created FLINK-14847: -- Summary: Support retrieving Hive PK constraints Key: FLINK-14847 URL: https://issues.apache.org/jira/browse/FLINK-14847 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-14846) Correct the default writerbuffer size documentation of RocksDB

2019-11-18 Thread Yun Tang (Jira)
Yun Tang created FLINK-14846: Summary: Correct the default writerbuffer size documentation of RocksDB Key: FLINK-14846 URL: https://issues.apache.org/jira/browse/FLINK-14846 Project: Flink

Re: [DISCUSS] Support configure remote flink jar

2019-11-18 Thread Thomas Weise
There is a related use case (not specific to HDFS) that I came across: It would be nice if the jar upload endpoint could accept the URL of a jar file as alternative to the jar file itself. Such URL could point to an artifactory or distributed file system. Thomas On Mon, Nov 18, 2019 at 7:40 PM

Re: [DISCUSS] Support configure remote flink jar

2019-11-18 Thread Yang Wang
Hi tison, Thanks for your starting this discussion. * For user customized flink-dist jar, it is an useful feature. Since it could avoid to upload the flink-dist jar every time. Especially in production environment, it could accelerate the submission process. * For the standard flink-dist jar,

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread Yang Wang
+1 (non-binding) It is great to have a new end-to-end test framework, even it is only for performance tests now. Best, Yang Jingsong Li 于2019年11月19日周二 上午9:54写道: > +1 (non-binding) > > Best, > Jingsong Lee > > On Mon, Nov 18, 2019 at 7:59 PM Becket Qin wrote: > > > +1 (binding) on having the

[jira] [Created] (FLINK-14845) Introduce data compression to blocking shuffle.

2019-11-18 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-14845: --- Summary: Introduce data compression to blocking shuffle. Key: FLINK-14845 URL: https://issues.apache.org/jira/browse/FLINK-14845 Project: Flink Issue Type:

[DISCUSS] Support configure remote flink jar

2019-11-18 Thread tison
Hi forks, Recently, our customers ask for a feature configuring remote flink jar. I'd like to reach to you guys to see whether or not it is a general need. ATM Flink only supports configures local file as flink jar via `-yj` option. If we pass a HDFS file path, due to implementation detail it

[jira] [Created] (FLINK-14844) ConnectorCatalogTable should not patch the row type with TimeAttributes for batch mode

2019-11-18 Thread Danny Chen (Jira)
Danny Chen created FLINK-14844: -- Summary: ConnectorCatalogTable should not patch the row type with TimeAttributes for batch mode Key: FLINK-14844 URL: https://issues.apache.org/jira/browse/FLINK-14844

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

2019-11-18 Thread Yijie Shen
Hi everyone, I've put the catalog part design in separate doc with more details for easier communication. https://docs.google.com/document/d/1LMnABtXn-wQedsmWv8hopvx-B-jbdr8-jHbIiDhdsoE/edit?usp=sharing I would love to hear your thoughts on this. Best, Yijie On Mon, Oct 21, 2019 at 11:15 AM

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread Jingsong Li
+1 (non-binding) Best, Jingsong Lee On Mon, Nov 18, 2019 at 7:59 PM Becket Qin wrote: > +1 (binding) on having the test suite. > > BTW, it would be good to have a few more details about the performance > tests. For example: > 1. How do the testing records look like? The size and key

Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)

2019-11-18 Thread Bowen Li
Hi Dawid, As title of this thread says, only features related to 1.10 release (MVP) is accepted, and the other part is still under discussion. I updated the FLIP's status field to reflect that. Also added it to https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals I will

[jira] [Created] (FLINK-14843) Streaming bucketing end-to-end test can fail with Output hash mismatch

2019-11-18 Thread Gary Yao (Jira)
Gary Yao created FLINK-14843: Summary: Streaming bucketing end-to-end test can fail with Output hash mismatch Key: FLINK-14843 URL: https://issues.apache.org/jira/browse/FLINK-14843 Project: Flink

RE: SQL for Avro GenericRecords on Parquet

2019-11-18 Thread Hanan Yehudai
HI Peter. Thanks. This is my code . I used one of the parquet / avro tests as a reference. The code will fail on Test testScan(ParquetTestCase) failed with: java.lang.UnsupportedOperationException at

Re: [DISCUSSION] Kafka Metrics Reporter

2019-11-18 Thread Yun Tang
Hi all Glad to see this topic in community. We at Alibaba also implemented a kafka metrics reporter and extend it to other message queues like Alibaba cloud log service [1] half a year ago. The reason why we not launch a similar discussion is that we previously thought we only provide a way

Re: SQL for Avro GenericRecords on Parquet

2019-11-18 Thread Peter Huang
Hi Hanan, Thanks for reporting the issue. Would you please attach your test code here? I may help to investigate. Best Regards Peter Huang On Mon, Nov 18, 2019 at 2:51 AM Hanan Yehudai wrote: > I have tried to persist Generic Avro records in a parquet file and then > read it via

[jira] [Created] (FLINK-14842) add logging for functions loaded from modules

2019-11-18 Thread Bowen Li (Jira)
Bowen Li created FLINK-14842: Summary: add logging for functions loaded from modules Key: FLINK-14842 URL: https://issues.apache.org/jira/browse/FLINK-14842 Project: Flink Issue Type:

Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)

2019-11-18 Thread Peter Huang
Hi David, Sorry for my misbehavior. FLINK-7151 should be the parent tickets of the function DDL effort. I recently created several tickets of it, currently including https://issues.apache.org/jira/browse/FLINK-14055 https://issues.apache.org/jira/browse/FLINK-14711

[jira] [Created] (FLINK-14841) Add create and drop function DDL in parser

2019-11-18 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-14841: - Summary: Add create and drop function DDL in parser Key: FLINK-14841 URL: https://issues.apache.org/jira/browse/FLINK-14841 Project: Flink Issue Type:

[ANNOUNCE] Launch of flink-packages.org: A website to foster the Flink Ecosystem

2019-11-18 Thread Robert Metzger
Hi all, I would like to announce that Ververica, with the permission of the Flink PMC, is launching a website called flink-packages.org. This goes back to an effort proposed earlier in 2019 [1] The idea of the site is to help developers building extensions / connectors / API etc. for Flink to get

Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)

2019-11-18 Thread Dawid Wysakowicz
Hey, I've just realized this FLIP was accepted but it was not reflected in the wiki page. Moreover I could not find a single jira issue that would link to the FLIP. I've seen though a few other issues and PRs being opened regarding this FLIP (FLINK-7151?, FLINK-14837, FLINK-14055). I was also a

Re: [DISCUSS] Release flink-shaded 9.0

2019-11-18 Thread Ufuk Celebi
@Chesnay: I know you said that you are pretty busy these days. If we can't find anybody else to work on this, when would you be available to create the first RC? On Sun, Nov 17, 2019 at 6:48 AM Hequn Cheng wrote: > Hi, > > Big +1 to release 9.0. > It would be good if we can solve these security

[jira] [Created] (FLINK-14840) Use new Executor interface in SQL cli

2019-11-18 Thread Aljoscha Krettek (Jira)
Aljoscha Krettek created FLINK-14840: Summary: Use new Executor interface in SQL cli Key: FLINK-14840 URL: https://issues.apache.org/jira/browse/FLINK-14840 Project: Flink Issue Type:

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread Becket Qin
+1 (binding) on having the test suite. BTW, it would be good to have a few more details about the performance tests. For example: 1. How do the testing records look like? The size and key distributions. 2. The resources for each task. 3. The intended configuration for the jobs. 4. What exact

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread Zhijiang
+1 (binding)! It is a good thing to enhance our testing work. Best, Zhijiang -- From:Hequn Cheng Send Time:2019 Nov. 18 (Mon.) 18:22 To:dev Subject:Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework +1

SQL for Avro GenericRecords on Parquet

2019-11-18 Thread Hanan Yehudai
I have tried to persist Generic Avro records in a parquet file and then read it via ParquetTablesource – using SQL. Seems that the SQL I not executed properly ! The persisted records are : Id , type 333,Type1 22,Type2 333,Type1 22,Type2 333,Type1 22,Type2 333,Type1

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread Hequn Cheng
+1 (binding)! I think this would be very helpful to detect regression problems. Best, Hequn On Mon, Nov 18, 2019 at 4:28 PM vino yang wrote: > +1 (non-binding) > > Best, > Vino > > jincheng sun 于2019年11月18日周一 下午2:31写道: > > > +1 (binding) > > > > OpenInx 于2019年11月18日周一 下午12:09写道: > > > > >

[jira] [Created] (FLINK-14839) JobGraph#userJars & JobGraph#classpaths are non-null

2019-11-18 Thread Zili Chen (Jira)
Zili Chen created FLINK-14839: - Summary: JobGraph#userJars & JobGraph#classpaths are non-null Key: FLINK-14839 URL: https://issues.apache.org/jira/browse/FLINK-14839 Project: Flink Issue Type:

Re: [DISCUSS] PyFlink User-Defined Function Resource Management

2019-11-18 Thread Yangze Guo
Thanks for driving this discussion, Dian! +1 for this proposal. It will help to reduce container failure due to the memory overuse. Some comments left in the design doc. Best, Yangze Guo On Mon, Nov 18, 2019 at 4:06 PM Xintong Song wrote: > > Sorry for the late reply. > > +1 for the general

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

2019-11-18 Thread vino yang
+1 (non-binding) Best, Vino jincheng sun 于2019年11月18日周一 下午2:31写道: > +1 (binding) > > OpenInx 于2019年11月18日周一 下午12:09写道: > > > +1 (non-binding) > > > > On Mon, Nov 18, 2019 at 11:54 AM aihua li wrote: > > > > > +1 (non-binding) > > > > > > Thanks Yu Li for driving on this. > > > > > > > 在

[jira] [Created] (FLINK-14838) Cleanup the description about container number config option in Scala and python shell doc

2019-11-18 Thread vinoyang (Jira)
vinoyang created FLINK-14838: Summary: Cleanup the description about container number config option in Scala and python shell doc Key: FLINK-14838 URL: https://issues.apache.org/jira/browse/FLINK-14838

[jira] [Created] (FLINK-14837) Support Function DDL in TableEnvironment

2019-11-18 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-14837: - Summary: Support Function DDL in TableEnvironment Key: FLINK-14837 URL: https://issues.apache.org/jira/browse/FLINK-14837 Project: Flink Issue Type:

[jira] [Created] (FLINK-14836) Unable to set yarn container number for scala shell in yarn mode

2019-11-18 Thread Jeff Zhang (Jira)
Jeff Zhang created FLINK-14836: -- Summary: Unable to set yarn container number for scala shell in yarn mode Key: FLINK-14836 URL: https://issues.apache.org/jira/browse/FLINK-14836 Project: Flink

Re: [DISCUSS] PyFlink User-Defined Function Resource Management

2019-11-18 Thread Xintong Song
Sorry for the late reply. +1 for the general proposal. And one remainder, to use UNKNOWN resource requirement, we need to make sure optimizer knowns which operators use off-heap managed memory, and compute and set a fraction to the operators. See FLIP-53[1] for more details, and I would suggest