[jira] [Created] (FLINK-20210) "RequestFactory" interface in flink elasticsearch connector doesn't support "routing" param

2020-11-18 Thread Oneway Chen (Jira)
Oneway Chen created FLINK-20210: --- Summary: "RequestFactory" interface in flink elasticsearch connector doesn't support "routing" param Key: FLINK-20210 URL: https://issues.apache.org/jira/browse/FLINK-20210

[jira] [Created] (FLINK-20209) Add missing checkpoint configuration to Flink UI

2020-11-18 Thread Peidian Li (Jira)
Peidian Li created FLINK-20209: -- Summary: Add missing checkpoint configuration to Flink UI Key: FLINK-20209 URL: https://issues.apache.org/jira/browse/FLINK-20209 Project: Flink Issue Type: Impr

[jira] [Created] (FLINK-20211) Can not get the JobManager web ip according to the document

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20211: - Summary: Can not get the JobManager web ip according to the document Key: FLINK-20211 URL: https://issues.apache.org/jira/browse/FLINK-20211 Project: Flink Issue

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Till Rohrmann
Big +1 for your proposal Robert. I think the decision to use bash for our e2e tests already fired back on us several times. Hence, I believe it is time for a serious effort to get away from it. The e2e test framework can be found here [1]. When using the e2e test framework, it should be possible

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Jark Wu
+1 to use the Java-based testing framework and +1 for using docker images in the future. IIUC, the Java-based testing framework refers to the `flink-end-to-end-tests-common` module. The java-based framework helped us a lot when debugging the unstable e2e tests. Best, Jark On Wed, 18 Nov 2020 at 1

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Arvid Heise
Also +1 for java-based testing framework; bash just disrupts development flow too much. +1 to use docker as much as possible, ideally with testcontainers [1]. [1] https://www.testcontainers.org/ On Wed, Nov 18, 2020 at 10:23 AM Jark Wu wrote: > +1 to use the Java-based testing framework and +1

[jira] [Created] (FLINK-20212) Fail command line interfaces when provided with an unrecognized option

2020-11-18 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20212: -- Summary: Fail command line interfaces when provided with an unrecognized option Key: FLINK-20212 URL: https://issues.apache.org/jira/browse/FLINK-20212 Project:

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Yang Wang
Thanks till and Jark for sharing the information. I am also +1 for this proposal and glad to wire the new introduced K8s HA e2e tests to java based framework. Best, Yang Jark Wu 于2020年11月18日周三 下午5:23写道: > +1 to use the Java-based testing framework and +1 for using docker images > in the future

[jira] [Created] (FLINK-20213) Partition commit is delayed when records keep coming

2020-11-18 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-20213: Summary: Partition commit is delayed when records keep coming Key: FLINK-20213 URL: https://issues.apache.org/jira/browse/FLINK-20213 Project: Flink Issue Ty

[jira] [Created] (FLINK-20214) Unnecessary warning log when starting a k8s session cluster

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20214: - Summary: Unnecessary warning log when starting a k8s session cluster Key: FLINK-20214 URL: https://issues.apache.org/jira/browse/FLINK-20214 Project: Flink Issue

[REMINDER] Build docs before merging

2020-11-18 Thread Till Rohrmann
Hi everyone, I noticed in the last couple of days that our docs were broken. The main reason was invalid links using the {% link %} tag. The best way to avoid this situation is to build the docs before merging them [1]. That way, we can ensure that they are not broken. [1] https://github.com/apac

[jira] [Created] (FLINK-20215) Keep the "Access the Flink UI " document same in “Kubernetes Setup” and "Native Kubernetes Setup Beta"

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20215: - Summary: Keep the "Access the Flink UI " document same in “Kubernetes Setup” and "Native Kubernetes Setup Beta" Key: FLINK-20215 URL: https://issues.apache.org/jira/browse/FLINK-20215

[jira] [Created] (FLINK-20216) Move "Configuration" and "Environment Variables" under "Python API"

2020-11-18 Thread Dian Fu (Jira)
Dian Fu created FLINK-20216: --- Summary: Move "Configuration" and "Environment Variables" under "Python API" Key: FLINK-20216 URL: https://issues.apache.org/jira/browse/FLINK-20216 Project: Flink Is

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Aljoscha Krettek
+1 And I want to second Arvid's mention of testcontainers [1]. [1] https://www.testcontainers.org/ On 18.11.20 10:43, Yang Wang wrote: Thanks till and Jark for sharing the information. I am also +1 for this proposal and glad to wire the new introduced K8s HA e2e tests to java based framework.

[jira] [Created] (FLINK-20217) More fine-grained timer processing

2020-11-18 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-20217: --- Summary: More fine-grained timer processing Key: FLINK-20217 URL: https://issues.apache.org/jira/browse/FLINK-20217 Project: Flink Issue Type: Improvement

Re: [REMINDER] Build docs before merging

2020-11-18 Thread Jark Wu
+1 for this. Would be better to run the `check_links.sh` for broken links. Btw, could we add the docs build and check into PR CI. I think it would be better to guarantee this in the process. Best, Jark On Wed, 18 Nov 2020 at 18:08, Till Rohrmann wrote: > Hi everyone, > > I noticed in the last c

[jira] [Created] (FLINK-20218) AttributeError: module 'urllib' has no attribute 'parse'

2020-11-18 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-20218: Summary: AttributeError: module 'urllib' has no attribute 'parse' Key: FLINK-20218 URL: https://issues.apache.org/jira/browse/FLINK-20218 Project: Flink Issu

[jira] [Created] (FLINK-20219) Rethink the Kubernetes HA related ConfigMap clean up for session cluster

2020-11-18 Thread Yang Wang (Jira)
Yang Wang created FLINK-20219: - Summary: Rethink the Kubernetes HA related ConfigMap clean up for session cluster Key: FLINK-20219 URL: https://issues.apache.org/jira/browse/FLINK-20219 Project: Flink

[jira] [Created] (FLINK-20220) DataSet.collect() uses TaskExecutionState for transferring user-payload

2020-11-18 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20220: -- Summary: DataSet.collect() uses TaskExecutionState for transferring user-payload Key: FLINK-20220 URL: https://issues.apache.org/jira/browse/FLINK-20220 Project:

[jira] [Created] (FLINK-20221) DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss

2020-11-18 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-20221: -- Summary: DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss Key: FLINK-20221 URL: https://issues.apache.org/jira/browse/FLINK-20221

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-11-18 Thread Gyula Fóra
Hi All! I have found the following issue today which might be considered a blocker for this release as well: https://issues.apache.org/jira/browse/FLINK-20221 Could someone please quickly provide a second set of eyes and confirm that this is indeed a big problem? :) Thank you! Gyula On Wed, No

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Leonard Xu
+1 to stop using bash scripts, and I also have experienced the bash scripts that is really hard to maintain and debug, thanks @Robert for the great work again. I think testcontainers is a nice candidate. Best, Leonard > 在 2020年11月18日,19:46,Aljoscha Krettek 写道: > > +1 > > And I want to seco

[jira] [Created] (FLINK-20222) The CheckpointCoordinator should reset the OperatorCoordinators when fail before the first checkpoint.

2020-11-18 Thread Jiangjie Qin (Jira)
Jiangjie Qin created FLINK-20222: Summary: The CheckpointCoordinator should reset the OperatorCoordinators when fail before the first checkpoint. Key: FLINK-20222 URL: https://issues.apache.org/jira/browse/FLINK-2

[jira] [Created] (FLINK-20223) The RecreateOnResetOperatorCoordinator and SourceCoordinator executor thread should use the user class loader.

2020-11-18 Thread Jiangjie Qin (Jira)
Jiangjie Qin created FLINK-20223: Summary: The RecreateOnResetOperatorCoordinator and SourceCoordinator executor thread should use the user class loader. Key: FLINK-20223 URL: https://issues.apache.org/jira/browse

[jira] [Created] (FLINK-20224) add username&password to provide a credential for es rest client

2020-11-18 Thread jiang7chengzitc (Jira)
jiang7chengzitc created FLINK-20224: --- Summary: add username&password to provide a credential for es rest client Key: FLINK-20224 URL: https://issues.apache.org/jira/browse/FLINK-20224 Project: Flink

[jira] [Created] (FLINK-20225) Break link in the document

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20225: - Summary: Break link in the document Key: FLINK-20225 URL: https://issues.apache.org/jira/browse/FLINK-20225 Project: Flink Issue Type: Improvement Compon

Re: [REMINDER] Build docs before merging

2020-11-18 Thread Till Rohrmann
I think automating the check is a good idea since everything which is not automated will be forgotten at some point. Cheers, Till On Wed, Nov 18, 2020 at 12:56 PM Jark Wu wrote: > +1 for this. Would be better to run the `check_links.sh` for broken links. > Btw, could we add the docs build and c

[jira] [Created] (FLINK-20226) Temporal Table Functions cannot be created with a view reference

2020-11-18 Thread Jira
Benoît Paris created FLINK-20226: Summary: Temporal Table Functions cannot be created with a view reference Key: FLINK-20226 URL: https://issues.apache.org/jira/browse/FLINK-20226 Project: Flink

[jira] [Created] (FLINK-20227) Kafka transaction IDs exceeding limit

2020-11-18 Thread John Mathews (Jira)
John Mathews created FLINK-20227: Summary: Kafka transaction IDs exceeding limit Key: FLINK-20227 URL: https://issues.apache.org/jira/browse/FLINK-20227 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-20228) ESsink accur thread blocked

2020-11-18 Thread liqi316 (Jira)
liqi316 created FLINK-20228: --- Summary: ESsink accur thread blocked Key: FLINK-20228 URL: https://issues.apache.org/jira/browse/FLINK-20228 Project: Flink Issue Type: Bug Components: API /

[jira] [Created] (FLINK-20229) duplicate "for" security-kerberos doc

2020-11-18 Thread zhouyongjin (Jira)
zhouyongjin created FLINK-20229: --- Summary: duplicate "for" security-kerberos doc Key: FLINK-20229 URL: https://issues.apache.org/jira/browse/FLINK-20229 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-20230) PARTITIONED TABLE, INSERT INTO EMPTY VALUES, THROW FileNotFoundException

2020-11-18 Thread liu (Jira)
liu created FLINK-20230: --- Summary: PARTITIONED TABLE, INSERT INTO EMPTY VALUES, THROW FileNotFoundException Key: FLINK-20230 URL: https://issues.apache.org/jira/browse/FLINK-20230 Project: Flink Issu

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Rui Li
Big +1 to java-based e2e tests. It'll be much easier to write/debug these tests. On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu wrote: > +1 to stop using bash scripts, > and I also have experienced the bash scripts that is really hard to > maintain and debug, thanks @Robert for the great work again.

[jira] [Created] (FLINK-20231) Sql UDTF subplan reuse on correlate

2020-11-18 Thread Junning Liang (Jira)
Junning Liang created FLINK-20231: - Summary: Sql UDTF subplan reuse on correlate Key: FLINK-20231 URL: https://issues.apache.org/jira/browse/FLINK-20231 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Xingbo Huang
Big +1 for java-based e2e tests. Currently PyFlink related tests each take ~15minutes in bash e2e tests because we are using a secured YARN cluster which is the only convenient way of starting a YARN cluster in the bash e2e tests. I think if we migrate these tests to the java-based testing framewor

[jira] [Created] (FLINK-20232) Handle object reuse properly in the operators of Python DataStream API

2020-11-18 Thread Dian Fu (Jira)
Dian Fu created FLINK-20232: --- Summary: Handle object reuse properly in the operators of Python DataStream API Key: FLINK-20232 URL: https://issues.apache.org/jira/browse/FLINK-20232 Project: Flink

Hive Streaming write compaction

2020-11-18 Thread Chen Qin
Hi there, We are testing out writing Kafka to hive table as parquet format. Currently, we have seen user has to choose to create lots of small files in min level folder to gain latency benefits. I recall FF2020 Global folks mentioned implement compaction logic during the checkpointing time. Wonder

Re: Hive Streaming write compaction

2020-11-18 Thread Kurt Young
We just added this feature to 1.12 [1][2], it would be great that you can download the 1.12 RC to test it out, and give us some feedback. In case you will wonder why I linked 2 jiras, it's because both FileSystem & Hive connector share the same option options and also the implementations. [1] htt

Re: Hive Streaming write compaction

2020-11-18 Thread Jingsong Li
Hi Chen, Table Filesystem/Hive sink file compaction has been merged into master, detail in [1]. It is included in Flink 1.12. Hope you can have a try and test. [1]https://issues.apache.org/jira/browse/FLINK-19345 Best, Jingsong On Thu, Nov 19, 2020 at 2:31 PM Chen Qin wrote: > Hi there, > >

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

2020-11-18 Thread Yun Tang
+1 for java-based E2E test as bash scripts lack of the power to handle more complicated cases. For the docker image improvement, I think we should be more cautious as developers at China might suffer with the network issue, or at least we should ensure or find some guides to speed up the image

[jira] [Created] (FLINK-20233) DataGeneratorSource should increment serialVersionUID

2020-11-18 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-20233: Summary: DataGeneratorSource should increment serialVersionUID Key: FLINK-20233 URL: https://issues.apache.org/jira/browse/FLINK-20233 Project: Flink Issue T