[jira] [Created] (FLINK-19389) Error determining python major version to start fileserver during e-2-e tests

2020-09-23 Thread Di Xu (Jira)
Di Xu created FLINK-19389: - Summary: Error determining python major version to start fileserver during e-2-e tests Key: FLINK-19389 URL: https://issues.apache.org/jira/browse/FLINK-19389 Project: Flink

Re: [DISCUSS] Move Hive document to "Table & SQL Connectors" from "Table API & SQL"

2020-09-23 Thread Jark Wu
+1 to move it there. On Thu, 24 Sep 2020 at 12:16, Jingsong Li wrote: > Hi devs and users: > > After the 1.11 release, I heard some voices recently: How can't Hive's > documents be found in the "Table & SQL Connectors". > > Actually, Hive's documents are in the "Table API & SQL". Since the

[DISCUSS] Move Hive document to "Table & SQL Connectors" from "Table API & SQL"

2020-09-23 Thread Jingsong Li
Hi devs and users: After the 1.11 release, I heard some voices recently: How can't Hive's documents be found in the "Table & SQL Connectors". Actually, Hive's documents are in the "Table API & SQL". Since the "Table & SQL Connectors" document was extracted separately, Hive is a little out of

[jira] [Created] (FLINK-19388) Streaming bucketing end-to-end test failed with "Number of running task managers has not reached 4 within a timeout of 40 sec"

2020-09-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-19388: --- Summary: Streaming bucketing end-to-end test failed with "Number of running task managers has not reached 4 within a timeout of 40 sec" Key: FLINK-19388 URL:

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-23 Thread Guowei Ma
Thanks Aljoscha for your suggestion. I have updated FLIP. Any comments are welcome. Best, Guowei On Wed, Sep 23, 2020 at 4:25 PM Aljoscha Krettek wrote: > Yes, that sounds good! I'll probably have some comments on the FLIP > about the names of generic parameters and the Javadoc but we can

[jira] [Created] (FLINK-19387) ConnectedComponentsWithDeferredUpdateITCase.testJobWithoutObjectReuse deadlock

2020-09-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-19387: --- Summary: ConnectedComponentsWithDeferredUpdateITCase.testJobWithoutObjectReuse deadlock Key: FLINK-19387 URL: https://issues.apache.org/jira/browse/FLINK-19387 Project: Flink

[jira] [Created] (FLINK-19386) Remove legacy table planner dependecy from connectors and formats

2020-09-23 Thread hailong wang (Jira)
hailong wang created FLINK-19386: Summary: Remove legacy table planner dependecy from connectors and formats Key: FLINK-19386 URL: https://issues.apache.org/jira/browse/FLINK-19386 Project: Flink

Re: How to run k8s-related end-to-end tests locally?

2020-09-23 Thread Di Xu
Hi Wang Yang, Thanks for your quick reply. I followed the discussion in FLINK-17857 and now understand the problem and solution. I will fix it locally and please let me know if there is anything I can help with this issue. Yang Wang 于2020年9月24日 周四上午12:12写道: > Hi Xu Di, > > Sorry for the

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #2

2020-09-23 Thread Seth Wiesman
+1 (non binding) - Verified signatures and checksums - Checked licenses and notices - Clean build from source - Executed all end to end tests - Deployed to K8s with Go SDK (no sdk changes required btw :)) - Simulated TM and remote function failure and verified recovery - Checked updated docs

[jira] [Created] (FLINK-19385) Channel recovery may deadlock

2020-09-23 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-19385: - Summary: Channel recovery may deadlock Key: FLINK-19385 URL: https://issues.apache.org/jira/browse/FLINK-19385 Project: Flink Issue Type: Bug

Re: Kafka connection issues

2020-09-23 Thread Ramya Ramamurthy
hi Kostas, Copy pasting this snippet where we see the fluctuations. Let me know if this helps. 2020-09-22 23:39:19,646 DEBUG org.apache.kafka.clients.NetworkClient - Node 3 disconnected. 2020-09-22 23:39:19,646 DEBUG org.apache.kafka.clients.NetworkClient -

Re: How to run k8s-related end-to-end tests locally?

2020-09-23 Thread Yang Wang
Hi Xu Di, Sorry for the inconvenience. This is a known issue FLINK-17857[1]. After we migrate all the e2e tests from flink-container/docker to apache/flink-docker, the docker and k8s related e2e tests could not work. I will revive this ticket and fix it ASAP. [1].

[jira] [Created] (FLINK-19384) Source API exception signatures are inconsistent

2020-09-23 Thread Stephan Ewen (Jira)
Stephan Ewen created FLINK-19384: Summary: Source API exception signatures are inconsistent Key: FLINK-19384 URL: https://issues.apache.org/jira/browse/FLINK-19384 Project: Flink Issue Type:

[jira] [Created] (FLINK-19383) Per Partition State

2020-09-23 Thread Theo Diefenthal (Jira)
Theo Diefenthal created FLINK-19383: --- Summary: Per Partition State Key: FLINK-19383 URL: https://issues.apache.org/jira/browse/FLINK-19383 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19382) Introducing ReplayableSourceStateBackend

2020-09-23 Thread Theo Diefenthal (Jira)
Theo Diefenthal created FLINK-19382: --- Summary: Introducing ReplayableSourceStateBackend Key: FLINK-19382 URL: https://issues.apache.org/jira/browse/FLINK-19382 Project: Flink Issue Type:

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #2

2020-09-23 Thread Igal Shilman
+1 (non binding) - Verified the signatures and the checksums - Built with JDK11 and JDK8 - Verified that the source distribution does not contain any binary data. - Run e2e tests. - Run few examples via docker-compose - Deployed to Kubernetes with checkpointing to S3, with failure scenarios: --

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-23 Thread admin
+1,it’s a good news > 2020年9月23日 下午6:22,Jingsong Li 写道: > > Hi all, > > I'd like to start a discussion about improving the new TableSource and > TableSink interfaces. > > Most connectors have been migrated to FLIP-95, but there are still the > Filesystem and Hive that have not been migrated.

[jira] [Created] (FLINK-19381) Fix docs about relocatable savepoints

2020-09-23 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-19381: --- Summary: Fix docs about relocatable savepoints Key: FLINK-19381 URL: https://issues.apache.org/jira/browse/FLINK-19381 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19380) Add support for a gRPC transport for the RequestReply protocol.

2020-09-23 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-19380: Summary: Add support for a gRPC transport for the RequestReply protocol. Key: FLINK-19380 URL: https://issues.apache.org/jira/browse/FLINK-19380 Project: Flink

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Kurt Young
I see, I missed the part that a row is either in positioned mode or nameed mode. I can live with this. Thanks. Best, Kurt On Wed, Sep 23, 2020 at 9:07 PM Timo Walther wrote: > But the examples you mentioned would not be different. > > By calling `Row.withNames()`, the row has no definition of

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Timo Walther
But the examples you mentioned would not be different. By calling `Row.withNames()`, the row has no definition of position. All position-based methods would throw an exception. The hashCode()/equals() would return true for: > Row row1 = Row.withNames(); > row.setField("a", 1); >

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Kurt Young
Thanks for the detailed response, 1-5 sounds good to me. For #6, I just think of another case which would also annoy users. Consider code like this: Row row = Row.withNames(); row.setField("a", 1); row.setField("b", 2); and for second time, he changes the sequence of setting method calls: Row

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Timo Walther
Hi Kurt, thanks for your feedback. 1. "moving Schema after DataStream": I don't have a strong opinion here. One could argue that the API would look similar to a CREATE TABLE statement: first schema then connector. I updated the FLIP. 2. "will we do some verification?" Yes, we will

[DISCUSS] Support registering custom JobStatusListeners when scheduling a job

2020-09-23 Thread 季文昊
Hi there, I'm working on a Flink platform in my corp, which provides a service to provision and manage multiple dedicated Flink clusters. The problem is that we want to sync a job status without delay after its submission through our platform as long as it has been changed. Since we want to

[jira] [Created] (FLINK-19379) Submitting job to running YARN session fails

2020-09-23 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19379: -- Summary: Submitting job to running YARN session fails Key: FLINK-19379 URL: https://issues.apache.org/jira/browse/FLINK-19379 Project: Flink Issue Type:

[jira] [Created] (FLINK-19378) Running "./bin/flink run" without HADOOP_CLASSPATH to submit job to running YARN session fails

2020-09-23 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19378: -- Summary: Running "./bin/flink run" without HADOOP_CLASSPATH to submit job to running YARN session fails Key: FLINK-19378 URL:

Re: Re: [DISCUSS] Support source/sink parallelism config in Flink sql

2020-09-23 Thread Jingsong Li
Hi , I have started a discussion about improving the new TableSource and TableSink: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-146-Improve-new-TableSource-and-TableSink-interfaces-td45161.html It includes parallelism setting, welcome to join the discussion and

[DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-23 Thread Jingsong Li
Hi all, I'd like to start a discussion about improving the new TableSource and TableSink interfaces. Most connectors have been migrated to FLIP-95, but there are still the Filesystem and Hive that have not been migrated. They have some requirements on table connector API. And users also have

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Kurt Young
Sorry for being late, I went through the design doc and here are my comments: 1. A minor one, how about moving Schema after DataStream in all affected APIs? Such as: StreamTableEnvironment.fromDataStream(Schema, DataStream): Table StreamTableEnvironment.createTemporaryView(String, Schema,

Re: Kafka connection issues

2020-09-23 Thread Kostas Kloudas
Hi Ramya, Unfortunately I cannot see them. Kostas On Wed, Sep 23, 2020 at 10:27 AM Ramya Ramamurthy wrote: > > Hi Kostas, > > Attaching the taskmanager logs regarding this issue. > I have attached the Kaka related metrics. I hope you can see it this time. > > Not sure why we get these many

[jira] [Created] (FLINK-19377) Task can swallow test exceptions which hides test failures

2020-09-23 Thread Aljoscha Krettek (Jira)
Aljoscha Krettek created FLINK-19377: Summary: Task can swallow test exceptions which hides test failures Key: FLINK-19377 URL: https://issues.apache.org/jira/browse/FLINK-19377 Project: Flink

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-23 Thread Stephan Ewen
I am confused now with the concerns here. This is very much from the user perspective (which is partially also the developer perspective which is the sign of an intuitive abstraction). Of course, there will be docs describing what JMCheckpointStorage and FsCheckpointStorage are. And having

[jira] [Created] (FLINK-19375) The `kubernetes.secrets` and `kubernetes.env.secretKeyRef`'s document is missing

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19375: - Summary: The `kubernetes.secrets` and `kubernetes.env.secretKeyRef`'s document is missing Key: FLINK-19375 URL: https://issues.apache.org/jira/browse/FLINK-19375 Project:

[jira] [Created] (FLINK-19374) update the `table.exec.state.ttl`'s documentation

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19374: - Summary: update the `table.exec.state.ttl`'s documentation Key: FLINK-19374 URL: https://issues.apache.org/jira/browse/FLINK-19374 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19373) `jmx.server.port`'s document is missing.

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19373: - Summary: `jmx.server.port`'s document is missing. Key: FLINK-19373 URL: https://issues.apache.org/jira/browse/FLINK-19373 Project: Flink Issue Type: Bug

How to run k8s-related end-to-end tests locally?

2020-09-23 Thread Di Xu
Hi, I tried to run single k8s e-2-e test locally on my macos laptop (a minikube vm has been started manually), and here is the command I run from the source root. ``` $ flink-end-to-end-tests/run-single-test.sh flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh ``` Then it failed

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-23 Thread Aljoscha Krettek
On 23.09.20 04:40, Yu Li wrote: To be specific, with the old API users don't need to set checkpoint storage, instead they only need to pass the checkpoint path w/o caring about the storage. The new APIs are forcing users to set the storage so they have to know the difference between different

[VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #2

2020-09-23 Thread Tzu-Li (Gordon) Tai
Hi everyone, Please review and vote on the release candidate #2 for the version 2.2.0 of Apache Flink Stateful Functions, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) ***Testing Guideline*** You can find here [1] a page in the

Re: Kafka connection issues

2020-09-23 Thread Ramya Ramamurthy
Hi Kostas, Attaching the taskmanager logs regarding this issue. I have attached the Kaka related metrics. I hope you can see it this time. Not sure why we get these many disconnects to Kafka. Maybe because of this interruptions, we seem to slow down on our processing. At some point the memory

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-23 Thread Aljoscha Krettek
Yes, that sounds good! I'll probably have some comments on the FLIP about the names of generic parameters and the Javadoc but we can address them later or during implementation. I also think that we probably need the FAIL,RETRY,SUCCESS result for globalCommit() but we can also do that as a

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Timo Walther
Hi Jark, thanks for your feedback. I removed `withNamesAndPositions` from the public API list and added a comment that this is only internal API for converters and serializers. I would start a new vote tomorrow if there are no objections. What do you think? Regards, Timo On 23.09.20

[CANCEL][VOTE] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Timo Walther
There is still a discussion ongoing. I will cancel the vote for now. Regards, Timo On 14.09.20 09:48, Dawid Wysakowicz wrote: +1 On 11/09/2020 16:19, Timo Walther wrote: Hi all, after the discussion in [1], I would like to open a voting thread for FLIP-136 [2] which covers different topic

[jira] [Created] (FLINK-19372) Support Pandas Batch Over Window Aggregation

2020-09-23 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-19372: Summary: Support Pandas Batch Over Window Aggregation Key: FLINK-19372 URL: https://issues.apache.org/jira/browse/FLINK-19372 Project: Flink Issue Type:

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #1

2020-09-23 Thread Tzu-Li (Gordon) Tai
Thanks for testing the RC and fixing so quickly Igal! The fixes have been merged now. I'll cancel this RC in favor of a new one.

[jira] [Created] (FLINK-19371) Support table transformations in SQL

2020-09-23 Thread Timo Walther (Jira)
Timo Walther created FLINK-19371: Summary: Support table transformations in SQL Key: FLINK-19371 URL: https://issues.apache.org/jira/browse/FLINK-19371 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Release flink-shaded 12.0

2020-09-23 Thread Yu Li
+1 Best Regards, Yu On Tue, 22 Sep 2020 at 17:49, Robert Metzger wrote: > No concerns from my side. > > On Fri, Sep 18, 2020 at 8:25 AM Chesnay Schepler > wrote: > > > Hello, > > > > I'd like to kickoff the next release of flink-shaded, which will contain > > a bump to netty (4.1.49) and

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-23 Thread Jark Wu
Hi Timo, Sorry for the late reply. I think it would be great if we can make `withNamesAndPositions` internal visible. This reduces the complexity of the public API. It's hard to come up with a perfect solution. So let's move on this FLIP. I don't have other concerns. Best, Jark On Fri, 18 Sep

[jira] [Created] (FLINK-19370) FileSourceTextLinesITCase.testContinuousTextFileSource failed

2020-09-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-19370: --- Summary: FileSourceTextLinesITCase.testContinuousTextFileSource failed Key: FLINK-19370 URL: https://issues.apache.org/jira/browse/FLINK-19370 Project: Flink Issue

[jira] [Created] (FLINK-19369) BlobClientTest.testGetFailsDuringStreamingForJobPermanentBlob hangs

2020-09-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-19369: --- Summary: BlobClientTest.testGetFailsDuringStreamingForJobPermanentBlob hangs Key: FLINK-19369 URL: https://issues.apache.org/jira/browse/FLINK-19369 Project: Flink