Re: Breakage in Flink CLI in 1.5.0

2018-06-19 Thread Sampath Bhat
Hi Chesnay If REST API (i.e. the web server) is mandatory for submitting jobs then why is there an option to set rest.port to -1? I think it should be mandatory to set some valid port for rest.port and make sure flink job manager does not come up if valid port is not set for rest.port? Or else

[jira] [Created] (FLINK-9619) Always close the task manager connection when the container is completed in YarnResourceManager

2018-06-19 Thread Sihua Zhou (JIRA)
Sihua Zhou created FLINK-9619: - Summary: Always close the task manager connection when the container is completed in YarnResourceManager Key: FLINK-9619 URL: https://issues.apache.org/jira/browse/FLINK-9619

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread zhangminglei
Thanks to Fabian and Timo, I watched the scalar udf and find it is very quick to implements a case when udf for the specify logic to meet my necessary Cheers Minglei > 在 2018年6月19日,下午10:52,Fabian Hueske 写道: > > I see, then this case wasn't covered by the fix that we added for Flink > 1.5.0.

[jira] [Created] (FLINK-9618) NullPointerException in FlinkKinesisProducer when aws.region is not set and aws.endpoint is set

2018-06-19 Thread Aaron Langford (JIRA)
Aaron Langford created FLINK-9618: - Summary: NullPointerException in FlinkKinesisProducer when aws.region is not set and aws.endpoint is set Key: FLINK-9618 URL: https://issues.apache.org/jira/browse/FLINK-9618

Re: Ordering of stream from different kafka partitions

2018-06-19 Thread Andrey Zagrebin
Hi Amol, I think you could try (based on your stack overflow code) org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor like this: DataStream streamSource = env .addSource(kafkaConsumer) .setParallelism(4) .assignTimestampsAndWatermarks( new

Re: Ordering of stream from different kafka partitions

2018-06-19 Thread Andrey Zagrebin
Hi Amol, I think you could try (based on your stack overflow code) org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor like this: DataStream streamSource = env .addSource(kafkaConsumer) .setParallelism(4) .assignTimestampsAndWatermarks(

[jira] [Created] (FLINK-9617) Provide alias for whole records in Table API

2018-06-19 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-9617: - Summary: Provide alias for whole records in Table API Key: FLINK-9617 URL: https://issues.apache.org/jira/browse/FLINK-9617 Project: Flink Issue Type:

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread Fabian Hueske
I see, then this case wasn't covered by the fix that we added for Flink 1.5.0. I guess the problem is that the code is needed to evaluate a single field. Implementing a scalar user-function is not very difficult [1]. However, you need to register it in the TableEnvironment before you can use it

[jira] [Created] (FLINK-9616) DatadogHttpReporter fails to be created due to missing shaded dependency

2018-06-19 Thread Addison Higham (JIRA)
Addison Higham created FLINK-9616: - Summary: DatadogHttpReporter fails to be created due to missing shaded dependency Key: FLINK-9616 URL: https://issues.apache.org/jira/browse/FLINK-9616 Project:

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread zhangminglei
Hi, Fabian, Absolutely, Flink 1.5.0 I am using for this. A big CASE WHEN statement. Is it hard to implement ? I am a new to flink table api & sql. Best Minglei. > 在 2018年6月19日,下午10:36,Fabian Hueske 写道: > > Hi, > > Which version are you using? We fixed a similar issue for Flink 1.5.0. > If

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread Fabian Hueske
Hi, Which version are you using? We fixed a similar issue for Flink 1.5.0. If you can't upgrade yet, you can also implement a user-defined function that evaluates the big CASE WHEN statement. Best, Fabian 2018-06-19 16:27 GMT+02:00 zhangminglei <18717838...@163.com>: > Hi, friends. > > When I

DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread zhangminglei
Hi, friends. When I execute a long sql and get the follow error, how can I have a quick fix ? org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at

Re: Please review FLINK-9610 Add Kafka partitioner that uses the key to partition by

2018-06-19 Thread Niels Basjes
I fixed the problem indicated in your comment and added an extra test for that. CI is currently running the tests. Niels On Tue, Jun 19, 2018 at 12:19 PM, Ted Yu wrote: > Interesting enhancement. > > I left a minor comment on the PR. > > Cheers > > On Tue, Jun 19, 2018 at 12:26 AM, Niels

[jira] [Created] (FLINK-9615) Add

2018-06-19 Thread JIRA
Dominik Wosiński created FLINK-9615: --- Summary: Add Key: FLINK-9615 URL: https://issues.apache.org/jira/browse/FLINK-9615 Project: Flink Issue Type: Improvement Reporter:

[jira] [Created] (FLINK-9614) Improve the error message for Compiler#compile

2018-06-19 Thread mingleizhang (JIRA)
mingleizhang created FLINK-9614: --- Summary: Improve the error message for Compiler#compile Key: FLINK-9614 URL: https://issues.apache.org/jira/browse/FLINK-9614 Project: Flink Issue Type:

Re: Breakage in Flink CLI in 1.5.0

2018-06-19 Thread Chesnay Schepler
In 1.5 we reworked the job-submission to go through the REST API instead of akka. I believe the jobmanager rpc port shouldn't be necessary anymore, the rpc address is still /required /due to some technical implementations; it may be that you can set this to some arbitrary value however. As

Re:Ordering of stream from different kafka partitions

2018-06-19 Thread sihua zhou
Hi Amol, I'm not sure whether this is impossible, especially when you need to operate the record in multi parallelism. IMO, in theroy, we can only get a ordered stream when there is a single partition of kafka and operate it with a single parallelism in flink. Even in this case, if you

Breakage in Flink CLI in 1.5.0

2018-06-19 Thread Sampath Bhat
Hello I'm using Flink 1.5.0 version and Flink CLI to submit jar to flink cluster. In flink 1.4.2 only job manager rpc address and job manager rpc port were sufficient for flink client to connect to job manager and submit the job. But in flink 1.5.0 the flink client additionally requires the

Ordering of stream from different kafka partitions

2018-06-19 Thread Amol S - iProgrammer
Hi, I have used flink streaming API in my application where the source of streaming is kafka. My kafka producer will publish data in ascending order of time in different partitions of kafka and consumer will read data from these partitions. However some kafka partitions may be slow due to some

Re: Please review FLINK-9610 Add Kafka partitioner that uses the key to partition by

2018-06-19 Thread Ted Yu
Interesting enhancement. I left a minor comment on the PR. Cheers On Tue, Jun 19, 2018 at 12:26 AM, Niels Basjes wrote: > Hi, > > Yesterday we ran into problems regarding the distribution of records across > Kafka where Flink was used as the producer. So we fixed this and realized > that the

[jira] [Created] (FLINK-9613) YARNSessionCapacitySchedulerITCase failed because YarnTestBase.checkClusterEmpty()

2018-06-19 Thread Sihua Zhou (JIRA)
Sihua Zhou created FLINK-9613: - Summary: YARNSessionCapacitySchedulerITCase failed because YarnTestBase.checkClusterEmpty() Key: FLINK-9613 URL: https://issues.apache.org/jira/browse/FLINK-9613 Project:

Re: Improvements to Mesos Deployments Using Docker

2018-06-19 Thread Till Rohrmann
Hi Addison, thanks for starting the discussion. My gut feeling is that we could solve FLINK-9611 and FLINK-9612 both with allowing the user to specify a custom AbstractContainerOverlay implementation. Thus, introducing an AbstractContainerOverlayFactory instead of the specific

Please review FLINK-9610 Add Kafka partitioner that uses the key to partition by

2018-06-19 Thread Niels Basjes
Hi, Yesterday we ran into problems regarding the distribution of records across Kafka where Flink was used as the producer. So we fixed this and realized that the code to do this would be useful to others. I put up a Jira ticket and pull request yesterday and it passes all automated tests.

Re: Why not put "FLIP: Network Stack Improvements" into Confluence page?

2018-06-19 Thread Tony Wei
Hi Piotrek, Thanks for your information. =) Best Regards, Tony Wei 2018-06-19 15:15 GMT+08:00 Piotr Nowojski : > Hi, > > Besides FLIP document describing network improvements there is not much > more and it is actually pretty up to date. > > I will link this document on wiki with FLIP

Re: Why not put "FLIP: Network Stack Improvements" into Confluence page?

2018-06-19 Thread Piotr Nowojski
Hi, Besides FLIP document describing network improvements there is not much more and it is actually pretty up to date. I will link this document on wiki with FLIP proposals. Piotrek > On 19 Jun 2018, at 06:22, Tony Wei wrote: > > Hi, > > I read Flink 1.5.0 release announcements[1] recently