[jira] [Created] (FLINK-13868) Job vertex add taskmanager id in rest api

lining created FLINK-13868: -- Summary: Job vertex add taskmanager id in rest api Key: FLINK-13868 URL: https://issues.apache.org/jira/browse/FLINK-13868 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Yun Gao, Thanks a lot for your clarification. Now that the notification of broadcast events requires alignment whose implementation, in my opinion, will affect the correctness of synchronous iterations, I prefer to postpone the discussion until you have completed the design of the new iteratio

[jira] [Created] (FLINK-13867) Write file only once when doing blocking broadcast shuffle

Kurt Young created FLINK-13867: -- Summary: Write file only once when doing blocking broadcast shuffle Key: FLINK-13867 URL: https://issues.apache.org/jira/browse/FLINK-13867 Project: Flink Issue

Flink 1.9 build failed

Hi all I’m trying to build flink 1.9 release branch, it raises the error like: Could not resolve dependencies for project org.apache.flink:flink-s3-fs-hadoop:jar:1.9-SNAPSHOT: Could not find artifact org.apache.flink:flink-fs-hadoop-shaded:jar:tests:1.9-SNAPSHOT in maven-ali (http://maven.al

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Xiaogang, Very thanks for also considering the iteration case! :) These points are really important for iteration. As a whole, we are implementing a new iteration library on top of Stream API. As a library, most of its implementation does not need to touch Runtime layer, but it re

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Jincheng, Thanks! It works. Thanks, Dian > 在 2019年8月27日，上午10:55，jincheng sun 写道： > > Hi Dian, can you check if you have edit access? :) > > > Dian Fu 于2019年8月26日周一 上午10:52写道： > >> Hi Jincheng, >> >> Appreciated for the kind tips and offering of help. Definitely need it! >> Could you gr

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Dian, can you check if you have edit access? :) Dian Fu 于2019年8月26日周一 上午10:52写道： > Hi Jincheng, > > Appreciated for the kind tips and offering of help. Definitely need it! > Could you grant me write permission for confluence? My Id: Dian Fu > > Thanks, > Dian > > > 在 2019年8月26日，上午9:53，jinche

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

Great Job Jark :) Best, Jincheng Kurt Young 于2019年8月26日周一 下午6:38写道： > Thanks for the updates, Jark! I have subscribed the ML and everything > looks good now. > > Best, > Kurt > > > On Mon, Aug 26, 2019 at 11:17 AM Jark Wu wrote: > > > Hi all, > > > > Sorry it take so long to get back. I have so

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, Yun Gao The discussion seems to move in a different direction, changing from supporting multicasting to implementing new iteration libraries on data streams. Regarding the broadcast events in iterations, many details of new iteration libraries are unclear, 1. How the iteration progress is det

[jira] [Created] (FLINK-13866) develop testing plan for many Hive versions that we support

Bowen Li created FLINK-13866: Summary: develop testing plan for many Hive versions that we support Key: FLINK-13866 URL: https://issues.apache.org/jira/browse/FLINK-13866 Project: Flink Issue Ty

[jira] [Created] (FLINK-13865) Support custom config in Flink docker image

Dagang Wei created FLINK-13865: -- Summary: Support custom config in Flink docker image Key: FLINK-13865 URL: https://issues.apache.org/jira/browse/FLINK-13865 Project: Flink Issue Type: Improveme

[jira] [Created] (FLINK-13864) StreamingFileSink: Allow inherited classes to extend StreamingFileSink correctly

Kailash Hassan Dayanand created FLINK-13864: --- Summary: StreamingFileSink: Allow inherited classes to extend StreamingFileSink correctly Key: FLINK-13864 URL: https://issues.apache.org/jira/browse/FLINK-1

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

Seems everyone is in favor in principle. - For public APIs, I would keep Time for now (to not break the API). Maybe add a Duration variant and deprecate the Time variant, but not remove it before Flink 1.0 - For all runtime Java code, switch to Java's Duration now - For all Scala code let's

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, Very thanks for all the points raised ! @Piotr For using another edge to broadcast the event, I think it may not be able to address the iteration case. The primary problem is that with two edges we cannot ensure the order of records. However, In the iteration case, the broadcasted event

[jira] [Created] (FLINK-13863) Update Operations Playground to Flink 1.9.0

Fabian Hueske created FLINK-13863: - Summary: Update Operations Playground to Flink 1.9.0 Key: FLINK-13863 URL: https://issues.apache.org/jira/browse/FLINK-13863 Project: Flink Issue Type: Tas

Re: [CODE-STYLE] Builder pattern

Hi, I agree with Dawid, modulo that I don’t have any preference about point 2 - I’m ok even with not enforcing this. One side note about point 4. There are use cases where passing obligatory parameters in the build method itself might make sense: I. - when those parameters can not be or can n

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Yes, glad to see that there is already a PR for such optimization. Best, Kurt On Mon, Aug 26, 2019 at 6:59 PM Piotr Nowojski wrote: > Hi, > > Xiaogang, those things worry me the most. > > 1. Regarding the broadcasting, doesn’t the BroadcastState [1] cover our > issues? Can not we construct a j

[jira] [Created] (FLINK-13862) Remove or rewrite Execution Plan docs

Stephan Ewen created FLINK-13862: Summary: Remove or rewrite Execution Plan docs Key: FLINK-13862 URL: https://issues.apache.org/jira/browse/FLINK-13862 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-13861) No new checkpoint trigged when canceling an expired checkpoint failed

Congxian Qiu(klion26) created FLINK-13861: - Summary: No new checkpoint trigged when canceling an expired checkpoint failed Key: FLINK-13861 URL: https://issues.apache.org/jira/browse/FLINK-13861

[DISCUSS] Builder dedicated for testing

Hi devs, I'd like to share an observation that we have too many @VisibleForTesting constructors that only used in test scope such as ExecutionGraph and RestClusterClient. It would be helpful if we introduce Builders in test scope for build such instance and remain the production code only necessa

Re: [CODE-STYLE] Builder pattern

Hi Gyula, Thanks for bringing this. I think it would be nice if we have a common approach to create builder pattern. Currently, we have a lot of builders but with different tastes. > 1. Creating the builder objects: I prefer option a) too. It would be easier for users to get the builder instance

Re: [CODE-STYLE] Builder pattern

Hi Gyula, A few comments from my side. Ad. 1 Personally I also prefer a static method in the "built" class. Not sure if I would be that strict about the "Builder" suffix, though. It is usually quite easy to realize the method returns a builder rather than the object itself. In my opinion the suff

[jira] [Created] (FLINK-13860) Flink Apache Kudu Connector

Joao Boto created FLINK-13860: - Summary: Flink Apache Kudu Connector Key: FLINK-13860 URL: https://issues.apache.org/jira/browse/FLINK-13860 Project: Flink Issue Type: New Feature Rep

[CODE-STYLE] Builder pattern

Hi All! I would like to start a code-style related discussion regarding how we implement the builder pattern in the Flink project. It would be the best to have a common approach, there are some aspects of the pattern that come to my mind please feel free to add anything I missed: 1. Creating the

[jira] [Created] (FLINK-13859) JSONDeserializationSchema spell error

limbo created FLINK-13859: - Summary: JSONDeserializationSchema spell error Key: FLINK-13859 URL: https://issues.apache.org/jira/browse/FLINK-13859 Project: Flink Issue Type: Improvement Com

Re: CiBot Update

Thanks Chesnay for the nice work, it's very helpful Best, Congxian Terry Wang 于2019年8月26日周一 下午6:59写道： > Very helpful! Thanks Chesnay! > Best, > Terry Wang > > > > > 在 2019年8月23日，下午11:47，Ethan Li 写道： > > > > Thank you very much Chesnay! This is helpful > > > >> On Aug 23, 2019, at 2:58 AM, Che

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

Before ARM CI is ready, I can close the CI test for each PR and let it only be triggered by PR comment. It's quite easy for OpenLab to do this. OpenLab have many job piplines[1]. Now I use `check` pipline in https://github.com/apache/flink/pull/9416. The job trigger contains github_action and gi

[jira] [Created] (FLINK-13858) Add flink-connector-elasticsearch6 Specify the field as the primary key id

hubin created FLINK-13858: - Summary: Add flink-connector-elasticsearch6 Specify the field as the primary key id Key: FLINK-13858 URL: https://issues.apache.org/jira/browse/FLINK-13858 Project: Flink

Re: CiBot Update

Very helpful! Thanks Chesnay! Best, Terry Wang > 在 2019年8月23日，下午11:47，Ethan Li 写道： > > Thank you very much Chesnay! This is helpful > >> On Aug 23, 2019, at 2:58 AM, Chesnay Schepler wrote: >> >> @Ethan Li The source for the CiBot is available here >> .

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, Xiaogang, those things worry me the most. 1. Regarding the broadcasting, doesn’t the BroadcastState [1] cover our issues? Can not we construct a job graph, where one operator has two outputs, one keyed another broadcasted, which are wired together back to the KeyedBroadcastProcessFunction

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

Thanks for the updates, Jark! I have subscribed the ML and everything looks good now. Best, Kurt On Mon, Aug 26, 2019 at 11:17 AM Jark Wu wrote: > Hi all, > > Sorry it take so long to get back. I have some good news. > > After some investigation and development and the help from Chesnay, we >

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

Adding CI builds for ARM makes only sense when we actually take them into account as "blocking a merge", otherwise there is no point in having them. So we would need to be prepared to do that. The cases where something runs in UNIX/x64 but fails on ARM are few cases and so far seem to have been re

[jira] [Created] (FLINK-13857) Remove remaining UdfAnalyzer configurations

Dawid Wysakowicz created FLINK-13857: Summary: Remove remaining UdfAnalyzer configurations Key: FLINK-13857 URL: https://issues.apache.org/jira/browse/FLINK-13857 Project: Flink Issue Typ

[jira] [Created] (FLINK-13856) Reduce the delete file api when the checkpoint is completed

andrew.D.lin created FLINK-13856: Summary: Reduce the delete file api when the checkpoint is completed Key: FLINK-13856 URL: https://issues.apache.org/jira/browse/FLINK-13856 Project: Flink

[jira] [Created] (FLINK-13855) Keep Travis build in sync with module structure

Till Rohrmann created FLINK-13855: - Summary: Keep Travis build in sync with module structure Key: FLINK-13855 URL: https://issues.apache.org/jira/browse/FLINK-13855 Project: Flink Issue Type:

[jira] [Created] (FLINK-13854) Support Aggregating in Join and CoGroup

Jiayi Liao created FLINK-13854: -- Summary: Support Aggregating in Join and CoGroup Key: FLINK-13854 URL: https://issues.apache.org/jira/browse/FLINK-13854 Project: Flink Issue Type: New Feature

Re: Watermarks not propagated to WebUI?

Hi Robert, I'd very much love to, but because I run my pipeline with Beam, I'm afraid I will have to wait a little longer, before Beam has runner for 1.9 [1]. I'm pretty sure that the watermarks disappeared with overall parallelism (over all operators) something above 2000. There was quite a

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

Sorry, maybe my words is misleading. We are just starting adding ARM support. So the CI is non-voting at this moment to avoid blocking normal Flink development. But once the ARM CI works well and stable enough. We should mark it as voting. It means that in the future, if the ARM test is failed in

[jira] [Created] (FLINK-13853) Running HA (file, async) end-to-end test failed on Travis

Till Rohrmann created FLINK-13853: - Summary: Running HA (file, async) end-to-end test failed on Travis Key: FLINK-13853 URL: https://issues.apache.org/jira/browse/FLINK-13853 Project: Flink I

Re: Watermarks not propagated to WebUI?

Jan, will you be able to test this issue on the now-released Flink 1.9 with the new UI? What parallelism is needed to reproduce the issue? On Thu, Aug 15, 2019 at 1:59 PM Chesnay Schepler wrote: > I remember an issue regarding the watermark fetch request from the WebUI > exceeding some HTTP si

[jira] [Created] (FLINK-13852) Support storing in-progress/pending files in different directories (StreamingFileSink)

Gyula Fora created FLINK-13852: -- Summary: Support storing in-progress/pending files in different directories (StreamingFileSink) Key: FLINK-13852 URL: https://issues.apache.org/jira/browse/FLINK-13852 Pr

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Hi Oytun, I think it intents to publish flink-queryable-state-client-java without scala suffix since it is scala-free. An artifact without scala suffix has been published [2]. See also [1]. Best, tison. [1] https://issues.apache.org/jira/browse/FLINK-12602 [2] https://mvnrepository.com/artifact

[jira] [Created] (FLINK-13851) Time unit of garbage collection in Flink Web UI is not displayed

Jeff Zhang created FLINK-13851: -- Summary: Time unit of garbage collection in Flink Web UI is not displayed Key: FLINK-13851 URL: https://issues.apache.org/jira/browse/FLINK-13851 Project: Flink

Re: [ANNOUNCE] Apache Flink 1.9.0 released

The missing support for the Scala shell with Scala 2.12 was documented in the 1.7 release notes [1]. @Oytun, the docker image should be updated in a bit. Sorry for the inconveniences. Thanks for the pointer that flink-queryable-state-client-java_2.11 hasn't been published. We'll upload this in a b

[jira] [Created] (FLINK-13850) Refactor part file configuration into a single method

Gyula Fora created FLINK-13850: -- Summary: Refactor part file configuration into a single method Key: FLINK-13850 URL: https://issues.apache.org/jira/browse/FLINK-13850 Project: Flink Issue Type:

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

>From SQL's perspective, distributed cross join is a valid feature but not very urgent. Actually this discuss reminds me about another useful feature (sorry for the distraction): when doing broadcast in batch shuffle mode, we can make each producer only write one copy of the output data, but not f