Re: [DISCUSS] Use Java's Duration instead of Flink's Time

Hi vino, I agree that it introduces extra complexity to replace Duration(Scala) with Duration(Java) *in Scala code*. We could separate the usage for each language and use a bridge when necessary. As a matter of fact, Scala concurrent APIs(including Duration) are used more than necessary at least

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Got it. Thanks Till & Zili. +1 for the release notes need to cover such issues. On Fri, Aug 23, 2019 at 11:01 PM Oytun Tez wrote: > Hi all, > > We also had to rollback our upgrade effort for 2 reasons: > > - Official Docker container is not ready yet > - This artefact is not published with >

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

+1 to replace the Time class provided by Flink with Java's Duration: - Java's Duration has better representation than the Flink's Time class; - As a built-in Java class, Duration class has a clear advantage over Java's Time class when interacting with other Java APIs and third-party

[jira] [Created] (FLINK-13841) Extend Hive version support to all 1.2 and 2.3 versions

Xuefu Zhang created FLINK-13841: --- Summary: Extend Hive version support to all 1.2 and 2.3 versions Key: FLINK-13841 URL: https://issues.apache.org/jira/browse/FLINK-13841 Project: Flink Issue

[VOTE] Release flink-shaded 8.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 8.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

I'm wondering what we are supposed to do if the build fails? We aren't providing and guides on setting up an arm dev environment; so reproducing it locally isn't possible. On 23/08/2019 17:55, Stephan Ewen wrote: Hi all! As part of the Flink on ARM effort, there is a pull request that

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Piotr, Very thanks for the suggestions! Totally agree with that we could first focus on the broadcast scenarios and exposing the broadcastEmit method first considering the semantics and performance. For the keyed stream, I also agree with that broadcasting keyed

[DISCUSS] Add ARM CI build to Flink (information-only)

Hi all! As part of the Flink on ARM effort, there is a pull request that triggers a build on OpenLabs CI for each push and runs tests on ARM machines. Currently that build is roughly equivalent to what the "core" and "tests" profiles do on Travis. The result will be posted to the PR comments,

Re: CiBot Update

Thank you very much Chesnay! This is helpful > On Aug 23, 2019, at 2:58 AM, Chesnay Schepler wrote: > > @Ethan Li The source for the CiBot is available here > . The implementation of this command is > tightly connected to how the CiBot works; but

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Hi all, We also had to rollback our upgrade effort for 2 reasons: - Official Docker container is not ready yet - This artefact is not published with scala: org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.9.0 --- Oytun Tez *M O T A W O R D* The World's Fastest Human

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

Hi Stephan, I like the idea unify usage of time/duration api. We actually use at least five different classes for this purposes(see below). One thing I'd like to pick up is that duration configuration in Flink is almost in pattern as "60 s" that fits in the pattern parsed by

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Hi Till, Did we mention this in release note(or maybe previous release note where we did the exclusion)? Best, tison. Till Rohrmann 于2019年8月23日周五 下午10:28写道： > Hi Gavin, > > if I'm not mistaken, then the community excluded the Scala FlinkShell > since a couple of versions for Scala 2.12. The

Re: [DISCUSS] Flink client api enhancement for downstream project

Hi Till, Thanks for your update. Nice to hear :-) Best, tison. Till Rohrmann 于2019年8月23日周五 下午10:39写道： > Hi Tison, > > just a quick comment concerning the class loading issues when using the per > job mode. The community wants to change it so that the > StandaloneJobClusterEntryPoint actually

[DISCUSS] Use Java's Duration instead of Flink's Time

Hi all! Many parts of the code use Flink's "Time" class. The Time really is a "time interval" or a "Duration". Since Java 8, there is a Java class "Duration" that is nice and flexible to use. I would suggest we start using Java Duration instead and drop Time as much as possible in the runtime

Re: [DISCUSS] Flink client api enhancement for downstream project

Hi Tison, just a quick comment concerning the class loading issues when using the per job mode. The community wants to change it so that the StandaloneJobClusterEntryPoint actually uses the user code class loader with child first class loading [1]. Hence, I hope that this problem will be resolved

[jira] [Created] (FLINK-13840) Let StandaloneJobClusterEntryPoint use user code class loader

Till Rohrmann created FLINK-13840: - Summary: Let StandaloneJobClusterEntryPoint use user code class loader Key: FLINK-13840 URL: https://issues.apache.org/jira/browse/FLINK-13840 Project: Flink

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, If the primary motivation is broadcasting (for the iterations) and we have no immediate need for multicast (cross join), I would prefer to first expose broadcast via the DataStream API and only later, once we finally need it, support multicast. As I wrote, multicast would be more

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Hi Gavin, if I'm not mistaken, then the community excluded the Scala FlinkShell since a couple of versions for Scala 2.12. The problem seems to be that some of the tests failed. See here [1] for more information. [1] https://issues.apache.org/jira/browse/FLINK-10911 Cheers, Till On Fri, Aug

[jira] [Created] (FLINK-13839) Support to set yarn node label for flink jobmanager and taskmanager container

Yang Wang created FLINK-13839: - Summary: Support to set yarn node label for flink jobmanager and taskmanager container Key: FLINK-13839 URL: https://issues.apache.org/jira/browse/FLINK-13839 Project:

[jira] [Created] (FLINK-13838) Support -yta(--yarnshipArchives) arguments in flink run command line

Yang Wang created FLINK-13838: - Summary: Support -yta(--yarnshipArchives) arguments in flink run command line Key: FLINK-13838 URL: https://issues.apache.org/jira/browse/FLINK-13838 Project: Flink

[jira] [Created] (FLINK-13837) Support --files and --libjars arguments in flink run command line

Yang Wang created FLINK-13837: - Summary: Support --files and --libjars arguments in flink run command line Key: FLINK-13837 URL: https://issues.apache.org/jira/browse/FLINK-13837 Project: Flink

Re: [DISCUSS] Flink client api enhancement for downstream project

Hi all, On the topic of web submission, I agree with Till that it only seems to complicate things. It is bad for security, job isolation (anybody can submit/cancel jobs), and its implementation complicates some parts of the code. So, if it were to redesign the WebUI, maybe this part could be left

[jira] [Created] (FLINK-13836) Improve support of java.util.UUID for JDBCTypeUtil

François Lacombe created FLINK-13836: Summary: Improve support of java.util.UUID for JDBCTypeUtil Key: FLINK-13836 URL: https://issues.apache.org/jira/browse/FLINK-13836 Project: Flink

[jira] [Created] (FLINK-13835) Add class for FeatureHasherMapper.

Xu Yang created FLINK-13835: --- Summary: Add class for FeatureHasherMapper. Key: FLINK-13835 URL: https://issues.apache.org/jira/browse/FLINK-13835 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13834) Add class for PolynomialExpansionMapper

Xu Yang created FLINK-13834: --- Summary: Add class for PolynomialExpansionMapper Key: FLINK-13834 URL: https://issues.apache.org/jira/browse/FLINK-13834 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13833) Incorrect Maven dependencies for Flink-Hive connector

Qi Kang created FLINK-13833: --- Summary: Incorrect Maven dependencies for Flink-Hive connector Key: FLINK-13833 URL: https://issues.apache.org/jira/browse/FLINK-13833 Project: Flink Issue Type: Bug

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Thanks Piotr, Users asked for this feature sometimes ago when they migrating batch jobs to Flink(Blink). It's not very urgent as they have taken some workarounds to solve it.(like partitioning data set to different job vertices) So it's fine to not make it top priority. Anyway, as a commonly

[jira] [Created] (FLINK-13832) DefaultRollingPolicy create() method should be renamed to builder()

Gyula Fora created FLINK-13832: -- Summary: DefaultRollingPolicy create() method should be renamed to builder() Key: FLINK-13832 URL: https://issues.apache.org/jira/browse/FLINK-13832 Project: Flink

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, Thanks for the answers :) Ok I understand the full picture now. +1 from my side on solving this issue somehow. But before we start discussing how to solve it one last control question: I guess this multicast is intended to be used in blink planner, right? Assuming that we implement the

[jira] [Created] (FLINK-13831) slots total display error

Yu Wang created FLINK-13831: --- Summary: slots total display error Key: FLINK-13831 URL: https://issues.apache.org/jira/browse/FLINK-13831 Project: Flink Issue Type: Bug Components:

[jira] [Created] (FLINK-13830) The Document about Cluster on yarn have some problems

zhangmeng created FLINK-13830: - Summary: The Document about Cluster on yarn have some problems Key: FLINK-13830 URL: https://issues.apache.org/jira/browse/FLINK-13830 Project: Flink Issue Type:

Re: 1.9 release uses docs of master branch

I’ve filed a ticket to track this. [1] https://issues.apache.org/jira/browse/FLINK-13829 Best, Paul Lam > 在 2019年8月23日，16:58，Paul Lam 写道： > > Hi devs, > > I've just noticed that the documentation of 1.9 release links to docs of > master

Re: [ANNOUNCE] Apache Flink 1.9.0 released

I used package on apache official site, with mirror [1], the difference is I used scala 2.12 version. I also tried to build from source for both scala 2.11 and 2.12, when build 2.12 the FlinkShell.class is in flink-dist jar file but after running mvn clean package -Dscala-2.12, this class was

[jira] [Created] (FLINK-13829) Incorrect doc version of 1.9 docs

Paul Lin created FLINK-13829: Summary: Incorrect doc version of 1.9 docs Key: FLINK-13829 URL: https://issues.apache.org/jira/browse/FLINK-13829 Project: Flink Issue Type: Task

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Piotr, Thanks a lot for sharing the thoughts! For the iteration, agree with that multicasting is not necessary. Exploring the broadcast interface to Output of the operators in some way should also solve this issue, and I think it should be even more convenient to have

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Piotr, Yes you are right it's a distributed cross join requirement. Broadcast join can help with cross join cases. But users cannot use it if the data set to join is too large to fit into one subtask. Sorry for left some details behind. Thanks, Zhu Zhu Piotr Nowojski 于2019年8月23日周五

1.9 release uses docs of master branch

Hi devs, I've just noticed that the documentation of 1.9 release links to docs of master branch. It would be appreciated if someone can fix this. Thanks a lot! Best, Paul Lam

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Yun and Zhu Zhu, Thanks for the more detailed example Zhu Zhu. As far as I understand for the iterations example we do not need multicasting. Regarding the Join example, I don’t fully understand it. The example that Zhu Zhu presented has a drawback of sending both tables to multiple nodes.

[jira] [Created] (FLINK-13828) Deprecate ConfigConstants.LOCAL_START_WEBSERVER

TisonKun created FLINK-13828: Summary: Deprecate ConfigConstants.LOCAL_START_WEBSERVER Key: FLINK-13828 URL: https://issues.apache.org/jira/browse/FLINK-13828 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Apache Flink 1.9.0 released

I downloaded 1.9.0 dist from here[1] and didn't see the issue. Where do you download it? Could you try to download the dist from [1] and see whether the problem last? Best, tison. [1] http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.9.0/flink-1.9.0-bin-scala_2.11.tgz Gavin Lee

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Thanks for your reply @Zili. I'm afraid it's not the same issue. I found that the FlinkShell.class was not included in flink dist jar file in 1.9.0 version. Nowhere can find this class file inside jar, either in opt or lib directory under root folder of flink distribution. On Fri, Aug 23, 2019

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Piotr, Thanks a lot for the suggestions! The core motivation of this discussion is to implement a new iteration library on the DataStream, and it requires to insert special records in the stream to notify the progress of the iteration. The mechanism of such records is

[jira] [Created] (FLINK-13827) shell variable should be escaped in start-scala-shell.sh

TisonKun created FLINK-13827: Summary: shell variable should be escaped in start-scala-shell.sh Key: FLINK-13827 URL: https://issues.apache.org/jira/browse/FLINK-13827 Project: Flink Issue Type:

[jira] [Created] (FLINK-13826) Support INSERT OVERWRITE for Hive connector

Rui Li created FLINK-13826: -- Summary: Support INSERT OVERWRITE for Hive connector Key: FLINK-13826 URL: https://issues.apache.org/jira/browse/FLINK-13826 Project: Flink Issue Type: Bug

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi Piotr, The case is about a broadcast join: A--\ +--(join)--> C B--/ Usually we can broadcast A(the result that JobVertex A produces) to all subtasks of C. But in this case the size of A is too large to fit in one subtask of C. Thus we have to partition A to (A_0, A_1, A_2, ..., A_m-1).

Re: CiBot Update

@Ethan Li The source for the CiBot is available here . The implementation of this command is tightly connected to how the CiBot works; but conceptually it looks at a PR, finds the most recent build that ran, and uses the Travis REST API to restart the

[jira] [Created] (FLINK-13824) Code duplication in tools/trawis_watchdog.sh to launch watchdog process

Alex created FLINK-13824: Summary: Code duplication in tools/trawis_watchdog.sh to launch watchdog process Key: FLINK-13824 URL: https://issues.apache.org/jira/browse/FLINK-13824 Project: Flink

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Why bin/start-scala-shell.sh local return following error? bin/start-scala-shell.sh local Error: Could not find or load main class org.apache.flink.api.scala.FlinkShell For flink 1.8.1 and previous ones, no such issues. On Fri, Aug 23, 2019 at 2:05 PM qi luo wrote: > Congratulations and

[jira] [Created] (FLINK-13823) Incorrect debug log in CompileUtils

wangsan created FLINK-13823: --- Summary: Incorrect debug log in CompileUtils Key: FLINK-13823 URL: https://issues.apache.org/jira/browse/FLINK-13823 Project: Flink Issue Type: Bug

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Hi, Yun: Thanks for proposing the idea. I have checked the document and left couple of questions there, but it might be better to answer them here. What is the exact motivation and what problems do you want to solve? We have dropped multicast support from the network stack [1] for two

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

Thanks Yun for starting this discussion. I think the multicasting can be very helpful in certain cases. I have received requirements from users that they want to do broadcast join, while the data set to broadcast is too large to fit in one task. Thus the requirement turned out to be to support

Re: [ANNOUNCE] Apache Flink 1.9.0 released

Congratulations and thanks for the hard work! Qi > On Aug 22, 2019, at 8:03 PM, Tzu-Li (Gordon) Tai wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.9.0, which is the latest major release. > > Apache Flink® is an open-source stream processing