[jira] [Commented] (FLINK-2337) Multiple SLF4J bindings using Storm compatibility layer
[ https://issues.apache.org/jira/browse/FLINK-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623710#comment-14623710 ] ASF GitHub Bot commented on FLINK-2337: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/903#issuecomment-120702067 +1 looks good Multiple SLF4J bindings using Storm compatibility layer --- Key: FLINK-2337 URL: https://issues.apache.org/jira/browse/FLINK-2337 Project: Flink Issue Type: Bug Components: flink-contrib Reporter: Matthias J. Sax Assignee: Matthias J. Sax Priority: Minor Storm depends on logback as slf4j implemenation but Flink uses log4j. The log shows the following conflict: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/cicero/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/cicero/.m2/repository/org/slf4j/slf4j-log4j12/1.7.7/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Need to exclude logback from storm dependencies to fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1737) Add statistical whitening transformation to machine learning library
[
https://issues.apache.org/jira/browse/FLINK-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623974#comment-14623974
]
Daniel Pape commented on FLINK-1737:
Could someone assign me the task? Would be willing to jump to work right after
this ;o)
Add statistical whitening transformation to machine learning library
Key: FLINK-1737
URL: https://issues.apache.org/jira/browse/FLINK-1737
Project: Flink
Issue Type: New Feature
Components: Machine Learning Library
Reporter: Till Rohrmann
Labels: ML, Starter
The statistical whitening transformation [1] is a preprocessing step for
different ML algorithms. It decorrelates the individual dimensions and sets
its variance to 1.
Statistical whitening should be implemented as a {{Transfomer}}.
Resources:
[1] [http://en.wikipedia.org/wiki/Whitening_transformation]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1085] Unnecessary failing of GroupReduc...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/854#discussion_r34425008
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/GroupReduceCombineDriver.java
---
@@ -170,7 +171,14 @@ public void run() throws Exception {
// write the value again
if (!this.sorter.write(value)) {
- throw new IOException(Cannot write record to
fresh sort buffer. Record too large.);
+ if (oversizedRecordCount == Long.MAX_VALUE) {
--- End diff --
I think we can skip this test. It's going to be very hard to process 2^63
oversized records in a single Flink thread. Might take a millennium or so ;-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[
https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623977#comment-14623977
]
ASF GitHub Bot commented on FLINK-1085:
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/854#discussion_r34425008
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/operators/GroupReduceCombineDriver.java
---
@@ -170,7 +171,14 @@ public void run() throws Exception {
// write the value again
if (!this.sorter.write(value)) {
- throw new IOException(Cannot write record to
fresh sort buffer. Record too large.);
+ if (oversizedRecordCount == Long.MAX_VALUE) {
--- End diff --
I think we can skip this test. It's going to be very hard to process 2^63
oversized records in a single Flink thread. Might take a millennium or so ;-)
Unnecessary failing of GroupReduceCombineDriver
---
Key: FLINK-1085
URL: https://issues.apache.org/jira/browse/FLINK-1085
Project: Flink
Issue Type: Bug
Components: Local Runtime
Affects Versions: 0.6.1-incubating, 0.7.0-incubating
Reporter: Fabian Hueske
Labels: starter
With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the
GroupReduceCombineDriver was changed to not use an asynchronous partial
sorter. Instead, the driver fills a sort buffer with records, sorts it,
combines them, clears the buffer, and continues to fill it again.
The GroupReduceCombineDriver fails if a record cannot be serialized into an
empty sort buffer, i.e., if the record is too large for the buffer.
Alternatively, we should emit a WARN message for the first record that is too
large and just forward all records which do not fit into the empty sort
buffer (maybe continue to count how many records were simply forwarded and
give a second WARN message with this statistic).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver
[ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624051#comment-14624051 ] ASF GitHub Bot commented on FLINK-1085: --- Github user dabaitu commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-120769914 ProcessFailureStreamingRecoveryITCase passes locally. Is the master branch broken? Unnecessary failing of GroupReduceCombineDriver --- Key: FLINK-1085 URL: https://issues.apache.org/jira/browse/FLINK-1085 Project: Flink Issue Type: Bug Components: Local Runtime Affects Versions: 0.6.1-incubating, 0.7.0-incubating Reporter: Fabian Hueske Labels: starter With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again. The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer. Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1085] Unnecessary failing of GroupReduc...
Github user dabaitu commented on the pull request: https://github.com/apache/flink/pull/854#issuecomment-120769914 ProcessFailureStreamingRecoveryITCase passes locally. Is the master branch broken? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2346) Mesos clustering
[ https://issues.apache.org/jira/browse/FLINK-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623809#comment-14623809 ] Stephan Ewen commented on FLINK-2346: - You mean running Flink on Mesos? That is a nice project. A little bigger, though ;-) The YARN integration taps into monitoring, configuration, container restarts, etc. Would be great if the Mesos integration behaved similar. Mesos clustering Key: FLINK-2346 URL: https://issues.apache.org/jira/browse/FLINK-2346 Project: Flink Issue Type: New Feature Reporter: Suminda Dharmasena -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2345) Zero Configuration Cluster Setup
[ https://issues.apache.org/jira/browse/FLINK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623815#comment-14623815 ] Suminda Dharmasena commented on FLINK-2345: --- Why not HA for standalone mode without ZK? You can have manual configuration. This can have a option to include or exclude auto discovered instances. You can have a discovery component which finds other instances on the network and configures them. You should be able to restrict the nodes which you look for security and when there are multiple setups. Zero Configuration Cluster Setup Key: FLINK-2345 URL: https://issues.apache.org/jira/browse/FLINK-2345 Project: Flink Issue Type: New Feature Reporter: Suminda Dharmasena Best for of local clustering is using auto discovery. Also automatic tuning of cluster configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2349) Instable (failing) Test
Matthias J. Sax created FLINK-2349: -- Summary: Instable (failing) Test Key: FLINK-2349 URL: https://issues.apache.org/jira/browse/FLINK-2349 Project: Flink Issue Type: Bug Components: Tests Reporter: Matthias J. Sax Instable Test fails regularly: - https://travis-ci.org/apache/flink/builds/70397048 - https://travis-ci.org/mjsax/flink/jobs/70432777 - https://travis-ci.org/mjsax/flink/jobs/70432616 - https://travis-ci.org/mjsax/flink/jobs/70386808 Failed tests: ProcessFailureStreamingRecoveryITCaseAbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure:198 The program encountered a ProgramInvocationException : The program execution failed: Job execution failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-2348) Instable (failing) Test
[ https://issues.apache.org/jira/browse/FLINK-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated FLINK-2348: --- Summary: Instable (failing) Test (was: Instable (failing) Tests) Instable (failing) Test --- Key: FLINK-2348 URL: https://issues.apache.org/jira/browse/FLINK-2348 Project: Flink Issue Type: Bug Components: Streaming Reporter: Matthias J. Sax This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec FAILURE! - in org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase) Time elapsed: 16.34 sec FAILURE! java.lang.AssertionError: Different number of lines in expected and obtained result. expected:18 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258) at org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38) Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase Results : Failed tests: IterateExampleITCaseStreamingProgramTestBase.testJobWithoutObjectReuse:109-postSubmit:38-TestBaseUtils.compareResultsByLinesInMemory:258-TestBaseUtils.compareResultsByLinesInMemory:272 Different number of lines in expected and obtained result. expected:18 but was:0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2343) Change default garbage collector in streaming environments
[
https://issues.apache.org/jira/browse/FLINK-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623832#comment-14623832
]
Yanping Wang commented on FLINK-2343:
-
Hi, Stephan
I'd like to promote G1 as default GC for Flink with following reasons
(1) CMS is no longer the development focus from Oracle Java and OpenJDK, all
the GC development is focusing on G1
(2) Based on jdk9 plan, jdk9's default collector for jdk9 release will be G1
(3) jdk8 update 45 is now the default jdk to be downloaded from
http://www.oracle.com/technetwork/java/javase/downloads/index.html In jdk8
update 45, G1 is in full production with many optimizations fit for Spark, etc
running 100GB plus heap.
(4) In addition, upcoming release of jdk8 update 60 will contain following
optimizations on:
• Refactor Remembered Set for Object Array
• Early reclaim dead humongous objects
• Hot card cache lock contention reduction
• ext root scan optimization (scanning the roots: globals, registers,
thread stacks and VM data structures)
• class unloading during the marking cycle
• object copy optimization
• G1 default value adjustment for some global flags
• Dynamic IHOP
• Fix PLAB sizing heuristic
I used jdk8 update 60 (downloaded from here:
https://jdk8.java.net/download.html ) measured 2X performance improvement
over jdk7 update 79 on one Spark graph computing workload running 4 node
clusters of Intel Ivybridge servers with 160GB java heap.
I was the one of several JVM runtime engineers who helped putting those
optimizations for Spark in jdk8 update 40 and above. Let me know if you have
any questions regarding G1 status and how to best run G1.
Change default garbage collector in streaming environments
--
Key: FLINK-2343
URL: https://issues.apache.org/jira/browse/FLINK-2343
Project: Flink
Issue Type: Improvement
Components: Start-Stop Scripts
Affects Versions: 0.10
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.10
When starting Flink, we don't pass any particular GC related JVM flags to the
system. That means, it uses the default garbage collectors, which are the
bulk parallel GCs for both old gen and new gen.
For streaming applications, this results in vastly fluctuating latencies.
Latencies are much more constant with either the {{CMS}} or {{G1}} GC.
I propose to make the CMS the default GC for streaming setups.
G1 may become the GC of choice in the future, but fro various articles I
found, it is still somewhat in beta status (see for example here:
http://jaxenter.com/kirk-pepperdine-on-the-g1-for-java-9-118190.html )
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user chiwanpark commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120707231 +1 for renaming. I confused the difference between the options and arguments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2218) Web client cannot distinguesh between Flink options and program arguments
[ https://issues.apache.org/jira/browse/FLINK-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623749#comment-14623749 ] ASF GitHub Bot commented on FLINK-2218: --- Github user chiwanpark commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120707231 +1 for renaming. I confused the difference between the options and arguments. Web client cannot distinguesh between Flink options and program arguments - Key: FLINK-2218 URL: https://issues.apache.org/jira/browse/FLINK-2218 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Assignee: Matthias J. Sax WebClient has only one input field for arguments. This field is used for Flink options (e.g., `-p`) and program arguments. Thus, supported Flink options restrict the possible program arguments. CliFrontend in contrast can distinguish both and thus `-p` can also be used as an program argument. Solution: add a second input field for Flink options to WebClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2346) Mesos clustering
[ https://issues.apache.org/jira/browse/FLINK-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623816#comment-14623816 ] Suminda Dharmasena commented on FLINK-2346: --- Best is you can have SPI interface with also supplied components which can spring up VMs as needed. This SPI can perhaps be used for Mesos also. In addition it can manage cloud deployments and manage clustering. You can grab this functionality from some other project and integrated it into flink. Maybe you can convince the other project to make this into a reusable component. (Maybe you can borrow from Ignite the clustering part and share the code and infrastructure.) Mesos clustering Key: FLINK-2346 URL: https://issues.apache.org/jira/browse/FLINK-2346 Project: Flink Issue Type: New Feature Reporter: Suminda Dharmasena -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2218) Web client cannot distinguesh between Flink options and program arguments
[ https://issues.apache.org/jira/browse/FLINK-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623740#comment-14623740 ] ASF GitHub Bot commented on FLINK-2218: --- Github user mjsax commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120706769 Right now, there is only `-c` that must be used if no Manifest (program-class or Main-Class argument) is given. We can also clarify via tooltips or help-popup, that options are Flink parameters and arguments go straight to the user program. As an alternative we could also rename both to Flink Options and Program Arguments to make the difference clearer. I was just using both terms because CLI uses them too. Web client cannot distinguesh between Flink options and program arguments - Key: FLINK-2218 URL: https://issues.apache.org/jira/browse/FLINK-2218 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Assignee: Matthias J. Sax WebClient has only one input field for arguments. This field is used for Flink options (e.g., `-p`) and program arguments. Thus, supported Flink options restrict the possible program arguments. CliFrontend in contrast can distinguish both and thus `-p` can also be used as an program argument. Solution: add a second input field for Flink options to WebClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2348) Instable (failing) Tests
Matthias J. Sax created FLINK-2348: -- Summary: Instable (failing) Tests Key: FLINK-2348 URL: https://issues.apache.org/jira/browse/FLINK-2348 Project: Flink Issue Type: Bug Components: Streaming Reporter: Matthias J. Sax This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872 ``` Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec FAILURE! - in org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase) Time elapsed: 16.34 sec FAILURE! java.lang.AssertionError: Different number of lines in expected and obtained result. expected:18 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258) at org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38) Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase Results : Failed tests: IterateExampleITCaseStreamingProgramTestBase.testJobWithoutObjectReuse:109-postSubmit:38-TestBaseUtils.compareResultsByLinesInMemory:258-TestBaseUtils.compareResultsByLinesInMemory:272 Different number of lines in expected and obtained result. expected:18 but was:0 ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-2348) Instable (failing) Tests
[ https://issues.apache.org/jira/browse/FLINK-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated FLINK-2348: --- Description: This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec FAILURE! - in org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase) Time elapsed: 16.34 sec FAILURE! java.lang.AssertionError: Different number of lines in expected and obtained result. expected:18 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258) at org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38) Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase Results : Failed tests: IterateExampleITCaseStreamingProgramTestBase.testJobWithoutObjectReuse:109-postSubmit:38-TestBaseUtils.compareResultsByLinesInMemory:258-TestBaseUtils.compareResultsByLinesInMemory:272 Different number of lines in expected and obtained result. expected:18 but was:0 was: This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872 ``` Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec FAILURE! - in org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase) Time elapsed: 16.34 sec FAILURE! java.lang.AssertionError: Different number of lines in expected and obtained result. expected:18 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258) at org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38) Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase Results : Failed tests: IterateExampleITCaseStreamingProgramTestBase.testJobWithoutObjectReuse:109-postSubmit:38-TestBaseUtils.compareResultsByLinesInMemory:258-TestBaseUtils.compareResultsByLinesInMemory:272 Different number of lines in expected and obtained result. expected:18 but was:0 ``` Instable (failing) Tests Key: FLINK-2348 URL: https://issues.apache.org/jira/browse/FLINK-2348 Project: Flink Issue Type: Bug Components: Streaming Reporter: Matthias J. Sax This test is not stable: See: https://travis-ci.org/mjsax/flink/jobs/70396872 Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.057 sec FAILURE! - in org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase testJobWithoutObjectReuse(org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase) Time elapsed: 16.34 sec FAILURE! java.lang.AssertionError: Different number of lines in expected and obtained result. expected:18 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:272) at org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:258) at org.apache.flink.streaming.test.exampleJavaPrograms.iteration.IterateExampleITCase.postSubmit(IterateExampleITCase.java:38) Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.827 sec - in org.apache.flink.streaming.test.exampleScalaPrograms.join.WindowJoinITCase Results : Failed tests: IterateExampleITCaseStreamingProgramTestBase.testJobWithoutObjectReuse:109-postSubmit:38-TestBaseUtils.compareResultsByLinesInMemory:258-TestBaseUtils.compareResultsByLinesInMemory:272 Different number of lines in expected and obtained result. expected:18 but was:0 --
[jira] [Commented] (FLINK-2349) Instable (failing) Test
[ https://issues.apache.org/jira/browse/FLINK-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623825#comment-14623825 ] Stephan Ewen commented on FLINK-2349: - I found the source of this problem and have a fix pending... Instable (failing) Test --- Key: FLINK-2349 URL: https://issues.apache.org/jira/browse/FLINK-2349 Project: Flink Issue Type: Bug Components: Tests Reporter: Matthias J. Sax Instable Test fails regularly: - https://travis-ci.org/apache/flink/builds/70397048 - https://travis-ci.org/mjsax/flink/jobs/70432777 - https://travis-ci.org/mjsax/flink/jobs/70432616 - https://travis-ci.org/mjsax/flink/jobs/70386808 Failed tests: ProcessFailureStreamingRecoveryITCaseAbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure:198 The program encountered a ProgramInvocationException : The program execution failed: Job execution failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2337] Multiple SLF4J bindings using Sto...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/903#issuecomment-120702067 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2218) Web client cannot distinguesh between Flink options and program arguments
[ https://issues.apache.org/jira/browse/FLINK-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623709#comment-14623709 ] ASF GitHub Bot commented on FLINK-2218: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120702048 Are there any other possible options for the options field except for the parallelism? I think users might be confused how the options and arguments differ. Web client cannot distinguesh between Flink options and program arguments - Key: FLINK-2218 URL: https://issues.apache.org/jira/browse/FLINK-2218 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Assignee: Matthias J. Sax WebClient has only one input field for arguments. This field is used for Flink options (e.g., `-p`) and program arguments. Thus, supported Flink options restrict the possible program arguments. CliFrontend in contrast can distinguish both and thus `-p` can also be used as an program argument. Solution: add a second input field for Flink options to WebClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120702048 Are there any other possible options for the options field except for the parallelism? I think users might be confused how the options and arguments differ. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-2141) Allow GSA's Gather to perform this operation in more than one direction
[ https://issues.apache.org/jira/browse/FLINK-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andra Lungu resolved FLINK-2141. Resolution: Fixed Fix Version/s: 0.10 Allow GSA's Gather to perform this operation in more than one direction --- Key: FLINK-2141 URL: https://issues.apache.org/jira/browse/FLINK-2141 Project: Flink Issue Type: New Feature Components: Gelly Affects Versions: 0.9 Reporter: Andra Lungu Assignee: Shivani Ghatge Fix For: 0.10 For the time being, a vertex only gathers information from its in-edges. Similarly to the vertex-centric approach, we would like to allow users to gather data from out and all edges as well. This property should be set using a setDirection() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user mjsax commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120706769 Right now, there is only `-c` that must be used if no Manifest (program-class or Main-Class argument) is given. We can also clarify via tooltips or help-popup, that options are Flink parameters and arguments go straight to the user program. As an alternative we could also rename both to Flink Options and Program Arguments to make the difference clearer. I was just using both terms because CLI uses them too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2200) Flink API with Scala 2.11 - Maven Repository
[ https://issues.apache.org/jira/browse/FLINK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623801#comment-14623801 ] ASF GitHub Bot commented on FLINK-2200: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/885#issuecomment-120720565 Let's wait for @rmetzger to give a +1 to this. He has done most of the POM magic and can probably give a good review... Flink API with Scala 2.11 - Maven Repository Key: FLINK-2200 URL: https://issues.apache.org/jira/browse/FLINK-2200 Project: Flink Issue Type: Wish Components: Build System, Scala API Reporter: Philipp Götze Assignee: Chiwan Park Priority: Trivial Labels: maven It would be nice if you could upload a pre-built version of the Flink API with Scala 2.11 to the maven repository. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2200] Add Flink with Scala 2.11 in Mave...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/885#issuecomment-120720565 Let's wait for @rmetzger to give a +1 to this. He has done most of the POM magic and can probably give a good review... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2345) Zero Configuration Cluster Setup
[ https://issues.apache.org/jira/browse/FLINK-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623806#comment-14623806 ] Stephan Ewen commented on FLINK-2345: - Can you give us a few more details about this effort? I assume it only affects the standalone mode, as the YARN mode does not need discovery (YARN manages nodes and containers). Also, in the high-availability mode that is coming up, discovers of the masters/leader will happen with zookeeper. This proposal would hence affect the standalone mode in non-HA setups. That is fair, such setups are common for experimenting. Can you outline how the mechanism should work: - How does it interoperate with the mode where people still configure the cluster manually (I think that mode should keep existing)? - What is the general mechanism? Multicast/broadcast messages? - How do nodes know which master to discover, if multiple Flink setups run in the same network? BTW: Here are some notes about how discovery happens in the upcoming master high-availability with zookeeper: https://cwiki.apache.org/confluence/display/FLINK/JobManager+High+Availability Zero Configuration Cluster Setup Key: FLINK-2345 URL: https://issues.apache.org/jira/browse/FLINK-2345 Project: Flink Issue Type: New Feature Reporter: Suminda Dharmasena Best for of local clustering is using auto discovery. Also automatic tuning of cluster configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2343) Change default garbage collector in streaming environments
[
https://issues.apache.org/jira/browse/FLINK-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623879#comment-14623879
]
Yanping Wang commented on FLINK-2343:
-
Hi, Stephan
There were some bug fixes and optimizations back ported to jdk7 update 60 and
later but not all, and not up to date. The most recent or last jdk7, which is
update 79,
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
that probably is the best G1 in jdk7 you can get. As jdk7 is no longer to be
updated, update 79 was marked the end of jdk7.
The most helpful optimizations and fixes were in upcoming jdk8 update 60
(scheduled to be default java download in sep/oct 2015) , and jdk9 a year later.
The spark graph computing workload performance I measured 2X performance gain
on 160GB heap was compare jdk7u79 vs. jdk8u60. Also there is an article on
tuning G1 for Spark was based on jdk8u40 from my fellow engineers working on
Spark (my name was listed at ACK section :-)
https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html
One good point by using G1 is, if we find problems, we can ask Oracle Java
developers to fix and to optimize. For CMS, there is no development efforts
there, as it will be replaced by G1.
By the way, in the experiment I did and also mentioned by the paper above, CMS
was bad because the Spark workload was a throughput workload with low latency
requirement. Throughput was run down so much when CMS running into full GC to
compact a large heap.
If you run G1 on Flink and see funning things, let me know I think I can try
making it better.
Thanks
-yanping
Change default garbage collector in streaming environments
--
Key: FLINK-2343
URL: https://issues.apache.org/jira/browse/FLINK-2343
Project: Flink
Issue Type: Improvement
Components: Start-Stop Scripts
Affects Versions: 0.10
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.10
When starting Flink, we don't pass any particular GC related JVM flags to the
system. That means, it uses the default garbage collectors, which are the
bulk parallel GCs for both old gen and new gen.
For streaming applications, this results in vastly fluctuating latencies.
Latencies are much more constant with either the {{CMS}} or {{G1}} GC.
I propose to make the CMS the default GC for streaming setups.
G1 may become the GC of choice in the future, but fro various articles I
found, it is still somewhat in beta status (see for example here:
http://jaxenter.com/kirk-pepperdine-on-the-g1-for-java-9-118190.html )
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2351) Deprecate config builders in InputFormats and Output formats
Stephan Ewen created FLINK-2351: --- Summary: Deprecate config builders in InputFormats and Output formats Key: FLINK-2351 URL: https://issues.apache.org/jira/browse/FLINK-2351 Project: Flink Issue Type: Bug Components: Core Affects Versions: 0.10 Reporter: Stephan Ewen Fix For: 0.10 Old APIs used to pass functions as classes and parameters as configs. To support that, all input- and output formats had config builders. As the record API is deprecated and about to be removed, these builders are no longer needed and should be deprecated and removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120731537 Yes, would be great if we could reuse CLI code., agreed... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-2350) Deprecate the stream open timeouts in FileInputFormat and FileOutputFormat
Stephan Ewen created FLINK-2350: --- Summary: Deprecate the stream open timeouts in FileInputFormat and FileOutputFormat Key: FLINK-2350 URL: https://issues.apache.org/jira/browse/FLINK-2350 Project: Flink Issue Type: Bug Components: Core Affects Versions: 0.10 Reporter: Stephan Ewen The stream open requests have the ability to fail after a certain timeout. By default, this timeout is deactivated, because loaded HDFS has in the past frequently exceeded this timeout. I think no one ever used this, and I am not sure it is useful, actually. We can remove it to reduce code complexity, in my opinion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user mjsax commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120731470 Btw: CliFontend support the follows options: -c / --class , -m / --jobmanager , -p / -- paralleslism To have the same options, we might want to extend CliFontend for those. What do you think? Furthermore, WebClient parses options -c and -p in an own code path. This results in code duplication and manual synchronization of options in CliFrontend and WebClient. We should reuse CliFrontend code in WebClient (if possible). What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2218) Web client cannot distinguesh between Flink options and program arguments
[ https://issues.apache.org/jira/browse/FLINK-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623860#comment-14623860 ] ASF GitHub Bot commented on FLINK-2218: --- Github user mjsax commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120731470 Btw: CliFontend support the follows options: -c / --class , -m / --jobmanager , -p / -- paralleslism To have the same options, we might want to extend CliFontend for those. What do you think? Furthermore, WebClient parses options -c and -p in an own code path. This results in code duplication and manual synchronization of options in CliFrontend and WebClient. We should reuse CliFrontend code in WebClient (if possible). What do you think? Web client cannot distinguesh between Flink options and program arguments - Key: FLINK-2218 URL: https://issues.apache.org/jira/browse/FLINK-2218 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Assignee: Matthias J. Sax WebClient has only one input field for arguments. This field is used for Flink options (e.g., `-p`) and program arguments. Thus, supported Flink options restrict the possible program arguments. CliFrontend in contrast can distinguish both and thus `-p` can also be used as an program argument. Solution: add a second input field for Flink options to WebClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2349) Instable (failing) Test
[ https://issues.apache.org/jira/browse/FLINK-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2349. - Resolution: Fixed Assignee: Stephan Ewen Fix Version/s: 0.10 Fixed via d0ecb9170e5edef48c7efd95764eeec7dbdf51a8 WARNING: tagged the commit accidentally with the wrong issue number. Instable (failing) Test --- Key: FLINK-2349 URL: https://issues.apache.org/jira/browse/FLINK-2349 Project: Flink Issue Type: Bug Components: Tests Reporter: Matthias J. Sax Assignee: Stephan Ewen Fix For: 0.10 Instable Test fails regularly: - https://travis-ci.org/apache/flink/builds/70397048 - https://travis-ci.org/mjsax/flink/jobs/70432777 - https://travis-ci.org/mjsax/flink/jobs/70432616 - https://travis-ci.org/mjsax/flink/jobs/70386808 Failed tests: ProcessFailureStreamingRecoveryITCaseAbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure:198 The program encountered a ProgramInvocationException : The program execution failed: Job execution failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2349) Instable (failing) Test
[ https://issues.apache.org/jira/browse/FLINK-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2349. --- Instable (failing) Test --- Key: FLINK-2349 URL: https://issues.apache.org/jira/browse/FLINK-2349 Project: Flink Issue Type: Bug Components: Tests Reporter: Matthias J. Sax Assignee: Stephan Ewen Fix For: 0.10 Instable Test fails regularly: - https://travis-ci.org/apache/flink/builds/70397048 - https://travis-ci.org/mjsax/flink/jobs/70432777 - https://travis-ci.org/mjsax/flink/jobs/70432616 - https://travis-ci.org/mjsax/flink/jobs/70386808 Failed tests: ProcessFailureStreamingRecoveryITCaseAbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure:198 The program encountered a ProgramInvocationException : The program execution failed: Job execution failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2218) Web client cannot distinguesh between Flink options and program arguments
[ https://issues.apache.org/jira/browse/FLINK-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623862#comment-14623862 ] ASF GitHub Bot commented on FLINK-2218: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120731537 Yes, would be great if we could reuse CLI code., agreed... Web client cannot distinguesh between Flink options and program arguments - Key: FLINK-2218 URL: https://issues.apache.org/jira/browse/FLINK-2218 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Assignee: Matthias J. Sax WebClient has only one input field for arguments. This field is used for Flink options (e.g., `-p`) and program arguments. Thus, supported Flink options restrict the possible program arguments. CliFrontend in contrast can distinguish both and thus `-p` can also be used as an program argument. Solution: add a second input field for Flink options to WebClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-2300) Links on FAQ page not rendered correctly
[
https://issues.apache.org/jira/browse/FLINK-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enrique Bautista Barahona updated FLINK-2300:
-
Attachment: fix_github_plugin.patch
The attached patch should fix the problem.
I have noticed there are other links to the GitHub repo in the website that
don't use the tag. Now that it works they could be modified to use it, in this
issue or a new one.
Links on FAQ page not rendered correctly
Key: FLINK-2300
URL: https://issues.apache.org/jira/browse/FLINK-2300
Project: Flink
Issue Type: Bug
Components: Project Website
Reporter: Robert Metzger
Priority: Minor
Labels: starter
Attachments: fix_github_plugin.patch
On the Flink website, the links using the github plugin are broken.
For example
{code}
{% github README.md master build instructions %}
{code}
renders to
{code}
https://github.com/apache/flink/tree/master/README.md
{code}
See:
http://flink.apache.org/faq.html#my-job-fails-early-with-a-javaioeofexception-what-could-be-the-cause
I was not able to resolve the issue by using {{a}} tags or markdown links.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
