[jira] [Commented] (FLINK-2017) Add predefined required parameters to ParameterTool
[
https://issues.apache.org/jira/browse/FLINK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970541#comment-14970541
]
ASF GitHub Bot commented on FLINK-2017:
---
Github user mliesenberg commented on the pull request:
https://github.com/apache/flink/pull/1097#issuecomment-150490748
I think that might just boil down to the question whether or not there is a
common enough use case for one without the other.
If so, we could change it to one method for check and one for populate
which would untangle the code a bit as well.
If there is no use case for one without the other, I'd agree to get rid of
the check.
I am unfortunately a bit pressed for time at the moment, but hope to find
time to address the other comments during the weekend and.
> Add predefined required parameters to ParameterTool
> ---
>
> Key: FLINK-2017
> URL: https://issues.apache.org/jira/browse/FLINK-2017
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 0.9
>Reporter: Robert Metzger
> Labels: starter
>
> In FLINK-1525 we've added the {{ParameterTool}}.
> During the PR review, there was a request for required parameters.
> This issue is about implementing a facility to define required parameters.
> The tool should also be able to print a help menu with a list of all
> parameters.
> This test case shows my initial ideas how to design the API
> {code}
> @Test
> public void requiredParameters() {
> RequiredParameters required = new RequiredParameters();
> Option input = required.add("input").alt("i").help("Path to
> input file or directory"); // parameter with long and short variant
> required.add("output"); // parameter only with long variant
> Option parallelism =
> required.add("parallelism").alt("p").type(Integer.class); // parameter with
> type
> Option spOption =
> required.add("sourceParallelism").alt("sp").defaultValue(12).help("Number
> specifying the number of parallel data source instances"); // parameter with
> default value, specifying the type.
> Option executionType =
> required.add("executionType").alt("et").defaultValue("pipelined").choices("pipelined",
> "batch");
> ParameterUtil parameter = ParameterUtil.fromArgs(new
> String[]{"-i", "someinput", "--output", "someout", "-p", "15"});
> required.check(parameter);
> required.printHelp();
> required.checkAndPopulate(parameter);
> String inputString = input.get();
> int par = parallelism.getInteger();
> String output = parameter.get("output");
> int sourcePar = parameter.getInteger(spOption.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2017] Add predefined required parameter...
Github user mliesenberg commented on the pull request: https://github.com/apache/flink/pull/1097#issuecomment-150490748 I think that might just boil down to the question whether or not there is a common enough use case for one without the other. If so, we could change it to one method for check and one for populate which would untangle the code a bit as well. If there is no use case for one without the other, I'd agree to get rid of the check. I am unfortunately a bit pressed for time at the moment, but hope to find time to address the other comments during the weekend and. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Mesos
Github user cfmcgrady closed the pull request at: https://github.com/apache/flink/pull/1295 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Mesos
GitHub user cfmcgrady opened a pull request: https://github.com/apache/flink/pull/1295 Mesos - remoe httpserver - add make-distribution.sh - tool for making binary distributions of flink You can merge this pull request into a Git repository by running: $ git pull https://github.com/cfmcgrady/flink mesos Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1295.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1295 commit d5d7b225f137f08f72bd13a0aa13fbe94da544dd Author: Ankur Chauhan Date: 2015-07-20T08:45:41Z wip commit bc62e47cbf3b82b6173ed1726bd53654993067b9 Author: Ankur Chauhan Date: 2015-07-25T19:18:42Z WIP - flink mesos integration initial commit commit ca46a26ae5af30efe52577eaef137bb468647376 Author: Ankur Chauhan Date: 2015-07-25T19:33:57Z Add todo for config changes/issues commit d6d045b269184525d611640ca53f796a2229856a Author: Ankur Chauhan Date: 2015-07-27T00:03:12Z Fix some stuff commit 09a3f75b686177e130ef78eb20ca5fbf55f607cf Author: Ankur Chauhan Date: 2015-07-28T01:41:20Z Add flink-mesos to dist, update scheduler and executor code commit 25fed5fc36012eef986ae3368df8849a758ac4d0 Author: Ankur Chauhan Date: 2015-07-28T03:14:16Z Clean up code + codestyle changes commit 1d7dfbc393ddafb2fa407753d5b25f654af25d65 Author: Ankur Chauhan Date: 2015-07-28T07:12:55Z Clean up executor, adjust resource defaults commit 057705750602ed055d302387a9e56fdd0c7e7b25 Author: Ankur Chauhan Date: 2015-07-28T20:56:24Z Remove status pinging and fix logging level in TaskManagerExecutor commit 5269ad1a6490ae8a98af3c5eb404664d57df1ca1 Author: Ankur Chauhan Date: 2015-07-28T22:33:54Z Code cleanup and refactoring of the conf classes commit 61b28fef345dfc0390df7911a68ed8e0764d5f32 Author: Ankur Chauhan Date: 2015-07-28T22:36:04Z Remove unused dependencies and plugins commit 038c74b59951c5b33fbcdeae0f224ef36d3016a8 Author: Ankur Chauhan Date: 2015-07-28T22:40:50Z Use tab based indents in pom files commit cd793892c8ac939824f675ab293af947ad66b211 Author: Ankur Chauhan Date: 2015-07-28T23:06:30Z Add comments for each of the properties commit 5a701eaab144d03146ed356097b96fb4b8a39992 Author: Ankur Chauhan Date: 2015-07-28T23:22:57Z Fix pom + Add flink-mesos entry to travis CI confiuration commit f75c3202e0a2589bde9af9d2c6388d9458977946 Author: Ankur Chauhan Date: 2015-07-28T23:54:29Z Remove unussed method commit 30efe5f0e4d18d452a3505c6f101d4322bc0af6c Author: Ankur Chauhan Date: 2015-07-30T23:40:52Z Remove yarn features while building for mesos commit 59f6e3995362461f4877aa037087e4f7d3aefa37 Author: Ankur Chauhan Date: 2015-08-02T04:50:42Z Revert changes to .travis.yml commit f6b20e74d3aead76abcc1acf2829a7e294e8453e Author: Ankur Chauhan Date: 2015-08-31T02:22:56Z [Review updates] Use http server to host log4j properties, added apache spark attribution commit 7486347187e5e73e4f800c3026131ed3ef4f8aab Author: Ankur Chauhan Date: 2015-08-31T02:39:01Z Compile failure with httpServer commit 4da1732249780b8604aabf2c899ff9754788952d Author: Ankur Chauhan Date: 2015-08-31T02:44:54Z Handle exceptions during driver startup of Scheduler commit 2f19c522a1b6f2ec0d6c11070667f42519561220 Author: Ankur Chauhan Date: 2015-08-31T02:47:19Z Update mesos java library commit 4d5f147da6ab4fd6c6f2f79d160ab4942ebe90dc Author: Ankur Chauhan Date: 2015-09-08T01:58:56Z Merge branch 'master' into flink-mesos * master: (27 commits) [FLINK-2372] Add new FlinkKafkaProducer bases on the new producer API [FLINK-2626] [core] Minor cleanups in AverageAccumulator and AverageAccumulatorTest [FLINK-2626] [core] Add an average accumulator [hotfix] [tests] Increase zookeeper session timeouts in tests them more robust [FLINK-2628] [tests] CoStreamCheckpointingITCase prints a warning when test is inconclusive [FLINK-2625] Pass configuration properly to JobGraphGenerator [FLINK-2605] [runtime] Unclosed RandomAccessFile may leak resource in StaticFileServerHandler [FLINK-2161] [scala shell] Modify start script to take additional argument (-a or --addclasspath ) for external libraries [FLINK-2475] Rename Flink Client log file [FLINK-2614][scala-shell] scala shell's default execution mode is broken [FLINK-2607][quickstart] ignore signature files when creating fat jar [hotfix][docs] Update Kafka section in streaming guide to match the renamed class names [hotfix] [gelly] Remove unintended calls to getExecutionEnvironment in VertexCentricIteration and GatherSumApplyIteration [FLINK-2448] [tests] Create new Test Environments on every call to getExecutionEnvironment and allow sharing of job execution re
[jira] [Assigned] (FLINK-2902) Web interface sort tasks newest first
[ https://issues.apache.org/jira/browse/FLINK-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sachin Goel reassigned FLINK-2902: -- Assignee: Sachin Goel > Web interface sort tasks newest first > - > > Key: FLINK-2902 > URL: https://issues.apache.org/jira/browse/FLINK-2902 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Sachin Goel >Priority: Minor > > Sort completed jobs in reverse order so the most recently finished (?) are at > the top of the list. With a long list of completed jobs the user must scroll > down to view recently completed jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[
https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970089#comment-14970089
]
ASF GitHub Bot commented on FLINK-1982:
---
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42818462
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -38,8 +37,7 @@
import org.junit.Assert;
import org.junit.Test;
-@SuppressWarnings("deprecation")
-public class MapTaskTest extends
DriverTestBase> {
+public class MapTaskTest extends DriverTestBase> {
--- End diff --
Small nit: should probably rename this to ``FlatMapTaskTest`` since now it
is actually using ``flatMap`` function instead?
> Remove dependencies on Record for Flink runtime and core
>
>
> Key: FLINK-1982
> URL: https://issues.apache.org/jira/browse/FLINK-1982
> Project: Flink
> Issue Type: Sub-task
> Components: Core
>Reporter: Henry Saputra
>Assignee: Fabian Hueske
>
> Seemed like there are several uses of Record API in core and runtime module
> that need to be updated before Record API could be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1982] [record-api] Remove dependencies ...
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42818462
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -38,8 +37,7 @@
import org.junit.Assert;
import org.junit.Test;
-@SuppressWarnings("deprecation")
-public class MapTaskTest extends
DriverTestBase> {
+public class MapTaskTest extends DriverTestBase> {
--- End diff --
Small nit: should probably rename this to ``FlatMapTaskTest`` since now it
is actually using ``flatMap`` function instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1982] [record-api] Remove dependencies ...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1294#issuecomment-150375992 Updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[ https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970036#comment-14970036 ] ASF GitHub Bot commented on FLINK-1982: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1294#issuecomment-150375992 Updated > Remove dependencies on Record for Flink runtime and core > > > Key: FLINK-1982 > URL: https://issues.apache.org/jira/browse/FLINK-1982 > Project: Flink > Issue Type: Sub-task > Components: Core >Reporter: Henry Saputra >Assignee: Fabian Hueske > > Seemed like there are several uses of Record API in core and runtime module > that need to be updated before Record API could be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[
https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969948#comment-14969948
]
ASF GitHub Bot commented on FLINK-1982:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42809588
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -39,7 +38,7 @@
import org.junit.Test;
@SuppressWarnings("deprecation")
--- End diff --
Good catch, thanks!
Will check the other classes as well.
> Remove dependencies on Record for Flink runtime and core
>
>
> Key: FLINK-1982
> URL: https://issues.apache.org/jira/browse/FLINK-1982
> Project: Flink
> Issue Type: Sub-task
> Components: Core
>Reporter: Henry Saputra
>Assignee: Fabian Hueske
>
> Seemed like there are several uses of Record API in core and runtime module
> that need to be updated before Record API could be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1982] [record-api] Remove dependencies ...
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42809588
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -39,7 +38,7 @@
import org.junit.Test;
@SuppressWarnings("deprecation")
--- End diff --
Good catch, thanks!
Will check the other classes as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[
https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969934#comment-14969934
]
ASF GitHub Bot commented on FLINK-1982:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42808744
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -39,7 +38,7 @@
import org.junit.Test;
@SuppressWarnings("deprecation")
--- End diff --
this suppression can be removed
> Remove dependencies on Record for Flink runtime and core
>
>
> Key: FLINK-1982
> URL: https://issues.apache.org/jira/browse/FLINK-1982
> Project: Flink
> Issue Type: Sub-task
> Components: Core
>Reporter: Henry Saputra
>Assignee: Fabian Hueske
>
> Seemed like there are several uses of Record API in core and runtime module
> that need to be updated before Record API could be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1982] [record-api] Remove dependencies ...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/1294#discussion_r42808744
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/MapTaskTest.java
---
@@ -39,7 +38,7 @@
import org.junit.Test;
@SuppressWarnings("deprecation")
--- End diff --
this suppression can be removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2901) Several flink-test ITCases depend on Record API features
[
https://issues.apache.org/jira/browse/FLINK-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969924#comment-14969924
]
Fabian Hueske commented on FLINK-2901:
--
I started to remove some of the dependencies and tests.
The current state is available in my Github repository:
https://github.com/fhueske/flink/tree/flinkTestsRecord
> Several flink-test ITCases depend on Record API features
>
>
> Key: FLINK-2901
> URL: https://issues.apache.org/jira/browse/FLINK-2901
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 0.10
>Reporter: Fabian Hueske
>
> There are several ITCases and utility classes in {{flink-tests}} that depend
> on the Record API including:
> - ITCases for Record API operators in
> {{flink-tests/src/test/java/org/apache/flink/test/operators}}
> - ITCases for Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
> - Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
> - Several ITCases for iterations in
> {{flink-tests/src/test/java/org/apache/flink/test/iterative}}
> - Tests for job canceling in
> {{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
> - Tests for failing jobs in
> {{flink-tests/src/test/java/org/apache/flink/test/failingPrograms}}
> - Optimizer tests in
> {{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
> and probably a few more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[ https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969911#comment-14969911 ] ASF GitHub Bot commented on FLINK-1982: --- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/1294 [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests Rename Match*Test to Join*Test You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink runtimeTestsRecord Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1294 commit 6367ad7e05eb81f1bf719beda0eeb972864ee568 Author: Fabian Hueske Date: 2015-10-22T19:10:41Z [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests Rename Match*Test to Join*Test > Remove dependencies on Record for Flink runtime and core > > > Key: FLINK-1982 > URL: https://issues.apache.org/jira/browse/FLINK-1982 > Project: Flink > Issue Type: Sub-task > Components: Core >Reporter: Henry Saputra >Assignee: Fabian Hueske > > Seemed like there are several uses of Record API in core and runtime module > that need to be updated before Record API could be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1982] [record-api] Remove dependencies ...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/1294 [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests Rename Match*Test to Join*Test You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink runtimeTestsRecord Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1294 commit 6367ad7e05eb81f1bf719beda0eeb972864ee568 Author: Fabian Hueske Date: 2015-10-22T19:10:41Z [FLINK-1982] [record-api] Remove dependencies on Record API from flink-runtime tests Rename Match*Test to Join*Test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-2904) Web interface truncated task counts
Greg Hogan created FLINK-2904: - Summary: Web interface truncated task counts Key: FLINK-2904 URL: https://issues.apache.org/jira/browse/FLINK-2904 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: 0.10 Reporter: Greg Hogan Priority: Minor Task counts have only three digits visible as the color square needs to dynamically expand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2900] [hadoop-compat] Remove Record API...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/1293 [FLINK-2900] [hadoop-compat] Remove Record API code from Hadoop Compat module You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink hadoopCompatRecord Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1293 commit c5548b92ddc1fbc8d65b7d141d11b8a4a16b8f3a Author: Fabian Hueske Date: 2015-10-22T19:12:15Z [FLINK-2900] [hadoop-compat] Remove Record API code from Hadoop Compat module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-2903) Web interface numeric localization
Greg Hogan created FLINK-2903: - Summary: Web interface numeric localization Key: FLINK-2903 URL: https://issues.apache.org/jira/browse/FLINK-2903 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: 0.10 Reporter: Greg Hogan Priority: Minor It would be nice to localize numbers in the web interface as 10+ digits is difficult to parse without separators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2900) Remove Record-API dependencies from Hadoop Compat module
[ https://issues.apache.org/jira/browse/FLINK-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969906#comment-14969906 ] ASF GitHub Bot commented on FLINK-2900: --- GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/1293 [FLINK-2900] [hadoop-compat] Remove Record API code from Hadoop Compat module You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink hadoopCompatRecord Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1293 commit c5548b92ddc1fbc8d65b7d141d11b8a4a16b8f3a Author: Fabian Hueske Date: 2015-10-22T19:12:15Z [FLINK-2900] [hadoop-compat] Remove Record API code from Hadoop Compat module > Remove Record-API dependencies from Hadoop Compat module > > > Key: FLINK-2900 > URL: https://issues.apache.org/jira/browse/FLINK-2900 > Project: Flink > Issue Type: Sub-task > Components: Hadoop Compatibility >Affects Versions: 0.10 >Reporter: Fabian Hueske >Assignee: Fabian Hueske > > The Hadoop Compat module includes wrappers for Hadoop Input/OutputFormat for > the Record API classes and a corresponding test. > These need to be removed before removing the Record API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2902) Web interface sort tasks newest first
Greg Hogan created FLINK-2902: - Summary: Web interface sort tasks newest first Key: FLINK-2902 URL: https://issues.apache.org/jira/browse/FLINK-2902 Project: Flink Issue Type: Improvement Components: Webfrontend Affects Versions: 0.10 Reporter: Greg Hogan Priority: Minor Sort completed jobs in reverse order so the most recently finished (?) are at the top of the list. With a long list of completed jobs the user must scroll down to view recently completed jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2901) Several flink-test ITCases depend on Record API features
Fabian Hueske created FLINK-2901:
Summary: Several flink-test ITCases depend on Record API features
Key: FLINK-2901
URL: https://issues.apache.org/jira/browse/FLINK-2901
Project: Flink
Issue Type: Sub-task
Components: Tests
Affects Versions: 0.10
Reporter: Fabian Hueske
There are several ITCases and utility classes in {{flink-tests}} that depend on
the Record API including:
- ITCases for Record API operators in
{{flink-tests/src/test/java/org/apache/flink/test/operators}}
- ITCases for Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
- Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
- Several ITCases for iterations in
{{flink-tests/src/test/java/org/apache/flink/test/iterative}}
- Tests for job canceling in
{{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
- Tests for failing jobs in
{{flink-tests/src/test/java/org/apache/flink/test/failingPrograms}}
- Optimizer tests in
{{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
and probably a few more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[ https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-1982: Assignee: Fabian Hueske > Remove dependencies on Record for Flink runtime and core > > > Key: FLINK-1982 > URL: https://issues.apache.org/jira/browse/FLINK-1982 > Project: Flink > Issue Type: Sub-task > Components: Core >Reporter: Henry Saputra >Assignee: Fabian Hueske > > Seemed like there are several uses of Record API in core and runtime module > that need to be updated before Record API could be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2900) Remove Record-API dependencies from Hadoop Compat module
Fabian Hueske created FLINK-2900: Summary: Remove Record-API dependencies from Hadoop Compat module Key: FLINK-2900 URL: https://issues.apache.org/jira/browse/FLINK-2900 Project: Flink Issue Type: Sub-task Components: Hadoop Compatibility Affects Versions: 0.10 Reporter: Fabian Hueske The Hadoop Compat module includes wrappers for Hadoop Input/OutputFormat for the Record API classes and a corresponding test. These need to be removed before removing the Record API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-2900) Remove Record-API dependencies from Hadoop Compat module
[ https://issues.apache.org/jira/browse/FLINK-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reassigned FLINK-2900: Assignee: Fabian Hueske > Remove Record-API dependencies from Hadoop Compat module > > > Key: FLINK-2900 > URL: https://issues.apache.org/jira/browse/FLINK-2900 > Project: Flink > Issue Type: Sub-task > Components: Hadoop Compatibility >Affects Versions: 0.10 >Reporter: Fabian Hueske >Assignee: Fabian Hueske > > The Hadoop Compat module includes wrappers for Hadoop Input/OutputFormat for > the Record API classes and a corresponding test. > These need to be removed before removing the Record API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-1988) Port Record API based tests in the common.io
[ https://issues.apache.org/jira/browse/FLINK-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-1988. Resolution: Invalid Yes, `common.io` is clean. > Port Record API based tests in the common.io > > > Key: FLINK-1988 > URL: https://issues.apache.org/jira/browse/FLINK-1988 > Project: Flink > Issue Type: Sub-task >Reporter: Henry Saputra > > As part of removing old Record API, need to remove more tests that relying on > Record API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1988) Port Record API based tests in the common.io
[ https://issues.apache.org/jira/browse/FLINK-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969822#comment-14969822 ] Henry Saputra commented on FLINK-1988: -- I think we could close this one. > Port Record API based tests in the common.io > > > Key: FLINK-1988 > URL: https://issues.apache.org/jira/browse/FLINK-1988 > Project: Flink > Issue Type: Sub-task >Reporter: Henry Saputra > > As part of removing old Record API, need to remove more tests that relying on > Record API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1982) Remove dependencies on Record for Flink runtime and core
[ https://issues.apache.org/jira/browse/FLINK-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969821#comment-14969821 ] Henry Saputra commented on FLINK-1982: -- I am seeing TeraSortITCase extends RecordAPITestBase but neither seemed to actually access Record API. > Remove dependencies on Record for Flink runtime and core > > > Key: FLINK-1982 > URL: https://issues.apache.org/jira/browse/FLINK-1982 > Project: Flink > Issue Type: Sub-task > Components: Core >Reporter: Henry Saputra > > Seemed like there are several uses of Record API in core and runtime module > that need to be updated before Record API could be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2897) Use distinct initial indices for OutputEmitter round-robin
[ https://issues.apache.org/jira/browse/FLINK-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969807#comment-14969807 ] ASF GitHub Bot commented on FLINK-2897: --- GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1292 [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2897_use_distinct_initial_indices_for_outputemitter_round_robin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1292 commit 6365e693a69810cf1597a2f19191d08b6e644da8 Author: Greg Hogan Date: 2015-10-22T14:43:26Z [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin > Use distinct initial indices for OutputEmitter round-robin > -- > > Key: FLINK-2897 > URL: https://issues.apache.org/jira/browse/FLINK-2897 > Project: Flink > Issue Type: Improvement > Components: Distributed Runtime >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Greg Hogan > > Currently, when performing a round-robin partitioning each task will > sequentially partition starting with partition "1". This is fine in the usual > case where the number of partitioned objects greatly exceeds the number of > channels. However, in the case where the number of objects is relatively few > (each, perhaps, requiring a large computation or access to an external > system) it would be much better to begin partitioning at distinct indices > (the task index). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2897] [runtime] Use distinct initial in...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1292 [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2897_use_distinct_initial_indices_for_outputemitter_round_robin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1292 commit 6365e693a69810cf1597a2f19191d08b6e644da8 Author: Greg Hogan Date: 2015-10-22T14:43:26Z [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-2899) The groupReduceOn* methods which take types as a parameter fail with TypeErasure
Andra Lungu created FLINK-2899: -- Summary: The groupReduceOn* methods which take types as a parameter fail with TypeErasure Key: FLINK-2899 URL: https://issues.apache.org/jira/browse/FLINK-2899 Project: Flink Issue Type: Bug Components: Gelly Affects Versions: 0.10 Reporter: Andra Lungu I tried calling groupReduceOnEdges (EdgesFunctionWithVertexValue edgesFunction, EdgeDirection direction, TypeInformation typeInfo) in order to make the vertex-centric version of the Triangle Count library method applicable to any kind of key and I got a TypeErasure Exception. After doing a bit of debugging (see the hack in https://github.com/andralungu/flink/tree/trianglecount-vertexcentric), I saw that actually the call to TypeExtractor.createTypeInfo(NeighborsFunctionWithVertexValue.class, in ApplyNeighborCoGroupFunction does not work properly, i.e. it returns null. From what I see, the coGroup in groupReduceOnNeighbors tries to infer a type before "returns" is called. I may be missing something, but that particular feature (groupReduceOn with types) is not documented or tested so we would also need some tests for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2898) Invert Travis CI build order
[ https://issues.apache.org/jira/browse/FLINK-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969765#comment-14969765 ] ASF GitHub Bot commented on FLINK-2898: --- Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1290#issuecomment-150336339 Thanks for catching that! The commit log is correct, so it is puzzling why the pull request would have a different title. > Invert Travis CI build order > > > Key: FLINK-2898 > URL: https://issues.apache.org/jira/browse/FLINK-2898 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > > The Travis CI builds generally perform fastest to slowest. When running > additional, concurrent Travis CI builds it would be preferable to have the > slowest tasks begin first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2898] [build] Invert Travis CI build or...
Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1290#issuecomment-150336339 Thanks for catching that! The commit log is correct, so it is puzzling why the pull request would have a different title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2634) Add a Vertex-centric Version of the Tringle Count Library Method
[ https://issues.apache.org/jira/browse/FLINK-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969754#comment-14969754 ] ASF GitHub Bot commented on FLINK-2634: --- GitHub user andralungu opened a pull request: https://github.com/apache/flink/pull/1291 [FLINK-2634] [gelly] [WIP] Vertex Centric Triangle Count This PR builds on the code presented in #1105. Basically, the reduceOn* calls are replaced with groupReduceOn* calls. As discussed back then, I made the lib method accept any kind of keys. While doing so, I found a bit of a bug (which is why I marked this as WIP). The groupReduceOnNeighbors function has a version that takes a type argument. The problem with that is: coGroup tries to build a type before the call to `returns()` which means that it cannot infer the type from the info it has. I'll explain this a bit better in a Jira. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andralungu/flink trianglecount-vertexcentric Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 1fbc09ca35e80f61f31cf7a6166f27f2a6f142c1 Author: andralungu Date: 2015-10-22T06:18:43Z [FLINK-2634] [gelly] Vertex Centric Triangle Count [FLINK-2634] [gelly] Fixed Type Erasure > Add a Vertex-centric Version of the Tringle Count Library Method > > > Key: FLINK-2634 > URL: https://issues.apache.org/jira/browse/FLINK-2634 > Project: Flink > Issue Type: Task > Components: Gelly >Affects Versions: 0.10 >Reporter: Andra Lungu >Assignee: Andra Lungu >Priority: Minor > > The vertex-centric version of this algorithm receives an undirected graph as > input and outputs the total number of triangles formed by the graph's edges. > The implementation consists of three phases: > 1). Select neighbours with id greater than the current vertex id. > 2). Propagate each received value to neighbours with higher id. > 3). Compute the number of Triangles by verifying if the final vertex contains > the sender's id in its list. > As opposed to the GAS version, all these three steps will be performed via > message passing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2634] [gelly] [WIP] Vertex Centric Tria...
GitHub user andralungu opened a pull request: https://github.com/apache/flink/pull/1291 [FLINK-2634] [gelly] [WIP] Vertex Centric Triangle Count This PR builds on the code presented in #1105. Basically, the reduceOn* calls are replaced with groupReduceOn* calls. As discussed back then, I made the lib method accept any kind of keys. While doing so, I found a bit of a bug (which is why I marked this as WIP). The groupReduceOnNeighbors function has a version that takes a type argument. The problem with that is: coGroup tries to build a type before the call to `returns()` which means that it cannot infer the type from the info it has. I'll explain this a bit better in a Jira. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andralungu/flink trianglecount-vertexcentric Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 1fbc09ca35e80f61f31cf7a6166f27f2a6f142c1 Author: andralungu Date: 2015-10-22T06:18:43Z [FLINK-2634] [gelly] Vertex Centric Triangle Count [FLINK-2634] [gelly] Fixed Type Erasure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2874) Certain Avro generated getters/setters not recognized
[
https://issues.apache.org/jira/browse/FLINK-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969628#comment-14969628
]
ASF GitHub Bot commented on FLINK-2874:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1252#issuecomment-150318989
Will merge this PR.
> Certain Avro generated getters/setters not recognized
> -
>
> Key: FLINK-2874
> URL: https://issues.apache.org/jira/browse/FLINK-2874
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 0.9.1, 0.9.0, 0.10
>Reporter: Ulf Karlsson
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> For Avro schemas where value null is not allowed, the field is unboxed e.g.
> int but the getter/setter methods provide the boxed Integer as interface:
> {code}
> {
> "fields": [
> {
>"type": "double",
>"name": "time"
> },
> }
> {code}
> This results in Java
> {code}
> private double time;
> public java.lang.Double getTime() {
> return time;
> }
> public void setTime(java.lang.Double value) {
> this.time = value;
> }
> {code}
> There is also a problem when there is an underscore in the Avro schema, e.g.:
> {code}
> {
>"default": null,
>"type": [
> "null",
> "long"
>],
>"name": "conn_id"
> },
> {code}
> This results in Java:
> {code}
> private java.lang.Long conn_id;
> public java.lang.Long getConnId() {
> return conn_id;
> }
> public void setConnId(java.lang.Long value) {
> this.conn_id = value;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2206) JobManager webinterface shows 5 finished jobs at most
[ https://issues.apache.org/jira/browse/FLINK-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969626#comment-14969626 ] ASF GitHub Bot commented on FLINK-2206: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1287#issuecomment-150318790 Thanks for the review. Will merge this. > JobManager webinterface shows 5 finished jobs at most > - > > Key: FLINK-2206 > URL: https://issues.apache.org/jira/browse/FLINK-2206 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 0.9, 0.10 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Blocker > Labels: starter > Fix For: 0.9, 0.10 > > > The webinterface of the JobManager shows at most 5 finished jobs. This is > because only the last 5 JobGraphs are remembered in the JobManager. > The same might apply to canceled and failed jobs. > I think this is very confusing to users and could be easily fixed. > We should add three simple counters in the JobManager to correctly track > these numbers and give the right counts to the webinterface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2668] Chained Projections are no longer...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1279#issuecomment-150318859 Will merge this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2668) ProjectOperator method to close projection
[
https://issues.apache.org/jira/browse/FLINK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969627#comment-14969627
]
ASF GitHub Bot commented on FLINK-2668:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1279#issuecomment-150318859
Will merge this
> ProjectOperator method to close projection
> --
>
> Key: FLINK-2668
> URL: https://issues.apache.org/jira/browse/FLINK-2668
> Project: Flink
> Issue Type: Improvement
> Components: Java API
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Priority: Minor
>
> I have come across an issue in my code where I called project(...) on a
> {{DataSet}} which was already a {{ProjectOperator}}. Instead of reducing the
> number of fields from 2 to 1 this instead increased the number of fields from
> 2 to 3 resulting in
> {{org.apache.flink.api.common.functions.InvalidTypesException: Input
> mismatch: Tuple arity '3' expected but was '1'.}} when processing the next
> operator.
> This can be resolved by adding an optional explicit call to conclude the
> projection, perhaps {{ProjectOperator.closeProjection()}}. Can this be done
> without creating a new no-op operator?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2874] Fix Avro getter/setter recognitio...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1252#issuecomment-150318989 Will merge this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2206] Fix incorrect counts of finished,...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1287#issuecomment-150318790 Thanks for the review. Will merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2559) Fix Javadoc Code Examples
[
https://issues.apache.org/jira/browse/FLINK-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969523#comment-14969523
]
Hubert Czerpak commented on FLINK-2559:
---
I'd be happy to help with this one if nobody is working on it.
> Fix Javadoc Code Examples
> -
>
> Key: FLINK-2559
> URL: https://issues.apache.org/jira/browse/FLINK-2559
> Project: Flink
> Issue Type: Improvement
>Reporter: Aljoscha Krettek
>Priority: Minor
> Labels: starter
>
> Many multiline Javadoc code examples are not correctly rendered. One of the
> problems is that an @ inside a code block breaks the rendering.
> This is an example that works:
> {code}
> * {@code
> * private static class MyIndexRequestBuilder implements
> IndexRequestBuilder {
> *
> * public IndexRequest createIndexRequest(String element,
> RuntimeContext ctx) {
> * Map json = new HashMap<>();
> * json.put("data", element);
> *
> * return Requests.indexRequest()
> * .index("my-index")
> * .type("my-type")
> * .source(json);
> * }
> * }
> * }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2897] [runtime] Use distinct initial in...
Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1290#issuecomment-150300806 Wrong title. :') --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2897) Use distinct initial indices for OutputEmitter round-robin
[ https://issues.apache.org/jira/browse/FLINK-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969518#comment-14969518 ] ASF GitHub Bot commented on FLINK-2897: --- GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1290 [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2898_invert_travis_ci_build_order Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1290 commit dfb482e14535870c1599895949668e4bc252479f Author: Greg Hogan Date: 2015-10-22T15:12:43Z [FLINK-2898] [build] Invert Travis CI build order > Use distinct initial indices for OutputEmitter round-robin > -- > > Key: FLINK-2897 > URL: https://issues.apache.org/jira/browse/FLINK-2897 > Project: Flink > Issue Type: Improvement > Components: Distributed Runtime >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Greg Hogan > > Currently, when performing a round-robin partitioning each task will > sequentially partition starting with partition "1". This is fine in the usual > case where the number of partitioned objects greatly exceeds the number of > channels. However, in the case where the number of objects is relatively few > (each, perhaps, requiring a large computation or access to an external > system) it would be much better to begin partitioning at distinct indices > (the task index). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2897] [runtime] Use distinct initial in...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1290 [FLINK-2897] [runtime] Use distinct initial indices for OutputEmitter round-robin You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2898_invert_travis_ci_build_order Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1290 commit dfb482e14535870c1599895949668e4bc252479f Author: Greg Hogan Date: 2015-10-22T15:12:43Z [FLINK-2898] [build] Invert Travis CI build order --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2897) Use distinct initial indices for OutputEmitter round-robin
[ https://issues.apache.org/jira/browse/FLINK-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969520#comment-14969520 ] ASF GitHub Bot commented on FLINK-2897: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1290#issuecomment-150300806 Wrong title. :') > Use distinct initial indices for OutputEmitter round-robin > -- > > Key: FLINK-2897 > URL: https://issues.apache.org/jira/browse/FLINK-2897 > Project: Flink > Issue Type: Improvement > Components: Distributed Runtime >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Greg Hogan > > Currently, when performing a round-robin partitioning each task will > sequentially partition starting with partition "1". This is fine in the usual > case where the number of partitioned objects greatly exceeds the number of > channels. However, in the case where the number of objects is relatively few > (each, perhaps, requiring a large computation or access to an external > system) it would be much better to begin partitioning at distinct indices > (the task index). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes f...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1289#issuecomment-150299847 +1 Code looks good, I actually solved it the same way but got stuck on the tests... :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969515#comment-14969515
]
ASF GitHub Bot commented on FLINK-2866:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1289#issuecomment-150299847
+1
Code looks good, I actually solved it the same way but got stuck on the
tests... :smile:
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969512#comment-14969512 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150299549 @mxm , I have addressed some of your comments. Two things still remain: batch + stream jobs, and better tests. Need advice on those. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150299549 @mxm , I have addressed some of your comments. Two things still remain: batch + stream jobs, and better tests. Need advice on those. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969503#comment-14969503
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42778575
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
Actually, these arguments are not needed. I will simply remove them.
Also, in `textGetExecutionPlan` where these values are used, the locations
are not actually accessed. Only plan is tested. So shouldn't be an issue.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42778575
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
Actually, these arguments are not needed. I will simply remove them.
Also, in `textGetExecutionPlan` where these values are used, the locations
are not actually accessed. Only plan is tested. So shouldn't be an issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes f...
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/1289 [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes for various release blocking issues Default state value is returned as copies in key value state. To circumvent this (if ever becomes a performance issue), one can use null as the default state and manually check/initialize to an immutable default value. Window user functions have access to key/value state even window is evaluated (part 2 - pert 1 was committed by Aljoscha earlier). The main part here is a rework of the PreAggregatingWindow tests to make them more meaningful. This also includes #1282 You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink multiple_fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1289 commit 0a14e9ff3c499ca339d275b5ca58bed545f71519 Author: tedyu Date: 2015-10-22T02:40:21Z [FLINK-2866] [runtime] Eagerly close FSDataInputStream in file state handle This closes #1282 commit acbcc98e9cb09554b879f60af390dbb1485f03d6 Author: Stephan Ewen Date: 2015-10-22T09:24:12Z [FLINK-2888] [streaming] State backends return copies of the default values commit fca433e1652840c0d8c8de1c57bd988428cc6218 Author: Stephan Ewen Date: 2015-10-22T16:58:29Z [FLINK-2891] [streaming] Set keys for key/value state in window evaluation of fast-path windows. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969484#comment-14969484
]
ASF GitHub Bot commented on FLINK-2866:
---
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1289
[FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes for various release blocking
issues
Default state value is returned as copies in key value state. To circumvent
this (if ever becomes a performance issue), one can use null as the default
state and manually check/initialize to an immutable default value.
Window user functions have access to key/value state even window is
evaluated (part 2 - pert 1 was committed by Aljoscha earlier). The main part
here is a rework of the PreAggregatingWindow tests to make them more meaningful.
This also includes #1282
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink multiple_fixes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1289.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1289
commit 0a14e9ff3c499ca339d275b5ca58bed545f71519
Author: tedyu
Date: 2015-10-22T02:40:21Z
[FLINK-2866] [runtime] Eagerly close FSDataInputStream in file state handle
This closes #1282
commit acbcc98e9cb09554b879f60af390dbb1485f03d6
Author: Stephan Ewen
Date: 2015-10-22T09:24:12Z
[FLINK-2888] [streaming] State backends return copies of the default values
commit fca433e1652840c0d8c8de1c57bd988428cc6218
Author: Stephan Ewen
Date: 2015-10-22T16:58:29Z
[FLINK-2891] [streaming] Set keys for key/value state in window evaluation
of fast-path windows.
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969448#comment-14969448
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774775
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
I was concerned about this too. Perhaps a better idea would be have these
classes in the test-jar?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969447#comment-14969447
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774659
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
+ assertEquals("The Job was submitted in detached mode.
Only one execution is allowed. " +
--- End diff --
Will do.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774775
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
I was concerned about this too. Perhaps a better idea would be have these
classes in the test-jar?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969446#comment-14969446
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774635
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
I was following the code in `testGetExecutionPlan`. Can you elaborate a bit
more?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774659
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
+ assertEquals("The Job was submitted in detached mode.
Only one execution is allowed. " +
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774635
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
I was following the code in `testGetExecutionPlan`. Can you elaborate a bit
more?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969435#comment-14969435 ] ASF GitHub Bot commented on FLINK-2797: --- Github user sachingoel0101 commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42774405 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- I tend to use `_field` appended names instead of `this.field`. But yeah. `last*` makes more sense here. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42774405 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- I tend to use `_field` appended names instead of `this.field`. But yeah. `last*` makes more sense here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969432#comment-14969432
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774255
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
Ah. I had not considered this possibility. What should be done in such a
case? Execute both programs? That does make sense, but what should be the
return value?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42774255
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
Ah. I had not considered this possibility. What should be done in such a
case? Execute both programs? That does make sense, but what should be the
return value?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969420#comment-14969420 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150283627 Thanks for the update @sachingoel0101. I've made some comments. I think the pull request needs a bit of cleanup to make things more explicit. The general approach looks good. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969416#comment-14969416
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772661
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
This should write to System.getProperty("java.io.tmpdir") and have some
other Random input to be cross-platform compatible.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969414#comment-14969414 ] ASF GitHub Bot commented on FLINK-2797: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42772634 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- Is there a reason for these odd names? `lastBatchPlan` and `lastStreamJobGraph` would be more appropriate. The first one is for batch programs, the latter for JobGraph created by the streaming API. > CLI: Missing option to submit jobs in detached mode > --- > > Key: FLINK-2797 > URL: https://issues.apache.org/jira/browse/FLINK-2797 > Project: Flink > Issue Type: Bug > Components: Command-line client >Affects Versions: 0.9, 0.10 >Reporter: Maximilian Michels >Assignee: Sachin Goel > Fix For: 0.10 > > > Jobs can only be submitted in detached mode using YARN but not on a > standalone installation. This has been requested by users who want to submit > a job, get the job id, and later query its status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969418#comment-14969418
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772695
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
These are no proper tests.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1214#issuecomment-150283627 Thanks for the update @sachingoel0101. I've made some comments. I think the pull request needs a bit of cleanup to make things more explicit. The general approach looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969417#comment-14969417
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772667
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
+ assertEquals("The Job was submitted in detached mode.
Only one execution is allowed. " +
--- End diff --
Can we make this message available somewhere such that it doesn't have to
be copied over and over?
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772695
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -304,4 +368,60 @@ public String getDescription() {
return "TestOptimizerPlan
";
}
}
+
+ public static final class TestExecuteTwice {
+
+ public static void main(String args[]) throws Exception {
--- End diff --
These are no proper tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969413#comment-14969413
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772606
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -172,5 +184,74 @@ public ExecutionEnvironment
createExecutionEnvironment() {
}
return env;
}
+
+ public static void setPlan(JobWithJars plan) {
+ if (_plan == null) {
+ _plan = plan;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one execution is allowed.
Please make sure your program doesn't call " +
+ "execute twice and/or doesn't
call an eager execution function[collect, " +
+ "print, printToErr, count].\n");
+ }
+ }
+
+ public static JobWithJars getPlan() {
+ return _plan;
+ }
+
+ public static JobGraph getGraph() {
+ return _graph;
+ }
+
+ public static void setGraph(JobGraph graph) {
+ if (_graph == null) {
+ _graph = graph;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one call to execute is
allowed.");
+ }
+ }
+ }
+
+ public static final class DetachedJobExecutionResult extends
JobExecutionResult {
+
+ public static final DetachedJobExecutionResult INSTANCE = new
DetachedJobExecutionResult();
+
+ public DetachedJobExecutionResult() {
--- End diff --
Constructor should be private to be a singleton.
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772667
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
+ out.runDetached(prg, 1);
+ } catch (ProgramInvocationException e) {
+ assertEquals("The Job was submitted in detached mode.
Only one execution is allowed. " +
--- End diff --
Can we make this message available somewhere such that it doesn't have to
be copied over and over?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772661
--- Diff:
flink-clients/src/test/java/org/apache/flink/client/program/ClientTest.java ---
@@ -139,6 +140,69 @@ public void shouldSubmitToJobClient() {
}
/**
+* Tests that invalid detached mode programs fail.
+*/
+ @Test
+ public void testDetachedMode() throws Exception{
+
jobManagerSystem.actorOf(Props.create(SuccessReturningActor.class),
JobManager.JOB_MANAGER_NAME());
+ Client out = new Client(config);
+
+ try {
+ PackagedProgram prg = new
PackagedProgram(TestExecuteTwice.class, "/dev/random", "/tmp");
--- End diff --
This should write to System.getProperty("java.io.tmpdir") and have some
other Random input to be cross-platform compatible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1214#discussion_r42772634 --- Diff: flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java --- @@ -161,6 +171,8 @@ public ContextEnvironmentFactory(Client client, List jarFilesToAttach, this.userCodeClassLoader = userCodeClassLoader; this.defaultParallelism = defaultParallelism; this.wait = wait; + _plan = null; + _graph = null; --- End diff -- Is there a reason for these odd names? `lastBatchPlan` and `lastStreamJobGraph` would be more appropriate. The first one is for batch programs, the latter for JobGraph created by the streaming API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772606
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java
---
@@ -172,5 +184,74 @@ public ExecutionEnvironment
createExecutionEnvironment() {
}
return env;
}
+
+ public static void setPlan(JobWithJars plan) {
+ if (_plan == null) {
+ _plan = plan;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one execution is allowed.
Please make sure your program doesn't call " +
+ "execute twice and/or doesn't
call an eager execution function[collect, " +
+ "print, printToErr, count].\n");
+ }
+ }
+
+ public static JobWithJars getPlan() {
+ return _plan;
+ }
+
+ public static JobGraph getGraph() {
+ return _graph;
+ }
+
+ public static void setGraph(JobGraph graph) {
+ if (_graph == null) {
+ _graph = graph;
+ } else {
+ throw new DetachedProgramException("The Job was
submitted in detached mode. Only " +
+ "one call to execute is
allowed.");
+ }
+ }
+ }
+
+ public static final class DetachedJobExecutionResult extends
JobExecutionResult {
+
+ public static final DetachedJobExecutionResult INSTANCE = new
DetachedJobExecutionResult();
+
+ public DetachedJobExecutionResult() {
--- End diff --
Constructor should be private to be a singleton.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
[
https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969411#comment-14969411
]
ASF GitHub Bot commented on FLINK-2797:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772489
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
What would happen if we created a streaming program and a batch program in
one jar? Then both would be set and only the batch program would be created...
> CLI: Missing option to submit jobs in detached mode
> ---
>
> Key: FLINK-2797
> URL: https://issues.apache.org/jira/browse/FLINK-2797
> Project: Flink
> Issue Type: Bug
> Components: Command-line client
>Affects Versions: 0.9, 0.10
>Reporter: Maximilian Michels
>Assignee: Sachin Goel
> Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a
> standalone installation. This has been requested by users who want to submit
> a job, get the job id, and later query its status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/1214#discussion_r42772489
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/program/Client.java ---
@@ -277,12 +278,20 @@ else if (prog.isUsingInteractiveMode()) {
// invoke here
try {
prog.invokeInteractiveModeForExecution();
+ // if we're here, it means the program is valid
for detached mode execution.
+ JobWithJars plan =
ContextEnvironment.ContextEnvironmentFactory.getPlan();
+ JobGraph graph =
ContextEnvironment.ContextEnvironmentFactory.getGraph();
+ if (plan != null && graph == null) {
--- End diff --
What would happen if we created a streaming program and a batch program in
one jar? Then both would be set and only the batch program would be created...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (FLINK-2800) kryo serialization problem
[ https://issues.apache.org/jira/browse/FLINK-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann reassigned FLINK-2800: Assignee: Till Rohrmann > kryo serialization problem > -- > > Key: FLINK-2800 > URL: https://issues.apache.org/jira/browse/FLINK-2800 > Project: Flink > Issue Type: Bug > Components: Type Serialization System >Affects Versions: 0.10 > Environment: linux ubuntu 12.04 LTS, Java 7 >Reporter: Stefano Bortoli >Assignee: Till Rohrmann > > Performing a cross of two dataset of POJOs I have got the exception below. > The first time I run the process, there was no problem. When I run it the > second time, I have got the exception. My guess is that it could be a race > condition related to the reuse of the Kryo serializer object. However, it > could also be "a bug where type registrations are not properly forwarded to > all Serializers", as suggested by Stephan. > > 2015-10-01 18:18:21 INFO JobClient:161 - 10/01/2015 18:18:21 Cross(Cross at > main(FlinkMongoHadoop2LinkPOI2CDA.java:160))(3/4) switched to FAILED > com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: > 114 > at > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) > at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:210) > at > org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:127) > at > org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:30) > at > org.apache.flink.runtime.operators.resettable.AbstractBlockResettableIterator.getNextRecord(AbstractBlockResettableIterator.java:180) > at > org.apache.flink.runtime.operators.resettable.BlockResettableMutableObjectIterator.next(BlockResettableMutableObjectIterator.java:111) > at > org.apache.flink.runtime.operators.CrossDriver.runBlockedOuterSecond(CrossDriver.java:309) > at > org.apache.flink.runtime.operators.CrossDriver.run(CrossDriver.java:162) > at > org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:489) > at > org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:354) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:581) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2895) Duplicate immutable object creation
[
https://issues.apache.org/jira/browse/FLINK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969303#comment-14969303
]
ASF GitHub Bot commented on FLINK-2895:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1288#issuecomment-150261354
Thanks for your work @greghogan.
LGTM, +1 for merging.
> Duplicate immutable object creation
> ---
>
> Key: FLINK-2895
> URL: https://issues.apache.org/jira/browse/FLINK-2895
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Inverse of FLINK-2724. When object reuse is disabled a few operators are
> creating and passing objects locally. In the case of immutable objects these
> will be discarded by the {{TypeSerializer}} when deserializing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2895] Duplicate immutable object creati...
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/1288#issuecomment-150261354 Thanks for your work @greghogan. LGTM, +1 for merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2895) Duplicate immutable object creation
[
https://issues.apache.org/jira/browse/FLINK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969289#comment-14969289
]
ASF GitHub Bot commented on FLINK-2895:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/1288
[FLINK-2895] Duplicate immutable object creation
Operators defer object creation when object reuse is disabled.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
2895_duplicate_immutable_object_creation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1288
commit f5750fcbffa74b05305b8259561dd6f6a2d76371
Author: Greg Hogan
Date: 2015-10-22T13:31:09Z
[FLINK-2895] Duplicate immutable object creation
Operators defer object creation when object reuse is disabled.
> Duplicate immutable object creation
> ---
>
> Key: FLINK-2895
> URL: https://issues.apache.org/jira/browse/FLINK-2895
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Inverse of FLINK-2724. When object reuse is disabled a few operators are
> creating and passing objects locally. In the case of immutable objects these
> will be discarded by the {{TypeSerializer}} when deserializing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2895] Duplicate immutable object creati...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/1288 [FLINK-2895] Duplicate immutable object creation Operators defer object creation when object reuse is disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 2895_duplicate_immutable_object_creation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1288.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1288 commit f5750fcbffa74b05305b8259561dd6f6a2d76371 Author: Greg Hogan Date: 2015-10-22T13:31:09Z [FLINK-2895] Duplicate immutable object creation Operators defer object creation when object reuse is disabled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2898) Invert Travis CI build order
[ https://issues.apache.org/jira/browse/FLINK-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969287#comment-14969287 ] Fabian Hueske commented on FLINK-2898: -- Good idea! +1 > Invert Travis CI build order > > > Key: FLINK-2898 > URL: https://issues.apache.org/jira/browse/FLINK-2898 > Project: Flink > Issue Type: Improvement > Components: Build System >Affects Versions: 0.10 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > > The Travis CI builds generally perform fastest to slowest. When running > additional, concurrent Travis CI builds it would be preferable to have the > slowest tasks begin first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2898) Invert Travis CI build order
Greg Hogan created FLINK-2898: - Summary: Invert Travis CI build order Key: FLINK-2898 URL: https://issues.apache.org/jira/browse/FLINK-2898 Project: Flink Issue Type: Improvement Components: Build System Affects Versions: 0.10 Reporter: Greg Hogan Assignee: Greg Hogan Priority: Trivial The Travis CI builds generally perform fastest to slowest. When running additional, concurrent Travis CI builds it would be preferable to have the slowest tasks begin first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2897) Use distinct initial indices for OutputEmitter round-robin
Greg Hogan created FLINK-2897: - Summary: Use distinct initial indices for OutputEmitter round-robin Key: FLINK-2897 URL: https://issues.apache.org/jira/browse/FLINK-2897 Project: Flink Issue Type: Improvement Components: Distributed Runtime Affects Versions: 0.10 Reporter: Greg Hogan Assignee: Greg Hogan Currently, when performing a round-robin partitioning each task will sequentially partition starting with partition "1". This is fine in the usual case where the number of partitioned objects greatly exceeds the number of channels. However, in the case where the number of objects is relatively few (each, perhaps, requiring a large computation or access to an external system) it would be much better to begin partitioning at distinct indices (the task index). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2891) Key for Keyed State is not set upon Window Evaluation
[ https://issues.apache.org/jira/browse/FLINK-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969198#comment-14969198 ] Aljoscha Krettek commented on FLINK-2891: - I also pushed the fix for the General Purpose (GP) Window Operator to the release-0.10 branch. > Key for Keyed State is not set upon Window Evaluation > - > > Key: FLINK-2891 > URL: https://issues.apache.org/jira/browse/FLINK-2891 > Project: Flink > Issue Type: Sub-task > Components: Streaming >Affects Versions: 0.10 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Fix For: 0.10 > > > In both the aligned and the general-purpose windows the key for the keyed > operator state is not set when evaluating the windows. This silently leads to > incorrect results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2891) Key for Keyed State is not set upon Window Evaluation
[ https://issues.apache.org/jira/browse/FLINK-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969197#comment-14969197 ] ASF GitHub Bot commented on FLINK-2891: --- Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1285 > Key for Keyed State is not set upon Window Evaluation > - > > Key: FLINK-2891 > URL: https://issues.apache.org/jira/browse/FLINK-2891 > Project: Flink > Issue Type: Sub-task > Components: Streaming >Affects Versions: 0.10 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Fix For: 0.10 > > > In both the aligned and the general-purpose windows the key for the keyed > operator state is not set when evaluating the windows. This silently leads to > incorrect results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2891] Set KV-State key upon Window Eval...
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1285 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2895) Duplicate immutable object creation
[
https://issues.apache.org/jira/browse/FLINK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969154#comment-14969154
]
Greg Hogan commented on FLINK-2895:
---
With immutable objects, when object reuse is enabled a single unused object is
created. When object reuse is disabled we are creating an unused object for
every object deserialization.
> Duplicate immutable object creation
> ---
>
> Key: FLINK-2895
> URL: https://issues.apache.org/jira/browse/FLINK-2895
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> Inverse of FLINK-2724. When object reuse is disabled a few operators are
> creating and passing objects locally. In the case of immutable objects these
> will be discarded by the {{TypeSerializer}} when deserializing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2896) Right-hand side menu items of website are missing in narrow browsers
Fabian Hueske created FLINK-2896: Summary: Right-hand side menu items of website are missing in narrow browsers Key: FLINK-2896 URL: https://issues.apache.org/jira/browse/FLINK-2896 Project: Flink Issue Type: Bug Components: Project Website Reporter: Fabian Hueske The right menu items (Blog, Community, Project) are missing if the browser window is too narrow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2895) Duplicate immutable object creation
Greg Hogan created FLINK-2895:
-
Summary: Duplicate immutable object creation
Key: FLINK-2895
URL: https://issues.apache.org/jira/browse/FLINK-2895
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime
Affects Versions: 0.10
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Minor
Inverse of FLINK-2724. When object reuse is disabled a few operators are
creating and passing objects locally. In the case of immutable objects these
will be discarded by the {{TypeSerializer}} when deserializing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2206) JobManager webinterface shows 5 finished jobs at most
[ https://issues.apache.org/jira/browse/FLINK-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969120#comment-14969120 ] ASF GitHub Bot commented on FLINK-2206: --- Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1287#issuecomment-150212141 +1 good to merge! > JobManager webinterface shows 5 finished jobs at most > - > > Key: FLINK-2206 > URL: https://issues.apache.org/jira/browse/FLINK-2206 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 0.9, 0.10 >Reporter: Fabian Hueske >Assignee: Fabian Hueske >Priority: Blocker > Labels: starter > Fix For: 0.9, 0.10 > > > The webinterface of the JobManager shows at most 5 finished jobs. This is > because only the last 5 JobGraphs are remembered in the JobManager. > The same might apply to canceled and failed jobs. > I think this is very confusing to users and could be easily fixed. > We should add three simple counters in the JobManager to correctly track > these numbers and give the right counts to the webinterface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2668) ProjectOperator method to close projection
[
https://issues.apache.org/jira/browse/FLINK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969122#comment-14969122
]
ASF GitHub Bot commented on FLINK-2668:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1279#issuecomment-150213016
Go ahead
> ProjectOperator method to close projection
> --
>
> Key: FLINK-2668
> URL: https://issues.apache.org/jira/browse/FLINK-2668
> Project: Flink
> Issue Type: Improvement
> Components: Java API
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Priority: Minor
>
> I have come across an issue in my code where I called project(...) on a
> {{DataSet}} which was already a {{ProjectOperator}}. Instead of reducing the
> number of fields from 2 to 1 this instead increased the number of fields from
> 2 to 3 resulting in
> {{org.apache.flink.api.common.functions.InvalidTypesException: Input
> mismatch: Tuple arity '3' expected but was '1'.}} when processing the next
> operator.
> This can be resolved by adding an optional explicit call to conclude the
> projection, perhaps {{ProjectOperator.closeProjection()}}. Can this be done
> without creating a new no-op operator?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2668] Chained Projections are no longer...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1279#issuecomment-150213016 Go ahead --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-2894) Flink does not allow to specify default serializer for Kryo
[
https://issues.apache.org/jira/browse/FLINK-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-2894.
Resolution: Not A Problem
> Flink does not allow to specify default serializer for Kryo
> ---
>
> Key: FLINK-2894
> URL: https://issues.apache.org/jira/browse/FLINK-2894
> Project: Flink
> Issue Type: Bug
>Affects Versions: 0.10
>Reporter: Till Rohrmann
>
> Currently, Flink only supports to specify Kryo {{Serializer}} for specific
> types but not default serializer for classes. A default serializer is used
> for the registered class and all its subclasses. That way one does not have
> to specify the serializer for each type individually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2206] Fix incorrect counts of finished,...
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1287#issuecomment-150212141 +1 good to merge! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2668] Chained Projections are no longer...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1279#issuecomment-150211310 Thanks for the update, looks good. I will merge this into `master` and `release-0.10` unless somebody objects. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2668) ProjectOperator method to close projection
[
https://issues.apache.org/jira/browse/FLINK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969111#comment-14969111
]
ASF GitHub Bot commented on FLINK-2668:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1279#issuecomment-150211310
Thanks for the update, looks good.
I will merge this into `master` and `release-0.10` unless somebody objects.
> ProjectOperator method to close projection
> --
>
> Key: FLINK-2668
> URL: https://issues.apache.org/jira/browse/FLINK-2668
> Project: Flink
> Issue Type: Improvement
> Components: Java API
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Priority: Minor
>
> I have come across an issue in my code where I called project(...) on a
> {{DataSet}} which was already a {{ProjectOperator}}. Instead of reducing the
> number of fields from 2 to 1 this instead increased the number of fields from
> 2 to 3 resulting in
> {{org.apache.flink.api.common.functions.InvalidTypesException: Input
> mismatch: Tuple arity '3' expected but was '1'.}} when processing the next
> operator.
> This can be resolved by adding an optional explicit call to conclude the
> projection, perhaps {{ProjectOperator.closeProjection()}}. Can this be done
> without creating a new no-op operator?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2894) Flink does not allow to specify default serializer for Kryo
Till Rohrmann created FLINK-2894:
Summary: Flink does not allow to specify default serializer for
Kryo
Key: FLINK-2894
URL: https://issues.apache.org/jira/browse/FLINK-2894
Project: Flink
Issue Type: Bug
Affects Versions: 0.10
Reporter: Till Rohrmann
Currently, Flink only supports to specify Kryo {{Serializer}} for specific
types but not default serializer for classes. A default serializer is used for
the registered class and all its subclasses. That way one does not have to
specify the serializer for each type individually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
