[jira] [Created] (FLINK-2919) Apply JMH on FieldAccessMinibenchmark class.
GaoLun created FLINK-2919: - Summary: Apply JMH on FieldAccessMinibenchmark class. Key: FLINK-2919 URL: https://issues.apache.org/jira/browse/FLINK-2919 Project: Flink Issue Type: Sub-task Components: Tests Reporter: GaoLun Assignee: GaoLun Priority: Minor JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks.Use JMH to replace the old micro benchmarks method in order to get much more accurate results.Modify the FieldAccessMinibenchmark class and move it to flink-benchmark module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2918) Add a method to be able to read SequenceFileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973465#comment-14973465
]
ASF GitHub Bot commented on FLINK-2918:
---
GitHub user smarthi opened a pull request:
https://github.com/apache/flink/pull/1299
[FLINK-2918] Add method to read a file of type SequenceFileInputFormat
Fixes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smarthi/flink work
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1299.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1299
commit 84762d6001cb16d7134d865d96503d55fd02701c
Author: smarthi
Date: 2015-10-25T12:42:31Z
Add method to read a file of SequenceFileInputFormat
> Add a method to be able to read SequenceFileInputFormat
> ---
>
> Key: FLINK-2918
> URL: https://issues.apache.org/jira/browse/FLINK-2918
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.9.1
>Reporter: Suneel Marthi
>Assignee: Suneel Marthi
>Priority: Minor
> Fix For: 0.10
>
>
> This is to add a method to ExecutionEnvironment.{java,scala} to be able to
> provide syntactic sugar to read a SequenceFileInputFormat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2918] Add method to read a file of type...
GitHub user smarthi opened a pull request: https://github.com/apache/flink/pull/1299 [FLINK-2918] Add method to read a file of type SequenceFileInputFormat Fixes You can merge this pull request into a Git repository by running: $ git pull https://github.com/smarthi/flink work Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1299.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1299 commit 84762d6001cb16d7134d865d96503d55fd02701c Author: smarthi Date: 2015-10-25T12:42:31Z Add method to read a file of SequenceFileInputFormat --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2891) Key for Keyed State is not set upon Window Evaluation
[ https://issues.apache.org/jira/browse/FLINK-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973463#comment-14973463 ] Stephan Ewen commented on FLINK-2891: - Second part (fast path processing time windows) fixed as of 3dc7423c4be8c15593a0d3d39cbac883f7c4cb7b > Key for Keyed State is not set upon Window Evaluation > - > > Key: FLINK-2891 > URL: https://issues.apache.org/jira/browse/FLINK-2891 > Project: Flink > Issue Type: Sub-task > Components: Streaming >Affects Versions: 0.10 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Fix For: 0.10 > > > In both the aligned and the general-purpose windows the key for the keyed > operator state is not set when evaluating the windows. This silently leads to > incorrect results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-2888) Default state not copied for AbstractHeapKvState
[ https://issues.apache.org/jira/browse/FLINK-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-2888. --- > Default state not copied for AbstractHeapKvState > > > Key: FLINK-2888 > URL: https://issues.apache.org/jira/browse/FLINK-2888 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 0.10 >Reporter: Gyula Fora >Assignee: Stephan Ewen >Priority: Blocker > Fix For: 0.10 > > > The Default state value needs to be copied before returned for mutable states > otherwise the map will be populated with the same object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-2866.
-
Resolution: Fixed
Assignee: Ted Yu
Fix Version/s: 0.10
Fixed via 2d647151c729f707681dd5e2d226e4bbe0329bba
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.10
>
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-2866.
---
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.10
>
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-2888) Default state not copied for AbstractHeapKvState
[ https://issues.apache.org/jira/browse/FLINK-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-2888. - Resolution: Fixed Assignee: Stephan Ewen Fix Version/s: 0.10 Fixed via 6424ce57b4012af64adc88ef37f341d6a5aec744 > Default state not copied for AbstractHeapKvState > > > Key: FLINK-2888 > URL: https://issues.apache.org/jira/browse/FLINK-2888 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 0.10 >Reporter: Gyula Fora >Assignee: Stephan Ewen >Priority: Blocker > Fix For: 0.10 > > > The Default state value needs to be copied before returned for mutable states > otherwise the map will be populated with the same object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2918) Add a method to be able to read SequenceFileInputFormat
Suneel Marthi created FLINK-2918:
Summary: Add a method to be able to read SequenceFileInputFormat
Key: FLINK-2918
URL: https://issues.apache.org/jira/browse/FLINK-2918
Project: Flink
Issue Type: Improvement
Components: Core
Affects Versions: 0.9.1
Reporter: Suneel Marthi
Assignee: Suneel Marthi
Priority: Minor
Fix For: 0.10
This is to add a method to ExecutionEnvironment.{java,scala} to be able to
provide syntactic sugar to read a SequenceFileInputFormat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973457#comment-14973457
]
ASF GitHub Bot commented on FLINK-2866:
---
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/1289
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973456#comment-14973456
]
ASF GitHub Bot commented on FLINK-2866:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1289#issuecomment-150979276
Manually merged into branches `master` and `release-0.10`
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes f...
Github user StephanEwen closed the pull request at: https://github.com/apache/flink/pull/1289 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes f...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1289#issuecomment-150979276 Manually merged into branches `master` and `release-0.10` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973455#comment-14973455
]
ASF GitHub Bot commented on FLINK-2866:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1282
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: FLINK-2866 Potential resource leak due to uncl...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1282 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2917) Make non-copying behaviour default for stream operator
[
https://issues.apache.org/jira/browse/FLINK-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973308#comment-14973308
]
Aljoscha Krettek commented on FLINK-2917:
-
We already have the switch: {{enableObjectReuse()}}
> Make non-copying behaviour default for stream operator
> --
>
> Key: FLINK-2917
> URL: https://issues.apache.org/jira/browse/FLINK-2917
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
>Reporter: Gyula Fora
>Priority: Blocker
>
> Currently chained streaming operators copy their output before invoking the
> process method of the chained output task.
> The community has agreed that this might be too big of a price for the
> immutable semantics and the default behaviour should be changed to not copy.
> I suggest to have an api level switch for the Operators to enable copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2917) Make non-copying behaviour default for stream operator
Gyula Fora created FLINK-2917: - Summary: Make non-copying behaviour default for stream operator Key: FLINK-2917 URL: https://issues.apache.org/jira/browse/FLINK-2917 Project: Flink Issue Type: Improvement Components: Streaming Reporter: Gyula Fora Priority: Blocker Currently chained streaming operators copy their output before invoking the process method of the chained output task. The community has agreed that this might be too big of a price for the immutable semantics and the default behaviour should be changed to not copy. I suggest to have an api level switch for the Operators to enable copying. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2901) Several flink-test ITCases depend on Record API features
[
https://issues.apache.org/jira/browse/FLINK-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973286#comment-14973286
]
Fabian Hueske commented on FLINK-2901:
--
The jobs can be removed as well. They are mainly used for the record API tests
which are removed. The remaining tests that depend on these jobs need to be
adapted.
> Several flink-test ITCases depend on Record API features
>
>
> Key: FLINK-2901
> URL: https://issues.apache.org/jira/browse/FLINK-2901
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 0.10
>Reporter: Fabian Hueske
>Assignee: Chesnay Schepler
>
> There are several ITCases and utility classes in {{flink-tests}} that depend
> on the Record API including:
> - ITCases for Record API operators in
> {{flink-tests/src/test/java/org/apache/flink/test/operators}}
> - ITCases for Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
> - Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
> - Several ITCases for iterations in
> {{flink-tests/src/test/java/org/apache/flink/test/iterative}}
> - Tests for job canceling in
> {{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
> - Test for failing jobs in
> {{flink-tests/src/test/java/org/apache/flink/test/failingPrograms/TaskFailureITCase}}
> - Optimizer tests in
> {{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
> - Accumulator test in
> {{flink-tests/src/test/java/org/apache/flink/test/accumulators/AccumulatorIterativeITCase}}
> - Broadcast test in
> {{flink-tests/src/test/java/org/apache/flink/test/broadcastvasr/BroadcastBranchingITCase}}
> - distributed cache test in
> {{flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest}}
> and probably a few more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2901) Several flink-test ITCases depend on Record API features
[
https://issues.apache.org/jira/browse/FLINK-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973272#comment-14973272
]
Chesnay Schepler commented on FLINK-2901:
-
What about {{recordJobs}}? Should all of them be ported as well?
> Several flink-test ITCases depend on Record API features
>
>
> Key: FLINK-2901
> URL: https://issues.apache.org/jira/browse/FLINK-2901
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 0.10
>Reporter: Fabian Hueske
>Assignee: Chesnay Schepler
>
> There are several ITCases and utility classes in {{flink-tests}} that depend
> on the Record API including:
> - ITCases for Record API operators in
> {{flink-tests/src/test/java/org/apache/flink/test/operators}}
> - ITCases for Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
> - Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
> - Several ITCases for iterations in
> {{flink-tests/src/test/java/org/apache/flink/test/iterative}}
> - Tests for job canceling in
> {{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
> - Test for failing jobs in
> {{flink-tests/src/test/java/org/apache/flink/test/failingPrograms/TaskFailureITCase}}
> - Optimizer tests in
> {{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
> - Accumulator test in
> {{flink-tests/src/test/java/org/apache/flink/test/accumulators/AccumulatorIterativeITCase}}
> - Broadcast test in
> {{flink-tests/src/test/java/org/apache/flink/test/broadcastvasr/BroadcastBranchingITCase}}
> - distributed cache test in
> {{flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest}}
> and probably a few more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-2901) Several flink-test ITCases depend on Record API features
[
https://issues.apache.org/jira/browse/FLINK-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-2901:
Description:
There are several ITCases and utility classes in {{flink-tests}} that depend on
the Record API including:
- ITCases for Record API operators in
{{flink-tests/src/test/java/org/apache/flink/test/operators}}
- ITCases for Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
- Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
- Several ITCases for iterations in
{{flink-tests/src/test/java/org/apache/flink/test/iterative}}
- Tests for job canceling in
{{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
- Test for failing jobs in
{{flink-tests/src/test/java/org/apache/flink/test/failingPrograms/TaskFailureITCase}}
- Optimizer tests in
{{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
- Accumulator test in
{{flink-tests/src/test/java/org/apache/flink/test/accumulators/AccumulatorIterativeITCase}}
- Broadcast test in
{{flink-tests/src/test/java/org/apache/flink/test/broadcastvasr/BroadcastBranchingITCase}}
- distributed cache test in
{{flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest}}
and probably a few more.
was:
There are several ITCases and utility classes in {{flink-tests}} that depend on
the Record API including:
- ITCases for Record API operators in
{{flink-tests/src/test/java/org/apache/flink/test/operators}}
- ITCases for Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
- Record API programs in
{{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
- Several ITCases for iterations in
{{flink-tests/src/test/java/org/apache/flink/test/iterative}}
- Tests for job canceling in
{{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
- Tests for failing jobs in
{{flink-tests/src/test/java/org/apache/flink/test/failingPrograms}}
- Optimizer tests in
{{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
and probably a few more.
> Several flink-test ITCases depend on Record API features
>
>
> Key: FLINK-2901
> URL: https://issues.apache.org/jira/browse/FLINK-2901
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Affects Versions: 0.10
>Reporter: Fabian Hueske
>Assignee: Chesnay Schepler
>
> There are several ITCases and utility classes in {{flink-tests}} that depend
> on the Record API including:
> - ITCases for Record API operators in
> {{flink-tests/src/test/java/org/apache/flink/test/operators}}
> - ITCases for Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobTests}}
> - Record API programs in
> {{flink-tests/src/test/java/org/apache/flink/test/recordJobs}}
> - Several ITCases for iterations in
> {{flink-tests/src/test/java/org/apache/flink/test/iterative}}
> - Tests for job canceling in
> {{flink-tests/src/test/java/org/apache/flink/test/cancelling}}
> - Test for failing jobs in
> {{flink-tests/src/test/java/org/apache/flink/test/failingPrograms/TaskFailureITCase}}
> - Optimizer tests in
> {{flink-tests/src/test/java/org/apache/flink/test/optimizer}}
> - Accumulator test in
> {{flink-tests/src/test/java/org/apache/flink/test/accumulators/AccumulatorIterativeITCase}}
> - Broadcast test in
> {{flink-tests/src/test/java/org/apache/flink/test/broadcastvasr/BroadcastBranchingITCase}}
> - distributed cache test in
> {{flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest}}
> and probably a few more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2591) Add configuration parameter for default number of yarn containers
[ https://issues.apache.org/jira/browse/FLINK-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973262#comment-14973262 ] ASF GitHub Bot commented on FLINK-2591: --- Github user willmiao commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-150928188 Sorry for the failed tests, I'll fix them. > Add configuration parameter for default number of yarn containers > - > > Key: FLINK-2591 > URL: https://issues.apache.org/jira/browse/FLINK-2591 > Project: Flink > Issue Type: Improvement > Components: YARN Client >Reporter: Robert Metzger >Assignee: Will Miao >Priority: Minor > Labels: starter > > A user complained about the requirement to always specify the number of yarn > containers (-n) when starting a job. > Adding a configuration value with a default value will allow users to set a > default ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user willmiao commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-150928188 Sorry for the failed tests, I'll fix them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user willmiao commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-150919886 Hi rmetzger, I have updated this PR. I generated a flink-conf.yaml in a directory, which I then used as the configuration directory for the test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2591) Add configuration parameter for default number of yarn containers
[ https://issues.apache.org/jira/browse/FLINK-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973216#comment-14973216 ] ASF GitHub Bot commented on FLINK-2591: --- Github user willmiao commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-150919886 Hi rmetzger, I have updated this PR. I generated a flink-conf.yaml in a directory, which I then used as the configuration directory for the test. > Add configuration parameter for default number of yarn containers > - > > Key: FLINK-2591 > URL: https://issues.apache.org/jira/browse/FLINK-2591 > Project: Flink > Issue Type: Improvement > Components: YARN Client >Reporter: Robert Metzger >Assignee: Will Miao >Priority: Minor > Labels: starter > > A user complained about the requirement to always specify the number of yarn > containers (-n) when starting a job. > Adding a configuration value with a default value will allow users to set a > default ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2017) Add predefined required parameters to ParameterTool
[
https://issues.apache.org/jira/browse/FLINK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973188#comment-14973188
]
ASF GitHub Bot commented on FLINK-2017:
---
Github user mliesenberg commented on a diff in the pull request:
https://github.com/apache/flink/pull/1097#discussion_r42943777
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/RequiredParameter.java
---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Facility to manage required parameters in user defined functions.
+ */
+public class RequiredParameter {
--- End diff --
sure, I can change that.
> Add predefined required parameters to ParameterTool
> ---
>
> Key: FLINK-2017
> URL: https://issues.apache.org/jira/browse/FLINK-2017
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 0.9
>Reporter: Robert Metzger
> Labels: starter
>
> In FLINK-1525 we've added the {{ParameterTool}}.
> During the PR review, there was a request for required parameters.
> This issue is about implementing a facility to define required parameters.
> The tool should also be able to print a help menu with a list of all
> parameters.
> This test case shows my initial ideas how to design the API
> {code}
> @Test
> public void requiredParameters() {
> RequiredParameters required = new RequiredParameters();
> Option input = required.add("input").alt("i").help("Path to
> input file or directory"); // parameter with long and short variant
> required.add("output"); // parameter only with long variant
> Option parallelism =
> required.add("parallelism").alt("p").type(Integer.class); // parameter with
> type
> Option spOption =
> required.add("sourceParallelism").alt("sp").defaultValue(12).help("Number
> specifying the number of parallel data source instances"); // parameter with
> default value, specifying the type.
> Option executionType =
> required.add("executionType").alt("et").defaultValue("pipelined").choices("pipelined",
> "batch");
> ParameterUtil parameter = ParameterUtil.fromArgs(new
> String[]{"-i", "someinput", "--output", "someout", "-p", "15"});
> required.check(parameter);
> required.printHelp();
> required.checkAndPopulate(parameter);
> String inputString = input.get();
> int par = parallelism.getInteger();
> String output = parameter.get("output");
> int sourcePar = parameter.getInteger(spOption.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2017) Add predefined required parameters to ParameterTool
[
https://issues.apache.org/jira/browse/FLINK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973187#comment-14973187
]
ASF GitHub Bot commented on FLINK-2017:
---
Github user mliesenberg commented on a diff in the pull request:
https://github.com/apache/flink/pull/1097#discussion_r42943773
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/RequiredParameter.java
---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Facility to manage required parameters in user defined functions.
+ */
+public class RequiredParameter {
+
+ private static final String HELP_TEXT_PARAM_DELIMITER = "\t";
+ private static final String HELP_TEXT_LINE_DELIMITER = "\n";
+
+ private HashMap data;
+
+ public RequiredParameter() {
+ this.data = new HashMap<>();
+ }
+
+ public void add(Option option) throws RequiredParameterException {
--- End diff --
We could add an additional method, though then we do have an implicit
coupling between that add method and the constructor of Option. If, and that is
probably a big if, the constructor were to change, we would have to update the
add method as well.
Only providing a String based version of the add method would lead to code
like that:
```
Option option = req.add("name")
option.type(...).values(...)
```
which in my opinion is a little bit awkward compared to instantiation,
setting the right type and values and then adding the object.
what do you think?
> Add predefined required parameters to ParameterTool
> ---
>
> Key: FLINK-2017
> URL: https://issues.apache.org/jira/browse/FLINK-2017
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 0.9
>Reporter: Robert Metzger
> Labels: starter
>
> In FLINK-1525 we've added the {{ParameterTool}}.
> During the PR review, there was a request for required parameters.
> This issue is about implementing a facility to define required parameters.
> The tool should also be able to print a help menu with a list of all
> parameters.
> This test case shows my initial ideas how to design the API
> {code}
> @Test
> public void requiredParameters() {
> RequiredParameters required = new RequiredParameters();
> Option input = required.add("input").alt("i").help("Path to
> input file or directory"); // parameter with long and short variant
> required.add("output"); // parameter only with long variant
> Option parallelism =
> required.add("parallelism").alt("p").type(Integer.class); // parameter with
> type
> Option spOption =
> required.add("sourceParallelism").alt("sp").defaultValue(12).help("Number
> specifying the number of parallel data source instances"); // parameter with
> default value, specifying the type.
> Option executionType =
> required.add("executionType").alt("et").defaultValue("pipelined").choices("pipelined",
> "batch");
> ParameterUtil parameter = ParameterUtil.fromArgs(new
> String[]{"-i", "someinput", "--output", "someout", "-p", "15"});
> required.check(parameter);
> required.printHelp();
> required.checkAndPopulate(parameter);
> String inputString = input.get();
> int par = parallelism.getInteger();
> String output = parameter.get("output");
> int sourcePar = parameter.getInteger(spOption.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2017] Add predefined required parameter...
Github user mliesenberg commented on a diff in the pull request:
https://github.com/apache/flink/pull/1097#discussion_r42943777
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/RequiredParameter.java
---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Facility to manage required parameters in user defined functions.
+ */
+public class RequiredParameter {
--- End diff --
sure, I can change that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2017] Add predefined required parameter...
Github user mliesenberg commented on a diff in the pull request:
https://github.com/apache/flink/pull/1097#discussion_r42943773
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/RequiredParameter.java
---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.utils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Facility to manage required parameters in user defined functions.
+ */
+public class RequiredParameter {
+
+ private static final String HELP_TEXT_PARAM_DELIMITER = "\t";
+ private static final String HELP_TEXT_LINE_DELIMITER = "\n";
+
+ private HashMap data;
+
+ public RequiredParameter() {
+ this.data = new HashMap<>();
+ }
+
+ public void add(Option option) throws RequiredParameterException {
--- End diff --
We could add an additional method, though then we do have an implicit
coupling between that add method and the constructor of Option. If, and that is
probably a big if, the constructor were to change, we would have to update the
add method as well.
Only providing a String based version of the add method would lead to code
like that:
```
Option option = req.add("name")
option.type(...).values(...)
```
which in my opinion is a little bit awkward compared to instantiation,
setting the right type and values and then adding the object.
what do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (FLINK-2916) Add more context information for the StateBackend
Gyula Fora created FLINK-2916: - Summary: Add more context information for the StateBackend Key: FLINK-2916 URL: https://issues.apache.org/jira/browse/FLINK-2916 Project: Flink Issue Type: Improvement Components: Streaming Reporter: Gyula Fora Assignee: Gyula Fora Priority: Minor The StateBackend currently only receives a very limited information about the context in which it was initialized (only the JobID). For more elaborate state backends (for instance db backends) it becomes necessary to have more information about the running job to allow more fine-grained handling or sharding of the stored state. Changes: - The initalizeForJob method should receive the Environment of the task from which it was called instead of just the jobId. This allows getting information about parallelism and subtask index for instance. - The createKvState method should additionally receive the unique id of the operator in the jobgraph along with the name of the created state name (jobid + operatorId + statename creates a unique identifier for the kvState) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2915) JobManagerProcessFailureBatchRecoveryITCase
Matthias J. Sax created FLINK-2915:
--
Summary: JobManagerProcessFailureBatchRecoveryITCase
Key: FLINK-2915
URL: https://issues.apache.org/jira/browse/FLINK-2915
Project: Flink
Issue Type: Bug
Components: Tests
Reporter: Matthias J. Sax
Priority: Critical
https://travis-ci.org/apache/flink/jobs/87193692
{noformat}
Failed tests:
JobManagerProcessFailureBatchRecoveryITCase>AbstractJobManagerProcessFailureRecoveryITCase.testJobManagerProcessFailure:259
JobManager did not start up within 291736881301 nanoseconds.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-2866] [FLINK-2888] [FLINK-2891] Fixes f...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1289#issuecomment-150900971 Ok, then we should merge it since this is blocking the release. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2866) Potential resource leak due to unclosed ObjectInputStream in FileSerializableStateHandle
[
https://issues.apache.org/jira/browse/FLINK-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973140#comment-14973140
]
ASF GitHub Bot commented on FLINK-2866:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1289#issuecomment-150900971
Ok, then we should merge it since this is blocking the release. :smile:
> Potential resource leak due to unclosed ObjectInputStream in
> FileSerializableStateHandle
>
>
> Key: FLINK-2866
> URL: https://issues.apache.org/jira/browse/FLINK-2866
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>
> Here is related code:
> {code}
> public T getState(ClassLoader classLoader) throws Exception {
> FSDataInputStream inStream = getFileSystem().open(getFilePath());
> ObjectInputStream ois = new
> InstantiationUtil.ClassLoaderObjectInputStream(inStream, classLoader);
> return (T) ois.readObject();
> {code}
> ObjectInputStream should be closed upon exit from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
