[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328014#comment-15328014
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/1626
Thank you @xhumanoid for reporting this! I will make the fix.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327660#comment-15327660
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user xhumanoid commented on the issue:
https://github.com/apache/flink/pull/1626
Hi, I know than issue closed, but he maybe have small bug
https://github.com/apache/flink/pull/1626/files#diff-4a133896fec62bcabc1120b0df8cb8daR205
in hashCode recursion on this.hashCode()
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168904#comment-15168904
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1626
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167956#comment-15167956
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-189004909
Pushed. Thanks for the explanation.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167871#comment-15167871
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-188985984
If you can quickly do it, yes.
Otherwise whoever merges this tomorrow, please remove the sysout statements.
We try to keep the maven output a bit cleaner that way. If you want to
output something into the logs for the tests, just use a regular slf4j logger...
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167864#comment-15167864
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-188982973
@StephanEwen just to clarify, since time is of the essence, would you like
me to update the pull request without the println's?
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167641#comment-15167641
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-188929386
This looks good to me.
+1 for merging, but removing the System.out.println
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158526#comment-15158526
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1692#issuecomment-187598869
Ooops, you are right. I'll fix it.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157671#comment-15157671
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1692#issuecomment-187376965
Is this FLINK-3440?
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157622#comment-15157622
]
ASF GitHub Bot commented on FLINK-3340:
---
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/1692
[FLINK-3340][Kafka 0.8] Commit also offsets retrieved from Kafka into the
OffsetStore (ZK)
Before this change, offsets were only committed to Zookeeper if something
was read from the TopicPartition.
Now, the start offsets retrieved from Kafka (based on `auto.offset.reset`)
are also put into the state so that they are also committed into ZK.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink3440
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1692.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1692
commit f7a7ae90f566faa38975c3f9a193090ae18ab37a
Author: Robert Metzger
Date: 2016-02-22T13:48:37Z
[FLINK-3340][Kafka 0.8] Commit also offsets retrieved from Kafka into the
OffsetStore (ZK)
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144761#comment-15144761
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-183390101
OK, after Stephan's comment on FLINK-3291, I agree.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144753#comment-15144753
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-183389185
`MergeIterator.next(reuse)` should allow object swapping. The source code
documentation should be improved to make this clearer, but this contract is
necessary to minimize the copying of objects
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144710#comment-15144710
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-183375884
Sorry, but I don't think it is independent, because the bug in the reduce
drivers could alternatively be fixed by saying that the bug is actually not in
the reduce drivers at all, but in `MergeIterator`, and fix it there instead (as
in
https://github.com/ggevay/flink/commit/b05e12075bb5d0518b6fe12cd8f7de1e6b16f805).
This depends on what we want the contract of `MergeIterator.next(reuse)` to be.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144619#comment-15144619
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-183348625
Hi, @ggevay, these fixes to the reduce drivers are independent of changes
to `MutableObjectIterator`. This patch is intended to fix the bug you reported
in [FLINK-3291](https://issues.apache.org/jira/browse/FLINK-3291).
+1 to improving the `MutableObjectIterator` API. Let's continue the
conversation over on
[FLINK-3291](https://issues.apache.org/jira/browse/FLINK-3291)!
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144481#comment-15144481
]
ASF GitHub Bot commented on FLINK-3340:
---
Github user ggevay commented on the pull request:
https://github.com/apache/flink/pull/1626#issuecomment-183287868
So, if I understand correctly, this is aiming to fix the problem that was
brought up by [1], but in a different way then proposed there.
I still have the same worry about this solution that I wrote in this
comment: [2]: `MutableObjectIterator.next(reuse)` is used at 271 places,
according to IntelliJ. If we accept the change in the object reuse contract of
`MergeIterator.next(reuse)` introduced by
0a8df6d513fa59d650ff875bdf3a1613d0f14af5, all the call sites can potentially
have the same problem as the reduce drivers, so we should check all of these
271 places. For example, just by skimming the list for a few minutes, all the 8
calls in `CrossDriver` seem suspicious at first glance, as well as the calls in
`UnionWithTempOperator`, `MutableHashTable.ProbeIterator.next`, and
`ReusingBuildFirstHashJoinIterator.callWithNextKey`.
My alternative solution [3] has the drawback that it would introduce an
extra copying in `MergeIterator.HeadStream.getHead`, but would change the
contract of `MergeIterator.next(reuse)` back to the same as it was before
0a8df6d513fa59d650ff875bdf3a1613d0f14af5.
[1] https://issues.apache.org/jira/browse/FLINK-3291
[2]
https://issues.apache.org/jira/browse/FLINK-3291?focusedCommentId=15130452&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15130452
[3]
https://github.com/ggevay/flink/commit/b05e12075bb5d0518b6fe12cd8f7de1e6b16f805
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143566#comment-15143566
]
ASF GitHub Bot commented on FLINK-3340:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/1626
[FLINK-3340] [runtime] Fix object juggling in drivers
A second attempt at object swapping in reduce drivers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
3340_fix_object_juggling_in_drivers
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1626.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1626
commit ae376abff90f2234bc67fed257e1bd4effd0114f
Author: Greg Hogan
Date: 2016-02-10T21:12:19Z
[FLINK-3340] [runtime] Fix object juggling in drivers
A second attempt at object swapping in reduce drivers.
> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are
> not properly tracking objects for reuse.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
