[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328014#comment-15328014
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/1626
  
Thank you @xhumanoid for reporting this! I will make the fix.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327660#comment-15327660
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user xhumanoid commented on the issue:

https://github.com/apache/flink/pull/1626
  
Hi, I know than issue closed, but he maybe have small bug


https://github.com/apache/flink/pull/1626/files#diff-4a133896fec62bcabc1120b0df8cb8daR205

in hashCode recursion on this.hashCode()


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167956#comment-15167956
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-189004909
  
Pushed. Thanks for the explanation.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167871#comment-15167871
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188985984
  
If you can quickly do it, yes.

Otherwise whoever merges this tomorrow, please remove the sysout statements.
We try to keep the maven output a bit cleaner that way. If you want to 
output something into the logs for the tests, just use a regular slf4j logger...


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167864#comment-15167864
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188982973
  
@StephanEwen just to clarify, since time is of the essence, would you like 
me to update the pull request without the println's?


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167641#comment-15167641
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-188929386
  
This looks good to me.

+1 for merging, but removing the System.out.println


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158526#comment-15158526
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1692#issuecomment-187598869
  
Ooops, you are right. I'll fix it.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157671#comment-15157671
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1692#issuecomment-187376965
  
Is this FLINK-3440?


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157622#comment-15157622
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1692

[FLINK-3340][Kafka 0.8] Commit also offsets retrieved from Kafka into the 
OffsetStore (ZK)

Before this change, offsets were only committed to Zookeeper if something 
was read from the TopicPartition.

Now, the start offsets retrieved from Kafka (based on `auto.offset.reset`) 
are also put into the state so that they are also committed into ZK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3440

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1692.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1692


commit f7a7ae90f566faa38975c3f9a193090ae18ab37a
Author: Robert Metzger 
Date:   2016-02-22T13:48:37Z

[FLINK-3340][Kafka 0.8] Commit also offsets retrieved from Kafka into the 
OffsetStore (ZK)




> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
> Fix For: 1.0.0
>
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144481#comment-15144481
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user ggevay commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-183287868
  
So, if I understand correctly, this is aiming to fix the problem that was 
brought up by [1], but in a different way then proposed there.

I still have the same worry about this solution that I wrote in this 
comment: [2]: `MutableObjectIterator.next(reuse)` is used at 271 places, 
according to IntelliJ. If we accept the change in the object reuse contract of 
`MergeIterator.next(reuse)` introduced by 
0a8df6d513fa59d650ff875bdf3a1613d0f14af5, all the call sites can potentially 
have the same problem as the reduce drivers, so we should check all of these 
271 places. For example, just by skimming the list for a few minutes, all the 8 
calls in `CrossDriver` seem suspicious at first glance, as well as the calls in 
`UnionWithTempOperator`, `MutableHashTable.ProbeIterator.next`, and 
`ReusingBuildFirstHashJoinIterator.callWithNextKey`.

My alternative solution [3] has the drawback that it would introduce an 
extra copying in `MergeIterator.HeadStream.getHead`, but would change the 
contract of `MergeIterator.next(reuse)` back to the same as it was before 
0a8df6d513fa59d650ff875bdf3a1613d0f14af5.

[1] https://issues.apache.org/jira/browse/FLINK-3291
[2] 
https://issues.apache.org/jira/browse/FLINK-3291?focusedCommentId=15130452=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15130452
[3] 
https://github.com/ggevay/flink/commit/b05e12075bb5d0518b6fe12cd8f7de1e6b16f805


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144619#comment-15144619
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-183348625
  
Hi, @ggevay, these fixes to the reduce drivers are independent of changes 
to `MutableObjectIterator`. This patch is intended to fix the bug you reported 
in [FLINK-3291](https://issues.apache.org/jira/browse/FLINK-3291).

+1 to improving the `MutableObjectIterator` API. Let's continue the 
conversation over on 
[FLINK-3291](https://issues.apache.org/jira/browse/FLINK-3291)!


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144761#comment-15144761
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user ggevay commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-183390101
  
OK, after Stephan's comment on FLINK-3291, I agree.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144710#comment-15144710
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user ggevay commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-183375884
  
Sorry, but I don't think it is independent, because the bug in the reduce 
drivers could alternatively be fixed by saying that the bug is actually not in 
the reduce drivers at all, but in `MergeIterator`, and fix it there instead (as 
in 
https://github.com/ggevay/flink/commit/b05e12075bb5d0518b6fe12cd8f7de1e6b16f805).
 This depends on what we want the contract of `MergeIterator.next(reuse)` to be.


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144753#comment-15144753
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

Github user greghogan commented on the pull request:

https://github.com/apache/flink/pull/1626#issuecomment-183389185
  
`MergeIterator.next(reuse)` should allow object swapping. The source code 
documentation should be improved to make this clearer, but this contract is 
necessary to minimize the copying of objects


> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3340) Fix object juggling in drivers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143566#comment-15143566
 ] 

ASF GitHub Bot commented on FLINK-3340:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/1626

[FLINK-3340] [runtime] Fix object juggling in drivers

A second attempt at object swapping in reduce drivers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
3340_fix_object_juggling_in_drivers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1626


commit ae376abff90f2234bc67fed257e1bd4effd0114f
Author: Greg Hogan 
Date:   2016-02-10T21:12:19Z

[FLINK-3340] [runtime] Fix object juggling in drivers

A second attempt at object swapping in reduce drivers.




> Fix object juggling in drivers
> --
>
> Key: FLINK-3340
> URL: https://issues.apache.org/jira/browse/FLINK-3340
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Critical
>
> {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are 
> not properly tracking objects for reuse.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)