[
https://issues.apache.org/jira/browse/SPARK-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068693#comment-15068693
]
Pete Robbins edited comment on SPARK-12470 at 12/22/15 9:47 PM:
----------------------------------------------------------------
I'm fairly sure the code in my PR is correct but it is causing an
ExchangeCoordinatorSuite test to fail. I'm struggling to see why this test is
failing with the change I made. The failure is:
determining the number of reducers: aggregate operator *** FAILED ***
3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)
putting some debug into the test I see that before my change the pre-shuffle
partition sizes are 600, 600, 600, 600, 600 an after my change are 800. 800.
800. 800. 720 but I have no idea why. I'd really appreciate anyone with
knowledge of this area a) checking my PR and b) helping explain the failing
test.
EDIT Please ignore. Merged with latest head including changes for SPARK-12388
now passes all tests
was (Author: robbinspg):
I'm fairly sure the code in my PR is correct but it is causing an
ExchangeCoordinatorSuite test to fail. I'm struggling to see why this test is
failing with the change I made. The failure is:
determining the number of reducers: aggregate operator *** FAILED ***
3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)
putting some debug into the test I see that before my change the pre-shuffle
partition sizes are 600, 600, 600, 600, 600 an after my change are 800. 800.
800. 800. 720 but I have no idea why. I'd really appreciate anyone with
knowledge of this area a) checking my PR and b) helping explain the failing
test.
> Incorrect calculation of row size in
> o.a.s.sql.catalyst.expressions.codegen.GenerateUnsafeRowJoiner
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-12470
> URL: https://issues.apache.org/jira/browse/SPARK-12470
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.2
> Reporter: Pete Robbins
> Priority: Minor
>
> While looking into https://issues.apache.org/jira/browse/SPARK-12319 I
> noticed that the row size is incorrectly calculated.
> The "sizeReduction" value is calculated in words:
> // The number of words we can reduce when we concat two rows together.
> // The only reduction comes from merging the bitset portion of the two
> rows, saving 1 word.
> val sizeReduction = bitset1Words + bitset2Words - outputBitsetWords
> but then it is subtracted from the size of the row in bytes:
> | out.pointTo(buf, ${schema1.size + schema2.size}, sizeInBytes -
> $sizeReduction);
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
