Github user a-roberts commented on the pull request:
https://github.com/apache/spark/pull/12327#issuecomment-222107940
> cacheSize1 and cacheSize2 are both the size after cleaning. The
difference is that, cacheSize1 is the size after cleaned the data with line
object reference, cacheSize2 is the size after cleaned the data without line
object reference.
Looking for clarity here, is it true that clean "with the reference" should
be bigger (cacheSize1) and clean "without the reference" should be smaller
(cacheSize2)?
OpenJDK, cacheSize1: 180392, cacheSize2: 187896 (bigger without the line
object reference)
IBM JDK, cacheSize1: 354692, cacheSize2: 263800 (smaller without the line
object reference)
What exactly does "without line object reference" mean and should
cacheSize1 be smaller or bigger than cacheSize2?
I know the SizeEstimator overestimates for IBM Java so our cached footprint
is much larger (handling this), so because of the larger difference we get this
test failing, OpenJDK **fails** with Kryo and IBM **passes** with Kryo for this
test.
A better check would be to run with and without the closure cleaner change
and to check the second result is less by the size of the line object, so based
on our cacheSize2 being smaller (without the line object reference), I'm
thinking that IBM Java functions as expected and OpenJDK doesn't - but this
depends on my questions above, interested to hear what you think @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
