[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2837/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-04 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95673 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95673/testReport)**
 for PR 22179 at commit 
[`d1dac1e`](https://github.com/apache/spark/commit/d1dac1e75d450ca26a460cb36f8ac0684a251c64).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-04 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
Thanks, @dongjoon-hyun 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-04 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22179
  
@wangyum

I made a [PR](https://github.com/wangyum/spark/pull/9) to your branch. 
Could you review and merge that? The content are the followings.
- Add a new test case for SPARK-23131 (Kryo raises StackOverflow during 
serializing GLR model)
- Update your test case for SPARK-25176 (ClassCastException when 
reading/writing a Map)

After merging the [PR](https://github.com/wangyum/spark/pull/9), please add 
`[SPARK-25176]` after `[SPARK-23131]` because we have test cases. We can 
resolve both issues under your name.

For the remaining SPARK-20389 (NegativeArraySizeException), there is no 
response on JIRA. So, let's simply exclude it from this PR description because 
we cannot claim that it's resolved.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95646/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95646 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95646/testReport)**
 for PR 22179 at commit 
[`0d78113`](https://github.com/apache/spark/commit/0d7811348e5746e1a7e1ce887d47ae4ba413c014).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95643/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95643 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95643/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class MapHolder `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95646 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95646/testReport)**
 for PR 22179 at commit 
[`0d78113`](https://github.com/apache/spark/commit/0d7811348e5746e1a7e1ce887d47ae4ba413c014).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2813/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2810/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95643 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95643/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95623/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95623 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95623/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class MapHolder `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2794/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95623 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95623/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95613/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95613 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95613/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class MapHolder `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
Sorry @dongjoon-hyun I only reproduce one test.
kryo-parametrized-type-inheritance related to language. It seems scala 
can't reproduce it:
```scala
val ser = new KryoSerializer(new 
SparkConf).newInstance().asInstanceOf[KryoSerializerInstance]

class BaseType[R] {}
class CollectionType(val child: BaseType[_]*) extends BaseType[Boolean] 
{
  val children: List[BaseType[_]] = child.toList
}
class ValueType[R](val v: R) extends BaseType[R] {}

val value = new CollectionType(new ValueType("hello"))

ser.serialize(value)
```

SPARK-23131 may be related to data. I can't reproduce it:
```scala
  def modelToString(model: GeneralizedLinearRegressionModel): (String, 
String) = {
val os: ByteArrayOutputStream = new ByteArrayOutputStream()
val zos = new GZIPOutputStream(os)
val oo: ObjectOutputStream = new ObjectOutputStream(zos)
oo.writeObject(model)
oo.close()
zos.close()
os.close()
(model.uid, DatatypeConverter.printBase64Binary(os.toByteArray))
  }
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95613 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95613/testReport)**
 for PR 22179 at commit 
[`f2fb28d`](https://github.com/apache/spark/commit/f2fb28da3eb272651530b77dbd4ea33511f0727d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2787/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-02 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22179
  
I pinged the first one, SPARK-20389, to find any chance to test on the 
original cluster. Although it's too old issue, but there is no harm for us to 
ping once more.

For the third one, SPARK-25176, the reporter provided the sample code. 
Although the code is open for us, but it's not Apache license. We can make our 
own test case for type-inheritance.
> You can find a simple test to reproduce the issue [4].
- [4] https://github.com/mpryahin/kryo-parametrized-type-inheritance
  - 
https://github.com/mpryahin/kryo-parametrized-type-inheritance/blob/master/src/main/java/io/cleverdata/dmpkit/BaseType.java


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-02 Thread srowen
Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22179
  
Tests are great of course; those three might be hard to test. The first and 
last one don't have info on the reproduction. The second one looks possibly 
reproducible though as it shows the custom serialization code used to write a 
GeneralizedLinearRegression model. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22179
  
Thanks, @wangyum !


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-01 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
@dongjoon-hyun I'm trying to add test cases.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-09-01 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22179
  
Ping, @wangyum . Could you add the test cases for SPARK-20389, SPARK-23131, 
and SPARK-25176?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22179
  
@wangyum . I know this is designed for a build-only PR. However, since this 
is claimed for bug fixes(SPARK-20389, SPARK-23131, and SPARK-25176), could you 
add test cases for them in order to verify those issues are really resolved?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/22179
  
For maximum breaking-change forgiveness, the "safe" option would be to punt 
it to 3.0; which seems likely to be the next release after this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread srowen
Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22179
  
Kryo 3.0.3 and Chill 0.8.2 already work with 2.12, so no issue particularly 
there. I don't see that this change would be visible to users, and tests 
suggest it doesn't affect Spark. I'm cautious that I'm maybe overlooking 
something though. And weighing the risk vs the upside of fixing some user 
issues and potential performance bumps.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/22179
  
If it doesn't expose any breaking changes to users, that seems reasonable. 
I assume there would be no integration problems with scala 2.12?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-30 Thread srowen
Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22179
  
https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0

This has a number of breaking changes. However, Spark actually doesn't use 
kryo itself, and excludes it. It uses kryo-shaded, via chill. chill otherwise 
has almost no changes from 0.8 to 0.9 except to accommodate Kryo 4. Therefore 
the update shouldn't actually force downstream apps to use a different kryo, 
which is most of the potential problem here.

The changes in serialized format should not be visible outside Spark, if I 
am not mistaken. It controls serialization within Spark components and this 
isn't compatible across versions anyway.

I'm inclined to merge this for 2.4 as it resolves a number of issues users 
are hitting.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-22 Thread srowen
Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22179
  
Yes, I think the question is whether anything breaks as well. Kryo 4.x is a 
breaking change IIRC. Is there a 3.x version that has these fixes?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-22 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
Thanks @srowen 
[SPARK-25176](https://issues.apache.org/jira/browse/SPARK-25176) has a detail 
description:

> I'm using the latest spark version spark-core_2.11:2.3.1 which 
transitively depends on com.esotericsoftware:kryo-shaded:3.0.3 via the
com.twitter:chill_2.11:0.8.0 dependency. This exact version of kryo 
serializer contains an issue [1,2] which results in throwing 
ClassCastExceptions when serialising parameterised type hierarchy.
This issue has been fixed in kryo version 4.0.0 [3]. It would be great to 
have this update in Spark as well. Could you please upgrade the version of 
com.twitter:chill_2.11 dependency from 0.8.0 up to 0.9.2?
You can find a simple test to reproduce the issue [4].
[1] https://github.com/EsotericSoftware/kryo/issues/384
[2] https://github.com/EsotericSoftware/kryo/issues/377
[3] https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0
[4] https://github.com/mpryahin/kryo-parametrized-type-inheritance


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-22 Thread srowen
Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22179
  
That looks like a major version bump -- the usual question here -- what are 
the key changes we need, what are possible incompatible changes?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread wangyum
Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/22179
  
cc @srowen


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95072/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95072 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95072/testReport)**
 for PR 22179 at commit 
[`8c28e07`](https://github.com/apache/spark/commit/8c28e078526445a31099ca5f1ae71ce76d782004).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22179
  
**[Test build #95072 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95072/testReport)**
 for PR 22179 at commit 
[`8c28e07`](https://github.com/apache/spark/commit/8c28e078526445a31099ca5f1ae71ce76d782004).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2412/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #22179: [SPARK-23131][BUILD] Upgrade Kryo to 4.0.2

2018-08-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22179
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org