[GitHub] spark pull request #19296: Branch 2.2

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19296


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19296: Branch 2.2

[rkp2916]

GitHub user rkp2916 opened a pull request:

https://github.com/apache/spark/pull/19296

Branch 2.2

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-2.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19296.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19296


commit a0bf5c47cb9c72d73616f876a4521ae80e2e4ecb
Author: Peng 
Date:   2017-05-22T14:42:37Z

[SPARK-20764][ML][PYSPARK] Fix visibility discrepancy with numInstances and 
degreesOfFreedom in LR and GLR - Python version

## What changes were proposed in this pull request?

SPARK-20097 exposed degreesOfFreedom in LinearRegressionSummary and 
numInstances in GeneralizedLinearRegressionSummary. Python API should be 
updated to reflect these changes.

## How was this patch tested?
The existing UT

Author: Peng 

Closes #18062 from mpjlu/spark-20764.

(cherry picked from commit cfca01136bd7443c1d9daf8e8e256635eec20ddc)
Signed-off-by: Yanbo Liang 

commit 2fd613875ae272c72628bb5006cb70b0964c077d
Author: Mark Grover 
Date:   2017-05-22T17:10:41Z

[SPARK-20756][YARN] yarn-shuffle jar references unshaded guava

and contains scala classes

## What changes were proposed in this pull request?
This change ensures that all references to guava from within the yarn 
shuffle jar pointed to the shaded guava class already provided in the jar.

Also, it explicitly excludes scala classes from being added to the jar.

## How was this patch tested?
Ran unit tests on the module and they passed.
javap now returns the expected result - reference to the shaded guava under 
`org/spark_project` (previously this was referring to `com.google...`
```
javap -cp 
common/network-yarn/target/scala-2.11/spark-2.3.0-SNAPSHOT-yarn-shuffle.jar -c 
org/apache/spark/network/yarn/YarnShuffleService | grep Lists
  57: invokestatic  #138// Method 
org/spark_project/guava/collect/Lists.newArrayList:()Ljava/util/ArrayList;
```

Guava is still shaded in the jar:
```
jar -tf 
common/network-yarn/target/scala-2.11/spark-2.3.0-SNAPSHOT-yarn-shuffle.jar | 
grep guava | head
META-INF/maven/com.google.guava/
META-INF/maven/com.google.guava/guava/
META-INF/maven/com.google.guava/guava/pom.properties
META-INF/maven/com.google.guava/guava/pom.xml
org/spark_project/guava/
org/spark_project/guava/annotations/
org/spark_project/guava/annotations/Beta.class
org/spark_project/guava/annotations/GwtCompatible.class
org/spark_project/guava/annotations/GwtIncompatible.class
org/spark_project/guava/annotations/VisibleForTesting.class
```
(not sure if the above META-INF/* is a problem or not)

I took this jar, deployed it on a yarn cluster with shuffle service 
enabled, and made sure the YARN node managers came up. An application with a 
shuffle was run and it succeeded.

Author: Mark Grover 

Closes #17990 from markgrover/spark-20756.

(cherry picked from commit 36309110046a89d749a7c9746eaa16997de26922)
Signed-off-by: Marcelo Vanzin 

commit d8328d8d1cdaeae5d8a5ebee3e4c807821c8ddf3
Author: Marcelo Vanzin 
Date:   2017-05-22T19:34:15Z

[SPARK-20814][MESOS] Restore support for spark.executor.extraClassPath.

Restore code that was removed as part of SPARK-17979, but instead of
using the deprecated env variable name to propagate the class path, use
a new one.

Verified by running "./bin/spark-class 
o.a.s.executor.CoarseGrainedExecutorBackend"
manually.

Author: Marcelo Vanzin 

Closes #18037 from vanzin/SPARK-20814.

(cherry picked from commit df64fa79d678e354efe3969a6ba25d964128ce18)
Signed-off-by: Marcelo Vanzin 

commit ddc199eefbf68223f817a4c756b243362c1a95ca
Author: James Shuster 
Date:   2017-05-23T04:41:11Z

[SPARK-20815][SPARKR] NullPointerException in 
RPackageUtils#checkManifestForR

## What changes were proposed in this pull request?

- Add a null check to RPackageUtils#checkManifestForR so that jars w/o