GitHub user engineeyao opened a pull request:
https://github.com/apache/spark/pull/19423
Branch 2.2
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-2.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19423
commit 9cbf39f1c74f16483865cd93d6ffc3c521e878a7
Author: Yanbo Liang
Date: 2017-05-25T12:15:15Z
[SPARK-19281][FOLLOWUP][ML] Minor fix for PySpark FPGrowth.
## What changes were proposed in this pull request?
Follow-up for #17218, some minor fix for PySpark ```FPGrowth```.
## How was this patch tested?
Existing UT.
Author: Yanbo Liang
Closes #18089 from yanboliang/spark-19281.
(cherry picked from commit 913a6bfe4b0eb6b80a03b858ab4b2767194103de)
Signed-off-by: Yanbo Liang
commit e01f1f222bcb7c469b1e1595e9338ed478d99894
Author: Yan Facai (é¢åæ)
Date: 2017-05-25T13:40:39Z
[SPARK-20768][PYSPARK][ML] Expose numPartitions (expert) param of PySpark
FPGrowth.
## What changes were proposed in this pull request?
Expose numPartitions (expert) param of PySpark FPGrowth.
## How was this patch tested?
+ [x] Pass all unit tests.
Author: Yan Facai (é¢åæ)
Closes #18058 from facaiy/ENH/pyspark_fpg_add_num_partition.
(cherry picked from commit 139da116f130ed21481d3e9bdee5df4b8d7760ac)
Signed-off-by: Yanbo Liang
commit 022a4957d8dc8d6049e0a8c9191fcfd1bd95a4a4
Author: Lior Regev
Date: 2017-05-25T16:08:19Z
[SPARK-20741][SPARK SUBMIT] Added cleanup of JARs archive generated by
SparkSubmit
## What changes were proposed in this pull request?
Deleted generated JARs archive after distribution to HDFS
## How was this patch tested?
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: Lior Regev
Closes #17986 from liorregev/master.
(cherry picked from commit 7306d556903c832984c7f34f1e8fe738a4b2343c)
Signed-off-by: Sean Owen
commit 5ae1c652147aba9c5087335b0c6916a1035090b2
Author: hyukjinkwon
Date: 2017-05-25T16:10:30Z
[SPARK-19707][SPARK-18922][TESTS][SQL][CORE] Fix test failures/the invalid
path check for sc.addJar on Windows
## What changes were proposed in this pull request?
This PR proposes two things:
- A follow up for SPARK-19707 (Improving the invalid path check for
sc.addJar on Windows as well).
```
org.apache.spark.SparkContextSuite:
- add jar with invalid path *** FAILED *** (32 milliseconds)
2 was not equal to 1 (SparkContextSuite.scala:309)
...
```
- Fix path vs URI related test failures on Windows.
```
org.apache.spark.storage.LocalDirsSuite:
- SPARK_LOCAL_DIRS override also affects driver *** FAILED *** (0
milliseconds)
new java.io.File("/NONEXISTENT_PATH").exists() was true
(LocalDirsSuite.scala:50)
...
- Utils.getLocalDir() throws an exception if any temporary directory
cannot be retrieved *** FAILED *** (15 milliseconds)
Expected exception java.io.IOException to be thrown, but no exception
was thrown. (LocalDirsSuite.scala:64)
...
```
```
org.apache.spark.sql.hive.HiveSchemaInferenceSuite:
- orc: schema should be inferred and saved when INFER_AND_SAVE is
specified *** FAILED *** (203 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-dae61ab3-a851-4dd3-bf4e-be97c501f254
...
- parquet: schema should be inferred and saved when INFER_AND_SAVE is
specified *** FAILED *** (203 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-fa3aff89-a66e-4376-9a37-2a9b87596939
...
- orc: schema should be inferred but not stored when INFER_ONLY is
specified *** FAILED *** (141 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-fb464e59-b049-481b-9c75-f53295c9fc2c
...
- parquet: schema should be inferred but not stored when INFER_ONLY is
specified *** FAILED *** (125 millis
