GitHub user rezacsedu opened a pull request: https://github.com/apache/spark/pull/12291
Branch 1.6 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/spark branch-1.6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12291 ---- commit a2d584ed9ab3c073df057bed5314bdf877a47616 Author: Timothy Hunter <timhun...@databricks.com> Date: 2015-12-16T18:12:33Z [SPARK-12324][MLLIB][DOC] Fixes the sidebar in the ML documentation This fixes the sidebar, using a pure CSS mechanism to hide it when the browser's viewport is too narrow. Credit goes to the original author Titan-C (mentioned in the NOTICE). Note that I am not a CSS expert, so I can only address comments up to some extent. Default view: <img width="936" alt="screen shot 2015-12-14 at 12 46 39 pm" src="https://cloud.githubusercontent.com/assets/7594753/11793597/6d1d6eda-a261-11e5-836b-6eb2054e9054.png"> When collapsed manually by the user: <img width="1004" alt="screen shot 2015-12-14 at 12 54 02 pm" src="https://cloud.githubusercontent.com/assets/7594753/11793669/c991989e-a261-11e5-8bf6-aecf3bdb6319.png"> Disappears when column is too narrow: <img width="697" alt="screen shot 2015-12-14 at 12 47 22 pm" src="https://cloud.githubusercontent.com/assets/7594753/11793607/7754dbcc-a261-11e5-8b15-e0d074b0e47c.png"> Can still be opened by the user if necessary: <img width="651" alt="screen shot 2015-12-14 at 12 51 15 pm" src="https://cloud.githubusercontent.com/assets/7594753/11793612/7bf82968-a261-11e5-9cc3-e827a7a6b2b0.png"> Author: Timothy Hunter <timhun...@databricks.com> Closes #10297 from thunterdb/12324. (cherry picked from commit a6325fc401f68d9fa30cc947c44acc9d64ebda7b) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit ac0e2ea7c712e91503b02ae3c12fa2fcf5079886 Author: Yanbo Liang <yblia...@gmail.com> Date: 2015-12-16T18:34:30Z [SPARK-12310][SPARKR] Add write.json and write.parquet for SparkR Add ```write.json``` and ```write.parquet``` for SparkR, and deprecated ```saveAsParquetFile```. Author: Yanbo Liang <yblia...@gmail.com> Closes #10281 from yanboliang/spark-12310. (cherry picked from commit 22f6cd86fc2e2d6f6ad2c3aae416732c46ebf1b1) Signed-off-by: Shivaram Venkataraman <shiva...@cs.berkeley.edu> commit 16edd933d7323f8b6861409bbd62bc1efe244c14 Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Date: 2015-12-16T18:43:45Z [SPARK-12215][ML][DOC] User guide section for KMeans in spark.ml cc jkbradley Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Closes #10244 from yu-iskw/SPARK-12215. (cherry picked from commit 26d70bd2b42617ff731b6e9e6d77933b38597ebe) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit f815127294c06320204d9affa4f35da7ec3a710d Author: Jeff Zhang <zjf...@apache.org> Date: 2015-12-16T18:32:32Z [SPARK-12318][SPARKR] Save mode in SparkR should be error by default shivaram Please help review. Author: Jeff Zhang <zjf...@apache.org> Closes #10290 from zjffdu/SPARK-12318. (cherry picked from commit 2eb5af5f0d3c424dc617bb1a18dd0210ea9ba0bc) Signed-off-by: Shivaram Venkataraman <shiva...@cs.berkeley.edu> commit e5b85713d8a0dbbb1a0a07481f5afa6c5098147f Author: Timothy Chen <tnac...@gmail.com> Date: 2015-12-16T18:54:15Z [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode. SPARK_HOME is now causing problem with Mesos cluster mode since spark-submit script has been changed recently to take precendence when running spark-class scripts to look in SPARK_HOME if it's defined. We should skip passing SPARK_HOME from the Spark client in cluster mode with Mesos, since Mesos shouldn't use this configuration but should use spark.executor.home instead. Author: Timothy Chen <tnac...@gmail.com> Closes #10332 from tnachen/scheduler_ui. (cherry picked from commit ad8c1f0b840284d05da737fb2cc5ebf8848f4490) Signed-off-by: Andrew Or <and...@databricks.com> commit e1adf6d7d1c755fb16a0030e66ce9cff348c3de8 Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Date: 2015-12-16T18:55:42Z [SPARK-6518][MLLIB][EXAMPLE][DOC] Add example code and user guide for bisecting k-means This PR includes only an example code in order to finish it quickly. I'll send another PR for the docs soon. Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Closes #9952 from yu-iskw/SPARK-6518. (cherry picked from commit 7b6dc29d0ebbfb3bb941130f8542120b6bc3e234) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 168c89e07c51fa24b0bb88582c739cec0acb44d7 Author: Patrick Wendell <pwend...@gmail.com> Date: 2015-12-16T19:23:41Z Preparing Spark release v1.6.0-rc3 commit aee88eb55b89bfdc763fd30f7574d2aa7de4bf39 Author: Patrick Wendell <pwend...@gmail.com> Date: 2015-12-16T19:23:52Z Preparing development version 1.6.0-SNAPSHOT commit dffa6100d7d96eb38bf8a56f546d66f7a884b03f Author: Joseph K. Bradley <jos...@databricks.com> Date: 2015-12-16T19:53:04Z [SPARK-11608][MLLIB][DOC] Added migration guide for MLlib 1.6 No known breaking changes, but some deprecations and changes of behavior. CC: mengxr Author: Joseph K. Bradley <jos...@databricks.com> Closes #10235 from jkbradley/mllib-guide-update-1.6. (cherry picked from commit 8148cc7a5c9f52c82c2eb7652d9aeba85e72d406) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 04e868b63bfda5afe5cb1a0d6387fb873ad393ba Author: Yanbo Liang <yblia...@gmail.com> Date: 2015-12-16T20:59:22Z [SPARK-12364][ML][SPARKR] Add ML example for SparkR We have DataFrame example for SparkR, we also need to add ML example under ```examples/src/main/r```. cc mengxr jkbradley shivaram Author: Yanbo Liang <yblia...@gmail.com> Closes #10324 from yanboliang/spark-12364. (cherry picked from commit 1a8b2a17db7ab7a213d553079b83274aeebba86f) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 552b38f87fc0f6fab61b1e5405be58908b7f5544 Author: Davies Liu <dav...@databricks.com> Date: 2015-12-16T23:48:11Z [SPARK-12380] [PYSPARK] use SQLContext.getOrCreate in mllib MLlib should use SQLContext.getOrCreate() instead of creating new SQLContext. Author: Davies Liu <dav...@databricks.com> Closes #10338 from davies/create_context. (cherry picked from commit 27b98e99d21a0cc34955337f82a71a18f9220ab2) Signed-off-by: Davies Liu <davies....@gmail.com> commit 638b89bc3b1c421fe11cbaf52649225662d3d3ce Author: Andrew Or <and...@databricks.com> Date: 2015-12-17T00:13:48Z [MINOR] Add missing interpolation in NettyRPCEnv ``` Exception in thread "main" org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply in ${timeout.duration}. This timeout is controlled by spark.rpc.askTimeout at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ``` Author: Andrew Or <and...@databricks.com> Closes #10334 from andrewor14/rpc-typo. (cherry picked from commit 861549acdbc11920cde51fc57752a8bc241064e5) Signed-off-by: Shixiong Zhu <shixi...@databricks.com> commit fb02e4e3bcc50a8f823dfecdb2eef71287225e7b Author: Imran Rashid <iras...@cloudera.com> Date: 2015-12-17T03:01:05Z [SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests `DAGSchedulerEventLoop` normally only logs errors (so it can continue to process more events, from other jobs). However, this is not desirable in the tests -- the tests should be able to easily detect any exception, and also shouldn't silently succeed if there is an exception. This was suggested by mateiz on https://github.com/apache/spark/pull/7699. It may have already turned up an issue in "zero split job". Author: Imran Rashid <iras...@cloudera.com> Closes #8466 from squito/SPARK-10248. (cherry picked from commit 38d9795a4fa07086d65ff705ce86648345618736) Signed-off-by: Andrew Or <and...@databricks.com> commit 4af64385b085002d94c54d11bbd144f9f026bbd8 Author: tedyu <yuzhih...@gmail.com> Date: 2015-12-17T03:02:12Z [SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called SPARK-9886 fixed ExternalBlockStore.scala This PR fixes the remaining references to Runtime.getRuntime.addShutdownHook() Author: tedyu <yuzhih...@gmail.com> Closes #10325 from ted-yu/master. (cherry picked from commit f590178d7a06221a93286757c68b23919bee9f03) Signed-off-by: Andrew Or <and...@databricks.com> Conflicts: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala commit 154567dca126d4992c9c9b08d71d22e9af43c995 Author: Rohit Agarwal <roh...@qubole.com> Date: 2015-12-17T03:04:33Z [SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting. Author: Rohit Agarwal <roh...@qubole.com> Closes #10180 from mindprince/SPARK-12186. (cherry picked from commit fdb38227564c1af40cbfb97df420b23eb04c002b) Signed-off-by: Andrew Or <and...@databricks.com> commit 4ad08035d28b8f103132da9779340c5e64e2d1c2 Author: Marcelo Vanzin <van...@cloudera.com> Date: 2015-12-17T03:47:49Z [SPARK-12386][CORE] Fix NPE when spark.executor.port is set. Author: Marcelo Vanzin <van...@cloudera.com> Closes #10339 from vanzin/SPARK-12386. (cherry picked from commit d1508dd9b765489913bc948575a69ebab82f217b) Signed-off-by: Andrew Or <and...@databricks.com> commit d509194b81abc3c7bf9563d26560d596e1415627 Author: Yin Huai <yh...@databricks.com> Date: 2015-12-17T07:18:53Z [SPARK-12057][SQL] Prevent failure on corrupt JSON records This PR makes JSON parser and schema inference handle more cases where we have unparsed records. It is based on #10043. The last commit fixes the failed test and updates the logic of schema inference. Regarding the schema inference change, if we have something like ``` {"f1":1} [1,2,3] ``` originally, we will get a DF without any column. After this change, we will get a DF with columns `f1` and `_corrupt_record`. Basically, for the second row, `[1,2,3]` will be the value of `_corrupt_record`. When merge this PR, please make sure that the author is simplyianm. JIRA: https://issues.apache.org/jira/browse/SPARK-12057 Closes #10043 Author: Ian Macalinao <m...@ian.pw> Author: Yin Huai <yh...@databricks.com> Closes #10288 from yhuai/handleCorruptJson. (cherry picked from commit 9d66c4216ad830812848c657bbcd8cd50949e199) Signed-off-by: Reynold Xin <r...@databricks.com> commit da7542f2408140a9a3b7ea245350976ac18676a5 Author: echo2mei <534384...@qq.com> Date: 2015-12-17T15:59:17Z Once driver register successfully, stop it to connect to master. This commit is to resolve SPARK-12396. Author: echo2mei <534384...@qq.com> Closes #10354 from echoTomei/master. (cherry picked from commit 5a514b61bbfb609c505d8d65f2483068a56f1f70) Signed-off-by: Davies Liu <davies....@gmail.com> commit a8466489ab01e59fe07ba20adfc3983ec6928157 Author: Davies Liu <davies....@gmail.com> Date: 2015-12-17T16:01:59Z Revert "Once driver register successfully, stop it to connect to master." This reverts commit da7542f2408140a9a3b7ea245350976ac18676a5. commit 1ebedb20f2c5b781eafa9bf2b5ab092d744cc4fd Author: Davies Liu <dav...@databricks.com> Date: 2015-12-17T16:04:11Z [SPARK-12395] [SQL] fix resulting columns of outer join For API DataFrame.join(right, usingColumns, joinType), if the joinType is right_outer or full_outer, the resulting join columns could be wrong (will be null). The order of columns had been changed to match that with MySQL and PostgreSQL [1]. This PR also fix the nullability of output for outer join. [1] http://www.postgresql.org/docs/9.2/static/queries-table-expressions.html Author: Davies Liu <dav...@databricks.com> Closes #10353 from davies/fix_join. (cherry picked from commit a170d34a1b309fecc76d1370063e0c4f44dc2142) Signed-off-by: Davies Liu <davies....@gmail.com> commit 41ad8aced2fc6c694c15e9465cfa34517b2395e8 Author: Yanbo Liang <yblia...@gmail.com> Date: 2015-12-17T17:19:46Z [SQL] Update SQLContext.read.text doc Since we rename the column name from ```text``` to ```value``` for DataFrame load by ```SQLContext.read.text```, we need to update doc. Author: Yanbo Liang <yblia...@gmail.com> Closes #10349 from yanboliang/text-value. (cherry picked from commit 6e0771665b3c9330fc0a5b2c7740a796b4cd712e) Signed-off-by: Reynold Xin <r...@databricks.com> commit 1fbca41200d6e73cb276d5949b894881c700323f Author: Shixiong Zhu <shixi...@databricks.com> Date: 2015-12-17T17:55:37Z [SPARK-12220][CORE] Make Utils.fetchFile support files that contain special characters This PR encodes and decodes the file name to fix the issue. Author: Shixiong Zhu <shixi...@databricks.com> Closes #10208 from zsxwing/uri. (cherry picked from commit 86e405f357711ae93935853a912bc13985c259db) Signed-off-by: Shixiong Zhu <shixi...@databricks.com> commit 881f2544e13679c185a7c34ddb82e885aaa79813 Author: Iulian Dragos <jagua...@gmail.com> Date: 2015-12-17T18:19:31Z [SPARK-12345][MESOS] Properly filter out SPARK_HOME in the Mesos REST server Fix problem with #10332, this one should fix Cluster mode on Mesos Author: Iulian Dragos <jagua...@gmail.com> Closes #10359 from dragos/issue/fix-spark-12345-one-more-time. (cherry picked from commit 8184568810e8a2e7d5371db2c6a0366ef4841f70) Signed-off-by: Kousuke Saruta <saru...@oss.nttdata.co.jp> commit 88bbb5429dd3efcff6b2835a70143247b08ae6b2 Author: Andrew Or <and...@databricks.com> Date: 2015-12-17T04:01:47Z [SPARK-12390] Clean up unused serializer parameter in BlockManager No change in functionality is intended. This only changes internal API. Author: Andrew Or <and...@databricks.com> Closes #10343 from andrewor14/clean-bm-serializer. Conflicts: core/src/main/scala/org/apache/spark/storage/BlockManager.scala commit c0ab14fbeab2a81d174c3643a4fcc915ff2902e8 Author: Shixiong Zhu <shixi...@databricks.com> Date: 2015-12-17T21:23:48Z [SPARK-12410][STREAMING] Fix places that use '.' and '|' directly in split String.split accepts a regular expression, so we should escape "." and "|". Author: Shixiong Zhu <shixi...@databricks.com> Closes #10361 from zsxwing/reg-bug. (cherry picked from commit 540b5aeadc84d1a5d61bda4414abd6bf35dc7ff9) Signed-off-by: Shixiong Zhu <shixi...@databricks.com> commit 48dcee48416d87bf9572ace0a82285bacfcbf46e Author: Reynold Xin <r...@databricks.com> Date: 2015-12-17T22:16:49Z [SPARK-12397][SQL] Improve error messages for data sources when they are not found Point users to spark-packages.org to find them. Author: Reynold Xin <r...@databricks.com> Closes #10351 from rxin/SPARK-12397. (cherry picked from commit e096a652b92fc64a7b3457cd0766ab324bcc980b) Signed-off-by: Michael Armbrust <mich...@databricks.com> commit 4df1dd403441a4e4ca056d294385d8d0d8a0c65d Author: Evan Chen <ch...@us.ibm.com> Date: 2015-12-17T22:22:30Z [SPARK-12376][TESTS] Spark Streaming Java8APISuite fails in assertOrderInvariantEquals method org.apache.spark.streaming.Java8APISuite.java is failing due to trying to sort immutable list in assertOrderInvariantEquals method. Author: Evan Chen <ch...@us.ibm.com> Closes #10336 from evanyc15/SPARK-12376-StreamingJavaAPISuite. commit 9177ea383a29653f0591a59e1ee2dff6b87d5a1c Author: jhu-chang <gt.hu.ch...@gmail.com> Date: 2015-12-18T01:53:15Z [SPARK-11749][STREAMING] Duplicate creating the RDD in file stream when recovering from checkpoint data Add a transient flag `DStream.restoredFromCheckpointData` to control the restore processing in DStream to avoid duplicate works: check this flag first in `DStream.restoreCheckpointData`, only when `false`, the restore process will be executed. Author: jhu-chang <gt.hu.ch...@gmail.com> Closes #9765 from jhu-chang/SPARK-11749. (cherry picked from commit f4346f612b6798517153a786f9172cf41618d34d) Signed-off-by: Shixiong Zhu <shixi...@databricks.com> commit df0231952e5542e9870f8dde9ecbd7ad9a50f847 Author: Michael Gummelt <mgumm...@mesosphere.io> Date: 2015-12-18T11:18:00Z [SPARK-12413] Fix Mesos ZK persistence I believe this fixes SPARK-12413. I'm currently running an integration test to verify. Author: Michael Gummelt <mgumm...@mesosphere.io> Closes #10366 from mgummelt/fix-zk-mesos. (cherry picked from commit 2bebaa39d9da33bc93ef682959cd42c1968a6a3e) Signed-off-by: Kousuke Saruta <saru...@oss.nttdata.co.jp> commit 1dc71ec777ff7cac5d3d7adb13f2d63ffe8909b6 Author: Yin Huai <yh...@databricks.com> Date: 2015-12-18T18:52:14Z [SPARK-12218][SQL] Invalid splitting of nested AND expressions in Data Source filter API JIRA: https://issues.apache.org/jira/browse/SPARK-12218 When creating filters for Parquet/ORC, we should not push nested AND expressions partially. Author: Yin Huai <yh...@databricks.com> Closes #10362 from yhuai/SPARK-12218. (cherry picked from commit 41ee7c57abd9f52065fd7ffb71a8af229603371d) Signed-off-by: Yin Huai <yh...@databricks.com> ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org