GitHub user somideshmukh opened a pull request: https://github.com/apache/spark/pull/9464
Branch 1.5 You can merge this pull request into a Git repository by running: $ git pull https://github.com/somideshmukh/spark branch-1.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9464.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9464 ---- commit b7c4ff144e783e80adc1efc2c28965c2e739dd5e Author: Josh Rosen <joshro...@databricks.com> Date: 2015-08-25T07:04:10Z [SPARK-9293] [SPARK-9813] Analysis should check that set operations are only performed on tables with equal numbers of columns This patch adds an analyzer rule to ensure that set operations (union, intersect, and except) are only applied to tables with the same number of columns. Without this rule, there are scenarios where invalid queries can return incorrect results instead of failing with error messages; SPARK-9813 provides one example of this problem. In other cases, the invalid query can crash at runtime with extremely confusing exceptions. I also performed a bit of cleanup to refactor some of those logical operators' code into a common `SetOperation` base class. Author: Josh Rosen <joshro...@databricks.com> Closes #7631 from JoshRosen/SPARK-9293. (cherry picked from commit 82268f07abfa658869df2354ae72f8d6ddd119e8) Signed-off-by: Michael Armbrust <mich...@databricks.com> commit 76d920f2b814304051dd76f0ca78301e872fc811 Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Date: 2015-08-25T07:28:51Z [SPARK-10214] [SPARKR] [DOCS] Improve SparkR Column, DataFrame API docs cc: shivaram ## Summary - Add name tags to each methods in DataFrame.R and column.R - Replace `rdname column` with `rdname {each_func}`. i.e. alias method : `rdname column` => `rdname alias` ## Generated PDF File https://drive.google.com/file/d/0B9biIZIU47lLNHN2aFpnQXlSeGs/view?usp=sharing ## JIRA [[SPARK-10214] Improve SparkR Column, DataFrame API docs - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-10214) Author: Yu ISHIKAWA <yuu.ishik...@gmail.com> Closes #8414 from yu-iskw/SPARK-10214. (cherry picked from commit d4549fe58fa0d781e0e891bceff893420cb1d598) Signed-off-by: Shivaram Venkataraman <shiva...@cs.berkeley.edu> commit 4841ebb1861025067a1108c11f64bb144427a308 Author: Sean Owen <so...@cloudera.com> Date: 2015-08-25T07:32:20Z [SPARK-6196] [BUILD] Remove MapR profiles in favor of hadoop-provided Follow up to https://github.com/apache/spark/pull/7047 pwendell mentioned that MapR should use `hadoop-provided` now, and indeed the new build script does not produce `mapr3`/`mapr4` artifacts anymore. Hence the action seems to be to remove the profiles, which are now not used. CC trystanleftwich Author: Sean Owen <so...@cloudera.com> Closes #8338 from srowen/SPARK-6196. (cherry picked from commit 57b960bf3706728513f9e089455a533f0244312e) Signed-off-by: Sean Owen <so...@cloudera.com> commit 2032d66706d165079550f06bf695e0b08be7e143 Author: Tathagata Das <tathagata.das1...@gmail.com> Date: 2015-08-25T07:35:51Z [SPARK-10210] [STREAMING] Filter out non-existent blocks before creating BlockRDD When write ahead log is not enabled, a recovered streaming driver still tries to run jobs using pre-failure block ids, and fails as the block do not exists in-memory any more (and cannot be recovered as receiver WAL is not enabled). This occurs because the driver-side WAL of ReceivedBlockTracker is recovers that past block information, and ReceiveInputDStream creates BlockRDDs even if those blocks do not exist. The solution in this PR is to filter out block ids that do not exist before creating the BlockRDD. In addition, it adds unit tests to verify other logic in ReceiverInputDStream. Author: Tathagata Das <tathagata.das1...@gmail.com> Closes #8405 from tdas/SPARK-10210. (cherry picked from commit 1fc37581a52530bac5d555dbf14927a5780c3b75) Signed-off-by: Tathagata Das <tathagata.das1...@gmail.com> commit e5cea566a32d254adc9424a2f9e79b92eda3e6e4 Author: Davies Liu <dav...@databricks.com> Date: 2015-08-25T08:00:44Z [SPARK-10177] [SQL] fix reading Timestamp in parquet from Hive We misunderstood the Julian days and nanoseconds of the day in parquet (as TimestampType) from Hive/Impala, they are overlapped, so can't be added together directly. In order to avoid the confusing rounding when do the converting, we use `2440588` as the Julian Day of epoch of unix timestamp (which should be 2440587.5). Author: Davies Liu <dav...@databricks.com> Author: Cheng Lian <l...@databricks.com> Closes #8400 from davies/timestamp_parquet. (cherry picked from commit 2f493f7e3924b769160a16f73cccbebf21973b91) Signed-off-by: Cheng Lian <l...@databricks.com> commit a0f22cf295a1d20814c5be6cc727e39e95a81c27 Author: Josh Rosen <joshro...@databricks.com> Date: 2015-08-25T08:06:36Z [SPARK-10195] [SQL] Data sources Filter should not expose internal types Spark SQL's data sources API exposes Catalyst's internal types through its Filter interfaces. This is a problem because types like UTF8String are not stable developer APIs and should not be exposed to third-parties. This issue caused incompatibilities when upgrading our `spark-redshift` library to work against Spark 1.5.0. To avoid these issues in the future we should only expose public types through these Filter objects. This patch accomplishes this by using CatalystTypeConverters to add the appropriate conversions. Author: Josh Rosen <joshro...@databricks.com> Closes #8403 from JoshRosen/datasources-internal-vs-external-types. (cherry picked from commit 7bc9a8c6249300ded31ea931c463d0a8f798e193) Signed-off-by: Reynold Xin <r...@databricks.com> commit 73f1dd1b5acf1c6c37045da25902d7ca5ab795e4 Author: Yin Huai <yh...@databricks.com> Date: 2015-08-25T08:19:34Z [SPARK-10197] [SQL] Add null check in wrapperFor (inside HiveInspectors). https://issues.apache.org/jira/browse/SPARK-10197 Author: Yin Huai <yh...@databricks.com> Closes #8407 from yhuai/ORCSPARK-10197. (cherry picked from commit 0e6368ffaec1965d0c7f89420e04a974675c7f6e) Signed-off-by: Cheng Lian <l...@databricks.com> commit 5d6840569761a42624f9852b942e33039d21f46a Author: Zhang, Liye <liye.zh...@intel.com> Date: 2015-08-25T10:48:55Z [DOC] add missing parameters in SparkContext.scala for scala doc Author: Zhang, Liye <liye.zh...@intel.com> Closes #8412 from liyezhang556520/minorDoc. (cherry picked from commit 5c14890159a5711072bf395f662b2433a389edf9) Signed-off-by: Sean Owen <so...@cloudera.com> commit bdcc8e608d9a1160db988faa76808149c28a3b50 Author: ehnalis <zoltan.zv...@gmail.com> Date: 2015-08-25T11:30:06Z Fixed a typo in DAGScheduler. Author: ehnalis <zoltan.zv...@gmail.com> Closes #8308 from ehnalis/master. (cherry picked from commit 7f1e507bf7e82bff323c5dec3c1ee044687c4173) Signed-off-by: Sean Owen <so...@cloudera.com> commit 0402f1297c697bfbe8b5c7bfc170fcdc6b2c9de5 Author: Michael Armbrust <mich...@databricks.com> Date: 2015-08-25T17:22:54Z [SPARK-10198] [SQL] Turn off partition verification by default Author: Michael Armbrust <mich...@databricks.com> Closes #8404 from marmbrus/turnOffPartitionVerification. (cherry picked from commit 5c08c86bfa43462fb2ca5f7c5980ddfb44dd57f8) Signed-off-by: Michael Armbrust <mich...@databricks.com> commit 742c82ed97ed3fc60d4f17c4363c52062829ea49 Author: Yuhao Yang <hhb...@gmail.com> Date: 2015-08-25T17:54:03Z [SPARK-8531] [ML] Update ML user guide for MinMaxScaler jira: https://issues.apache.org/jira/browse/SPARK-8531 Update ML user guide for MinMaxScaler Author: Yuhao Yang <hhb...@gmail.com> Author: unknown <yuhao...@yuhaoyan-mobl1.ccr.corp.intel.com> Closes #7211 from hhbyyh/minmaxdoc. (cherry picked from commit b37f0cc1b4c064d6f09edb161250fa8b783de52a) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit c740f5dd20459b491a8c088383c19c11a76c225d Author: Feynman Liang <fli...@databricks.com> Date: 2015-08-25T18:58:47Z [SPARK-10230] [MLLIB] Rename optimizeAlpha to optimizeDocConcentration See [discussion](https://github.com/apache/spark/pull/8254#discussion_r37837770) CC jkbradley Author: Feynman Liang <fli...@databricks.com> Closes #8422 from feynmanliang/SPARK-10230. (cherry picked from commit 881208a8e849facf54166bdd69d3634407f952e7) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 5a32ed75c939dc42886ea940aba2b14b89e9f40e Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-25T19:16:23Z [SPARK-10231] [MLLIB] update @Since annotation for mllib.classification Update `Since` annotation in `mllib.classification`: 1. add version to classes, objects, constructors, and public variables declared in constructors 2. correct some versions 3. remove `Since` on `toString` MechCoder dbtsai Author: Xiangrui Meng <m...@databricks.com> Closes #8421 from mengxr/SPARK-10231 and squashes the following commits: b2dce80 [Xiangrui Meng] update @Since annotation for mllib.classification (cherry picked from commit 16a2be1a84c0a274a60c0a584faaf58b55d4942b) Signed-off-by: DB Tsai <d...@netflix.com> commit 95e44b4df81b09803be2fde8c4e2566be0c8fdbc Author: Feynman Liang <fli...@databricks.com> Date: 2015-08-25T20:21:05Z [SPARK-9800] Adds docs for GradientDescent$.runMiniBatchSGD alias * Adds doc for alias of runMIniBatchSGD documenting default value for convergeTol * Cleans up a note in code Author: Feynman Liang <fli...@databricks.com> Closes #8425 from feynmanliang/SPARK-9800. (cherry picked from commit c0e9ff1588b4d9313cc6ec6e00e5c7663eb67910) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 186326df21daf8d8271a522f2569eb5cd7be1442 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-25T20:22:38Z [SPARK-10237] [MLLIB] update since versions in mllib.fpm Same as #8421 but for `mllib.fpm`. cc feynmanliang Author: Xiangrui Meng <m...@databricks.com> Closes #8429 from mengxr/SPARK-10237. (cherry picked from commit c619c7552f22d28cfa321ce671fc9ca854dd655f) Signed-off-by: Xiangrui Meng <m...@databricks.com> commit 055387c087989c8790b6761429b68416ecee3a33 Author: Feynman Liang <fli...@databricks.com> Date: 2015-08-25T20:23:15Z [SPARK-9797] [MLLIB] [DOC] StreamingLinearRegressionWithSGD.setConvergenceTol default value Adds default convergence tolerance (0.001, set in `GradientDescent.convergenceTol`) to `setConvergenceTol`'s scaladoc Author: Feynman Liang <fli...@databricks.com> Closes #8424 from feynmanliang/SPARK-9797. (cherry picked from commit 9205907876cf65695e56c2a94bedd83df3675c03) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit 6f05b7aebd66a00e2556a29b35084e81ac526406 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-25T21:11:38Z [SPARK-10239] [SPARK-10244] [MLLIB] update since versions in mllib.pmml and mllib.util Same as #8421 but for `mllib.pmml` and `mllib.util`. cc dbtsai Author: Xiangrui Meng <m...@databricks.com> Closes #8430 from mengxr/SPARK-10239 and squashes the following commits: a189acf [Xiangrui Meng] update since versions in mllib.pmml and mllib.util (cherry picked from commit 00ae4be97f7b205432db2967ba6d506286ef2ca6) Signed-off-by: DB Tsai <d...@netflix.com> commit 8925896b1eb0a13d723d38fb263d3bec0a01ec10 Author: Davies Liu <dav...@databricks.com> Date: 2015-08-25T21:55:34Z [SPARK-10245] [SQL] Fix decimal literals with precision < scale In BigDecimal or java.math.BigDecimal, the precision could be smaller than scale, for example, BigDecimal("0.001") has precision = 1 and scale = 3. But DecimalType require that the precision should be larger than scale, so we should use the maximum of precision and scale when inferring the schema from decimal literal. Author: Davies Liu <dav...@databricks.com> Closes #8428 from davies/smaller_decimal. (cherry picked from commit ec89bd840a6862751999d612f586a962cae63f6d) Signed-off-by: Yin Huai <yh...@databricks.com> commit ab7d46d1d6e7e6705a3348a0cab2d05fe62951cf Author: Davies Liu <dav...@databricks.com> Date: 2015-08-25T22:19:41Z [SPARK-10215] [SQL] Fix precision of division (follow the rule in Hive) Follow the rule in Hive for decimal division. see https://github.com/apache/hive/blob/ac755ebe26361a4647d53db2a28500f71697b276/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDivide.java#L113 cc chenghao-intel Author: Davies Liu <dav...@databricks.com> Closes #8415 from davies/decimal_div2. (cherry picked from commit 7467b52ed07f174d93dfc4cb544dc4b69a2c2826) Signed-off-by: Yin Huai <yh...@databricks.com> commit 727771352855dbb780008c449a877f5aaa5fc27a Author: Patrick Wendell <pwend...@gmail.com> Date: 2015-08-25T22:56:37Z Preparing Spark release v1.5.0-rc2 commit 4c03cb4da846bf3ea4cd99f593d74c4a817a7d2d Author: Patrick Wendell <pwend...@gmail.com> Date: 2015-08-25T22:56:44Z Preparing development version 1.5.1-SNAPSHOT commit 5cf266fdeb6632622642e5d9bc056a76680b1970 Author: Feynman Liang <fli...@databricks.com> Date: 2015-08-26T00:39:20Z [SPARK-9888] [MLLIB] User guide for new LDA features * Adds two new sections to LDA's user guide; one for each optimizer/model * Documents new features added to LDA (e.g. topXXXperXXX, asymmetric priors, hyperpam optimization) * Cleans up a TODO and sets a default parameter in LDA code jkbradley hhbyyh Author: Feynman Liang <fli...@databricks.com> Closes #8254 from feynmanliang/SPARK-9888. (cherry picked from commit 125205cdb35530cdb4a8fff3e1ee49cf4a299583) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> commit af98e51f273d95e0fc19da1eca32a5f87a8c5576 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T01:17:54Z [SPARK-10233] [MLLIB] update since version in mllib.evaluation Same as #8421 but for `mllib.evaluation`. cc avulanov Author: Xiangrui Meng <m...@databricks.com> Closes #8423 from mengxr/SPARK-10233. (cherry picked from commit 8668ead2e7097b9591069599fbfccf67c53db659) Signed-off-by: Xiangrui Meng <m...@databricks.com> commit 46750b912781433b6ce0845ac22805cde975361e Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T03:07:56Z [SPARK-10238] [MLLIB] update since versions in mllib.linalg Same as #8421 but for `mllib.linalg`. cc dbtsai Author: Xiangrui Meng <m...@databricks.com> Closes #8440 from mengxr/SPARK-10238 and squashes the following commits: b38437e [Xiangrui Meng] update since versions in mllib.linalg (cherry picked from commit ab431f8a970b85fba34ccb506c0f8815e55c63bf) Signed-off-by: DB Tsai <d...@netflix.com> commit b7766699aef65586b0c3af96fb625efaa218d2b2 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T05:31:23Z [SPARK-10240] [SPARK-10242] [MLLIB] update since versions in mlilb.random and mllib.stat The same as #8241 but for `mllib.stat` and `mllib.random`. cc feynmanliang Author: Xiangrui Meng <m...@databricks.com> Closes #8439 from mengxr/SPARK-10242. (cherry picked from commit c3a54843c0c8a14059da4e6716c1ad45c69bbe6c) Signed-off-by: Xiangrui Meng <m...@databricks.com> commit be0c9915c0084a187933f338e51e606dc68e93af Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T05:33:48Z [SPARK-10234] [MLLIB] update since version in mllib.clustering Same as #8421 but for `mllib.clustering`. cc feynmanliang yu-iskw Author: Xiangrui Meng <m...@databricks.com> Closes #8435 from mengxr/SPARK-10234. (cherry picked from commit d703372f86d6a59383ba8569fcd9d379849cffbf) Signed-off-by: Xiangrui Meng <m...@databricks.com> commit 6d8ebc801799714d297c83be6935b37e26dc2df7 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T05:35:49Z [SPARK-10243] [MLLIB] update since versions in mllib.tree Same as #8421 but for `mllib.tree`. cc jkbradley Author: Xiangrui Meng <m...@databricks.com> Closes #8442 from mengxr/SPARK-10236. (cherry picked from commit fb7e12fe2e14af8de4c206ca8096b2e8113bfddc) Signed-off-by: Xiangrui Meng <m...@databricks.com> commit 08d390f457f80ffdc2dfce61ea579d9026047f12 Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T05:49:33Z [SPARK-10235] [MLLIB] update since versions in mllib.regression Same as #8421 but for `mllib.regression`. cc freeman-lab dbtsai Author: Xiangrui Meng <m...@databricks.com> Closes #8426 from mengxr/SPARK-10235 and squashes the following commits: 6cd28e4 [Xiangrui Meng] update since versions in mllib.regression (cherry picked from commit 4657fa1f37d41dd4c7240a960342b68c7c591f48) Signed-off-by: DB Tsai <d...@netflix.com> commit 21a10a86d20ec1a6fea42286b4d2aae9ce7e848d Author: Xiangrui Meng <m...@databricks.com> Date: 2015-08-26T06:45:41Z [SPARK-10236] [MLLIB] update since versions in mllib.feature Same as #8421 but for `mllib.feature`. cc dbtsai Author: Xiangrui Meng <m...@databricks.com> Closes #8449 from mengxr/SPARK-10236.feature and squashes the following commits: 0e8d658 [Xiangrui Meng] remove unnecessary comment ad70b03 [Xiangrui Meng] update since versions in mllib.feature (cherry picked from commit 321d7759691bed9867b1f0470f12eab2faa50aff) Signed-off-by: DB Tsai <d...@netflix.com> commit 5220db9e352b5d5eae59cead9478ca0a9f73f16b Author: felixcheung <felixcheun...@hotmail.com> Date: 2015-08-26T06:48:16Z [SPARK-9316] [SPARKR] Add support for filtering using `[` (synonym for filter / select) Add support for ``` df[df$name == "Smith", c(1,2)] df[df$age %in% c(19, 30), 1:2] ``` shivaram Author: felixcheung <felixcheun...@hotmail.com> Closes #8394 from felixcheung/rsubset. (cherry picked from commit 75d4773aa50e24972c533e8b48697fde586429eb) Signed-off-by: Shivaram Venkataraman <shiva...@cs.berkeley.edu> ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org