[GitHub] spark pull request: [SPARK-15364][ML][PySpark] Implement PySpark p...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/13219#issuecomment-221480232 @viirya I think we need a test for the picklers. See https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/mllib/api/python/PythonMLLibAPISuite.scala#L29. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12922][SparkR][WIP] Implement gapply() ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/12836#issuecomment-221478783 Lets keep it as `dapply` - The specific choice of applying on a partition as a data frame is built into its semantics. If we do build a single row UDF then we can make it match the Python / Scala UDF API or try to see if it matches `apply` in terms of semantics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478629 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221478643 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59257/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478607 **[Test build #59256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59256/consoleFull)** for PR 13284 at commit [`5ac9b9b`](https://github.com/apache/spark/commit/5ac9b9ba3094714f1d153a3f96690fa312b794ae). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221478631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59256/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221478641 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221478570 **[Test build #59257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59257/consoleFull)** for PR 12875 at commit [`218ecba`](https://github.com/apache/spark/commit/218ecba594f15fafa8c1424a04e780e4af49b9a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477942 as for subset - maybe tighter signature matching checks in R 3.3.0? I couldn't find any reference on that though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477791 so startsWith endsWith were definitely added in 3.3.0. I think we should be able to change the generic to match the signature to get it to work (have not tried it yet, since I don't have 3.3.0) ``` startsWith(x, prefix) endsWith(x, suffix) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12922][SparkR][WIP] Implement gapply() ...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/12836#issuecomment-221477758 @NarineK, mapply does not apply here, but we can later add something like lapply, which applies a function to each row in a DataFrame. I suddenly realized that it could be better to rename dapply to dapplyPartition @shivaram --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477450 Ok - so that explains one of the problems. Does anybody know what the problem in `subset` is ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221477146 **[Test build #59257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59257/consoleFull)** for PR 12875 at commit [`218ecba`](https://github.com/apache/spark/commit/218ecba594f15fafa8c1424a04e780e4af49b9a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221477072 @shivaram I will make the change with R version check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476748 Continuing from my previous message, we can't add or remove `endsWith` and `startsWith` as we want to support all R versions from 3.1.0 onwards. We could get the R version at run time (using ` R.Version()$major`) and only add them for 3.3.0 and higher --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476475 **[Test build #59256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59256/consoleFull)** for PR 13284 at commit [`5ac9b9b`](https://github.com/apache/spark/commit/5ac9b9ba3094714f1d153a3f96690fa312b794ae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13850] Force the sorter to Spill when n...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/13107#issuecomment-221476424 TimSort require a temporary buffer to store the shorter part, which could be half of the size of pointer array in worst case. This depends on the original order of rows, it's pretty hard to reproduce. I hit that twice and have a patch, but can't reproduce it anymore (without the patch). The better solution should be only use 2/3 of the pointer array, left 1/3 as temporary buffer for TimSort. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15491][SQL]fix assertion failure for JD...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13287#issuecomment-221476178 Thanks for the pull request. Can you format the description better? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476116 Hmm looks like `startsWith` and `endsWith` were added in R 3.3.0 - See http://www.r-statistics.com/2016/05/r-3-3-0-is-released/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221476011 @shivaram The pipedRDD one seems working when using sudo in Linux. My mac does not work though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12071][Doc] Document the behaviour of N...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13268 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15412][PySpark][SparkR][DOCS] Improve l...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13199 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221475769 @felixcheung > conflicts(detail = TRUE) $.GlobalEnv [1] "df" $`package:SparkR` [1] "alias" "cov" "filter""fitted" [5] "glm" "lag" "na.omit" "predict" [9] "sd""var" "window""head" [13] "str" "show" "%in%" "as.data.frame" [17] "atan2" "attach""colnames" "colnames<-" [21] "drop" "endsWith" "factorial" "ifelse" [25] "intersect" "mean" "merge" "ncol" [29] "nrow" "rank" "rbind" "sample" [33] "startsWith""subset""substr""summary" [37] "transform" "unique""with" $`package:stats` [1] "alias" "cov" "df" "filter" "fitted" "glm" "lag" [8] "na.omit" "predict" "sd" "var" "window" $`package:utils` [1] "head" "str" $`package:methods` [1] "show" "body<-""kronecker" $`package:base` [1] "%in%" "as.data.frame" "atan2" "attach" [5] "body<-""colnames" "colnames<-""drop" [9] "endsWith" "factorial" "ifelse""intersect" [13] "kronecker" "mean" "merge" "ncol" [17] "nrow" "rank" "rbind" "sample" [21] "startsWith""subset""substr""summary" [25] "transform" "unique""with" See the $`package:SparkR` above --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12071][Doc] Document the behaviour of N...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13268#issuecomment-221475713 LGTM. Merging this to master and branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15412][PySpark][SparkR][DOCS] Improve l...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13199#issuecomment-221475488 LGTM. Merging this to master and branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user wangmiao1981 commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221475397 R version 3.3.0 (2016-05-03) -- "Supposedly Educational" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-apple-darwin13.4.0 (64-bit) > installed.packages() Package SparkR "SparkR" base "base" boot "boot" class "class" cluster"cluster" codetools "codetools" compiler "compiler" crayon "crayon" datasets "datasets" digest "digest" foreign"foreign" graphics "graphics" grDevices "grDevices" grid "grid" KernSmooth "KernSmooth" lattice"lattice" magrittr "magrittr" MASS "MASS" Matrix "Matrix" memoise"memoise" methods"methods" mgcv "mgcv" nlme "nlme" nnet "nnet" parallel "parallel" praise "praise" R6 "R6" rpart "rpart" spatial"spatial" splines"splines" stats "stats" stats4 "stats4" survival "survival" tcltk "tcltk" testthat "testthat" tools "tools" utils "utils" LibPath SparkR "/Users/mwang/spark_ws_0904/R/lib" base "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" boot "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" class "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" cluster"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" codetools "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" compiler "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" crayon "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" datasets "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" digest "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" foreign"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" graphics "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" grDevices "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" grid "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" KernSmooth "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" lattice"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" magrittr "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" MASS "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" Matrix "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" memoise"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" methods"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" mgcv "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" nlme "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" nnet "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" parallel "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" praise "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" R6 "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" rpart "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" spatial"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" splines"/Library/Frameworks/R.framework/Versions/3.3/Resources/library" stats "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" stats4 "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" survival "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" tcltk "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" testthat "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" tools "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" utils "/Library/Frameworks/R.framework/Versions/3.3/Resources/library" Version Priority SparkR "2.0.0" NA base "3.3.0" "base" boot "1.3-18" "recommended" class "7.3-14" "recommended" cluster"2.0.4" "recommended" codetools "0.2-14" "recommended" compiler "3.3.0" "base" crayon "1.3.1" NA datasets "3.3.0" "base" digest "0.6.9" NA foreign"0.8-66" "recommended" graphics "3.3.0" "base" grDevices "3.3.0" "base" grid
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221474796 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221474798 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59255/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221474776 **[Test build #59255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59255/consoleFull)** for PR 12875 at commit [`39e5361`](https://github.com/apache/spark/commit/39e5361215a1107d954faecd0c15ab16ad23a580). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15508][Streaming][Tests]Fix flaky test:...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13281 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15008][ML][PySpark] Add integration tes...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12875#issuecomment-221473671 **[Test build #59255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59255/consoleFull)** for PR 12875 at commit [`39e5361`](https://github.com/apache/spark/commit/39e5361215a1107d954faecd0c15ab16ad23a580). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15508][Streaming][Tests]Fix flaky test:...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/13281#issuecomment-221473561 Thanks for reviewing. Merging to master / 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13272#issuecomment-221473050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59252/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13272#issuecomment-221473047 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13272#issuecomment-221472935 **[Test build #59252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59252/consoleFull)** for PR 13272 at commit [`636658e`](https://github.com/apache/spark/commit/636658ee19a581598c421432c8f5961b45f954fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14998][SQL]fix ArrayIndexOutOfBoundsExc...
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12772#issuecomment-221470057
Sorry, I am unable to reproduce it. Without the fix, the following test
case works well.
```scala
val data = Seq(("A\tB\tC\tD\t\t"), ("E\tF\tD\tE\t"))
data.toDF("d1").createOrReplaceTempView("script_trans")
checkAnswer(
sql("SELECT TRANSFORM (d1) USING 'cat' AS (a, b, c, d) FROM
script_trans"),
Row("A", "B", "C", "D") :: Row("E", "F", "D", "E") :: Nil)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513464
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
--- End diff --
I think we should keep the ..., like this
`x[, select, drop = drop, ...]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15498][TESTS] fix slow tests
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13273 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513468
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
+} else {
+x[subset, select, drop = drop]
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13284#discussion_r64513429
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1445,7 +1445,11 @@ setMethod("[", signature(x = "SparkDataFrame"),
#' }
setMethod("subset", signature(x = "SparkDataFrame"),
function(x, subset, select, drop = F, ...) {
-x[subset, select, drop = drop]
+if (missing(subset)) {
+x[, select, drop = drop]
+} else {
--- End diff --
could you pls fix the extra space: `else {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15498][TESTS] fix slow tests
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/13273#issuecomment-221469762 Merging to master and branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221469622 I'm seeing the error with `subset` as well. As for "endsWith", "startsWith" - could you check what you have in your environment - you could run `conflicts(detail = TRUE)` - AFAIK I'm not seeing this. Which version of R do you have? And package versions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15493][SQL] Allow setting the quoteEsca...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13267#issuecomment-221469255 Yea I agree with escapeQuotes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15493][SQL] Allow setting the quoteEsca...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13267#issuecomment-221469276 @jurriaan want to do the change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15493][SQL] Allow setting the quoteEsca...
Github user falaki commented on the pull request: https://github.com/apache/spark/pull/13267#issuecomment-221468338 @rxin and @jurriaan I agree to keep it set by default. However, I think it is better to leave it configurable. In two cases before, I assumed a reasonable default value is good enough, but ended up exposing them in options. Also, I suggest a simpler name like `escapeQuotes` or `enableQuoteEscaping`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15365] [SQL]: When table size statistic...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13150 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15365] [SQL]: When table size statistic...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13150#issuecomment-221467220 Merging in master/2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15365] [SQL]: When table size statistic...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13150#issuecomment-221467315 @Parth-Brahmbhatt you should add the email address you used in your commit to your github profile, so the commit is associated with your account. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518] Rename various scheduler backend...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13288 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518] Rename various scheduler backend...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13288#issuecomment-221466892 Thanks - merging in master/2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-10903] [SPARKR] R - Simplify SQLC...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221466855 Thanks - look forward to it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466629 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59249/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466626 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466522 **[Test build #59249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59249/consoleFull)** for PR 13283 at commit [`f6a4517`](https://github.com/apache/spark/commit/f6a45177609c3c2ab7cd376dd6eb1a4527e693e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-10903] [SPARKR] R - Simplify SQLC...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9192#issuecomment-221466080 @rxin - [ ] fix failed tests (hiveContext related) - [ ] check on roxygen2 generated doc - [ ] update code examples in doc to remove sqlContext (I'd like that to be a separate PR - less conflicts) As for SparkSession, perhaps that could be a different PR, I think that should be fairly independent to this and much more isolated. I still need to wrap my head around what API would make sense in R - any suggestions would be greatly appreciated! Sorry this is taking longer. I should be able to get the checklist done tonight. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466099 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59254/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221466000 **[Test build #59254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59254/consoleFull)** for PR 13283 at commit [`c313ffc`](https://github.com/apache/spark/commit/c313ffc88488229c45e74cde97210c8686e1c745). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221465283 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59250/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221465280 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221465200 **[Test build #59250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59250/consoleFull)** for PR 13271 at commit [`8d507bf`](https://github.com/apache/spark/commit/8d507bf6bb29e1e7f11e1776e9ea51d1387a39d7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15517][SQL][STREAMING] Add support for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13286#issuecomment-221464878 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15517][SQL][STREAMING] Add support for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13286#issuecomment-221464882 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59253/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15517][SQL][STREAMING] Add support for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13286#issuecomment-221464810 **[Test build #59253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59253/consoleFull)** for PR 13286 at commit [`074299c`](https://github.com/apache/spark/commit/074299ca9bf04a3b14d9c54ba7fc2cd2b4bce94b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/12943#discussion_r64510350 --- Diff: python/pyspark/ml/classification.py --- @@ -1063,6 +1064,13 @@ class MultilayerPerceptronClassifier(JavaEstimator, HasFeaturesCol, HasLabelCol, True >>> model.weights == model2.weights True +>>> mlp2 = mlp2.setInitialWeights([2, 5, 1, -7, -5, -10, 0, 0.6, -1, 2, -2, 1, 2, -7, -1, -2, +...2, 1, -1, 9, -9, 3, -3, -3, 3.0, 0, -1]) --- End diff -- We could, but this way its matching the tests in Scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518][SQL] Rename various scheduler ba...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/13288#issuecomment-221463656 nit: [SQL] -> [Core] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15365] [SQL]: When table size statistic...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13150#issuecomment-221463522 **[Test build #3017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3017/consoleFull)** for PR 13150 at commit [`ff69f91`](https://github.com/apache/spark/commit/ff69f91e273580547b6d86a6dfc87f2a94066507). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/13284#issuecomment-221462859 @wangmiao1981 Thanks for investigating this. Do you know why these are not failing in Jenkins though ? (the subset test and the pipedRDD one) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15345][SQL][PYSPARK]. SparkSession's co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13160#issuecomment-221462569 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15345][SQL][PYSPARK]. SparkSession's co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13160#issuecomment-221462571 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59243/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15345][SQL][PYSPARK]. SparkSession's co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13160#issuecomment-221462448 **[Test build #59243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59243/consoleFull)** for PR 13160 at commit [`071839f`](https://github.com/apache/spark/commit/071839f7255c9136d0b094f38cd0e98bf6f23a53). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518][SQL] Rename various scheduler ba...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13288#issuecomment-221462328 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518][SQL] Rename various scheduler ba...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13288#issuecomment-221462330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59245/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15518][SQL] Rename various scheduler ba...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13288#issuecomment-221462207 **[Test build #59245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59245/consoleFull)** for PR 13288 at commit [`01f5f96`](https://github.com/apache/spark/commit/01f5f96735012e067df6174784925afbf6806100). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221462008 Thank you, @rxin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221460289 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59244/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221460288 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221460191 **[Test build #59244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59244/consoleFull)** for PR 13271 at commit [`b43ac76`](https://github.com/apache/spark/commit/b43ac765eda23eaf58deb11889a32f33280236f8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13484][SQL] Prevent illegal NULL propag...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/11371#discussion_r64507505
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1443,6 +1445,32 @@ class Analyzer(
}
/**
+ * Corrects attribute references in an expression tree of some operators
(e.g., filters and
+ * projects) if these operators have a join as a child and the
references point to columns on the
+ * input relation of the join. This is because some joins change the
nullability of input columns
+ * and this could cause illegal optimization (e.g., NULL propagation)
and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object SolveIllegalReferences extends Rule[LogicalPlan] {
+
+private def replaceReferences(e: Expression, attrMap:
AttributeMap[Attribute]) = e.transform {
+ case a: AttributeReference => attrMap.get(a).getOrElse(a)
+}
+
+def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case q: LogicalPlan =>
+q.transform {
+ case f @ Filter(filterCondition,
ExtractJoinOutputAttributes(join, joinOutputMap)) =>
+f.copy(condition = replaceReferences(filterCondition,
joinOutputMap))
--- End diff --
okay, I wait your ping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15388][SQL] Fix spark sql CREATE FUNCTI...
Github user wangyang1992 commented on the pull request: https://github.com/apache/spark/pull/13177#issuecomment-221457800 Thanks @rxin . Added it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9876][SQL]: Update Parquet to 1.8.1.
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/13280#issuecomment-221457588 I had once tried to upgrade Parquet to 1.8.1, and one more change needs to be done for the upgrade: https://github.com/apache/spark/pull/9225/files#diff-b4108187503e0f3ac64c1630d266b122R115 For the performance regression, here is the full thread of previous discussion: https://mail-archives.apache.org/mod_mbox/parquet-dev/201510.mbox/<56169ffb.9060...@gmail.com> I had tried to bisect, but failed to find anything useful. I thought 1.8.2 would probably be release soon at that time, so didn't try hard to dig it... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221457452 Just realized this PR is related to https://github.com/apache/spark/pull/13201. Also tried to improve the exception reporting in Python through hiding the long java stack trace. @zsxwing @marmbrus Could you please help me review the changes? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221457242 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221457208 **[Test build #59254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59254/consoleFull)** for PR 13283 at commit [`c313ffc`](https://github.com/apache/spark/commit/c313ffc88488229c45e74cde97210c8686e1c745). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221457243 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59241/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15517][SQL][STREAMING] Add support for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13286#issuecomment-221457219 **[Test build #59253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59253/consoleFull)** for PR 13286 at commit [`074299c`](https://github.com/apache/spark/commit/074299ca9bf04a3b14d9c54ba7fc2cd2b4bce94b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13271#issuecomment-221457151 **[Test build #59241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59241/consoleFull)** for PR 13271 at commit [`d188179`](https://github.com/apache/spark/commit/d188179d739a6a3a44db3ec3c4ae22269358ad45). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13484][SQL] Prevent illegal NULL propag...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11371#discussion_r64506496
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1443,6 +1445,32 @@ class Analyzer(
}
/**
+ * Corrects attribute references in an expression tree of some operators
(e.g., filters and
+ * projects) if these operators have a join as a child and the
references point to columns on the
+ * input relation of the join. This is because some joins change the
nullability of input columns
+ * and this could cause illegal optimization (e.g., NULL propagation)
and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object SolveIllegalReferences extends Rule[LogicalPlan] {
+
+private def replaceReferences(e: Expression, attrMap:
AttributeMap[Attribute]) = e.transform {
+ case a: AttributeReference => attrMap.get(a).getOrElse(a)
+}
+
+def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case q: LogicalPlan =>
+q.transform {
+ case f @ Filter(filterCondition,
ExtractJoinOutputAttributes(join, joinOutputMap)) =>
+f.copy(condition = replaceReferences(filterCondition,
joinOutputMap))
--- End diff --
How about we use a `q.transformUp` to fix the nullability in a bottom-up
way? For every node, we create an `AttributeMap` using the output of its child.
Then, we use `transformExpressions` to fix the nullability if necessary. Let
me try it out and ping you when I have a version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15269][SQL] Removes unexpected empty ta...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13270#issuecomment-221456715 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59246/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15269][SQL] Removes unexpected empty ta...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13270#issuecomment-221456714 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Log warnings for numIterations * miniBatchFrac...
Github user Hydrotoast commented on a diff in the pull request:
https://github.com/apache/spark/pull/13265#discussion_r64506341
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala
---
@@ -197,6 +197,11 @@ object GradientDescent extends Logging {
"< 1.0 can be unstable because of the stochasticity in sampling.")
}
+if (numIterations * miniBatchFraction < 1.0) {
+ logWarning("Not all examples will be used if numIterations *
miniBatchFraction " +
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15269][SQL] Removes unexpected empty ta...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13270#issuecomment-221456607 **[Test build #59246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59246/consoleFull)** for PR 13270 at commit [`193e005`](https://github.com/apache/spark/commit/193e0059be83006ddcce30a56c2b6ae09a2cac31). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221456373 **[Test build #59251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59251/consoleFull)** for PR 13283 at commit [`26d72ae`](https://github.com/apache/spark/commit/26d72ae697b57fb8ad1a4f0357af0b726a76e443). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ClassNotFoundException(CapturedException):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221456376 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221456377 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59251/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13283#issuecomment-221455992 **[Test build #59251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59251/consoleFull)** for PR 13283 at commit [`26d72ae`](https://github.com/apache/spark/commit/26d72ae697b57fb8ad1a4f0357af0b726a76e443). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13272#issuecomment-221455988 **[Test build #59252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59252/consoleFull)** for PR 13272 at commit [`636658e`](https://github.com/apache/spark/commit/636658ee19a581598c421432c8f5961b45f954fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15474][SQL]ORC data source fails to wri...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/13257#issuecomment-221455213 @sbcd90 I currently can't think of other alternatives and it seems that's why it has not been enabled again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221454949 **[Test build #59248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59248/consoleFull)** for PR 13289 at commit [`d972a4a`](https://github.com/apache/spark/commit/d972a4ae90d5736c912a18aad9c1ce9543835b7e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221455015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59248/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
