[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70617197 [Test build #25808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25808/consoleFull) for PR 4015 at commit [`1c6edfa`](https://github.com/apache/spark/commit/1c6edfa759ee3d19fefd85e72193efefb123203b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class SQLDialect ` * `class DefaultSQLDialect extends SQLDialect ` * `class HiveQLDialect extends SQLDialect ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4114#issuecomment-70617182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25793/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4945] [SQL] Add overwrite option suppor...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3780#discussion_r23207087
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala ---
@@ -151,15 +151,20 @@ private[sql] object ParquetRelation {
*
* @param pathString The directory the Parquetfile will be stored in.
* @param attributes The schema of the relation.
+ * @param overwrite Overwrite the existed file path,
+ * If it's false, an exception will raise if the
path already existed,
--- End diff --
the alignment is off here. Just add a space after overwrite, and don't do
vertical alignment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4114#issuecomment-70617173 [Test build #25793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25793/consoleFull) for PR 4114 at commit [`977c103`](https://github.com/apache/spark/commit/977c10327c377d11830efdda70fffc10c9782fb4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4955]With executor dynamic scaling enab...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3962#issuecomment-70616605 [Test build #25809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25809/consoleFull) for PR 3962 at commit [`265c36d`](https://github.com/apache/spark/commit/265c36d3c6c0feccf6db6298e99021f2d14a4060). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5009] [SQL] Long keyword support in SQL...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3926#issuecomment-70616591 [Test build #25810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25810/consoleFull) for PR 3926 at commit [`686660f`](https://github.com/apache/spark/commit/686660f7a4d34228a244ac886b89410b5cbbe35e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4945] [SQL] Add overwrite option suppor...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3780#discussion_r23206838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala ---
@@ -179,9 +184,24 @@ private[sql] object ParquetRelation {
throw new IllegalArgumentException(
s"Unable to create ParquetRelation: incorrectly formatted path
$pathStr")
}
+
val path = origPath.makeQualified(fs)
-if (!allowExisting && fs.exists(path)) {
- sys.error(s"File $pathStr already exists.")
+val pathExisted = fs.exists(path)
+
+if (pathExisted) {
+ if (overwrite) {
+try {
+ fs.delete(path, true)
+} catch {
+ case e: IOException =>
+throw new IOException(
+ s"Unable to clear output directory ${path}")
--- End diff --
put this on the previous line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5009] [SQL] Long keyword support in SQL...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/3926#issuecomment-70616374 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70616282 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25806/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70616276 [Test build #25806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25806/consoleFull) for PR 4115 at commit [`c2a`](https://github.com/apache/spark/commit/c2a17d0b8ab98266da0b1313c3f250f150fa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Row extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4107#issuecomment-70616194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25796/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4107#issuecomment-70616187 [Test build #25796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25796/consoleFull) for PR 4107 at commit [`fce0849`](https://github.com/apache/spark/commit/fce084931ddc087e2c4e0a9a4a591a274a26ffdc). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` class FileSinkDescWrapper(val dirName: String, var tableInfo: TableDesc, var compressed: Boolean)` * ` trait ReflectedCall[T] ` * ` case class Construct[T](clazz: Class[T], args: (Class[_], AnyRef)*) extends ReflectedCall[T] ` * ` case class Invoke[U](clazz: Class[_], obj: AnyRef, methodName: String, args: (Class[_], AnyRef)*)` * ` case class InvokeStatic[U](clazz: Class[_], methodName: String, args: (Class[_], AnyRef)*)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70616103 [Test build #25808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25808/consoleFull) for PR 4015 at commit [`1c6edfa`](https://github.com/apache/spark/commit/1c6edfa759ee3d19fefd85e72193efefb123203b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70615542 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25807/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70615541 [Test build #25807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25807/consoleFull) for PR 4015 at commit [`d958589`](https://github.com/apache/spark/commit/d9585890d3c8c25c104a4c936fa49700ba1b3fbf). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class SQLDialect ` * `class DefaultSQLDialect extends SQLDialect ` * `class HiveQLDialect extends SQLDialect ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4687. [WIP] Add an addDirectory API
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3670#issuecomment-70615279 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25795/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4687. [WIP] Add an addDirectory API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3670#issuecomment-70615275 [Test build #25795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25795/consoleFull) for PR 3670 at commit [`3d7af57`](https://github.com/apache/spark/commit/3d7af57c5a412f21849e58181165aea6cd0c7628). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] Add a log4j file for catalyst tes...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4117#issuecomment-70615203 [Test build #25805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25805/consoleFull) for PR 4117 at commit [`8ad610b`](https://github.com/apache/spark/commit/8ad610b139b1069d29e3ddcf203c248f07d033de). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70615178 [Test build #25806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25806/consoleFull) for PR 4115 at commit [`c2a`](https://github.com/apache/spark/commit/c2a17d0b8ab98266da0b1313c3f250f150fa). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70615213 [Test build #25807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25807/consoleFull) for PR 4015 at commit [`d958589`](https://github.com/apache/spark/commit/d9585890d3c8c25c104a4c936fa49700ba1b3fbf). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] Add a log4j file for catalyst tes...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/4117 [SQL][minor] Add a log4j file for catalyst test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark catalyst-test-log4j Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4117 commit 8ad610b139b1069d29e3ddcf203c248f07d033de Author: Reynold Xin Date: 2015-01-20T07:36:14Z [SQL][minor] Add a log4j file for catalyst test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an Arti...
Github user lovmoy commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-70615067 Hi, @bgreeven What is the active function of the hidden layer? Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [Minor] Remove deprecated parquet tests
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4116#issuecomment-70614638 [Test build #25804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25804/consoleFull) for PR 4116 at commit [`abd7033`](https://github.com/apache/spark/commit/abd7033dd0ddce5ac7075efb5ee431f1bec9bb9f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70614537 [Test build #25803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25803/consoleFull) for PR 4015 at commit [`983d53c`](https://github.com/apache/spark/commit/983d53cac47315bdf66a58b440391fb300afaefc). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70614539 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25803/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4585. Spark dynamic executor allocation ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4051#issuecomment-70614222 [Test build #25802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25802/consoleFull) for PR 4051 at commit [`d1dd039`](https://github.com/apache/spark/commit/d1dd039c1807cab11f998b76b698d19e2565ec53). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5213] [SQL] [WIP] Sql Parser dialect su...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4015#issuecomment-70614230 [Test build #25803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25803/consoleFull) for PR 4015 at commit [`983d53c`](https://github.com/apache/spark/commit/983d53cac47315bdf66a58b440391fb300afaefc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [Minor] Remove deprecated parquet tests
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4116#issuecomment-70614232 [Test build #25801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25801/consoleFull) for PR 4116 at commit [`20a7f9b`](https://github.com/apache/spark/commit/20a7f9b1cf8300706e31f2209b94c19e09b36e42). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] [Minor] Remove deprecated parquet tests
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/4116 [SQL] [Minor] Remove deprecated parquet tests This PR removes the deprecated `ParquetQuerySuite`, renamed `ParquetQuerySuite2` to `ParquetQuerySuite`, and fixed several typos. It is a follow-up of #3644. Notice that test cases in the old `ParquetQuerySuite` have already been well covered by other test suites introduced in #3644. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark remove-deprecated-parquet-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4116.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4116 commit 687d4fed83821b4a2c1a4b73c8f9787c8036789c Author: Cheng Lian Date: 2015-01-20T00:02:58Z Removes the deprecated ParquetQuerySuite commit 20a7f9b1cf8300706e31f2209b94c19e09b36e42 Author: Cheng Lian Date: 2015-01-20T07:20:44Z Removes the deprecated old ParquetQuerySuite Renames ParquetQuerySuite2 to ParquetQuerySuite, and fixed several typos. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4585. Spark dynamic executor allocation ...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4051#issuecomment-70613708 I updated the patch to add a `spark.dynamicAllocation.initialExecutors` property. I also removed the requirement to set min/maxExecutors, so the user now only needs to configure any numbers if they want to tune for performance. The min now defaults to 0 and the max to Integer.MAX_VALUE. I looked through the code and it seems like this should be safe, but @andrewor14 let me know if there are reasons I missed why these needed to be required. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4113#issuecomment-70613325 [Test build #25794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25794/consoleFull) for PR 4113 at commit [`a5e84ca`](https://github.com/apache/spark/commit/a5e84ca9f493569daaf359c95f1872295592a30e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][Minor] Refactors deeply nested FP style ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4091#issuecomment-70613374 Hey @scwf, would you mind to take a look at this? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4113#issuecomment-70613334 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25794/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4107#issuecomment-70612792 [Test build #25800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25800/consoleFull) for PR 4107 at commit [`cb2edfb`](https://github.com/apache/spark/commit/cb2edfb4abd70aac289abbb9d713cbe51da821c2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70612751 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25798/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70612747 [Test build #25798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25798/consoleFull) for PR 4115 at commit [`cd6d46a`](https://github.com/apache/spark/commit/cd6d46a1e33040b3d6eb62b15c4cc38bb434bcd0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Row extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5315][Streaming] Fix reduceByWindow Jav...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/4104#discussion_r23205557
--- Diff: project/MimaExcludes.scala ---
@@ -82,6 +82,10 @@ object MimaExcludes {
// SPARK-5166 Spark SQL API stabilization
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.Transformer.transform"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.Estimator.fit")
+ ) ++ Seq(
+// SPARK-5315 Spark Streaming Java API returns Scala DStream
+ProblemFilters.exclude[MissingMethodProblem](
+
"org.apache.spark.streaming.api.java.JavaDStreamLike.reduceByWindow")
--- End diff --
Hi Patrick, Mima test complains about this, so I added it here.
Besides, what's your opinion about deprecating the old API or modifying it
if this API is actually not correct? I'm not sure which is the good way, but
@srowen suggested me to deprecate rather than modify.
Thanks a lot :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70612122 [Test build #25797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25797/consoleFull) for PR 4115 at commit [`5c6fa9c`](https://github.com/apache/spark/commit/5c6fa9c59dfc8a5e7d41c91ebbf4eb9707e9b00e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70612126 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25797/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70612117 [Test build #25798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25798/consoleFull) for PR 4115 at commit [`cd6d46a`](https://github.com/apache/spark/commit/cd6d46a1e33040b3d6eb62b15c4cc38bb434bcd0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5321] Support for transposing local mat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4109#issuecomment-70612131 [Test build #25799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25799/consoleFull) for PR 4109 at commit [`a01bd5f`](https://github.com/apache/spark/commit/a01bd5f75bcdea21d98541299afcf04bc4f26d14). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4107#discussion_r23205342
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveCompat.scala ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive
+
+import java.math.{BigDecimal => JBigDecimal}
+import java.util.{Properties, Set => JSet}
+
+import scala.collection.JavaConversions._
+import scala.language.{existentials, implicitConversions}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hive.common.`type`.HiveDecimal
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.plan.{FileSinkDesc, TableDesc}
+import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
+import org.apache.hadoop.hive.serde2.{ColumnProjectionUtils, Deserializer}
+import org.apache.hadoop.mapred.InputFormat
+
+import org.apache.spark.Logging
+
+/**
+ * A utility object used to cope with Hive compatibility issues.
+ */
+object HiveCompat {
+ def createDefaultDBIfNeeded(context: HiveContext) = {
+context.runSqlHive("CREATE DATABASE IF NOT EXISTS default")
+context.runSqlHive("USE default")
+ }
+
+ def newTableDesc(
+ serdeClass: Class[_ <: Deserializer],
+ inputFormatClass: Class[_ <: InputFormat[_, _]],
+ outputFormatClass: Class[_],
+ properties: Properties) = callWithAlternatives(
+// For Hive 0.13.1
+Construct(classOf[TableDesc],
+ classOf[Class[_ <: InputFormat[_, _]]] -> inputFormatClass,
+ classOf[Class[_]] -> outputFormatClass,
+ classOf[Properties] -> properties),
+
+// For Hive 0.12.0
+Construct(classOf[TableDesc],
+ classOf[Class[_ <: Deserializer]] -> serdeClass,
+ classOf[Class[_ <: InputFormat[_, _]]] -> inputFormatClass,
+ classOf[Class[_]] -> outputFormatClass,
+ classOf[Properties] -> properties))
+
+ def getCommandProcessor(cmd: Array[String], conf: HiveConf) = {
+callWithAlternatives[CommandProcessor](
+ // For Hive 0.13.1
+ InvokeStatic[CommandProcessor](classOf[CommandProcessorFactory],
"get",
+classOf[Array[String]] -> cmd,
+classOf[HiveConf] -> conf),
+
+ // For Hive 0.12.0
+ InvokeStatic[CommandProcessor](classOf[CommandProcessorFactory],
"get",
+classOf[String] -> cmd(0),
+classOf[HiveConf] -> conf))
+ }
+
+ def getAllPartitionsOf(client: Hive, tbl: Table) =
callWithAlternatives[JSet[Partition]](
+// For Hive 0.13.1
+Invoke[JSet[Partition]](classOf[Hive], client, "getAllPartitionsOf",
+ classOf[Table] -> tbl),
+
+// For Hive 0.12.0
+Invoke[JSet[Partition]](classOf[Hive], client,
"getAllPartitionsForPruner",
+ classOf[Table] -> tbl)
+ )
+
+ def newHiveDecimal(bd: JBigDecimal) = callWithAlternatives(
--- End diff --
I probably shouldn't reimplement this one via reflection, as it can be
performance sensitive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5315][Streaming] Fix reduceByWindow Jav...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/4104#discussion_r23205253
--- Diff: project/MimaExcludes.scala ---
@@ -82,6 +82,10 @@ object MimaExcludes {
// SPARK-5166 Spark SQL API stabilization
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.Transformer.transform"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.Estimator.fit")
+ ) ++ Seq(
+// SPARK-5315 Spark Streaming Java API returns Scala DStream
+ProblemFilters.exclude[MissingMethodProblem](
+
"org.apache.spark.streaming.api.java.JavaDStreamLike.reduceByWindow")
--- End diff --
Is this needed since you didn't remove the old API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5270 [CORE] Elegantly check if RDD is em...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4074 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5270 [CORE] Elegantly check if RDD is em...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4074#issuecomment-70611752 @srowen Thanks Sean, I committed this with a minor re-word of the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5297][Streaming] Fix Java file stream t...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4101#issuecomment-70611624 This LGTM in the current form, but awaiting feedback from @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70611539 [Test build #25797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25797/consoleFull) for PR 4115 at commit [`5c6fa9c`](https://github.com/apache/spark/commit/5c6fa9c59dfc8a5e7d41c91ebbf4eb9707e9b00e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4115#issuecomment-70611495 cc @yhuai @marmbrus @liancheng you should probably all take a look at this since it changes so many things. Our test cases are really hard to change :( --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5323][SQL] Remove Row's Seq inheritance...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/4115 [SPARK-5323][SQL] Remove Row's Seq inheritance. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark row-seq Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4115.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4115 commit 5c6fa9c59dfc8a5e7d41c91ebbf4eb9707e9b00e Author: Reynold Xin Date: 2015-01-20T06:45:18Z [SPARK-5323][SQL] Remove Row's Seq inheritance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5297][Streaming] Fix Java file stream t...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4101#issuecomment-70611362 So on this one, it took me a minute to figure out why this is breaking. The issue is that our workaround of casting to classtag[Any] doesn't work because we actually do need to pass a specific class when the `NewHadoopRDD` is created and down the chain Spark grabs this from the ClassTag: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L765 In terms of removing this, from what I can tell, this definitely would immediately fail for anyone who wanted to use this at runtime since it will always assume `Object` is the input format. So it seems fine to just remove the old method. Would you be okay with removing it @srowen? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5325] [SQL] Shrink the Hive shim layer
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4107#issuecomment-70611196 [Test build #25796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25796/consoleFull) for PR 4107 at commit [`fce0849`](https://github.com/apache/spark/commit/fce084931ddc087e2c4e0a9a4a591a274a26ffdc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4585. Spark dynamic executor allocation ...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/4051#discussion_r23205045
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -73,12 +73,12 @@ private[spark] class ClientArguments(args:
Array[String], sparkConf: SparkConf)
.orNull
// If dynamic allocation is enabled, start at the max number of
executors
if (isDynamicAllocationEnabled) {
- val maxExecutorsConf = "spark.dynamicAllocation.maxExecutors"
- if (!sparkConf.contains(maxExecutorsConf)) {
+ val minExecutorsConf = "spark.dynamicAllocation.maxExecutors"
+ if (!sparkConf.contains(minExecutorsConf)) {
--- End diff --
I think the rationale may have been that the `ExecutorAllocationManager`
and client can run in different processes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4687. [WIP] Add an addDirectory API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3670#issuecomment-70610559 [Test build #25795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25795/consoleFull) for PR 3670 at commit [`3d7af57`](https://github.com/apache/spark/commit/3d7af57c5a412f21849e58181165aea6cd0c7628). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/4110#issuecomment-70610348 Thanks @kayousterhout ! I left a minor comment. Functionality looks fine to me, but unfortunately I am not very familiar with the UI code. Maybe @andrewor14 could take a look ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/4110#discussion_r23204735
--- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala ---
@@ -24,8 +24,10 @@ private[spark] object ToolTips {
scheduler delay is large, consider decreasing the size of tasks or
decreasing the size
of task results."""
- val TASK_DESERIALIZATION_TIME =
-"""Time spent deserializating the task closure on the executor."""
+ val TASK_DESERIALIZATION_TIME = "Time spent deserializing the task
closure on the executor."
+
+ val SHUFFLE_READ_BLOCKED_TIME =
+"Time that the task spent blocked waiting for shuffle data to be read."
--- End diff --
Might want to say something about remote wait -- i.e. "blocked for data to
be read from other machines" ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL]narrow the scope of synchroni...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4001#issuecomment-70609080 @baishuo Would you mind closing this PR? The `Driver.plan` field is not properly synchronized; otherwise we wouldn't need the synchronization at all. And other parts of `Driver` are not obviously thread-safe. Considering DDL operations is infrequent, synchronize the whole function is both safe and performance insensitive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4113#issuecomment-70608970 [Test build #25794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25794/consoleFull) for PR 4113 at commit [`a5e84ca`](https://github.com/apache/spark/commit/a5e84ca9f493569daaf359c95f1872295592a30e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4114#issuecomment-70608966 [Test build #25793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25793/consoleFull) for PR 4114 at commit [`977c103`](https://github.com/apache/spark/commit/977c10327c377d11830efdda70fffc10c9782fb4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct default classloader in...
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/3840#issuecomment-70608810 @pwendell here are pull requests for 1.2 and master branches respectively: https://github.com/apache/spark/pull/4114 https://github.com/apache/spark/pull/4113 @pkolaczk is on vacation so he asked me to prepare these PRs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
GitHub user jacek-lewandowski opened a pull request: https://github.com/apache/spark/pull/4114 SPARK-4660: Use correct class loader in JavaSerializer (copy of PR #3840... ... by Piotr Kolaczkowski) You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacek-lewandowski/spark SPARK-4660-1.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4114 commit 977c10327c377d11830efdda70fffc10c9782fb4 Author: Jacek Lewandowski Date: 2015-01-20T05:50:53Z SPARK-4660: Use correct class loader in JavaSerializer (copy of PR #3840 by Piotr Kolaczkowski) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4660: Use correct class loader in JavaSe...
GitHub user jacek-lewandowski opened a pull request: https://github.com/apache/spark/pull/4113 SPARK-4660: Use correct class loader in JavaSerializer (copy of PR #3840... ... by Piotr Kolaczkowski) You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacek-lewandowski/spark SPARK-4660-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4113 commit a5e84ca9f493569daaf359c95f1872295592a30e Author: Jacek Lewandowski Date: 2015-01-20T05:37:16Z SPARK-4660: Use correct class loader in JavaSerializer (copy of PR #3840 by Piotr Kolaczkowski) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3288] All fields in TaskMetrics should ...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4020#issuecomment-70607997 @pwendell sorry, was out for the weekend, but this LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5100][SQL] add thriftserver-ui support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3946#issuecomment-70607861 [Test build #25792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25792/consoleFull) for PR 3946 at commit [`32332cb`](https://github.com/apache/spark/commit/32332cb7830fbf108c6697fb66929a28ee914b3a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5100][SQL] add thriftserver-ui support
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3946#issuecomment-70607865 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25792/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5019 [MLlib] - GaussianMixtureModel expo...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/4088#issuecomment-70607809 @tgaloppo Thank you! LGTM CC: @mengxr --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70607289 [Test build #25791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25791/consoleFull) for PR 4112 at commit [`bcc0fa9`](https://github.com/apache/spark/commit/bcc0fa99a7620ca03e7042ef14faa61856059a94). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70607296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25791/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4585. Spark dynamic executor allocation ...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4051#issuecomment-70606552 when we set initial number to min, there is a delay to request more executors. because in ExecutorAllocationManager's addExecutors numExecutorsToAdd starts from 1, and next minute is double. so when we request more executors, there is very slow and it spends several minutes.so maybe we need to change this scaling mechanism? another situation is: when stage has skew task and individual tasks need to run for a long time, number of executor will be to a small value, and then next stage need many executors to run tasks, so ExecutorAllocationManager need to request many executors. at this time it also need to spend a few minutes for reaching number of pending tasks. @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5321] Support for transposing local mat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4109#issuecomment-70605249 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25790/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5321] Support for transposing local mat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4109#issuecomment-70605244 [Test build #25790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25790/consoleFull) for PR 4109 at commit [`2a63593`](https://github.com/apache/spark/commit/2a63593bc8fb375af1676c2ba4c2fa6aa6680e32). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5100][SQL] add thriftserver-ui support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3946#issuecomment-70603912 [Test build #25792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25792/consoleFull) for PR 3946 at commit [`32332cb`](https://github.com/apache/spark/commit/32332cb7830fbf108c6697fb66929a28ee914b3a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4226][SQL] SparkSQL - Add support for s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3249#issuecomment-70603746 [Test build #25788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25788/consoleFull) for PR 3249 at commit [`0347f02`](https://github.com/apache/spark/commit/0347f027cef940d987cd079357059c7695e9400f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SubqueryExpression(subquery: LogicalPlan) extends Expression ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4226][SQL] SparkSQL - Add support for s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3249#issuecomment-70603747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25788/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5190] Allow SparkListeners to be regist...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4111#issuecomment-70603505 [Test build #25787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25787/consoleFull) for PR 4111 at commit [`25988f3`](https://github.com/apache/spark/commit/25988f377fc0cbeadbddd630f9564697b07ef877). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkContext(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5190] Allow SparkListeners to be regist...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4111#issuecomment-70603508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25787/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70603239 [Test build #25791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25791/consoleFull) for PR 4112 at commit [`bcc0fa9`](https://github.com/apache/spark/commit/bcc0fa99a7620ca03e7042ef14faa61856059a94). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5326] Show fetch wait time as optional ...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4110#issuecomment-70602347 I think this is very useful, will it backport to branch-1.2? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70601141 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25789/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70601139 [Test build #25789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25789/consoleFull) for PR 4112 at commit [`86a3b95`](https://github.com/apache/spark/commit/86a3b9516b33eac98241db1e0c8e66d286f5c0ad). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5321] Support for transposing local mat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4109#issuecomment-70601090 [Test build #25790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25790/consoleFull) for PR 4109 at commit [`2a63593`](https://github.com/apache/spark/commit/2a63593bc8fb375af1676c2ba4c2fa6aa6680e32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4112#issuecomment-70601086 [Test build #25789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25789/consoleFull) for PR 4112 at commit [`86a3b95`](https://github.com/apache/spark/commit/86a3b9516b33eac98241db1e0c8e66d286f5c0ad). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5329][WebUI] UIWorkloadGenerator should...
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/4112 [SPARK-5329][WebUI] UIWorkloadGenerator should stop SparkContext. UIWorkloadGenerator don't stop SparkContext. I ran UIWorkloadGenerator and try to watch the result at WebUI but Jobs are marked as finished. It's because SparkContext is not stopped. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark SPARK-5329 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4112 commit 86a3b9516b33eac98241db1e0c8e66d286f5c0ad Author: Kousuke Saruta Date: 2015-01-20T03:51:38Z Fixed UIWorkloadGenerator to stop SparkContext in it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70600230 @tnachen Ok, I see. It happened when executor couldn't get launched, doesn't it? I'll change that setting first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5019 [MLlib] - GaussianMixtureModel expo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4088#issuecomment-7051 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25786/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3270] Spark API for Application Extensi...
Github user mmalohlava commented on the pull request: https://github.com/apache/spark/pull/2691#issuecomment-70600014 Sure, let me create PR for this feature. In meantime, here is a prototype (tailored to our context) - a private `InvokeOnAllNodesRDD` (https://github.com/h2oai/sparkling-water/blob/master/core/src/main/scala/org/apache/spark/h2o/InvokeOnNodesRDD.scala) which is holding information about location of individual partitions. Nevertheless, in this case i have to provide list of executors in advance - it is collected based on list of blockmanagers and a few nasty round trips around actual cluster (not show in the code). Does it fit to your idea? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5019 [MLlib] - GaussianMixtureModel expo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4088#issuecomment-70599988 [Test build #25786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25786/consoleFull) for PR 4088 at commit [`3ef6c7f`](https://github.com/apache/spark/commit/3ef6c7f21de7ccd86a26cb6dee5845ab00c9fab6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4226][SQL] SparkSQL - Add support for s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3249#issuecomment-70599526 [Test build #25788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25788/consoleFull) for PR 3249 at commit [`0347f02`](https://github.com/apache/spark/commit/0347f027cef940d987cd079357059c7695e9400f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5190] Allow SparkListeners to be regist...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4111#issuecomment-70599178 [Test build #25787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25787/consoleFull) for PR 4111 at commit [`25988f3`](https://github.com/apache/spark/commit/25988f377fc0cbeadbddd630f9564697b07ef877). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5190] Allow SparkListeners to be regist...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/4111 [SPARK-5190] Allow SparkListeners to be registered before SparkContext is initialized This patch extends the SparkContext constructor to accept a list of SparkListeners to register before initializing the ListenerBus. This is necessary in order for custom listeners to be able to receive certain events, such as SparkListenerEnvironmentUpdate. I also added a `addSparkListener` methods to JavaSparkContext. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark SPARK-5190-register-sparklistener-in-sc-constructor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4111 commit 163ba19f04bd93247d56e79a6cfa5950427d25e3 Author: Josh Rosen Date: 2015-01-20T03:18:36Z [SPARK-5190] Allow SparkListeners to be registered before SparkContext starts. commit 25988f377fc0cbeadbddd630f9564697b07ef877 Author: Josh Rosen Date: 2015-01-20T03:22:27Z Add addSparkListener to JavaSparkContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5190] Allow SparkListeners to be regist...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4111#issuecomment-70598943 /cc @ksakellis, who originally requested this feature. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4003#issuecomment-70598670 [Test build #25785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25785/consoleFull) for PR 4003 at commit [`1f56897`](https://github.com/apache/spark/commit/1f5689715dcba758bd866a43ac23ec243fc9b8ab). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5202] [SQL] Add hql variable substituti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4003#issuecomment-70598676 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25785/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70598518 Yes when the executor couldn't get launched then all the tasks assigned to it is LOST. IMO this is really a configuration problem, and not a normal failure that should occur. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4894][mllib] Added Bernoulli option to ...
Github user leahmcguire commented on the pull request: https://github.com/apache/spark/pull/4087#issuecomment-70597399 Thanks for the comments! The JIRA for the python API is: https://issues.apache.org/jira/browse/SPARK-5328 I will get the rest fixed tonight or tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4961] [CORE] Put HadoopRDD.getPartition...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3794#issuecomment-70597066 [Test build #25784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25784/consoleFull) for PR 3794 at commit [`aed530b`](https://github.com/apache/spark/commit/aed530b31481e3f8ed007ee0abf99a9180d4342d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70597055 @tnachen In my case - above logs -, task 34 and 63 are assigned to same executor and also same container on same node. Task 34 occurs error about registration timeout, and task 34 is terminated and task 63, which is on queue, is also exited because mesos_containerizer destroys that container which contain task 34 and 63. I think this is a bug. As you told, an executor is responsible to manage tasks and executor can terminate all tasks which are running or queueing. I think only one task is terminated if the task fails. I think a task must not be influenced by any other task. What do you think of this situation? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4961] [CORE] Put HadoopRDD.getPartition...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3794#issuecomment-70597070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25784/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70595793 The executor is responsible to launch and wait for tasks, but it is entirely depends on how the executor is implemented if any task interfere with another. In spark case the executor lives through tasks so if one task fail it won't interfere with another one. Feel free to try it your self --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5019 [MLlib] - GaussianMixtureModel expo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4088#issuecomment-70595366 [Test build #25786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25786/consoleFull) for PR 4088 at commit [`3ef6c7f`](https://github.com/apache/spark/commit/3ef6c7f21de7ccd86a26cb6dee5845ab00c9fab6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5019 [MLlib] - GaussianMixtureModel expo...
Github user tgaloppo commented on the pull request: https://github.com/apache/spark/pull/4088#issuecomment-70595369 @jkbradley I considered making those plural for the initial commit. I guess I should have. Update has been made. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
