[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58775690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21660/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58775688 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21660/consoleFull) for PR 2538 at commit [`64561e4`](https://github.com/apache/spark/commit/64561e4e503eafb958f6769383ba3b37edbe5fa2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class StreamingContext(object):` * `class DStream(object):` * `class TransformedDStream(DStream):` * `class TransformFunction(object):` * `class TransformFunctionSerializer(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...
Github user tigerquoll commented on the pull request: https://github.com/apache/spark/pull/2516#issuecomment-58775682 Hi @vanzin, I've implemented your suggestions, tidied up the code more, and also added more unit tests to flesh out the test coverage. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58775578 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21659/consoleFull) for PR 2773 at commit [`93cd7f6`](https://github.com/apache/spark/commit/93cd7f62bf31c9015f30eb25439d368bac3c57c5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FlumeStreamSuite extends FunSuite with BeforeAndAfter with Matchers with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58775579 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21659/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58775530 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21661/consoleFull) for PR 2762 at commit [`cb97576`](https://github.com/apache/spark/commit/cb975761b23bc479ba1c84716e01631b0fdea2c3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-58774750 Should we create a JIRA for this task? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58774562 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/352/consoleFull) for PR 2538 at commit [`6db00da`](https://github.com/apache/spark/commit/6db00da9595e38eccff7bfb5683b32cee3ac6263). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58774569 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/353/consoleFull) for PR 2538 at commit [`6db00da`](https://github.com/apache/spark/commit/6db00da9595e38eccff7bfb5683b32cee3ac6263). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/2516#discussion_r18745897
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1479,6 +1479,14 @@ private[spark] object Utils extends Logging {
PropertyConfigurator.configure(pro)
}
+ /**
+ * Flatten a map of maps out into a single map, later maps in the
propList
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58774536 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21660/consoleFull) for PR 2538 at commit [`64561e4`](https://github.com/apache/spark/commit/64561e4e503eafb958f6769383ba3b37edbe5fa2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58774530 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21659/consoleFull) for PR 2773 at commit [`93cd7f6`](https://github.com/apache/spark/commit/93cd7f62bf31c9015f30eb25439d368bac3c57c5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-5877 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58774404 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2736#issuecomment-58774056 Hi, @ravipesala, you don't need create a new PR, you can update you pr here(use git push to update this branch) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21658/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773977 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21658/consoleFull) for PR 2607 at commit [`3b8ffc0`](https://github.com/apache/spark/commit/3b8ffc00e9854ca323f0c8772784bf1337eec562). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773403 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21655/consoleFull) for PR 2607 at commit [`bdca43a`](https://github.com/apache/spark/commit/bdca43a4679f194c22209733852515cfa09bf407). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773732 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21657/consoleFull) for PR 2607 at commit [`8e10c63`](https://github.com/apache/spark/commit/8e10c6364cf9ac0fffbfc63e1234a8e44c77a640). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21657/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773560 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21656/consoleFull) for PR 2607 at commit [`f62bc48`](https://github.com/apache/spark/commit/f62bc48491bd3cbb9dc99eee8490ca949287238d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773562 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21656/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773406 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21655/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773337 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21658/consoleFull) for PR 2607 at commit [`3b8ffc0`](https://github.com/apache/spark/commit/3b8ffc00e9854ca323f0c8772784bf1337eec562). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773258 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21654/consoleFull) for PR 2607 at commit [`2fbc9c7`](https://github.com/apache/spark/commit/2fbc9c74885617ffc61c2fa9add1148e28acaf91). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773260 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21654/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773142 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21657/consoleFull) for PR 2607 at commit [`8e10c63`](https://github.com/apache/spark/commit/8e10c6364cf9ac0fffbfc63e1234a8e44c77a640). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21653/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58773003 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21656/consoleFull) for PR 2607 at commit [`f62bc48`](https://github.com/apache/spark/commit/f62bc48491bd3cbb9dc99eee8490ca949287238d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772988 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21653/consoleFull) for PR 2607 at commit [`6dd4dd8`](https://github.com/apache/spark/commit/6dd4dd82ac6d42be0edb3f7498b300cc121a8021). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GradientBoosting (` * `case class BoostingStrategy(` * `trait Loss extends Serializable ` * `class GradientBoostingModel(trees: Array[DecisionTreeModel], algo: Algo) extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772868 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21655/consoleFull) for PR 2607 at commit [`bdca43a`](https://github.com/apache/spark/commit/bdca43a4679f194c22209733852515cfa09bf407). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-58772867 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21651/consoleFull)** for PR 2753 at commit [`9d9b4e1`](https://github.com/apache/spark/commit/9d9b4e1199bdeab7e454878bda61f0b5aecc79ad) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-58772869 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21651/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3890][Docs]remove redundant spark.execu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2745#issuecomment-58772832 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21652/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3890][Docs]remove redundant spark.execu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2745#issuecomment-58772831 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21652/consoleFull) for PR 2745 at commit [`fdbdb1f`](https://github.com/apache/spark/commit/fdbdb1f66a85e210f18a98a9349057902e8c93fd). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772720 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21654/consoleFull) for PR 2607 at commit [`2fbc9c7`](https://github.com/apache/spark/commit/2fbc9c74885617ffc61c2fa9add1148e28acaf91). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3586][streaming]Support nested director...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2765#discussion_r18745609
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
---
@@ -240,6 +260,31 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <:
NewInputFormat[K,V] : Clas
true
}
}
+
+ private[streaming]
+ class SubPathFilter extends PathFilter {
--- End diff --
No need to wrap this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3586][streaming]Support nested director...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2765#discussion_r18745606
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
---
@@ -240,6 +260,31 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <:
NewInputFormat[K,V] : Clas
true
}
}
+
+ private[streaming]
+ class SubPathFilter extends PathFilter {
+
+def accept(path: Path): Boolean = {
+ try {
+if(fs.getFileStatus(path).isDirectory()){
--- End diff --
Nit: spaces before `(` and `{`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772417 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21653/consoleFull) for PR 2607 at commit [`6dd4dd8`](https://github.com/apache/spark/commit/6dd4dd82ac6d42be0edb3f7498b300cc121a8021). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] SPARK-1547: Adding Gradient Boos...
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-58772394 @mengxr The test failures seem to be due to lack of permission. Could you please enable it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3890][Docs]remove redundant spark.execu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2745#issuecomment-58771859 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21652/consoleFull) for PR 2745 at commit [`fdbdb1f`](https://github.com/apache/spark/commit/fdbdb1f66a85e210f18a98a9349057902e8c93fd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-58771033 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21651/consoleFull) for PR 2753 at commit [`9d9b4e1`](https://github.com/apache/spark/commit/9d9b4e1199bdeab7e454878bda61f0b5aecc79ad). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58770566 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21650/consoleFull) for PR 2773 at commit [`93cd7f6`](https://github.com/apache/spark/commit/93cd7f62bf31c9015f30eb25439d368bac3c57c5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FlumeStreamSuite extends FunSuite with BeforeAndAfter with Matchers with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58770568 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21650/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3909][PySpark][Doc] A corrupted format ...
Github user cocoatomo commented on the pull request: https://github.com/apache/spark/pull/2766#issuecomment-58769648 Thank you for the comment. I'm also happy that Jenkins checks this kind of issue rather than me. -W option of sphinx-build would help us to check and report errors strictly. http://sphinx-doc.org/invocation.html#cmdoption-sphinx-build-W It is used through a make command such like: ```bash $ SPHINXOPTS=-W make clean html ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2757#issuecomment-58769459 Wow, there are a lot of changes in this PR. Perhaps this was the most ignored style rule, eh? cc @pwendell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2773#issuecomment-58769438 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21650/consoleFull) for PR 2773 at commit [`93cd7f6`](https://github.com/apache/spark/commit/93cd7f62bf31c9015f30eb25439d368bac3c57c5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58769389 > I quite like standardizing style, but doesn't this have the same problem mentioned before, that it's going to break a lot of potential merge commits? When I [did this for Python](https://github.com/apache/spark/pull/1744), I fixed all outstanding style problems as part of the same PR that introduced that check. It forced some people to rebase their open PRs, but it was a once-and-done thing. Are we opposed to doing that here? Trying to ease this check in by enforcing it only on new code is a good idea, but why not just get the style cleanup over with in this PR? It looks like @sarutak has done just that. Some people will have to rebase once and this style problem is done with. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58769373 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21649/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58769371 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21649/consoleFull) for PR 2712 at commit [`f85d24c`](https://github.com/apache/spark/commit/f85d24c954be419045236cfabc5613aab4c2a169). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3912][Streaming] Fixed flakyFlumeStream...
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/2773 [SPARK-3912][Streaming] Fixed flakyFlumeStreamSuite @harishreedharan @pwendell See JIRA for diagnosis of the problem https://issues.apache.org/jira/browse/SPARK-3912 The solution was to reimplement it. 1. Find a free port (by binding and releasing a server-scoket), and then use that port 2. Remove thread.sleep()s, instead repeatedly try to create a sender and send data and check whether data was sent. Use eventually() to minimize waiting time. 3. Check whether all the data was received, without caring about batches. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tdas/spark flume-test-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2773.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2773 commit 93cd7f62bf31c9015f30eb25439d368bac3c57c5 Author: Tathagata Das Date: 2014-10-12T00:04:01Z Reimplimented FlumeStreamSuite to be more robust. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58768088 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21649/consoleFull) for PR 2712 at commit [`f85d24c`](https://github.com/apache/spark/commit/f85d24c954be419045236cfabc5613aab4c2a169). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user james64 commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58767976 Sorry for the test name. Now it should be all fine including commets. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2315] Implement drop, dropRight and dro...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1839#issuecomment-58767527 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21648/consoleFull) for PR 1839 at commit [`af73e1f`](https://github.com/apache/spark/commit/af73e1f3ffab0909acaebdca154889030f1187f7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FanInDep[T: ClassTag](rdd: RDD[T]) extends NarrowDependency[T](rdd) ` * `class DropRDDFunctions[T : ClassTag](self: RDD[T]) extends Logging with Serializable ` * `class FanOutDep[T: ClassTag](rdd: RDD[T]) extends NarrowDependency[T](rdd) ` * `class PromisePartition extends Partition ` * `class PromiseRDD[V: ClassTag](expr: => (TaskContext => V),` * `class PromiseArgPartition(p: Partition, argv: Seq[PromiseRDD[_]]) extends Partition ` * `class PromiseRDDFunctions[T : ClassTag](self: RDD[T]) extends Logging with Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2315] Implement drop, dropRight and dro...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1839#issuecomment-58767530 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21648/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2315] Implement drop, dropRight and dro...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1839#issuecomment-58765937 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21648/consoleFull) for PR 1839 at commit [`af73e1f`](https://github.com/apache/spark/commit/af73e1f3ffab0909acaebdca154889030f1187f7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
Github user ravipesala closed the pull request at: https://github.com/apache/spark/pull/2736 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2772#issuecomment-58765710 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
Github user ravipesala commented on the pull request: https://github.com/apache/spark/pull/2736#issuecomment-58765712 Since this PR has conflicts , I created new PR https://github.com/apache/spark/pull/2772 and handled review comments in it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2772#issuecomment-58765716 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise & does not work in H...
GitHub user ravipesala opened a pull request: https://github.com/apache/spark/pull/2772 [SPARK-3814][SQL] Bitwise & does not work in Hive Currently there is no support of Bitwise & , | in Spark HiveQl and Spark SQL as well. So this PR support the same. I am closing https://github.com/apache/spark/pull/2736 as it has conflicts to merge. And I handled all review comments in that PR. Author : ravipesala ravindra.pes...@huawei.com You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/spark SPARK-3814-NEW1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2772.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2772 commit a73367c11dfaffef4a7f95460569d6707c95f731 Author: ravipesala Date: 2014-10-11T21:34:15Z Supporting Bitwise &, | in Spark SQL and HiveQl --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58761838 This looks good to me; sorry for my earlier confusion. If you add a comment and change the name of the test, I'll merge this and cherry-pick it back into `branch-1.1` and `branch-1.0`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2712#discussion_r18743890
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1409,7 +1410,9 @@ object SparkContext extends Logging {
simpleWritableConverter[Boolean, BooleanWritable](_.get)
implicit def bytesWritableConverter(): WritableConverter[Array[Byte]] = {
-simpleWritableConverter[Array[Byte], BytesWritable](_.getBytes)
+simpleWritableConverter[Array[Byte], BytesWritable](bw =>
+ Arrays.copyOfRange(bw.getBytes, 0, bw.getLength)
--- End diff --
Could you add a one-line comment here that explains why we need to make
this copy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58761660 Actually, ignore my earlier (deleted) comments; this looks like a valid issue (see [HADOOP-6298: "BytesWritable#getBytes is a bad name that leads to programming mistakes"](https://issues.apache.org/jira/browse/HADOOP-6298)). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58761280 Actually, I don't think that this is a bug. Instead, I think that the behavior that you're seeing could be an instance of [SPARK-1018](https://issues.apache.org/jira/browse/SPARK-1018), where calling `take()` or `collect()` on a non-transformed HadoopRDD returns the same element several times because the same `Writable` object is re-used. There's actually a note about this in the `sequenceFile()` Java/Scaladoc (added by https://github.com/apache/spark/commit/7101017803a70f3267381498594c0e8c604f932c): ```scala /** Get an RDD for a Hadoop SequenceFile with given key and value types. * * '''Note:''' Because Hadoop's RecordReader class re-uses the same Writable object for each * record, directly caching the returned RDD will create many references to the same object. * If you plan to directly cache Hadoop writable objects, you should first copy them using * a `map` function. * */ ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58760950 It looks like the original implementation of this converter was added in 2604939f643bca125f5e2fb53e3221202996d41b, all the way back in 2011, so I believe that this would affect every released version of Spark. How does this error manifest itself in the wild? Does it lead to silent corruption when reading / writing binary data? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2712#discussion_r18743728
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import org.scalatest.FunSuite
+
+import org.apache.hadoop.io.BytesWritable
+
+class SparkContextSuite extends FunSuite {
+ test("test of writing spark scala test") {
--- End diff --
This test could use a better name. I'd also add a comment, like `//
Regression test for SPARK-3121` to help readers link this back to the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58760768 That particular Flume test is known to be flaky; I think that TD is working on a rewrite / fix for that test suite. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3896] Pool#checkSpeculatableTasks fask ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2751#issuecomment-58760669 Hi @shijinkui, can you provide some motivation / context for this change? The scheduler is a complicated piece of code and even simple changes like this can have unanticipated consequences. There are two types of Schedulables: pools and TaskSetManagers. It looks like `TaskSetManager.checkSpeculatableTasks()` has the side-effect of updating some internal data-structures in TaskSetManager (such as the `speculatableTasks` HashSet), so your PR's early-termination means that only the first TaskSetManager in the pool will have its fields updated. I haven't checked too closely to see how `checkSpeculatableTasks` is called, but I'm concerned that this change might either cause us to miss out on scheduling some speculatable tasks or may result in more calls / loops through `checkSpeculatableTasks` (meaning that it wouldn't be a huge efficiency gain). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3909][PySpark][Doc] A corrupted format ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2766 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3909][PySpark][Doc] A corrupted format ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2766#issuecomment-58760169 This looks good to me; thanks! I'm happy to merge this PR, but I fear that we're going to continue this cycle of re-introducing and fixing doctest / Sphinx problems until we have Jenkins configured to automatically test the docs. As a short-term fix, I'll make sure to test the docs for compilation errors before cutting new Spark release candidates. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58759963 It would be neat if there was some way to restrict the style checker to only check new/changed lines introduced by a PR. This could be hard to integrate with local development workflows, though: I might be developing some code locally and periodically running scalastyle before opening a pull request, so we'd need to make sure that we don't emit tons of warnings from existing code. Maybe one approach would be to find all of the style warnings, then check whether the commits that introduced the lines that triggered the warnings are present in either `origin/master` or `origin/branch-1-1`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18743600
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorScalingManager.scala ---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
+
+/**
+ * An agent that dynamically scales the number of executors based on the
workload.
+ *
+ * The add policy depends on the number of pending tasks. If the queue of
pending tasks has not
+ * been drained for N seconds, then new executors are added. If the queue
persists for another M
+ * seconds, then more executors are added and so on. The number added in
each round increases
+ * exponentially from the previous round until an upper bound on the
number of executors has
+ * been reached.
+ *
+ * The rationale for the exponential increase is twofold: (1) Executors
should be added slowly
+ * in the beginning in case the number of extra executors needed turns out
to be small. Otherwise,
+ * we may add more executors than we need just to remove them later. (2)
Executors should be added
+ * quickly over time in case the maximum number of executors is very high.
Otherwise, it will take
+ * a long time to ramp up under heavy workloads.
+ *
+ * The remove policy is simpler: If an executor has been idle, meaning it
has not been scheduled
+ * to run any tasks, for K seconds, then it is removed. This requires
starting a timer on each
+ * executor instead of just starting a global one as in the add case.
+ *
+ * The relevant Spark properties include the following:
+ * spark.dynamicAllocation.enabled - Whether this feature is enabled
+ * spark.dynamicAllocation.minExecutors - Lower bound on the number of
executors
+ * spark.dynamicAllocation.maxExecutors - Upper bound on the number of
executors
+ * spark.dynamicAllocation.addExecutorThreshold - How long before new
executors are added (N)
+ * spark.dynamicAllocation.addExecutorInterval - How often to add new
executors (M)
+ * spark.dynamicAllocation.removeExecutorThreshold - How long before an
executor is removed (K)
+ *
+ * Synchronization: Because the schedulers in Spark are single-threaded,
contention only arises
+ * if the application itself runs multiple jobs concurrently. Under normal
circumstances, however,
+ * synchronizing each method on this class should not be expensive
assuming biased locking is
+ * enabled in the JVM (on by default for Java 6+). Tighter locks are also
used where possible.
+ *
+ * Note: This is part of a larger implementation (SPARK-3174) and
currently does not actually
+ * request to add or remove executors. The mechanism to actually do this
will be added separately,
+ * e.g. in SPARK-3822 for Yarn.
+ */
+private[scheduler] class ExecutorScalingManager(scheduler:
TaskSchedulerImpl) extends Logging {
--- End diff --
All of those sound good to me. The second one if I had to choose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-58759844 Awesome, sounds good, will hold off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867][PySpark] ./python/run-tests faile...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2759 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867][PySpark] ./python/run-tests faile...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58759299 This looks good to me. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2771#issuecomment-58756583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21647/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2771#issuecomment-58756581 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21647/consoleFull) for PR 2771 at commit [`cc3091e`](https://github.com/apache/spark/commit/cc3091e073781035680cb96923361a685fedeba7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2771#issuecomment-58755001 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21647/consoleFull) for PR 2771 at commit [`cc3091e`](https://github.com/apache/spark/commit/cc3091e073781035680cb96923361a685fedeba7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3907][sql] add truncate table support
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2770#issuecomment-58754918 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...
GitHub user chenghao-intel opened a pull request: https://github.com/apache/spark/pull/2771 [SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding ``` explain extended select cos(null) from src limit 1; ``` outputs: ``` Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5] MetastoreRelation default, src, None == Optimized Logical Plan == Limit 1 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5] MetastoreRelation default, src, None == Physical Plan == Limit 1 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5] HiveTableScan [], (MetastoreRelation default, src, None), None ``` After patching this PR it outputs ``` == Parsed Logical Plan == Limit 1 Project ['cos(null) AS c_0#0] UnresolvedRelation None, src, None == Analyzed Logical Plan == Limit 1 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#0] MetastoreRelation default, src, None == Optimized Logical Plan == Limit 1 Project [null AS c_0#0] MetastoreRelation default, src, None == Physical Plan == Limit 1 Project [null AS c_0#0] HiveTableScan [], (MetastoreRelation default, src, None), None ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenghao-intel/spark hive_udf_constant_folding Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2771.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2771 commit cc3091e073781035680cb96923361a685fedeba7 Author: Cheng Hao Date: 2014-10-11T16:09:32Z support constant folding for HiveSimpleUdf --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3907][sql] add truncate table support
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2770#issuecomment-58754916 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2771#issuecomment-58754914 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3907][sql] add truncate table support
GitHub user wangxiaojing opened a pull request: https://github.com/apache/spark/pull/2770 [spark-3907][sql] add truncate table support JIRA issue: [SPARK-3907]https://issues.apache.org/jira/browse/SPARK-3907 add turncate table support TRUNCATE TABLE table_name [PARTITION partition_spec]; partition_spec: : (partition_col = partition_col_value, partition_col = partiton_col_value, ...) Removes all rows from a table or partition(s). Currently target table should be native/managed table or exception will be thrown. User can specify partial partition_spec for truncating multiple partitions at once and omitting partition_spec will truncate all partitions in the table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangxiaojing/spark spark-3907 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2770.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2770 commit 77b1f2022d1a5b287fe43e9823f6d8b7934a969e Author: wangxiaojing Date: 2014-10-11T16:08:26Z add truncate table support --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1715: Ensure actor is self-contained in ...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/637#issuecomment-58754089 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1715: Ensure actor is self-contained in ...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/637#issuecomment-58754084 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] make i...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-58754077 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Stats collection
Github user DeathByTape commented on the pull request: https://github.com/apache/spark/pull/2769#issuecomment-58750992 Wrong PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Stats collection
Github user DeathByTape closed the pull request at: https://github.com/apache/spark/pull/2769 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Stats collection
GitHub user DeathByTape opened a pull request: https://github.com/apache/spark/pull/2769 Stats collection Implementation of basic stats collection for updated scheduler. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DeathByTape/spark Stats_Collection Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2769.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2769 commit 4db1b55ba0800185975cf55dd960c13d02963b7d Author: Dennis J. McWherter Jr Date: 2014-10-09T04:49:22Z Added NodeStats calculation helper class for stats reporting. commit ff544fd272bd158c191e3559f6288d7c261df75b Author: Dennis J. McWherter Jr Date: 2014-10-09T13:55:14Z Updated code to be scalastyle compliant (i.e. Apache open source header). commit 380331899491d0a02c94c67826e289213a92bcc9 Author: Dennis J. McWherter Jr Date: 2014-10-10T02:21:44Z Added static/initial stats sending upon connection. commit 565b8a4e0754ccf30ea2ea19a010e27f8a0c45ef Author: Dennis J. McWherter Jr Date: 2014-10-10T03:37:04Z Added node stats to web UI for worker. commit 2fb4741779f221d412c35318f038635844830f8d Author: Dennis J. McWherter Jr Date: 2014-10-10T13:18:28Z Added support for Latency and LeanStatistics. commit b3d1e23e7efa713a53785eec31264420853c3640 Author: Dennis J. McWherter Jr Date: 2014-10-10T21:28:01Z Threading stats through to clients-- may need to look into the further to make sure they're getting updated in correct places. commit 4db0892fbdc4c93e07ac9d3872e4aa3495962f0b Author: Dennis J. McWherter Jr Date: 2014-10-11T14:19:30Z Node stat transmission fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58749849 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21646/consoleFull) for PR 2768 at commit [`4f9517a`](https://github.com/apache/spark/commit/4f9517a2c11d13f439f3ed7ea447a4559f9e9088). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Sqrt(child: Expression) extends UnaryExpression with SignedFunction[Sqrt] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58749850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21646/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58748912 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21645/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58748909 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21645/consoleFull) for PR 2762 at commit [`335a952`](https://github.com/apache/spark/commit/335a95298d00fd43da4af85ba10ef399aeb11f7e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58748576 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21646/consoleFull) for PR 2768 at commit [`4f9517a`](https://github.com/apache/spark/commit/4f9517a2c11d13f439f3ed7ea447a4559f9e9088). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2768#issuecomment-58748502 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2863: [SQL] Add facilities for function-...
GitHub user willb opened a pull request: https://github.com/apache/spark/pull/2768 SPARK-2863: [SQL] Add facilities for function-argument coercion This commit adds the `SignedFunction` trait and modifies the `Sqrt` expression class to use it for coercing its argument to `DoubleType`. `SignedFunction` represents a fixed-arity function whose arguments should be casted to particular types. Expression classes extending SignedFunction must provide `formalTypes`, a List of expected types for formal parameters, `actualParams`, a list of Expressions corresponding to actual parameters, and create, which creates an instance of that expression class from a list of expressions corresponding to actuals. The type parameter for SignedFunction should be the expression class extending it. See the Sqrt class for a concrete example. This trait (or one or several abstract classes extending this trait) could be exposed to code outside `sql` in the future. You can merge this pull request into a Git repository by running: $ git pull https://github.com/willb/spark spark-2863 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2768.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2768 commit 4f9517a2c11d13f439f3ed7ea447a4559f9e9088 Author: William Benton Date: 2014-10-11T12:40:10Z Adds SignedFunction trait and type coercion rules SignedFunction represents a fixed-arity function whose arguments should be casted to particular types. Expression classes extending SignedFunction must provide `formalTypes`, a List of expected types for formal parameters, `actualParams`, a list of Expressions corresponding to actual parameters, and create, which creates an instance of that expression class from a list of expressions corresponding to actuals. The type parameter for SignedFunction should be the expression class extending it. See the Sqrt class for a concrete example. This trait (or one or several abstract classes extending this trait) could be exposed to code outside `sql` in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58748067 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21645/consoleFull) for PR 2762 at commit [`335a952`](https://github.com/apache/spark/commit/335a95298d00fd43da4af85ba10ef399aeb11f7e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/2762#discussion_r18741363
--- Diff:
sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
---
@@ -578,6 +578,7 @@ class HiveCompatibilitySuite extends HiveQueryFileTest
with BeforeAndAfter {
"multi_join_union",
"multiMapJoin1",
"multiMapJoin2",
+"udf_named_struct",
--- End diff --
Thanks, moved.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867][PySpark] ./python/run-tests faile...
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58746696 +1 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
