[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58740484 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21637/consoleFull) for PR 2762 at commit [`06581e3`](https://github.com/apache/spark/commit/06581e31aaef055c89a0d89ddaac657a9609d571). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58740444 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58740447 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-58740431 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...
GitHub user chenghao-intel opened a pull request:
https://github.com/apache/spark/pull/2762
[SPARK-3904] [SQL] add constant objectinspector support for udfs
In HQL, we convert all of the data type into normal `ObjectInspector`s for
UDFs, most of cases it work, however, some of the UDF actually requires the
input `ObjectInspector` to be the `ConstantObjectInspector`, which will cause
exception.
e.g.
select named_struct("x", "str") from src limit 1;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chenghao-intel/spark udf_coi
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2762.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2762
commit 06581e31aaef055c89a0d89ddaac657a9609d571
Author: Cheng Hao
Date: 2014-10-11T06:34:24Z
add constant objectinspector support for udfs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740227 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21628/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58740225 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21628/consoleFull) for PR 2760 at commit [`ff28e49`](https://github.com/apache/spark/commit/ff28e49d990577635fa148bd57461a387bd3466d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class JavaFutureActionWrapper[S, T](futureAction: FutureAction[S], converter: S => T)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58740083 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21636/consoleFull) for PR 2761 at commit [`d80d71a`](https://github.com/apache/spark/commit/d80d71abc4cf3d85a2585729719b35a5eca84551). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58739903 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21635/consoleFull) for PR 2570 at commit [`3774bd4`](https://github.com/apache/spark/commit/3774bd4617cb4dec3f78a08bdf42653b682102fd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58739870 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21625/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58739867 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21625/consoleFull) for PR 2759 at commit [`f068eb5`](https://github.com/apache/spark/commit/f068eb508c7f0e6991d296f4473eb754c7d5090f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58739815 Seems the failure is not related to this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3343] [SQL] Add serde support for CTAS
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2570#issuecomment-58739817 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739777 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21634/consoleFull) for PR 2761 at commit [`64b2c46`](https://github.com/apache/spark/commit/64b2c46474a48fc0906f140edf310c46eb63). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkSpaceBeforeLeftBraceChecker extends ScalariformChecker ` * `class SparkRunnerSettings(error: String => Unit) extends Settings(error) ` * `trait ActorHelper extends Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739778 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21634/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58739745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21626/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58739744 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21626/consoleFull) for PR 2675 at commit [`1c384b7`](https://github.com/apache/spark/commit/1c384b7bc8b0b8d5b9b6bf294f399de5bb8a9976). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739709 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21634/consoleFull) for PR 2761 at commit [`64b2c46`](https://github.com/apache/spark/commit/64b2c46474a48fc0906f140edf310c46eb63). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58739685 @marmbrus This should be ready to go once Jenkins nods. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user lirui-intel commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739690 Looks great! I think it's very useful to have these async APIs in java :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2574#issuecomment-58739666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21624/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2574#issuecomment-58739664 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21624/consoleFull) for PR 2574 at commit [`4fee5a8`](https://github.com/apache/spark/commit/4fee5a8400e87f7bb33363194cc3039feb3dbed6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739545 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21633/consoleFull) for PR 2761 at commit [`86c63e0`](https://github.com/apache/spark/commit/86c63e04c392b97a0b629e719bb42424992cffd1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkSpaceBeforeLeftBraceChecker extends ScalariformChecker ` * `class SparkRunnerSettings(error: String => Unit) extends Settings(error) ` * `trait ActorHelper extends Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739546 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21633/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739525 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21633/consoleFull) for PR 2761 at commit [`86c63e0`](https://github.com/apache/spark/commit/86c63e04c392b97a0b629e719bb42424992cffd1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/2344#discussion_r18739665
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -220,20 +220,44 @@ trait HiveTypeCoercion {
case a: BinaryArithmetic if a.right.dataType == StringType =>
a.makeCopy(Array(a.left, Cast(a.right, DoubleType)))
+ // we should cast all timestamp/date/string compare into string
compare
+ case p: BinaryPredicate if p.left.dataType == StringType
+&& p.right.dataType == DateType =>
+p.makeCopy(Array(p.left, Cast(p.right, StringType)))
+ case p: BinaryPredicate if p.left.dataType == DateType
+&& p.right.dataType == StringType =>
+p.makeCopy(Array(Cast(p.left, StringType), p.right))
case p: BinaryPredicate if p.left.dataType == StringType
&& p.right.dataType == TimestampType =>
-p.makeCopy(Array(Cast(p.left, TimestampType), p.right))
+p.makeCopy(Array(p.left, Cast(p.right, StringType)))
case p: BinaryPredicate if p.left.dataType == TimestampType
&& p.right.dataType == StringType =>
-p.makeCopy(Array(p.left, Cast(p.right, TimestampType)))
+p.makeCopy(Array(Cast(p.left, StringType), p.right))
+ case p: BinaryPredicate if p.left.dataType == TimestampType
+&& p.right.dataType == DateType =>
+p.makeCopy(Array(Cast(p.left, StringType), Cast(p.right,
StringType)))
+ case p: BinaryPredicate if p.left.dataType == DateType
+&& p.right.dataType == TimestampType =>
+p.makeCopy(Array(Cast(p.left, StringType), Cast(p.right,
StringType)))
--- End diff --
So Michael agreed to leave the whole ordering and comparing stuff in a
separated PR :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739516 Oh, I didn't run scalastyle for yarn-alpha. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739473 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21632/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Docs] logNormalGraph missing partition parame...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2523#issuecomment-58739482 @elmalto It looks like GitHub says that this PR was opened from "unknown repository", which might explain why you're not able to update its code. If that's the case, could you close this PR and open a new one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739472 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21632/consoleFull) for PR 2761 at commit [`86c63e0`](https://github.com/apache/spark/commit/86c63e04c392b97a0b629e719bb42424992cffd1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkSpaceBeforeLeftBraceChecker extends ScalariformChecker ` * `class SparkRunnerSettings(error: String => Unit) extends Settings(error) ` * `trait ActorHelper extends Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739441 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21632/consoleFull) for PR 2761 at commit [`86c63e0`](https://github.com/apache/spark/commit/86c63e04c392b97a0b629e719bb42424992cffd1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2344#discussion_r18739656
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -220,20 +220,44 @@ trait HiveTypeCoercion {
case a: BinaryArithmetic if a.right.dataType == StringType =>
a.makeCopy(Array(a.left, Cast(a.right, DoubleType)))
+ // we should cast all timestamp/date/string compare into string
compare
+ case p: BinaryPredicate if p.left.dataType == StringType
+&& p.right.dataType == DateType =>
+p.makeCopy(Array(p.left, Cast(p.right, StringType)))
+ case p: BinaryPredicate if p.left.dataType == DateType
+&& p.right.dataType == StringType =>
+p.makeCopy(Array(Cast(p.left, StringType), p.right))
case p: BinaryPredicate if p.left.dataType == StringType
&& p.right.dataType == TimestampType =>
-p.makeCopy(Array(Cast(p.left, TimestampType), p.right))
+p.makeCopy(Array(p.left, Cast(p.right, StringType)))
case p: BinaryPredicate if p.left.dataType == TimestampType
&& p.right.dataType == StringType =>
-p.makeCopy(Array(p.left, Cast(p.right, TimestampType)))
+p.makeCopy(Array(Cast(p.left, StringType), p.right))
+ case p: BinaryPredicate if p.left.dataType == TimestampType
+&& p.right.dataType == DateType =>
+p.makeCopy(Array(Cast(p.left, StringType), Cast(p.right,
StringType)))
+ case p: BinaryPredicate if p.left.dataType == DateType
+&& p.right.dataType == TimestampType =>
+p.makeCopy(Array(Cast(p.left, StringType), Cast(p.right,
StringType)))
--- End diff --
OK... verified this behavior with Hive, I've no idea about this :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739434 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739394 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21629/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739375 Jenkins, add to whitelist. This is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739341 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21630/consoleFull) for PR 2760 at commit [`6f8f6ac`](https://github.com/apache/spark/commit/6f8f6ac668d74a3164bcf037f09c8353134b53f6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58739352 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21631/consoleFull) for PR 2538 at commit [`64561e4`](https://github.com/apache/spark/commit/64561e4e503eafb958f6769383ba3b37edbe5fa2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58739325 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/350/consoleFull) for PR 2538 at commit [`6db00da`](https://github.com/apache/spark/commit/6db00da9595e38eccff7bfb5683b32cee3ac6263). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58739234 @tdas it's my mistake, the updateStateByKey() was used in another tests, it's fixed now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2761#issuecomment-58739200 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3854] Scala style: require spaces befor...
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2761
[SPARK-3854] Scala style: require spaces before `{`
This PR is a solution proposal of SPARK-3854.
Following is quoted from SPARK-3854:
We should require spaces before opening curly braces. This isn't in the
style guide, but it probably should be:
// Correct:
if (true) {
println("Wow!")
}
// Incorrect:
if (true){
println("Wow!")
}
See https://github.com/apache/spark/pull/1658#discussion-diff-18611791 for
an example "in the wild."
git grep "){" shows only a few occurrences of this style.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-3854
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2761.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2761
commit 8131d284dd7a718dd4fbbf31d3cadf6a3195680a
Author: Kousuke Saruta
Date: 2014-10-09T10:21:26Z
Added SparkSpaceBeforeLeftBraceChecker to check spaces before "{"
commit 69716ec48b4f05b4ce705c32c44f6d2b6cff8ebc
Author: Kousuke Saruta
Date: 2014-10-11T04:08:04Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3854
commit 4014be060ddf09de2e974a716d3763050a8597bd
Author: Kousuke Saruta
Date: 2014-10-11T05:44:13Z
Fixed styles
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2760#discussion_r18739624 --- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java --- @@ -20,7 +20,9 @@ import java.io.*; import java.net.URI; import java.util.*; +import java.util.concurrent.*; +import org.apache.spark.api.java.*; --- End diff -- Whoops, IntelliJ messed up the import ordering :(. I'll fix this now so that it doesn't have to be addressed later once we add import-order style-checking. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58739136 @tdas The failure looked wired, updater() take exactly two arguments, let's test it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739601
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
---
@@ -575,16 +575,49 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]]
extends Serializable {
def name(): String = rdd.name
/**
- * :: Experimental ::
- * The asynchronous version of the foreach action.
- *
- * @param f the function to apply to all the elements of the RDD
- * @return a FutureAction for the action
+ * The asynchronous version of `count`, which returns a
+ * future for counting the number of elements in this RDD.
*/
- @Experimental
- def foreachAsync(f: VoidFunction[T]): FutureAction[Unit] = {
--- End diff --
yea i think this is fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739598
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
---
@@ -575,16 +575,49 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]]
extends Serializable {
def name(): String = rdd.name
/**
- * :: Experimental ::
- * The asynchronous version of the foreach action.
- *
- * @param f the function to apply to all the elements of the RDD
- * @return a FutureAction for the action
+ * The asynchronous version of `count`, which returns a
+ * future for counting the number of elements in this RDD.
*/
- @Experimental
- def foreachAsync(f: VoidFunction[T]): FutureAction[Unit] = {
--- End diff --
Unfortunately, my PR breaks compatibility for this experimental Java API.
However, the previous version of this method hasn't been shipped in any Spark
releases yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58739062 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21628/consoleFull) for PR 2760 at commit [`ff28e49`](https://github.com/apache/spark/commit/ff28e49d990577635fa148bd57461a387bd3466d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739588
--- Diff: core/src/main/scala/org/apache/spark/FutureAction.scala ---
@@ -70,6 +70,11 @@ trait FutureAction[T] extends Future[T] {
override def isCompleted: Boolean
/**
+ * Returns whether the action has been cancelled.
+ */
+ def isCancelled: Boolean
--- End diff --
This method is new; I addd it to try to maintain feature parity between the
Java and Scala futures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2760#discussion_r18739587
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaFutureAction.java ---
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.api.java;
+
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+public interface JavaFutureAction extends Future {
--- End diff --
I think that it makes sense to expose an extended version of the Java
`Future` API to users, since there may be a number of existing libraries for
consuming these standard future types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2760#issuecomment-58738999 /cc's for review: - @rxin, who wrote the original AsyncRDDActions - @lirui-intel, who added an experimental Java API for `foreachAsync` in #2176, and - @vanzin, who added the `jobIds` method to expose job ids from FutureAction in #2337. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3902] Stabilize AsynRDDActions and add ...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/2760 [SPARK-3902] Stabilize AsynRDDActions and add Java API This PR adds a Java API for AsyncRDDActions and promotes the API from `@Experimental` to stable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark async-rdd-actions-in-java Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2760.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2760 commit 346e46ed8789ab72c709bec40c728568fd7294e5 Author: Josh Rosen Date: 2014-10-11T02:16:49Z [SPARK-3902] Stabilize AsyncRDDActions; add Java API. commit ff28e49d990577635fa148bd57461a387bd3466d Author: Josh Rosen Date: 2014-10-11T05:32:57Z Add MiMa excludes and fix a scalastyle error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58738863 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21627/consoleFull) for PR 2538 at commit [`331ecce`](https://github.com/apache/spark/commit/331ecced6f61ad5183da5830f94f584bcc74e479). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58738860 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21626/consoleFull) for PR 2675 at commit [`1c384b7`](https://github.com/apache/spark/commit/1c384b7bc8b0b8d5b9b6bf294f399de5bb8a9976). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58738838 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/349/consoleFull) for PR 2538 at commit [`6db00da`](https://github.com/apache/spark/commit/6db00da9595e38eccff7bfb5683b32cee3ac6263). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3892][SQL] Map type should have typeNam...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2747#issuecomment-58738829 This LGTM. Please rename the PR title to reflect the actual changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58738651 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21625/consoleFull) for PR 2759 at commit [`f068eb5`](https://github.com/apache/spark/commit/f068eb508c7f0e6991d296f4473eb754c7d5090f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58738607 Jenkins, add to whitelist. This is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2574#issuecomment-58738534 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21623/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3719][CORE][UI]:"complete/failed stages...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2574#issuecomment-58738484 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21624/consoleFull) for PR 2574 at commit [`4fee5a8`](https://github.com/apache/spark/commit/4fee5a8400e87f7bb33363194cc3039feb3dbed6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2757#issuecomment-58738330 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21621/consoleFull) for PR 2757 at commit [`753e98d`](https://github.com/apache/spark/commit/753e98d1dcfb3881ce4c254e2327291bf9210894). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ImportOrderChecker extends ScalariformChecker ` * `case class InSet(value: Expression, hset: HashSet[Any], child: Seq[Expression])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2757#issuecomment-58738331 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21621/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user giwa commented on a diff in the pull request: https://github.com/apache/spark/pull/2538#discussion_r18739441 --- Diff: examples/src/main/python/streaming/stateful_network_wordcount.py --- @@ -0,0 +1,57 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" + Counts words in UTF8 encoded, '\n' delimited text received from the + network every second. + + Usage: stateful_network_wordcount.py +and describe the TCP server that Spark Streaming +would connect to receive data. + + To run this on your local machine, you need to first run a Netcat server +`$ nc -lk ` + and then run the example +`$ bin/spark-submit examples/src/main/python/streaming/stateful_network_wordcount.py \ +localhost ` +""" + +import sys + +from pyspark import SparkContext +from pyspark.streaming import StreamingContext + +if __name__ == "__main__": +if len(sys.argv) != 3: +print >> sys.stderr, "Usage: stateful_network_wordcount.py " +exit(-1) +sc = SparkContext(appName="PythonStreamingNetworkWordCount") --- End diff -- appName could be "PythonStreamingStatefulNetworkWordCount" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58738102 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21622/consoleFull) for PR 2612 at commit [`96a5a52`](https://github.com/apache/spark/commit/96a5a52ba57d87ac7294a3e34dde6a7d7d7a75b1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58738103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21622/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2647#issuecomment-58737847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21618/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2647#issuecomment-58737844 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21618/consoleFull) for PR 2647 at commit [`c81806b`](https://github.com/apache/spark/commit/c81806bda4744382d2657441404cbb1206c3aa8a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3870] EOL character enforcement
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2726#issuecomment-58737824 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21619/consoleFull) for PR 2726 at commit [`7407515`](https://github.com/apache/spark/commit/7407515804e90596fab0e6e8a35399eef9f736b5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3870] EOL character enforcement
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2726#issuecomment-58737826 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21619/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-5873 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21617/consoleFull) for PR 2520 at commit [`b43d01f`](https://github.com/apache/spark/commit/b43d01fc872bc2126003feb57c43b531deec651e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-58737780 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21617/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-58737472 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21616/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-58737471 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21616/consoleFull) for PR 2746 at commit [`b3c7d44`](https://github.com/apache/spark/commit/b3c7d446160747b79e6afbd844f9c8b6d0158781). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB] topic modeling on Gra...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58736906 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21614/consoleFull) for PR 2388 at commit [`daf0787`](https://github.com/apache/spark/commit/daf07871fabaefb798c7c3f8dc91211246af). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TopicModelingKryoRegistrator extends KryoRegistrator ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB] topic modeling on Gra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58736908 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21614/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58736892 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21622/consoleFull) for PR 2612 at commit [`96a5a52`](https://github.com/apache/spark/commit/96a5a52ba57d87ac7294a3e34dde6a7d7d7a75b1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-58736899 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21615/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-58736896 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21615/consoleFull) for PR 2344 at commit [`f15074a`](https://github.com/apache/spark/commit/f15074a614281d3fe4de4f0529ddc53994b4c0d9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2757#issuecomment-58736712 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21621/consoleFull) for PR 2757 at commit [`753e98d`](https://github.com/apache/spark/commit/753e98d1dcfb3881ce4c254e2327291bf9210894). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3892][SQL] Map type should have typeNam...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2747#issuecomment-58736694 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21613/consoleFull) for PR 2747 at commit [`2824216`](https://github.com/apache/spark/commit/2824216f6a7b09374bb0aef0af3fa129dae7efb8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3892][SQL] Map type should have typeNam...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2747#issuecomment-58736697 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21613/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58736661 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21620/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58736660 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21620/consoleFull) for PR 2612 at commit [`894daf8`](https://github.com/apache/spark/commit/894daf8f263269962206f8f5e42c0fa330d85549). * This patch **fails Script style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2757#issuecomment-58736638 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2647#issuecomment-58736625 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21618/consoleFull) for PR 2647 at commit [`c81806b`](https://github.com/apache/spark/commit/c81806bda4744382d2657441404cbb1206c3aa8a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3758] Script style checking
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2612#issuecomment-58736621 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21620/consoleFull) for PR 2612 at commit [`894daf8`](https://github.com/apache/spark/commit/894daf8f263269962206f8f5e42c0fa330d85549). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3870] EOL character enforcement
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2726#issuecomment-58736618 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21619/consoleFull) for PR 2726 at commit [`7407515`](https://github.com/apache/spark/commit/7407515804e90596fab0e6e8a35399eef9f736b5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-58736603 @sryza Thanks for the comments. Unfortunately I have made significant changes recently and much of the code is now outdated. In my original design I went with a callback-based approach rather than a polling approach because I wanted the semantics of the former. In particular, I wanted to add/remove executors only if the respective condition has been satisfied without interruption for a certain duration, and this is difficult to guarantee precisely with polling. HOWEVER, the significant advantage in polling is that we only need one extra thread rather than one for each timer. I am convinced that the latter approach is probably both simpler and more scalable, and I'll likely make the changes shortly. Please hold off reviewing this PR for now until I make the relevant changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-58736523 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21617/consoleFull) for PR 2520 at commit [`b43d01f`](https://github.com/apache/spark/commit/b43d01fc872bc2126003feb57c43b531deec651e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18739007
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorScalingManager.scala ---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
+
+/**
+ * An agent that dynamically scales the number of executors based on the
workload.
+ *
+ * The add policy depends on the number of pending tasks. If the queue of
pending tasks has not
+ * been drained for N seconds, then new executors are added. If the queue
persists for another M
+ * seconds, then more executors are added and so on. The number added in
each round increases
+ * exponentially from the previous round until an upper bound on the
number of executors has
+ * been reached.
+ *
+ * The rationale for the exponential increase is twofold: (1) Executors
should be added slowly
+ * in the beginning in case the number of extra executors needed turns out
to be small. Otherwise,
+ * we may add more executors than we need just to remove them later. (2)
Executors should be added
+ * quickly over time in case the maximum number of executors is very high.
Otherwise, it will take
+ * a long time to ramp up under heavy workloads.
+ *
+ * The remove policy is simpler: If an executor has been idle, meaning it
has not been scheduled
+ * to run any tasks, for K seconds, then it is removed. This requires
starting a timer on each
+ * executor instead of just starting a global one as in the add case.
+ *
+ * The relevant Spark properties include the following:
+ * spark.dynamicAllocation.enabled - Whether this feature is enabled
+ * spark.dynamicAllocation.minExecutors - Lower bound on the number of
executors
+ * spark.dynamicAllocation.maxExecutors - Upper bound on the number of
executors
+ * spark.dynamicAllocation.addExecutorThreshold - How long before new
executors are added (N)
+ * spark.dynamicAllocation.addExecutorInterval - How often to add new
executors (M)
+ * spark.dynamicAllocation.removeExecutorThreshold - How long before an
executor is removed (K)
+ *
+ * Synchronization: Because the schedulers in Spark are single-threaded,
contention only arises
+ * if the application itself runs multiple jobs concurrently. Under normal
circumstances, however,
+ * synchronizing each method on this class should not be expensive
assuming biased locking is
+ * enabled in the JVM (on by default for Java 6+). Tighter locks are also
used where possible.
+ *
+ * Note: This is part of a larger implementation (SPARK-3174) and
currently does not actually
+ * request to add or remove executors. The mechanism to actually do this
will be added separately,
+ * e.g. in SPARK-3822 for Yarn.
+ */
+private[scheduler] class ExecutorScalingManager(scheduler:
TaskSchedulerImpl) extends Logging {
--- End diff --
Though I think we need some notion of `executor` in there.
`DynamicExecutorAllocationManager`? `ExecutorAllocationManager`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58736405 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21612/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2377] Python API for Streaming
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2538#issuecomment-58736402 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21612/consoleFull)** for PR 2538 at commit [`3e2492b`](https://github.com/apache/spark/commit/3e2492b9b95e0cc0e3427265f71f069000cc43f7) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58736227 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2759#issuecomment-58736225 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2746#issuecomment-58736135 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21616/consoleFull) for PR 2746 at commit [`b3c7d44`](https://github.com/apache/spark/commit/b3c7d446160747b79e6afbd844f9c8b6d0158781). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3867] ./python/run-tests failed when it...
GitHub user cocoatomo opened a pull request: https://github.com/apache/spark/pull/2759 [SPARK-3867] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed ./python/run-tests search a Python 2.6 executable on PATH and use it if available. When using Python 2.6, it is going to import unittest2 module which is not a standard library in Python 2.6, so it fails with ImportError. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cocoatomo/spark issues/3867-unittest2-import-error Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2759.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2759 commit f068eb508c7f0e6991d296f4473eb754c7d5090f Author: cocoatomo Date: 2014-10-11T03:05:22Z [SPARK-3867] ./python/run-tests failed when it run with Python 2.6 and unittest2 is not installed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18738931
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorScalingManager.scala ---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
+
+/**
+ * An agent that dynamically scales the number of executors based on the
workload.
+ *
+ * The add policy depends on the number of pending tasks. If the queue of
pending tasks has not
+ * been drained for N seconds, then new executors are added. If the queue
persists for another M
+ * seconds, then more executors are added and so on. The number added in
each round increases
+ * exponentially from the previous round until an upper bound on the
number of executors has
+ * been reached.
+ *
+ * The rationale for the exponential increase is twofold: (1) Executors
should be added slowly
+ * in the beginning in case the number of extra executors needed turns out
to be small. Otherwise,
+ * we may add more executors than we need just to remove them later. (2)
Executors should be added
+ * quickly over time in case the maximum number of executors is very high.
Otherwise, it will take
+ * a long time to ramp up under heavy workloads.
+ *
+ * The remove policy is simpler: If an executor has been idle, meaning it
has not been scheduled
+ * to run any tasks, for K seconds, then it is removed. This requires
starting a timer on each
+ * executor instead of just starting a global one as in the add case.
+ *
+ * The relevant Spark properties include the following:
+ * spark.dynamicAllocation.enabled - Whether this feature is enabled
+ * spark.dynamicAllocation.minExecutors - Lower bound on the number of
executors
+ * spark.dynamicAllocation.maxExecutors - Upper bound on the number of
executors
+ * spark.dynamicAllocation.addExecutorThreshold - How long before new
executors are added (N)
+ * spark.dynamicAllocation.addExecutorInterval - How often to add new
executors (M)
+ * spark.dynamicAllocation.removeExecutorThreshold - How long before an
executor is removed (K)
+ *
+ * Synchronization: Because the schedulers in Spark are single-threaded,
contention only arises
+ * if the application itself runs multiple jobs concurrently. Under normal
circumstances, however,
+ * synchronizing each method on this class should not be expensive
assuming biased locking is
+ * enabled in the JVM (on by default for Java 6+). Tighter locks are also
used where possible.
+ *
+ * Note: This is part of a larger implementation (SPARK-3174) and
currently does not actually
+ * request to add or remove executors. The mechanism to actually do this
will be added separately,
+ * e.g. in SPARK-3822 for Yarn.
+ */
+private[scheduler] class ExecutorScalingManager(scheduler:
TaskSchedulerImpl) extends Logging {
+ private val conf = scheduler.conf
+
+ // Lower and upper bounds on the number of executors. These are required.
+ private val minNumExecutors =
conf.getInt("spark.dynamicAllocation.minExecutors", -1)
+ private val maxNumExecutors =
conf.getInt("spark.dynamicAllocation.maxExecutors", -1)
+ if (minNumExecutors < 0 || maxNumExecutors < 0) {
+throw new SparkException("spark.dynamicAllocation.{min/max}Executors
must be set!")
+ }
+
+ // How frequently to add and remove executors
+ private val addExecutorThreshold =
+conf.getLong("spark.dynamicAllocation.addExecutorThreshold", 60) // s
+ private val addExecutorInterval =
+conf.getLong("spark.dynamicAllocation.addExecutorInterval",
addExecutorThreshold) // s
+ private val removeExecutorThreshold =
+conf.getLong("spark.dynamicAllocation.remov
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-58735889 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21615/consoleFull) for PR 2344 at commit [`f15074a`](https://github.com/apache/spark/commit/f15074a614281d3fe4de4f0529ddc53994b4c0d9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3407][SQL]Add Date type support
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2344#issuecomment-58735806 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB] topic modeling on Gra...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58735791 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21614/consoleFull) for PR 2388 at commit [`daf0787`](https://github.com/apache/spark/commit/daf07871fabaefb798c7c3f8dc91211246af). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB] topic modeling on Gra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2388#issuecomment-58735730 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB] topic modeling on Gra...
GitHub user witgo reopened a pull request: https://github.com/apache/spark/pull/2388 [WIP][SPARK-1405][MLLIB] topic modeling on Graphx This PR relies on #2631 - [X] Topic de-duplication - [X] Support 10 topics - [X] Asymmetric Dirichlet priors - [ ] Add the documentation - [X] Add infer interface - [X] Add unit tests - [X] Add the performance test - [X] Optimizing the infer interface performance - [ ] Verifying the correctness of the algorithm The performance test: `2000` topics: Item | value | - The cluster resource | 36 executors(36 cores, 216g memory) The corpus size | 253064 document, 29696335 words The number of iterations | `105` The number of distinct term | 75496 The number of topics | `2000` alpha | 0.01 beta | 0.01 The running time | 37.1 minutes `1` topics: Item | value | - The cluster resource | 36 executors(36 cores, 216g memory) The corpus size | 253064 document, 29696335 words The number of iterations | `105` The number of distinct term | 75496 The number of topics | `1` alpha | 0.01 beta | 0.01 The running time | 49 minutes `10` topics: Item | value | - The cluster resource | 36 executors(36 cores, 216g memory) The corpus size | 253064 document, 29696335 words The number of iterations | `105` The number of distinct term | 75496 The number of topics | `10` alpha | 0.1 beta | 0.01 The running time | 268.9 minutes conf/spark-defaults.conf: ``` spark.akka.frameSize 20 spark.executor.instances 36 spark.rdd.compress true spark.executor.memory 6g spark.default.parallelism 72 spark.broadcast.blockSize 8192 spark.storage.memoryFraction 0.4 spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryo.registrator org.apache.spark.mllib.feature.TopicModelingKryoRegistrator ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark graphx_lda Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2388.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2388 commit ca8e6f296a2f7ed674dd3a5cde49d4301d3d6d14 Author: GuoQiang Li Date: 2014-10-08T08:10:12Z topic modeling on Graphx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
