[GitHub] spark pull request: [SPARK-9835] [PySpark] fix install pylint
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7721#issuecomment-125475844 Note that the code to actually run the PyLint checks is still commented out, so you might want to revert that commit as part of this PR if you're trying to re-enable those checks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Hotfix][Examples] fix a Javadoc of a preforma...
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/7708#issuecomment-125475153
Hm, I do see many more uses of `{@code }` in the project than `` for
this purpose, which is good. It would be reasonable to fix, at least, all the
Java examples in one go here. Or, even fix all 14 instances I see in the code
at once, everywhere. If so, make a JIRA please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [PySpark] fix install pylint
GitHub user davies opened a pull request: https://github.com/apache/spark/pull/7721 [SPARK-9835] [PySpark] fix install pylint You can merge this pull request into a Git repository by running: $ git pull https://github.com/davies/spark fix_install_pylint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7721.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7721 commit dac4db3c89b88773414e8b727718f7ea68542d3e Author: Davies Liu Date: 2015-07-28T06:57:42Z fix install pylint --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9835] [PySpark] fix install pylint
Github user davies commented on the pull request: https://github.com/apache/spark/pull/7721#issuecomment-125475355 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7711#discussion_r35618667
--- Diff: dev/sparktestsupport/modules.py ---
@@ -29,7 +29,7 @@ class Module(object):
changed.
"""
-def __init__(self, name, dependencies, source_file_regexes,
build_profile_flags=(),
+def __init__(self, name, dependencies, source_file_regexes,
build_profile_flags=(), environs={},
--- End diff --
Minor naming nit: I think `environ` would be _slightly_ more idiomatic,
given that Python calls the dictionary itself `os.environ`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8206][SQL][WIP]Add function round
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6836 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125474257 After a cursory glance this seems reasonable to me; thanks for adding such detailed code comments to explain what's going on here. I'm cool with merging this but let's make sure that we add a "test Kinesis" item to our release checklists so that we run this test right before packaging the release candidates. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8231][SQL][WIP] Add array_contains
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7580#issuecomment-125473405 [Test build #38662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38662/console) for PR 7580 at commit [`c4e6e43`](https://github.com/apache/spark/commit/c4e6e437569c1b721ff72c4c0811cec51e9e0f9c). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class RFormula(override val uid: String) extends Estimator[RFormulaModel] with RFormulaBase ` * `abstract class InternalRow extends Serializable with SpecializedGetters ` * `case class ArrayContains(left: Expression, right: Expression)` * `case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes ` * `case class NextDay(startDate: Expression, dayOfWeek: Expression)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Hotfix][Examples] fix a Javadoc of a preforma...
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/7708#issuecomment-125471931
@srowen how about changing all the `` to `{@code...}`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125472010 [Test build #38675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38675/consoleFull) for PR 7718 at commit [`616a425`](https://github.com/apache/spark/commit/616a425d053665f6484aa78cbdfa0e6e790a2d26). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8881][SPARK-9260] Fix algorithm for sch...
Github user nishkamravi2 commented on the pull request: https://github.com/apache/spark/pull/7274#issuecomment-125471627 Hey @andrewor14, thanks for taking care of this! Sorry, couldn't respond sooner, was out for a couple of days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7720#issuecomment-125471286 [Test build #1216 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1216/consoleFull) for PR 7720 at commit [`d9757f5`](https://github.com/apache/spark/commit/d9757f54165a62494755708897b250ebb89a0756). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9225] [MLlib] LDASuite needs unit tests...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7620#issuecomment-125470862 [Test build #38676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38676/consoleFull) for PR 7620 at commit [`ad55665`](https://github.com/apache/spark/commit/ad55665f083025074bf3f350a1a66baab9d4a4be). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125470413 [Test build #38677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38677/consoleFull) for PR 7564 at commit [`7f463f8`](https://github.com/apache/spark/commit/7f463f81575b73c65f52757539ae30e813e851d5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9225] [MLlib] LDASuite needs unit tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7620#issuecomment-125470117 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125470071 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125470118 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9225] [MLlib] LDASuite needs unit tests...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7620#issuecomment-125470063 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125469955 OK I deleted that too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7720#issuecomment-125470087 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125470050 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125469952 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125470099 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125469705 [Test build #132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/132/consoleFull) for PR 6644 at commit [`d94a716`](https://github.com/apache/spark/commit/d94a7164174eb826b35a7b7caa8a3b9de240f5d2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125469560 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9225] [MLlib] LDASuite needs unit tests...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/7620#issuecomment-125469429 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125469505 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Hotfix][Examples] fix a Javadoc of a preforma...
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/7708#issuecomment-125469263
Agree, this is not a fix. The modern way to write code javadoc is ...
```
{@code
val foo = ...
}
```
... but for consistency with other javadoc this is good enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9388] [yarn] Make executor info log mes...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7706#issuecomment-125468830 Looks OK to me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7710#discussion_r35618064
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -874,15 +874,15 @@
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C
}
def matchSerDe(clause: Seq[ASTNode])
- : (Seq[(String, String)], String, Seq[(String, String)]) =
clause match {
+ : (Seq[(String, String)], Option[String], Seq[(String,
String)]) = clause match {
case Token("TOK_SERDEPROPS", propsClause) :: Nil =>
val rowFormat = propsClause.map {
case Token(name, Token(value, Nil) :: Nil) => (name,
value)
}
-(rowFormat, "", Nil)
+(rowFormat, None, Nil)
--- End diff --
We could but I feel that's a bit less clear and more error-prone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125468536 There's another sql.DatetimeExpressionsSuite, should be moved to sql.DateFunctionsSuite. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8003][SQL] Added virtual column support...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7478#discussion_r35618076
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -140,7 +142,14 @@ class SQLContext(@transient val sparkContext:
SparkContext)
// TODO how to handle the temp function per user session?
@transient
- protected[sql] lazy val functionRegistry: FunctionRegistry =
FunctionRegistry.builtin
+ protected[sql] lazy val functionRegistry: FunctionRegistry = {
--- End diff --
actually i think there is one more problem here: HiveContext doesn't load
whatever is added here. Can you add a unit test to UDFSuite in Hive package,
and fix this for Hive as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125468461 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125468479 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/7710#discussion_r35618000 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala --- @@ -68,9 +71,44 @@ case class ScriptTransformation( val errorStream = proc.getErrorStream val reader = new BufferedReader(new InputStreamReader(inputStream)) - val (outputSerde, outputSoi) = ioschema.initOutputSerDe(output) + // TODO make the 2048 configurable? --- End diff -- I see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-125468342 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125467977 [Test build #38673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38673/consoleFull) for PR 7711 at commit [`ef84a08`](https://github.com/apache/spark/commit/ef84a08f48afb4341317ac4900ec808b0c1389a2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125467679 [Test build #1215 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1215/consoleFull) for PR 7711 at commit [`ef84a08`](https://github.com/apache/spark/commit/ef84a08f48afb4341317ac4900ec808b0c1389a2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-9202] capping maximum number of ex...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7714#issuecomment-125467607 The basic approach looks okay to me, so this is on the right track. Thanks for choosing to work on this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-9202] capping maximum number of ex...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7714#discussion_r35617836
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -614,6 +621,22 @@ private[worker] class Worker(
webUi.stop()
metricsSystem.stop()
}
+
+ private def trimFinishedExecutorsIfNecessary(): Unit = {
+if (finishedExecutors.size > retainedExecutors) {
+ finishedExecutors.take(math.max(finishedExecutors.size / 10,
1)).foreach{
--- End diff --
Minor style nit: space after `foreach`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-9202] capping maximum number of ex...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7714#discussion_r35617851
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -614,6 +621,22 @@ private[worker] class Worker(
webUi.stop()
metricsSystem.stop()
}
+
+ private def trimFinishedExecutorsIfNecessary(): Unit = {
+if (finishedExecutors.size > retainedExecutors) {
+ finishedExecutors.take(math.max(finishedExecutors.size / 10,
1)).foreach{
+case (executorId, _) => finishedExecutors.remove(executorId)}
--- End diff --
Minor style nit: the closing `}` should be on its own line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125467508 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125467492 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-9202] capping maximum number of ex...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7714#issuecomment-125467324 In principle, doesn't the Master also have similar problems with retained applications? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125467341 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Streaming][Tests]Make sure the te...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7663 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9397] DataFrame should provide an API t...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7717#discussion_r35617769
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -1546,6 +1547,21 @@ class DataFrame private[sql](
}
}
+ /**
+ * Returns a best-effort snapshot of the files that compose this
DataFrame. This method simply
+ * asks each constituent BaseRelation for its respective files and takes
the union of all results.
+ * Depending on the source relations, this may not find all input paths.
Duplicates are removed.
+ */
+ def sourcePaths: Array[String] = {
--- End diff --
how about inputFiles?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7575][ml][doc] Example code for OneVsRe...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/6115#discussion_r35617723 --- Diff: examples/src/main/java/org/apache/spark/examples/ml/JavaOneVsRestExample.java --- @@ -0,0 +1,236 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.examples.ml; + +import org.apache.commons.cli.*; --- End diff -- Yeah, I suppose I couldn't recommend removing the flags now that they're here, or rewriting this without the library just to remove the dependency (it's just a test scope dependency and already comes in, evidently). Let's add it in https://github.com/apache/spark/pull/7697 instead, which is also using it. There, it might make more sense to curb the use of flags and make it more of an example; the issue here is that this example doesn't show me how to run it -- just says to set the options to something appropriate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7710#discussion_r35617688 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala --- @@ -68,9 +71,44 @@ case class ScriptTransformation( val errorStream = proc.getErrorStream val reader = new BufferedReader(new InputStreamReader(inputStream)) - val (outputSerde, outputSoi) = ioschema.initOutputSerDe(output) + // TODO make the 2048 configurable? --- End diff -- This is a carryover from the old code; I ended up moving a bunch of code around without updating comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
Github user JoshRosen closed the pull request at: https://github.com/apache/spark/pull/7719 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7720#issuecomment-125466905 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7720#issuecomment-125466910 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7719#issuecomment-125466862 Actually, I change my mind on this; there's no way that the `Option` can have any significant overhead given that it's always going to return `None`. Going to close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466750 [Test build #131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/131/consoleFull) for PR 7692 at commit [`612c18c`](https://github.com/apache/spark/commit/612c18c845f1a1840af316af92d43056cc85c502). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466892 [Test build #38671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38671/consoleFull) for PR 7692 at commit [`612c18c`](https://github.com/apache/spark/commit/612c18c845f1a1840af316af92d43056cc85c502). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466707 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7719#issuecomment-125466771 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7720#issuecomment-125466682 cc @JoshRosen this should fix all the problems you mentioned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/7710#discussion_r35617644
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala
---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.hadoop.hive.serde2.`lazy`.LazySimpleSerDe
+import org.scalatest.exceptions.TestFailedException
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.SQLContext
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
+import org.apache.spark.sql.execution.{UnaryNode, SparkPlan, SparkPlanTest}
+import org.apache.spark.sql.hive.test.TestHive
+import org.apache.spark.sql.types.StringType
+
+class ScriptTransformationSuite extends SparkPlanTest {
--- End diff --
Great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9373][SQL] follow up for StructType sup...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/7720 [SPARK-9373][SQL] follow up for StructType support in Tungsten projection. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark struct-followup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7720.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7720 commit d9757f54165a62494755708897b250ebb89a0756 Author: Reynold Xin Date: 2015-07-28T06:30:23Z [SPARK-9373][SQL] follow up for StructType support in Tungsten projection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466688 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Streaming][Tests]Make sure the te...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/7663#issuecomment-125466561 I am merging this fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/7710#discussion_r35617564 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala --- @@ -68,9 +71,44 @@ case class ScriptTransformation( val errorStream = proc.getErrorStream val reader = new BufferedReader(new InputStreamReader(inputStream)) - val (outputSerde, outputSoi) = ioschema.initOutputSerDe(output) + // TODO make the 2048 configurable? --- End diff -- Too much configurations is also not a good thing. I think it's fine here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/7710#discussion_r35617474
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -874,15 +874,15 @@
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C
}
def matchSerDe(clause: Seq[ASTNode])
- : (Seq[(String, String)], String, Seq[(String, String)]) =
clause match {
+ : (Seq[(String, String)], Option[String], Seq[(String,
String)]) = clause match {
case Token("TOK_SERDEPROPS", propsClause) :: Nil =>
val rowFormat = propsClause.map {
case Token(name, Token(value, Nil) :: Nil) => (name,
value)
}
-(rowFormat, "", Nil)
+(rowFormat, None, Nil)
--- End diff --
Could we use `null` here without changing the type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user carsonwang commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466146 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7719#issuecomment-125466205 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7719#issuecomment-125466217 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7710#discussion_r35617428
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanTest.scala ---
@@ -98,7 +100,7 @@ class SparkPlanTest extends SparkFunSuite {
planFunction: Seq[SparkPlan] => SparkPlan,
expectedAnswer: Seq[Row],
sortAnswers: Boolean = true): Unit = {
-SparkPlanTest.checkAnswer(input, planFunction, expectedAnswer,
sortAnswers) match {
+SparkPlanTest.checkAnswer(input, planFunction, expectedAnswer,
sortAnswers, sqlContext) match {
--- End diff --
This is a call on the `SparkPlanTest` companion object, which doesn't have
that field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466225 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2016][WebUI]RDD partition table paginat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7692#issuecomment-125466209 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/7710#discussion_r35617380
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanTest.scala ---
@@ -98,7 +100,7 @@ class SparkPlanTest extends SparkFunSuite {
planFunction: Seq[SparkPlan] => SparkPlan,
expectedAnswer: Seq[Row],
sortAnswers: Boolean = true): Unit = {
-SparkPlanTest.checkAnswer(input, planFunction, expectedAnswer,
sortAnswers) match {
+SparkPlanTest.checkAnswer(input, planFunction, expectedAnswer,
sortAnswers, sqlContext) match {
--- End diff --
If `sqlContext` is already a member of SparkPlanTest, is it possible that
we don't need to pass it to `checkAnswer`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125465865 [Test build #38669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38669/consoleFull) for PR 7718 at commit [`faa78cf`](https://github.com/apache/spark/commit/faa78cf8bfed740fead9b6c163185174fe5811f4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9394][SQL] Handle parentheses in CodeFo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7712#issuecomment-125465630 [Test build #1214 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1214/consoleFull) for PR 7712 at commit [`c2b1c5f`](https://github.com/apache/spark/commit/c2b1c5f6269fb1263205ca831341d3cc321d2561). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125465377 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125465336 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9399] Minor performance optimizations i...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/7719 [SPARK-9399] Minor performance optimizations in PythonRDD This patch implements three trivial performance optimizations in PythonRDD: - Replace a .`foreach()` call with a while() loop. - Returns nulls instead of Options to avoid allocating a new Option once per read() call. - Call `.size()` instead of `.length()`, thereby avoiding an implicit Java -> Scala collections conversion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark pyspark-read-optimization Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7719.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7719 commit 13f7e2f65fc23f0bd5fcbb3a5f449479138f37c5 Author: Josh Rosen Date: 2015-07-28T06:19:20Z Minor optimizations in PythonRDD --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125465150 cc @adrian-wang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8232][SQL] Add sort_array support
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/7581#discussion_r35617061 --- Diff: python/pyspark/sql/functions.py --- @@ -840,6 +841,20 @@ def size(col): return Column(sc._jvm.functions.size(_to_java_column(col))) +@since(1.5) +def sort_array(col): +""" +Collection function: sorts the input array for the given column in ascending order. --- End diff -- Are you still working on this? If not, we could leave this a follow up PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125464826 [Test build #38668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38668/consoleFull) for PR 7718 at commit [`6c4fb6a`](https://github.com/apache/spark/commit/6c4fb6a3ab164f386a8a2d2c0a9c49c5030d02a5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125464710 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user suyanNone commented on a diff in the pull request: https://github.com/apache/spark/pull/6644#discussion_r35617005 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -60,7 +60,8 @@ class ExecutorSummary private[spark]( val totalShuffleRead: Long, val totalShuffleWrite: Long, val maxMemory: Long, -val executorLogs: Map[String, String]) +val executorLogs: Map[String, String], +val isRemoved: Boolean) --- End diff -- Ok, I will refine that, and it's nice to tell me about MimaExcludes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125464538 [Test build #38667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38667/consoleFull) for PR 7564 at commit [`8946cb9`](https://github.com/apache/spark/commit/8946cb96cbe1c435198891432648fc4c262a583a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125464692 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7718#issuecomment-125464673 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8196][SQL] Fix null handling & document...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/7718 [SPARK-8196][SQL] Fix null handling & documentation for next_day. The original patch didn't handle nulls correctly for next_day. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark next_day Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7718.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7718 commit 6c4fb6a3ab164f386a8a2d2c0a9c49c5030d02a5 Author: Reynold Xin Date: 2015-07-28T06:16:25Z [SPARK-8196][SQL] Fix null handling & documentation for next_day. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125463318 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9225] [MLlib] LDASuite needs unit tests...
Github user rotationsymmetry commented on the pull request: https://github.com/apache/spark/pull/7620#issuecomment-125463408 @mengxr Thank you for OK to test. I checked the Jerkins log. The failure is caused by git unable to fetch the repo. Anything on my end to fix it? cc @feynmanliang @yu-iskw From Jenkins log: ``` GitHub pull request #7620 of commit ad55665f083025074bf3f350a1a66baab9d4a4be automatically merged. [EnvInject] - Loading node environment variables. Building remotely on amp-jenkins-worker-07 (centos) in workspace /home/jenkins/workspace/SparkPullRequestBuilder > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/spark.git # timeout=10 Fetching upstream changes from https://github.com/apache/spark.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/7620/*:refs/remotes/origin/pr/7620/* # timeout=15 ERROR: Timeout after 15 minutes ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016) at hudson.scm.SCM.checkout(SCM.java:485) at hudson.model.AbstractProject.checkout(AbstractProject.java:1282) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:381) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/7620/*:refs/remotes/origin/pr/7620/*" returned status code 143: stdout: stderr: error: RPC failed; result=18, HTTP code = 200 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1591) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1379) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:325) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ..remote call to amp-jenkins-worker-07(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1361) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:753) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145) at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131) at com.sun.proxy.$Proxy54.execute(Unknown Source) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733) ... 11 more ERROR: Error fetching remote repo 'origin' Archiving artifacts Recording test results ERROR: Publisher 'Publish JUnit test result report' failed: Test reports were found but none of them are new. Did tests run? For example, /home/jenkins/workspace/SparkPullRequestBuilder/bagel/target/test-reports/org.apache.spark.bagel.BagelSuite.xml is 2 hr 19 min old Finished: FAILURE ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125463378 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8232][SQL] Add sort_array support
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/7581#discussion_r35616625
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -35,3 +36,45 @@ case class Size(child: Expression) extends
UnaryExpression with ExpectsInputType
nullSafeCodeGen(ctx, ev, c => s"${ev.primitive} = ($c).size();")
}
}
+
+/**
+ * Sorts the input array in ascending order according to the natural
ordering of
+ * the array elements and returns it.
+ */
+case class SortArray(child: Expression)
+ extends UnaryExpression with ExpectsInputTypes with CodegenFallback {
+
+ override def dataType: DataType = child.dataType
+ override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType)
+
+ override def checkInputDataTypes(): TypeCheckResult = child.dataType
match {
+case _ @ ArrayType(n: AtomicType, _) =>
TypeCheckResult.TypeCheckSuccess
+case other => TypeCheckResult.TypeCheckFailure(
--- End diff --
`other` could be ArrayType of non-atomic types, or not a ArrayType, we
could be have different error messages for them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8232][SQL] Add sort_array support
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/7581#discussion_r35616590
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -35,3 +36,45 @@ case class Size(child: Expression) extends
UnaryExpression with ExpectsInputType
nullSafeCodeGen(ctx, ev, c => s"${ev.primitive} = ($c).size();")
}
}
+
+/**
+ * Sorts the input array in ascending order according to the natural
ordering of
+ * the array elements and returns it.
+ */
+case class SortArray(child: Expression)
+ extends UnaryExpression with ExpectsInputTypes with CodegenFallback {
+
+ override def dataType: DataType = child.dataType
+ override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType)
+
+ override def checkInputDataTypes(): TypeCheckResult = child.dataType
match {
--- End diff --
Once we have `checkInputDataTypes`, do we still need `ExpectsInputTypes`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125461403 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9335][Tests]Enable Kinesis tests only w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7711#issuecomment-125461335 **[Test build #38646 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38646/console)** for PR 7711 at commit [`f691028`](https://github.com/apache/spark/commit/f6910284aa02d3cb85b0d946e9afc14e188c22c0) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9393] [SQL] Fix several error-handling ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7710#issuecomment-125460983 @davies, it might be good to have you take a look at this given the similarity of this code to PySpark's similar error-handling techniques. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR][SQL]Support mutable expression unit te...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7566 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125460683 Lots of changes are temporarily merged in here while I test on top of Reynold's struct type patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125460656 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8850] [SQL] [WIP] Enable Unsafe mode by...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7564#issuecomment-125460669 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR][SQL]Support mutable expression unit te...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/7566#issuecomment-125460646 Merged into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9397] DataFrame should provide an API t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7717#issuecomment-125460590 [Test build #38665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38665/consoleFull) for PR 7717 at commit [`0acd3ad`](https://github.com/apache/spark/commit/0acd3ad929f18706e05b5deba0e5acae8067c7a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9368][SQL] Support get(ordinal, dataTyp...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7682#discussion_r35616321
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
---
@@ -236,6 +236,41 @@ public Object get(int ordinal) {
}
@Override
+ public Object get(int ordinal, DataType dataType) {
+if (dataType instanceof NullType) {
--- End diff --
Actually, it looks like our existing row behavior is to just return the
zero-value of the given type for null inputs (e.g. getFloat on a null column
returns 0.0f whereas the generic getter returns null). For some reason, it
looks like UnsafeRow was returning NaN instead of 0 in those cases, leading to
a confusing bug. I'm going to fix this inconsistency in a separate patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
