[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17711 **[Test build #76020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76020/testReport)** for PR 17711 at commit [`72d1ae1`](https://github.com/apache/spark/commit/72d1ae191c64d5f8063750ff64daef6c91dfacf4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17711: [SPARK-19951][SQL] Add string concatenate operato...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/17711#discussion_r112627257
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1483,4 +1483,16 @@ class SparkSqlAstBuilder(conf: SQLConf) extends
AstBuilder {
query: LogicalPlan): LogicalPlan = {
RepartitionByExpression(expressions, query, conf.numShufflePartitions)
}
+
+ /**
+ * Create a [[Concat]] expression for pipeline concatenation.
+ */
+ override def visitConcat(ctx: ConcatContext): Expression = {
--- End diff --
Can you move this to the `CatalystSqlParser`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112627226
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
okay, I'll fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17688: [MINOR][DOCS][PYTHON] Adding missing boolean type for re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17688 **[Test build #3669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3669/testReport)** for PR 17688 at commit [`acae068`](https://github.com/apache/spark/commit/acae068bcb6523d4296d18752523b34625169fd7). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17688: [MINOR][DOCS][PYTHON] Adding missing boolean type for re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17688 **[Test build #3669 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3669/testReport)** for PR 17688 at commit [`acae068`](https://github.com/apache/spark/commit/acae068bcb6523d4296d18752523b34625169fd7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17709: Small rewording about history server use case
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17709 **[Test build #3668 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3668/testReport)** for PR 17709 at commit [`2560555`](https://github.com/apache/spark/commit/2560555a32159987b545b9965d83812711231e0e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/14638 Also a useful feature for me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17711 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17682: [SPARK-20385][WEB-UI]'Submitted Time' field, the date fo...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17682 @guoxiaolongzte, I just tested the backquotes on Windows with Chrome as below:   It is a backquote `` ` `` not a single-quote `'`. Could you try again? I see you made many PRs and I would like them to be more readable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17684: [SPARK-20341][SQL] Support BigInt's value that do...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17684#discussion_r112623415
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -132,14 +132,22 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
}
/**
- * Set this Decimal to the given BigInteger value. Will have precision
38 and scale 0.
+ * Set this Decimal to the given BigInteger value. Will have precision
38 and scale 0
+ * if value is fit into long value range. Otherwise, use BigDecimal
--- End diff --
Thanks, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17710: [SPARK-20420][SQL] Add events to the external cat...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17710 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17684: [SPARK-20341][SQL] Support BigInt's value that does not ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17684 **[Test build #76019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76019/testReport)** for PR 17684 at commit [`6425239`](https://github.com/apache/spark/commit/6425239bf6ffad6319e18225b6311cabd622dc54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17711 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76016/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17706: [SPARK-20423][ML] fix MLOR coeffs centering when reg == ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17706 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17711 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17706: [SPARK-20423][ML] fix MLOR coeffs centering when reg == ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17706 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76018/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17700: [SPARK-20391][Core] Rename memory related fields in Exec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17700 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76015/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17700: [SPARK-20391][Core] Rename memory related fields in Exec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17700 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17710: [SPARK-20420][SQL] Add events to the external catalog
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17710 Merging in master/branch-2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17711 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112622098
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala
---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
can we create a new instance instead so this is immutable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76014/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17711 **[Test build #76014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76014/testReport)** for PR 17711 at commit [`0701f87`](https://github.com/apache/spark/commit/0701f876a6d05513b8ca25d521f0370a4621a15a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15078: [SPARK-17523] Support to generate build info file when b...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15078 @guoxiaolongzte, Let's ask a question to mailing list. Up to my knowledge, we should build first before running tests properly and maybe the errors are related with it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17684: [SPARK-20341][SQL] Support BigInt's value that does not ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17684 LGTM except one comment. cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17684: [SPARK-20341][SQL] Support BigInt's value that do...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17684#discussion_r112620153
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -132,14 +132,22 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
}
/**
- * Set this Decimal to the given BigInteger value. Will have precision
38 and scale 0.
+ * Set this Decimal to the given BigInteger value. Will have precision
38 and scale 0
+ * if value is fit into long value range. Otherwise, use BigDecimal
--- End diff --
How about
> If the value is not in the range of long, convert it to BigDecimal and
the precision and scale is based on the converted value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15078: [SPARK-17523] Support to generate build info file when b...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/15078 **In Linux** mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.rest.StandaloneRestSubmitSuite test I want to know why?thank you!@HyukjinKwon **Report errors:** StandaloneRestSubmitSuite: *** RUN ABORTED *** java.lang.ExceptionInInitializerError: at org.apache.spark.package$.(package.scala:91) at org.apache.spark.package$.(package.scala) at org.apache.spark.deploy.rest.RestSubmissionClient.constructSubmitRequest(RestSubmissionClient.scala:179) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply$mcV$sp(StandaloneRestSubmitSuite.scala:58) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply(StandaloneRestSubmitSuite.scala:54) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply(StandaloneRestSubmitSuite.scala:54) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) ... Cause: org.apache.spark.SparkException: Error while locating file spark-version-info.properties at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:75) at org.apache.spark.package$SparkBuildInfo$.(package.scala:61) at org.apache.spark.package$SparkBuildInfo$.(package.scala) at org.apache.spark.package$.(package.scala:91) at org.apache.spark.package$.(package.scala) at org.apache.spark.deploy.rest.RestSubmissionClient.constructSubmitRequest(RestSubmissionClient.scala:179) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply$mcV$sp(StandaloneRestSubmitSuite.scala:58) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply(StandaloneRestSubmitSuite.scala:54) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply(StandaloneRestSubmitSuite.scala:54) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) ... Cause: java.lang.NullPointerException: at java.util.Properties$LineReader.readLine(Properties.java:434) at java.util.Properties.load0(Properties.java:353) at java.util.Properties.load(Properties.java:341) at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:64) at org.apache.spark.package$SparkBuildInfo$.(package.scala:61) at org.apache.spark.package$SparkBuildInfo$.(package.scala) at org.apache.spark.package$.(package.scala:91) at org.apache.spark.package$.(package.scala) at org.apache.spark.deploy.rest.RestSubmissionClient.constructSubmitRequest(RestSubmissionClient.scala:179) at org.apache.spark.deploy.rest.StandaloneRestSubmitSuite$$anonfun$1.apply$mcV$sp(StandaloneRestSubmitSuite.scala:58) ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17706: [SPARK-20423][ML] fix MLOR coeffs centering when reg == ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17706 **[Test build #76018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76018/testReport)** for PR 17706 at commit [`a694d13`](https://github.com/apache/spark/commit/a694d13095a37c8fbcdea2ed51ca01569f4ef6af). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operat...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17540#discussion_r112615142
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala ---
@@ -353,11 +353,19 @@ class SQLListener(conf: SparkConf) extends
SparkListener with Logging {
}
val driverUpdates = executionUIData.driverAccumUpdates.toSeq
-val totalUpdates = (accumulatorUpdates ++ driverUpdates).filter {
- case (id, _) => executionUIData.accumulatorMetrics.contains(id)
+// filter out updates that aren't Longs
+val totalUpdates = (accumulatorUpdates ++ driverUpdates).filter {
update =>
+ update._2.isInstanceOf[Long]
}
+// Hack alert!
+// metrics may be missing from the executionUIData because metrics
are taken from the
+// SparkPlan passed to withNewExecutionId, but some nodes in that
plan link in LogicalPlan
+// nodes that don't expose the actual SparkPlan or metrics because
the SparkPlan is built
--- End diff --
If these metrics are not linked with any `SparkPlan`s, will them be showed
in Spark UI?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112614624 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -83,7 +83,8 @@ class KinesisBackedBlockRDD[T: ClassTag]( @transient private val isBlockIdValid: Array[Boolean] = Array.empty, val retryTimeoutMs: Int = 1, val messageHandler: Record => T = KinesisInputDStream.defaultMessageHandler _, -val kinesisCreds: SparkAWSCredentials = DefaultCredentials +val kinesisCreds: SparkAWSCredentials = DefaultCredentials, +val kinesisConf: Map[String, String] = Map.empty --- End diff -- +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17596: [SPARK-12837][CORE] Do not send the accumulator name to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17596 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17596: [SPARK-12837][CORE] Do not send the accumulator name to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76017/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17596: [SPARK-12837][CORE] Do not send the accumulator name to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17596 **[Test build #76017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76017/testReport)** for PR 17596 at commit [`e8633f9`](https://github.com/apache/spark/commit/e8633f9ef48638ea241a137f9a1608256f435152). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AccumulatorMetadata(id: Long, countFailedValues: Boolean) extends Serializable` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...
Github user ouyangxiaochen commented on the issue: https://github.com/apache/spark/pull/17681 OK. After that, leave me a message, i will work on this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
