[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93374/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)** for PR 21488 at commit [`1738642`](https://github.com/apache/spark/commit/17386429150d26d838f6895ec9698b7176765ffc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21764: [SPARK-24802] Optimization Rule Exclusion
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21764#discussion_r204202735
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -175,6 +182,44 @@ abstract class Optimizer(sessionCatalog:
SessionCatalog)
* Override to provide additional rules for the operator optimization
batch.
*/
def extendedOperatorOptimizationRules: Seq[Rule[LogicalPlan]] = Nil
+
+ override def batches: Seq[Batch] = {
+val excludedRules =
+
SQLConf.get.optimizerExcludedRules.toSeq.flatMap(_.split(",").map(_.trim).filter(!_.isEmpty))
--- End diff --
You can use `Utils.stringToSeq`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93373/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21802 **[Test build #93373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93373/testReport)** for PR 21802 at commit [`2ca1230`](https://github.com/apache/spark/commit/2ca12302e08d60ab9534d7d65fad9854fe1d6f28). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class RandomIndicesGenerator(randomSeed: Long) ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20861: [SPARK-23599][SQL] Use RandomUUIDGenerator in Uui...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20861#discussion_r204201930
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1994,6 +1996,20 @@ class Analyzer(
}
}
+ /**
+ * Set the seed for random number generation in Uuid expressions.
+ */
+ object ResolvedUuidExpressions extends Rule[LogicalPlan] {
+private lazy val random = new Random()
+
+override def apply(plan: LogicalPlan): LogicalPlan = plan.transformUp {
+ case p if p.resolved => p
+ case p => p transformExpressionsUp {
+case Uuid(None) => Uuid(Some(random.nextLong()))
--- End diff --
hmm, if we want to make it deterministic between re-tries of same query. I
think we should do it. I can make a PR for it, WDYT?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21826 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21826 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93378/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21826 **[Test build #93378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93378/testReport)** for PR 21826 at commit [`fb98029`](https://github.com/apache/spark/commit/fb98029c451023789a2c7fa0e758c6c8790bbaea). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21827: [SPARK-24873]Increase switch to shielding frequent inter...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21827 @hejiefang, looks indeed a duplicate. Mind closing this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 Also, mind adding `[SPARK-24873]` in the PR title since the JIRA happened to be open anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @wangyum, mind adding Closes #21784 here? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21826 **[Test build #93378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93378/testReport)** for PR 21826 at commit [`fb98029`](https://github.com/apache/spark/commit/fb98029c451023789a2c7fa0e758c6c8790bbaea). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21784: [SPARK-24182][YARN][FOLLOW-UP] Turn off noisy log output
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21784 @vanzin WDYT about this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21826 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21826 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21826 Looks it's gonna make a compilation failure and I see potential references referring this field. @httfighter, I think manual build and tests are required. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Test failure was in Hive test, not related to this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21826: [SPARK-24872] Remove the symbol “||” of the “OR”...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21826 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93372/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)** for PR 21488 at commit [`241878c`](https://github.com/apache/spark/commit/241878c886f206dabc44fd5d55d3fe6908a35a3b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93370/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21822 **[Test build #93370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93370/testReport)** for PR 21822 at commit [`38980ad`](https://github.com/apache/spark/commit/38980ad066d26327387673910e0dfd981102cab9). * This patch **fails from timeout after a configured wait of \`300m\`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...
Github user Achuth17 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21608#discussion_r204200662
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
---
@@ -47,15 +48,27 @@ object CommandUtils extends Logging {
}
}
- def calculateTotalSize(sessionState: SessionState, catalogTable:
CatalogTable): BigInt = {
+ def calculateTotalSize(spark: SparkSession, catalogTable: CatalogTable):
BigInt = {
+val sessionState = spark.sessionState
if (catalogTable.partitionColumnNames.isEmpty) {
calculateLocationSize(sessionState, catalogTable.identifier,
catalogTable.storage.locationUri)
} else {
// Calculate table size as a sum of the visible partitions. See
SPARK-21079
val partitions =
sessionState.catalog.listPartitions(catalogTable.identifier)
- partitions.map { p =>
-calculateLocationSize(sessionState, catalogTable.identifier,
p.storage.locationUri)
- }.sum
+ val paths = partitions.map(x => new Path(x.storage.locationUri.get))
+ val stagingDir =
sessionState.conf.getConfString("hive.exec.stagingdir", ".hive-staging")
+ val pathFilter = new PathFilter with Serializable {
+override def accept(path: Path): Boolean = {
+ val fileName = path.getName
+ (!fileName.startsWith(stagingDir) &&
+// Ignore metadata files starting with "_"
+!fileName.startsWith("_"))
--- End diff --
Done.
Also, we are not doing this check when `calculateLocationSize` is called
directly. I will file a different PR for this as this is not related to
AnalyzeTableCommand.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #93377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93377/testReport)** for PR 21608 at commit [`27b68d3`](https://github.com/apache/spark/commit/27b68d3a561001cfd0ab85fd41abb8ef11fc5105). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20146 @HyukjinKwon Yeah, looks like re-triggering the AppVeyor build passes. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21798: [SPARK-24836][SQL] New option for Avro datasource...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21798 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21798: [SPARK-24836][SQL] New option for Avro datasource - igno...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21798 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21832: [SPARK-24879][SQL] Fix NPE in Hive partition prun...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21832 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21832: [SPARK-24879][SQL] Fix NPE in Hive partition pruning fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21832 LGTM Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21320 @mallman I still think we need to split it to two PRs. To resolve the issues you mentioned above, how about creating a separate PR? Only 10 days left before the code freeze of Spark 2.4. We plan to merge the main logic of nested column pruning to Spark 2.4 release first and then address the other parts in the next release. WDYT? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTes...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21833 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21653: [SPARK-13343] speculative tasks that didn't commi...
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21653#discussion_r204199580
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -723,6 +723,21 @@ private[spark] class TaskSetManager(
def handleSuccessfulTask(tid: Long, result: DirectTaskResult[_]): Unit =
{
val info = taskInfos(tid)
val index = info.index
+// Check if any other attempt succeeded before this and this attempt
has not been handled
+if (successful(index) && killedByOtherAttempt.contains(tid)) {
+ calculatedTasks -= 1
+
+ val resultSizeAcc = result.accumUpdates.find(a =>
+a.name == Some(InternalAccumulator.RESULT_SIZE))
+ if (resultSizeAcc.isDefined) {
+totalResultSize -=
resultSizeAcc.get.asInstanceOf[LongAccumulator].value
--- End diff --
I agree, I dont see a better option.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21831 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93368/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21831 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21831 **[Test build #93368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93368/testReport)** for PR 21831 at commit [`980d30c`](https://github.com/apache/spark/commit/980d30c8964c92f3965e725063fd27b5c4e60922). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21823: [SPARK-24870][SQL]Cache can't work normally if there are...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21823 **[Test build #93376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93376/testReport)** for PR 21823 at commit [`f2091a4`](https://github.com/apache/spark/commit/f2091a45b88b0a1bc57ec2ec9cf91a915827). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21823: [SPARK-24870][SQL]Cache can't work normally if there are...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21823 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21823: [SPARK-24870][SQL]Cache can't work normally if th...
Github user eatoncys commented on a diff in the pull request:
https://github.com/apache/spark/pull/21823#discussion_r204199119
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SameResultSuite.scala ---
@@ -58,4 +61,16 @@ class SameResultSuite extends QueryTest with
SharedSQLContext {
val df4 = spark.range(10).agg(sumDistinct($"id"))
assert(df3.queryExecution.executedPlan.sameResult(df4.queryExecution.executedPlan))
}
+
+ test("Canonicalized result is not case-insensitive") {
+val a = AttributeReference("A", IntegerType)()
+val b = AttributeReference("B", IntegerType)()
+val planUppercase = Project(Seq(a, b), LocalRelation(a))
--- End diff --
Ok,thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21823: [SPARK-24870][SQL]Cache can't work normally if there are...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21823 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1196/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1195/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21822 **[Test build #93375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93375/testReport)** for PR 21822 at commit [`38980ad`](https://github.com/apache/spark/commit/38980ad066d26327387673910e0dfd981102cab9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21822 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user guozhangwang commented on the issue: https://github.com/apache/spark/pull/21488 > 1.1.1 has been released, maybe we can upgrade to that. +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93367/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19194: [SPARK-20589] Allow limiting task concurrency per stage
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19194 **[Test build #93367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93367/testReport)** for PR 19194 at commit [`aac8a6a`](https://github.com/apache/spark/commit/aac8a6a619c8d60f66f9ddb072e0c4f9a7782621). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93366/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21822 **[Test build #93366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93366/testReport)** for PR 21822 at commit [`38980ad`](https://github.com/apache/spark/commit/38980ad066d26327387673910e0dfd981102cab9). * This patch **fails from timeout after a configured wait of \`300m\`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21320: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21320 > Could we move the changes made in ParquetReadSupport.scala to a separate PR? Then, we can merge this PR very quickly. If I remove the changes to `ParquetReadSupport.scala`, then four tests fail in `ParquetSchemaPruningSuite.scala`. I don't think we should/can proceed without addressing the issue of reading from two parquet files with identical column names and types but different ordering of those columns in their respective file schema. Personally, I think the fact that the Spark parquet reader appears to assume the same column order in otherwise compatible schema across files is a bug. I think column selection should be by name, not index. The parquet-mr reader behaves that way. As a stop-gap alternative, I suppose we could disable the built-in reader if parquet schema pruning is turned on. But I think that would be a rather ugly, invasive and confusing hack. Of course I'm open to other ideas as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1194/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)** for PR 21488 at commit [`1738642`](https://github.com/apache/spark/commit/17386429150d26d838f6895ec9698b7176765ffc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Thanks for the reminder, @ijuma Updated pom.xml and title accordingly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 1.1.1 has been released, maybe we can upgrade to that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19999: JDBC support date/timestamp type as partitionColumn
Github user maropu commented on the issue: https://github.com/apache/spark/pull/1 ok, I will. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21608#discussion_r204196291
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
---
@@ -47,15 +48,27 @@ object CommandUtils extends Logging {
}
}
- def calculateTotalSize(sessionState: SessionState, catalogTable:
CatalogTable): BigInt = {
+ def calculateTotalSize(spark: SparkSession, catalogTable: CatalogTable):
BigInt = {
+val sessionState = spark.sessionState
if (catalogTable.partitionColumnNames.isEmpty) {
calculateLocationSize(sessionState, catalogTable.identifier,
catalogTable.storage.locationUri)
} else {
// Calculate table size as a sum of the visible partitions. See
SPARK-21079
val partitions =
sessionState.catalog.listPartitions(catalogTable.identifier)
- partitions.map { p =>
-calculateLocationSize(sessionState, catalogTable.identifier,
p.storage.locationUri)
- }.sum
+ val paths = partitions.map(x => new Path(x.storage.locationUri.get))
+ val stagingDir =
sessionState.conf.getConfString("hive.exec.stagingdir", ".hive-staging")
+ val pathFilter = new PathFilter with Serializable {
+override def accept(path: Path): Boolean = {
+ val fileName = path.getName
+ (!fileName.startsWith(stagingDir) &&
+// Ignore metadata files starting with "_"
+!fileName.startsWith("_"))
--- End diff --
How about `DataSourceUtils`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21802 **[Test build #93373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93373/testReport)** for PR 21802 at commit [`2ca1230`](https://github.com/apache/spark/commit/2ca12302e08d60ab9534d7d65fad9854fe1d6f28). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1193/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21802: [SPARK-23928][SQL] Add shuffle collection function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21802 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21832: [SPARK-24879][SQL] Fix NPE in Hive partition pruning fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21832 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93369/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21832: [SPARK-24879][SQL] Fix NPE in Hive partition pruning fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21832 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21832: [SPARK-24879][SQL] Fix NPE in Hive partition pruning fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21832 **[Test build #93369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93369/testReport)** for PR 21832 at commit [`ce86fbe`](https://github.com/apache/spark/commit/ce86fbeda06eb2448ecd2c425982aacca3d66b45). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21826: [SPARK-24872] Remove the symbol “||” of the �...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21826#discussion_r204192991 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -442,8 +442,6 @@ case class Or(left: Expression, right: Expression) extends BinaryOperator with P override def inputType: AbstractDataType = BooleanType - override def symbol: String = "||" --- End diff -- If you remove it, it will not compile. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)** for PR 21488 at commit [`241878c`](https://github.com/apache/spark/commit/241878c886f206dabc44fd5d55d3fe6908a35a3b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1192/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Ryan: Thanks for the reminder. I have disabled that test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21118 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93365/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21118 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21118: SPARK-23325: Use InternalRow when reading with DataSourc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21118 **[Test build #93365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93365/testReport)** for PR 21118 at commit [`d1fa32e`](https://github.com/apache/spark/commit/d1fa32e201e73f281a87d46a3510f0e3082c1d35). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21831 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21748: [SPARK-23146][K8S] Support client mode.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21748 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21748: [SPARK-23146][K8S] Support client mode.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21748 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93357/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21831 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93362/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21748: [SPARK-23146][K8S] Support client mode.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21748 **[Test build #93357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93357/testReport)** for PR 21748 at commit [`086747e`](https://github.com/apache/spark/commit/086747e12f0af16c3479b07e59934d42ced4004b). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21831: [SPARK-24880][BUILD]Fix the group id for spark-kubernete...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21831 **[Test build #93362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93362/testReport)** for PR 21831 at commit [`4345139`](https://github.com/apache/spark/commit/4345139cd45e1506ac788dc55a4d9ed420ca6b78). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21826: [SPARK-24872] Remove the symbol “||” of the �...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21826#discussion_r204190916 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -442,8 +442,6 @@ case class Or(left: Expression, right: Expression) extends BinaryOperator with P override def inputType: AbstractDataType = BooleanType - override def symbol: String = "||" --- End diff -- I think this won't be compiled? ``` class Or needs to be abstract, since method symbol in class BinaryOperator of type => String is not defined ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r204190696 --- Diff: dev/create-release/releaseutils.py --- @@ -149,7 +152,11 @@ def get_commits(tag): if not is_valid_author(author): author = github_username # Guard against special characters -author = unidecode.unidecode(unicode(author, "UTF-8")).strip() +try: # Python 2 +author = unicode(author, "UTF-8") +except NameError: # Python 3 +author = str(author) +author = unidecode.unidecode(author).strip() --- End diff -- My thought was that we are first casting `author` this to unicode already with `unicode(author)` and it doesn't really matter if it is "UTF-8" or not because we then immediately decode it into ASCII with `unidecode`, which can handle it even it it wasn't "UTF-8", so the end result should be the same I believe. It was just to clean up a little, so not a big deal either way. The way it is now replicates the old behavior, so it's probably safer. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/21822#discussion_r204190535
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -231,10 +231,11 @@ class Analyzer(
* Substitute child plan with WindowSpecDefinitions.
*/
object WindowsSubstitution extends Rule[LogicalPlan] {
-def apply(plan: LogicalPlan): LogicalPlan = plan.transformUp {
+def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperators {
// Lookup WindowSpecDefinitions. This rule works with unresolved
children.
case WithWindowDefinition(windowDefinitions, child) =>
-child.transform {
+// TODO(rxin): Check with Herman whether the next line is OK.
--- End diff --
It is good. The earlier `resolveOperators` makes sure we don't overwrite a
window spec, with a similarly named one defined higher up the tree. BTW I don't
think we have a test that covers this (it is pretty rare).
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @tedyu I forgot one place: https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala#L32 Could you also disable it? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21676: [SPARK-24699][SS][WIP] Watermark / Append mode should wo...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21676 hey @c-horn , I am ready to merge your PR, and to add you as coauthor i think i need to know your email address i the github account. Can you provide me that? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @tedyu could you update the PR title and description to reflect the latest changes, such as `[SPARK-18057][SS] Update Kafka client version from 0.10.0.1 to 1.1.0`? Otherwise, LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 Okey. In Kafka 1.1.0, deleting a topic when a Kafka client is running may make the client hang at this line forever: https://github.com/apache/kafka/blob/1.1.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1428 The fix is https://issues.apache.org/jira/browse/KAFKA-6979. Before we upgrade to Kafka 2.0.0, we have to ignore these tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTests
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21833 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTests
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21833 **[Test build #93371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93371/testReport)** for PR 21833 at commit [`c4f664b`](https://github.com/apache/spark/commit/c4f664bd49f701773ea52751ee135915af973014). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTests
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93371/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21508: [SPARK-24488] [SQL] Fix issue when generator is a...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21508 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20272: [SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to ...
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20272 @felixcheung I think yes and with https://github.com/apache/spark/pull/21748, users should be able to run the Thrift server in a pod. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTests
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21833 **[Test build #93371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93371/testReport)** for PR 21833 at commit [`c4f664b`](https://github.com/apache/spark/commit/c4f664bd49f701773ea52751ee135915af973014). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17520: [WIP][SPARK-19712][SQL] Move PullupCorrelatedPredicates ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17520 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21746 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17520: [WIP][SPARK-19712][SQL] Move PullupCorrelatedPredicates ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17520 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1191/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21833: [PYSPARK] [TEST] [MINOR] Fix UDFInitializationTests
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21833 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
