[GitHub] [spark] mccheah commented on a change in pull request #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
mccheah commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r306562340
##
File path:
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
##
@@ -99,74 +104,82 @@
BypassMergeSortShuffleWriter(
BlockManager blockManager,
- IndexShuffleBlockResolver shuffleBlockResolver,
BypassMergeSortShuffleHandle handle,
int mapId,
+ long mapTaskAttemptId,
SparkConf conf,
- ShuffleWriteMetricsReporter writeMetrics) {
+ ShuffleWriteMetricsReporter writeMetrics,
+ ShuffleWriteSupport shuffleWriteSupport) {
// Use getSizeAsKb (not bytes) to maintain backwards compatibility if no
units are provided
this.fileBufferSize = (int) (long)
conf.get(package$.MODULE$.SHUFFLE_FILE_BUFFER_SIZE()) * 1024;
this.transferToEnabled = conf.getBoolean("spark.file.transferTo", true);
this.blockManager = blockManager;
final ShuffleDependency dep = handle.dependency();
this.mapId = mapId;
+this.mapTaskAttemptId = mapTaskAttemptId;
this.shuffleId = dep.shuffleId();
this.partitioner = dep.partitioner();
this.numPartitions = partitioner.numPartitions();
this.writeMetrics = writeMetrics;
this.serializer = dep.serializer();
-this.shuffleBlockResolver = shuffleBlockResolver;
+this.shuffleWriteSupport = shuffleWriteSupport;
}
@Override
public void write(Iterator> records) throws IOException {
assert (partitionWriters == null);
-if (!records.hasNext()) {
- partitionLengths = new long[numPartitions];
- shuffleBlockResolver.writeIndexFileAndCommit(shuffleId, mapId,
partitionLengths, null);
- mapStatus = MapStatus$.MODULE$.apply(blockManager.shuffleServerId(),
partitionLengths);
- return;
-}
-final SerializerInstance serInstance = serializer.newInstance();
-final long openStartTime = System.nanoTime();
-partitionWriters = new DiskBlockObjectWriter[numPartitions];
-partitionWriterSegments = new FileSegment[numPartitions];
-for (int i = 0; i < numPartitions; i++) {
- final Tuple2 tempShuffleBlockIdPlusFile =
-blockManager.diskBlockManager().createTempShuffleBlock();
- final File file = tempShuffleBlockIdPlusFile._2();
- final BlockId blockId = tempShuffleBlockIdPlusFile._1();
- partitionWriters[i] =
-blockManager.getDiskWriter(blockId, file, serInstance, fileBufferSize,
writeMetrics);
-}
-// Creating the file to write to and creating a disk writer both involve
interacting with
-// the disk, and can take a long time in aggregate when we open many
files, so should be
-// included in the shuffle write time.
-writeMetrics.incWriteTime(System.nanoTime() - openStartTime);
-
-while (records.hasNext()) {
- final Product2 record = records.next();
- final K key = record._1();
- partitionWriters[partitioner.getPartition(key)].write(key, record._2());
-}
+ShuffleMapOutputWriter mapOutputWriter = shuffleWriteSupport
+.createMapOutputWriter(shuffleId, mapId, mapTaskAttemptId,
numPartitions, writeMetrics);
+try {
+ if (!records.hasNext()) {
+partitionLengths = new long[numPartitions];
+mapOutputWriter.commitAllPartitions();
+mapStatus = MapStatus$.MODULE$.apply(
+blockManager.shuffleServerId(),
+partitionLengths);
+return;
+ }
+ final SerializerInstance serInstance = serializer.newInstance();
+ final long openStartTime = System.nanoTime();
+ partitionWriters = new DiskBlockObjectWriter[numPartitions];
+ partitionWriterSegments = new FileSegment[numPartitions];
+ for (int i = 0; i < numPartitions; i++) {
+final Tuple2 tempShuffleBlockIdPlusFile =
+blockManager.diskBlockManager().createTempShuffleBlock();
+final File file = tempShuffleBlockIdPlusFile._2();
+final BlockId blockId = tempShuffleBlockIdPlusFile._1();
+partitionWriters[i] =
+blockManager.getDiskWriter(blockId, file, serInstance,
fileBufferSize, writeMetrics);
+ }
+ // Creating the file to write to and creating a disk writer both involve
interacting with
+ // the disk, and can take a long time in aggregate when we open many
files, so should be
+ // included in the shuffle write time.
+ writeMetrics.incWriteTime(System.nanoTime() - openStartTime);
-for (int i = 0; i < numPartitions; i++) {
- try (DiskBlockObjectWriter writer = partitionWriters[i]) {
-partitionWriterSegments[i] = writer.commitAndGet();
+ while (records.hasNext()) {
+final Product2 record = records.next();
+final K key = record._1();
+partitionWriters[partitioner.getPartition(key)].write(key,
record._2());
}
-}
-File output = shuffleBlockResolver.getDataFile(sh
[GitHub] [spark] mccheah commented on a change in pull request #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
mccheah commented on a change in pull request #25007:
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
URL: https://github.com/apache/spark/pull/25007#discussion_r306562006
##
File path:
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
##
@@ -99,74 +104,82 @@
BypassMergeSortShuffleWriter(
BlockManager blockManager,
- IndexShuffleBlockResolver shuffleBlockResolver,
BypassMergeSortShuffleHandle handle,
int mapId,
+ long mapTaskAttemptId,
SparkConf conf,
- ShuffleWriteMetricsReporter writeMetrics) {
+ ShuffleWriteMetricsReporter writeMetrics,
+ ShuffleWriteSupport shuffleWriteSupport) {
// Use getSizeAsKb (not bytes) to maintain backwards compatibility if no
units are provided
this.fileBufferSize = (int) (long)
conf.get(package$.MODULE$.SHUFFLE_FILE_BUFFER_SIZE()) * 1024;
this.transferToEnabled = conf.getBoolean("spark.file.transferTo", true);
this.blockManager = blockManager;
final ShuffleDependency dep = handle.dependency();
this.mapId = mapId;
+this.mapTaskAttemptId = mapTaskAttemptId;
this.shuffleId = dep.shuffleId();
this.partitioner = dep.partitioner();
this.numPartitions = partitioner.numPartitions();
this.writeMetrics = writeMetrics;
this.serializer = dep.serializer();
-this.shuffleBlockResolver = shuffleBlockResolver;
+this.shuffleWriteSupport = shuffleWriteSupport;
}
@Override
public void write(Iterator> records) throws IOException {
assert (partitionWriters == null);
-if (!records.hasNext()) {
- partitionLengths = new long[numPartitions];
- shuffleBlockResolver.writeIndexFileAndCommit(shuffleId, mapId,
partitionLengths, null);
- mapStatus = MapStatus$.MODULE$.apply(blockManager.shuffleServerId(),
partitionLengths);
- return;
-}
-final SerializerInstance serInstance = serializer.newInstance();
-final long openStartTime = System.nanoTime();
-partitionWriters = new DiskBlockObjectWriter[numPartitions];
-partitionWriterSegments = new FileSegment[numPartitions];
-for (int i = 0; i < numPartitions; i++) {
- final Tuple2 tempShuffleBlockIdPlusFile =
-blockManager.diskBlockManager().createTempShuffleBlock();
- final File file = tempShuffleBlockIdPlusFile._2();
- final BlockId blockId = tempShuffleBlockIdPlusFile._1();
- partitionWriters[i] =
-blockManager.getDiskWriter(blockId, file, serInstance, fileBufferSize,
writeMetrics);
-}
-// Creating the file to write to and creating a disk writer both involve
interacting with
-// the disk, and can take a long time in aggregate when we open many
files, so should be
-// included in the shuffle write time.
-writeMetrics.incWriteTime(System.nanoTime() - openStartTime);
-
-while (records.hasNext()) {
- final Product2 record = records.next();
- final K key = record._1();
- partitionWriters[partitioner.getPartition(key)].write(key, record._2());
-}
+ShuffleMapOutputWriter mapOutputWriter = shuffleWriteSupport
+.createMapOutputWriter(shuffleId, mapId, mapTaskAttemptId,
numPartitions, writeMetrics);
+try {
+ if (!records.hasNext()) {
+partitionLengths = new long[numPartitions];
+mapOutputWriter.commitAllPartitions();
+mapStatus = MapStatus$.MODULE$.apply(
+blockManager.shuffleServerId(),
+partitionLengths);
+return;
+ }
+ final SerializerInstance serInstance = serializer.newInstance();
+ final long openStartTime = System.nanoTime();
+ partitionWriters = new DiskBlockObjectWriter[numPartitions];
+ partitionWriterSegments = new FileSegment[numPartitions];
+ for (int i = 0; i < numPartitions; i++) {
+final Tuple2 tempShuffleBlockIdPlusFile =
+blockManager.diskBlockManager().createTempShuffleBlock();
+final File file = tempShuffleBlockIdPlusFile._2();
+final BlockId blockId = tempShuffleBlockIdPlusFile._1();
+partitionWriters[i] =
+blockManager.getDiskWriter(blockId, file, serInstance,
fileBufferSize, writeMetrics);
+ }
+ // Creating the file to write to and creating a disk writer both involve
interacting with
+ // the disk, and can take a long time in aggregate when we open many
files, so should be
+ // included in the shuffle write time.
+ writeMetrics.incWriteTime(System.nanoTime() - openStartTime);
-for (int i = 0; i < numPartitions; i++) {
- try (DiskBlockObjectWriter writer = partitionWriters[i]) {
-partitionWriterSegments[i] = writer.commitAndGet();
+ while (records.hasNext()) {
+final Product2 record = records.next();
+final K key = record._1();
+partitionWriters[partitioner.getPartition(key)].write(key,
record._2());
}
-}
-File output = shuffleBlockResolver.getDataFile(sh
[GitHub] [spark] SparkQA commented on issue #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
SparkQA commented on issue #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables. URL: https://github.com/apache/spark/pull/25040#issuecomment-514408366 **[Test build #108063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108063/testReport)** for PR 25040 at commit [`c396d32`](https://github.com/apache/spark/commit/c396d32d9bea1a8dfb0659849b7f2042281f935e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514400993 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13169/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins removed a comment on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514400989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514400993 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13169/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
AmplabJenkins commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514400989 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514400974 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13169/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mccheah commented on a change in pull request #25040: [SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables.
mccheah commented on a change in pull request #25040: [SPARK-28238][SQL]
Implement DESCRIBE TABLE for Data Source V2 Tables.
URL: https://github.com/apache/spark/pull/25040#discussion_r306553187
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.DescribeTableSchemas
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.sources.v2.Table
+
+case class DescribeTableExec(
+catalog: TableCatalog,
+ident: Identifier,
+isExtended: Boolean) extends LeafExecNode {
+
+ import DescribeTableExec._
+
+ override def output: Seq[AttributeReference] =
DescribeTableSchemas.DESCRIBE_TABLE_ATTRIBUTES
+
+ override protected def doExecute(): RDD[InternalRow] = {
+val rows = new ArrayBuffer[InternalRow]()
+if (catalog.tableExists(ident)) {
+ val table = catalog.loadTable(ident)
+ addSchema(rows, table)
+
+ if (isExtended) {
+addPartitioning(rows, table)
+addProperties(rows, table)
+ }
+
+} else {
+ rows += toCatalystRow(s"Table $ident does not exist.", "", "")
+}
+sparkContext.parallelize(rows)
+ }
+
+ private def addSchema(rows: ArrayBuffer[InternalRow], table: Table): Unit = {
+rows ++= table.schema.map{ column =>
+ toCatalystRow(
+column.name, column.dataType.simpleString,
column.getComment().getOrElse(""))
+}
+ }
+
+ private def addPartitioning(rows: ArrayBuffer[InternalRow], table: Table):
Unit = {
+rows += EMPTY_ROW
+rows += toCatalystRow(" Partitioning", "", "")
+rows += toCatalystRow("--", "", "")
+if (table.partitioning.isEmpty) {
+ rows += toCatalystRow("Not partitioned", "", "")
+} else {
+ rows ++= table.partitioning.zipWithIndex.map {
+case (transform, index) => toCatalystRow(s"Part $index",
transform.describe(), "")
+ }
+}
+ }
+
+ private def addProperties(rows: ArrayBuffer[InternalRow], table: Table):
Unit = {
+rows += EMPTY_ROW
+rows += toCatalystRow(" Table Property", " Value", "")
+rows += toCatalystRow("", "---", "")
+rows ++= table.properties.asScala.toList.sortBy(_._1).map {
+ case (key, value) => toCatalystRow(key, value, "")
+}
+ }
+}
+
+private object DescribeTableExec {
+ private val ENCODER = RowEncoder(DescribeTableSchemas.DESCRIBE_TABLE_SCHEMA)
+ private val EMPTY_ROW = toCatalystRow("", "", "")
+
+ private def toCatalystRow(strs: String*): InternalRow = {
+ENCODER.resolveAndBind().toRow(
Review comment:
I don't necessarily think so, but it could also be how this class is built.
I think the encoder's state needs to be reset. When I don't resolveAndBind
every time, the tests yield wrong results entirely.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514396323 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13169/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
SparkQA commented on issue #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236#issuecomment-514394123 **[Test build #108062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108062/testReport)** for PR 25236 at commit [`3d20f08`](https://github.com/apache/spark/commit/3d20f08390e3f805ec8cd6d7d672194d4eff7839). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin opened a new pull request #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S.
vanzin opened a new pull request #25236: [SPARK-28487][k8s] More responsive dynamic allocation with K8S. URL: https://github.com/apache/spark/pull/25236 This change implements a few changes to the k8s pod allocator so that it behaves a little better when dynamic allocation is on. (i) Allow the application to ramp up immediately when there's a change in the target number of executors. Without this change, scaling would only trigger when a change happened in the state of the cluster, e.g. an executor going down, or when the periodical snapshot was taken (default every 30s). (ii) Get rid of pending pod requests, both acknowledged (i.e. Spark knows that a pod is pending resource allocation) and unacknowledged (i.e. Spark has requested the pod but the API server hasn't created it yet), when they're not needed anymore. This avoids starting those executors to just remove them after the idle timeout, wasting resources in the meantime. (iii) Re-work some of the code to avoid unnecessary logging. While not bad without dynamic allocation, the existing logging was very chatty when dynamic allocation was on. With the changes, all the useful information is still there, but only when interesting changes happen. (iv) Gracefully shut down executors when they become idle. Just deleting the pod causes a lot of ugly logs to show up, so it's better to ask pods to exit nicely. That also allows Spark to respect the "don't delete pods" option when dynamic allocation is on. Tested on a small k8s cluster running different TPC-DS workloads. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514385727 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108061/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514385720 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514385727 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108061/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514385720 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
SparkQA removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514328323 **[Test build #108061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108061/testReport)** for PR 25231 at commit [`53d59fc`](https://github.com/apache/spark/commit/53d59fcd82c5390bb8142342c7f048893d82). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
SparkQA commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514385107 **[Test build #108061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108061/testReport)** for PR 25231 at commit [`53d59fc`](https://github.com/apache/spark/commit/53d59fcd82c5390bb8142342c7f048893d82). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514364448 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514364456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108060/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514364456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108060/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514364448 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
SparkQA removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514277881 **[Test build #108060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108060/testReport)** for PR 25073 at commit [`a008c81`](https://github.com/apache/spark/commit/a008c81eddff1bd252326f2acbb6b77605529556). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
SparkQA commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514363754 **[Test build #108060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108060/testReport)** for PR 25073 at commit [`a008c81`](https://github.com/apache/spark/commit/a008c81eddff1bd252326f2acbb6b77605529556). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514329154 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514329159 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108057/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514329159 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108057/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514329154 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514328514 **[Test build #108057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108057/testReport)** for PR 25204 at commit [`33441a3`](https://github.com/apache/spark/commit/33441a32412eacbe84f2b188f046b2959a8b60ee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514232452 **[Test build #108057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108057/testReport)** for PR 25204 at commit [`33441a3`](https://github.com/apache/spark/commit/33441a32412eacbe84f2b188f046b2959a8b60ee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
SparkQA commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514328323 **[Test build #108061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108061/testReport)** for PR 25231 at commit [`53d59fc`](https://github.com/apache/spark/commit/53d59fcd82c5390bb8142342c7f048893d82). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514327549 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514327554 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13168/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514327554 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13168/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514327549 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514327037 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108059/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514327022 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514327037 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108059/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
AmplabJenkins removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514327022 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
SparkQA removed a comment on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514250535 **[Test build #108059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108059/testReport)** for PR 25204 at commit [`110a39e`](https://github.com/apache/spark/commit/110a39e5b0eaf30a5e9321e8dd40cd8f2b96c5bc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
dongjoon-hyun closed pull request #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
SparkQA commented on issue #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#issuecomment-514326656 **[Test build #108059 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108059/testReport)** for PR 25204 at commit [`110a39e`](https://github.com/apache/spark/commit/110a39e5b0eaf30a5e9321e8dd40cd8f2b96c5bc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
AmplabJenkins removed a comment on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-513820735 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
dongjoon-hyun commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514325603 ok to test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
dongjoon-hyun commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514325042 Thank you, @cloud-fan and @viirya ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #25037: [SPARK-27489][WEBUI] UI updates to show executor resource information
tgravescs commented on a change in pull request #25037: [SPARK-27489][WEBUI] UI
updates to show executor resource information
URL: https://github.com/apache/spark/pull/25037#discussion_r306458317
##
File path: core/src/main/resources/org/apache/spark/ui/static/executorspage.js
##
@@ -39,6 +39,19 @@ function formatStatus(status, type, row) {
return "Dead"
}
+function formatResourceCells(resources) {
Review comment:
I was trying to take up less space. the linked PR adds in checkboxes to show
columns so we could break up more if its not on by default, but I think at this
point we should start with something small and then enhance later as people
figure out what is useful.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514306226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108054/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514306219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514306226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108054/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
AmplabJenkins commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514306219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA removed a comment on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514208675 **[Test build #108054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108054/testReport)** for PR 25198 at commit [`fdaee89`](https://github.com/apache/spark/commit/fdaee8919b64728b145fec07b053317e3e421a04). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType
SparkQA commented on issue #25198: [SPARK-28443][SQL] Spark sql add exception when create field type NullType URL: https://github.com/apache/spark/pull/25198#issuecomment-514305542 **[Test build #108054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108054/testReport)** for PR 25198 at commit [`fdaee89`](https://github.com/apache/spark/commit/fdaee8919b64728b145fec07b053317e3e421a04). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514287133 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108053/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514287126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514287133 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108053/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
AmplabJenkins commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514287126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
SparkQA removed a comment on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514185900 **[Test build #108053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108053/testReport)** for PR 23531 at commit [`80d2ed9`](https://github.com/apache/spark/commit/80d2ed900a05b809f4d84e497e8ae84c50ad81de). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query
SparkQA commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-514286281 **[Test build #108053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108053/testReport)** for PR 23531 at commit [`80d2ed9`](https://github.com/apache/spark/commit/80d2ed900a05b809f4d84e497e8ae84c50ad81de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514284365 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514284375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108052/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514284375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108052/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514284365 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
SparkQA removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514183285 **[Test build #108052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108052/testReport)** for PR 25233 at commit [`6a983f8`](https://github.com/apache/spark/commit/6a983f80b07a0c6ff22aa9bde5117c488e775c2e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
SparkQA commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514283670 **[Test build #108052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108052/testReport)** for PR 25233 at commit [`6a983f8`](https://github.com/apache/spark/commit/6a983f80b07a0c6ff22aa9bde5117c488e775c2e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
AmplabJenkins removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514281150 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108058/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
AmplabJenkins removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514281140 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
AmplabJenkins commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514281140 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
AmplabJenkins commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514281150 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108058/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
SparkQA removed a comment on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514235944 **[Test build #108058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108058/testReport)** for PR 25234 at commit [`673fef6`](https://github.com/apache/spark/commit/673fef6b6e77d461ed2299d25f279889bb74f423). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
SparkQA commented on issue #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue URL: https://github.com/apache/spark/pull/25234#issuecomment-514280642 **[Test build #108058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108058/testReport)** for PR 25234 at commit [`673fef6`](https://github.com/apache/spark/commit/673fef6b6e77d461ed2299d25f279889bb74f423). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Mister-Meeseeks commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README
Mister-Meeseeks commented on issue #25231: [SPARK-28473][DOC] Stylistic consistency of build command in README URL: https://github.com/apache/spark/pull/25231#issuecomment-514280574 I scoured the documentation for offending examples. I believe I found and corrected every command example not conforming to the `./` prepended convention. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514280218 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514280218 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514280226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13167/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
AmplabJenkins removed a comment on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514280226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13167/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
SparkQA commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514277881 **[Test build #108060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108060/testReport)** for PR 25073 at commit [`a008c81`](https://github.com/apache/spark/commit/a008c81eddff1bd252326f2acbb6b77605529556). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation
gatorsmile commented on issue #25073: [SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation URL: https://github.com/apache/spark/pull/25073#issuecomment-514276860 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #25111: [SPARK-28346][SQL] clone the query plan between analyzer, optimizer and planner
gatorsmile closed pull request #25111: [SPARK-28346][SQL] clone the query plan between analyzer, optimizer and planner URL: https://github.com/apache/spark/pull/25111 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514275178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108050/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514275166 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514275166 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
AmplabJenkins commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514275178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108050/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
SparkQA removed a comment on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514176274 **[Test build #108050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108050/testReport)** for PR 25233 at commit [`6870764`](https://github.com/apache/spark/commit/68707647675e92ce28239b738ef67e528c2518d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base
SparkQA commented on issue #25233: [SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base URL: https://github.com/apache/spark/pull/25233#issuecomment-514274419 **[Test build #108050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108050/testReport)** for PR 25233 at commit [`6870764`](https://github.com/apache/spark/commit/68707647675e92ce28239b738ef67e528c2518d3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514273434 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108056/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514273422 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514273422 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514273434 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108056/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514218472 **[Test build #108056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108056/testReport)** for PR 25235 at commit [`6f25e25`](https://github.com/apache/spark/commit/6f25e25962d01a8d71743c679861d493da4d6699). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25111: [SPARK-28346][SQL] clone the query plan between analyzer, optimizer and planner
gatorsmile commented on issue #25111: [SPARK-28346][SQL] clone the query plan between analyzer, optimizer and planner URL: https://github.com/apache/spark/pull/25111#issuecomment-514273248 LGTM Thanks! Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514271859 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108055/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514272963 **[Test build #108056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108056/testReport)** for PR 25235 at commit [`6f25e25`](https://github.com/apache/spark/commit/6f25e25962d01a8d71743c679861d493da4d6699). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514271854 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514271859 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108055/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
AmplabJenkins commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514271854 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
SparkQA removed a comment on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514215132 **[Test build #108055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108055/testReport)** for PR 25235 at commit [`ad579f1`](https://github.com/apache/spark/commit/ad579f12739a8dbfe1a0982965ea6efa052ffabe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages
SparkQA commented on issue #25235: [WIP][SPARK-28483][Core] Fix canceling a spark job using barrier mode but tasks still being printing messages URL: https://github.com/apache/spark/pull/25235#issuecomment-514271459 **[Test build #108055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108055/testReport)** for PR 25235 at commit [`ad579f1`](https://github.com/apache/spark/commit/ad579f12739a8dbfe1a0982965ea6efa052ffabe). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] DylanGuedes commented on issue #24881: [SPARK-23160][SQL][TEST] Port window.sql
DylanGuedes commented on issue #24881: [SPARK-23160][SQL][TEST] Port window.sql URL: https://github.com/apache/spark/pull/24881#issuecomment-514268697 @dongjoon-hyun I've been trying to make filtering works for a while but without success. For instance, in the following snippet ``` SELECT sum(salary), row_number() OVER (ORDER BY depname), sum( sum(salary) FILTER (WHERE enroll_date > '2007-01-01') ) FROM empsalary GROUP BY depname; ``` I already tried moving the filter to different places and using different syntaxes (I even used lambda), but none of them work for this case. Since I don't know If it is just me not knowing the API well enough or if Spark has any limitation in this side, I didn't created a JIRA for cases like this. What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery
viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python]
Fix error when non-foldable expression is used in correlated scalar subquery
URL: https://github.com/apache/spark/pull/25204#discussion_r306391020
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -465,19 +519,33 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
val alwaysTrueRef = AttributeReference(ALWAYS_TRUE_COLNAME,
BooleanType)(exprId = alwaysTrueExprId)
- val aggValRef = query.output.head
-
if (havingNode.isEmpty) {
// CASE 2: Subquery with no HAVING clause
+
+// We replace original expression id with a new one. The added
Alias column
+// must use expr id of original output. If we don't replace old
expr id in the
+// query, the added Project in potential Project-Filter-Project
can be removed
+// by removeProjectBeforeFilter in ColumnPruning.
Review comment:
Worth trying, right now not sure if any other thing will be affected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25234: [SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
dongjoon-hyun commented on a change in pull request #25234:
[SPARK-28054][SQL][followup] move the bug fix closer to where causes the issue
URL: https://github.com/apache/spark/pull/25234#discussion_r306388453
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##
@@ -186,10 +188,15 @@ case class InsertIntoHiveTable(
}
val partitionAttributes =
partitionColumnNames.takeRight(numDynamicPartitions).map { name =>
- query.resolve(name :: Nil,
sparkSession.sessionState.analyzer.resolver).getOrElse {
+ val attr = query.resolve(name :: Nil,
sparkSession.sessionState.analyzer.resolver).getOrElse {
throw new AnalysisException(
s"Unable to resolve $name given
[${query.output.map(_.name).mkString(", ")}]")
}.asInstanceOf[Attribute]
+ // SPARK-28054: Hive metastore is not case preserving and keeps
partition columns
+ // with lower cased names. Hive will validate the column names in the
partition directories
+ // during `loadDynamicPartitions`. Spark needs to write partition
directories with lower-cased
+ // column names in order to make `loadDynamicPartitions` work.
+ attr.withName(name.toLowerCase(Locale.ROOT))
Review comment:
This move looks nicer and also cleaner due to the removal of `scalastyle:off
caselocale`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
