[GitHub] [spark] SparkQA commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
SparkQA commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548688472 **[Test build #113070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113070/testReport)** for PR 26331 at commit [`62974f8`](https://github.com/apache/spark/commit/62974f8a7170ae682ff7a5fcdf64134c971dc148). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
SparkQA removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548663478 **[Test build #113070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113070/testReport)** for PR 26331 at commit [`62974f8`](https://github.com/apache/spark/commit/62974f8a7170ae682ff7a5fcdf64134c971dc148). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
AmplabJenkins commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548688878 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
AmplabJenkins commented on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548688875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
SparkQA commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548688871 **[Test build #113077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113077/testReport)** for PR 26353 at commit [`e86b15a`](https://github.com/apache/spark/commit/e86b15a51b8af06844fcfd5fa75c1115145bd88e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548687376 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17996/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548687372 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
AmplabJenkins removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548688878 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113070/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable
AmplabJenkins removed a comment on issue #26331: [SPARK-24203][core] Make executor's bindAddress configurable URL: https://github.com/apache/spark/pull/26331#issuecomment-548688875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #26124: [SPARK-29224][ML]Implement Factorization Machines as a ml-pipeline component
zhengruifeng commented on a change in pull request #26124: [SPARK-29224][ML]Implement Factorization Machines as a ml-pipeline component URL: https://github.com/apache/spark/pull/26124#discussion_r341470080 ## File path: mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala ## @@ -0,0 +1,360 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.ml.classification + +import org.apache.hadoop.fs.Path + +import org.apache.spark.annotation.Since +import org.apache.spark.internal.Logging +import org.apache.spark.ml.linalg._ +import org.apache.spark.ml.param._ +import org.apache.spark.ml.regression._ +import org.apache.spark.ml.util._ +import org.apache.spark.ml.util.Instrumentation.instrumented +import org.apache.spark.mllib.linalg.{Vector => OldVector} +import org.apache.spark.mllib.linalg.VectorImplicits._ +import org.apache.spark.mllib.optimization.GradientDescent +import org.apache.spark.mllib.regression.{LabeledPoint => OldLabeledPoint} +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.{Dataset, Row} +import org.apache.spark.sql.functions.col +import org.apache.spark.storage.StorageLevel + +/** + * Params for FMClassifier. + */ +private[classification] trait FMClassifierParams extends ProbabilisticClassifierParams + with FactorizationMachinesParams { +} + +/** + * Factorization Machines learning algorithm for classification. + * It supports normal gradient descent and AdamW solver. + * + * The implementation is based upon: + * https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf";> + * S. Rendle. "Factorization machines" 2010. + * + * FM is able to estimate interactions even in problems with huge sparsity + * (like advertising and recommendation system). + * FM formula is: + * {{{ + * y = w_0 + \sum\limits^n_{i-1} w_i x_i + + * \sum\limits^n_{i=1} \sum\limits^n_{j=i+1} \langle v_i, v_j \rangle x_i x_j + * }}} + * First two terms denote global bias and linear term (as same as linear regression), + * and last term denotes pairwise interactions term. {{{v_i}}} describes the i-th variable + * with k factors. + * + * FM classification model uses logistic loss which can be solved by gradient descent method, and + * regularization terms like L2 are usually added to the loss function to prevent overfitting. + * + * @note Multiclass labels are not currently supported. + */ +@Since("3.0.0") +class FMClassifier @Since("3.0.0") ( + @Since("3.0.0") override val uid: String) + extends ProbabilisticClassifier[Vector, FMClassifier, FMClassifierModel] + with FMClassifierParams with DefaultParamsWritable with Logging { + + import org.apache.spark.ml.regression.BaseFactorizationMachinesGradient.{LogisticLoss, parseLoss} + import org.apache.spark.ml.regression.FMRegressor.initCoefficients + + @Since("3.0.0") + def this() = this(Identifiable.randomUID("fmc")) + + /** + * Set the dimensionality of the factors. + * Default is 8. + * + * @group setParam + */ + @Since("3.0.0") + def setNumFactors(value: Int): this.type = set(numFactors, value) + setDefault(numFactors -> 8) + + /** + * Set whether to fit global bias term. + * Default is true. + * + * @group setParam + */ + @Since("3.0.0") + def setFitBias(value: Boolean): this.type = set(fitBias, value) + setDefault(fitBias -> true) + + /** + * Set whether to fit linear term. + * Default is true. + * + * @group setParam + */ + @Since("3.0.0") + def setFitLinear(value: Boolean): this.type = set(fitLinear, value) + setDefault(fitLinear -> true) + + /** + * Set the L2 regularization parameter. + * Default is 0.0. + * + * @group setParam + */ + @Since("3.0.0") + def setRegParam(value: Double): this.type = set(regParam, value) + setDefault(regParam -> 0.0) + + /** + * Set the mini-batch fraction parameter. + * Default is 1.0. + * + * @group setParam + */ + @Since("3.0.0") + def setMiniBatchFraction(value: Double): this.type = { +require(value > 0 && value <= 1.0, + s"Fraction for mini-batch SGD must be in range (0, 1] but got $value") +set(miniBatchFraction, value) + } + setDefault(miniBatchFraction -> 1.0) + + /** + * Set the standard d
[GitHub] [spark] zhengruifeng commented on a change in pull request #26124: [SPARK-29224][ML]Implement Factorization Machines as a ml-pipeline component
zhengruifeng commented on a change in pull request #26124: [SPARK-29224][ML]Implement Factorization Machines as a ml-pipeline component URL: https://github.com/apache/spark/pull/26124#discussion_r341470080 ## File path: mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala ## @@ -0,0 +1,360 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.ml.classification + +import org.apache.hadoop.fs.Path + +import org.apache.spark.annotation.Since +import org.apache.spark.internal.Logging +import org.apache.spark.ml.linalg._ +import org.apache.spark.ml.param._ +import org.apache.spark.ml.regression._ +import org.apache.spark.ml.util._ +import org.apache.spark.ml.util.Instrumentation.instrumented +import org.apache.spark.mllib.linalg.{Vector => OldVector} +import org.apache.spark.mllib.linalg.VectorImplicits._ +import org.apache.spark.mllib.optimization.GradientDescent +import org.apache.spark.mllib.regression.{LabeledPoint => OldLabeledPoint} +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.{Dataset, Row} +import org.apache.spark.sql.functions.col +import org.apache.spark.storage.StorageLevel + +/** + * Params for FMClassifier. + */ +private[classification] trait FMClassifierParams extends ProbabilisticClassifierParams + with FactorizationMachinesParams { +} + +/** + * Factorization Machines learning algorithm for classification. + * It supports normal gradient descent and AdamW solver. + * + * The implementation is based upon: + * https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf";> + * S. Rendle. "Factorization machines" 2010. + * + * FM is able to estimate interactions even in problems with huge sparsity + * (like advertising and recommendation system). + * FM formula is: + * {{{ + * y = w_0 + \sum\limits^n_{i-1} w_i x_i + + * \sum\limits^n_{i=1} \sum\limits^n_{j=i+1} \langle v_i, v_j \rangle x_i x_j + * }}} + * First two terms denote global bias and linear term (as same as linear regression), + * and last term denotes pairwise interactions term. {{{v_i}}} describes the i-th variable + * with k factors. + * + * FM classification model uses logistic loss which can be solved by gradient descent method, and + * regularization terms like L2 are usually added to the loss function to prevent overfitting. + * + * @note Multiclass labels are not currently supported. + */ +@Since("3.0.0") +class FMClassifier @Since("3.0.0") ( + @Since("3.0.0") override val uid: String) + extends ProbabilisticClassifier[Vector, FMClassifier, FMClassifierModel] + with FMClassifierParams with DefaultParamsWritable with Logging { + + import org.apache.spark.ml.regression.BaseFactorizationMachinesGradient.{LogisticLoss, parseLoss} + import org.apache.spark.ml.regression.FMRegressor.initCoefficients + + @Since("3.0.0") + def this() = this(Identifiable.randomUID("fmc")) + + /** + * Set the dimensionality of the factors. + * Default is 8. + * + * @group setParam + */ + @Since("3.0.0") + def setNumFactors(value: Int): this.type = set(numFactors, value) + setDefault(numFactors -> 8) + + /** + * Set whether to fit global bias term. + * Default is true. + * + * @group setParam + */ + @Since("3.0.0") + def setFitBias(value: Boolean): this.type = set(fitBias, value) + setDefault(fitBias -> true) + + /** + * Set whether to fit linear term. + * Default is true. + * + * @group setParam + */ + @Since("3.0.0") + def setFitLinear(value: Boolean): this.type = set(fitLinear, value) + setDefault(fitLinear -> true) + + /** + * Set the L2 regularization parameter. + * Default is 0.0. + * + * @group setParam + */ + @Since("3.0.0") + def setRegParam(value: Double): this.type = set(regParam, value) + setDefault(regParam -> 0.0) + + /** + * Set the mini-batch fraction parameter. + * Default is 1.0. + * + * @group setParam + */ + @Since("3.0.0") + def setMiniBatchFraction(value: Double): this.type = { +require(value > 0 && value <= 1.0, + s"Fraction for mini-batch SGD must be in range (0, 1] but got $value") +set(miniBatchFraction, value) + } + setDefault(miniBatchFraction -> 1.0) + + /** + * Set the standard d
[GitHub] [spark] SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690511 **[Test build #113078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113078/testReport)** for PR 25287 at commit [`3e58b81`](https://github.com/apache/spark/commit/3e58b81569ba3a6de7f39642effc45a67e73bfb8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690859 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
SparkQA removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690511 **[Test build #113078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113078/testReport)** for PR 25287 at commit [`3e58b81`](https://github.com/apache/spark/commit/3e58b81569ba3a6de7f39642effc45a67e73bfb8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690859 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690865 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113078/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690854 **[Test build #113078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113078/testReport)** for PR 25287 at commit [`3e58b81`](https://github.com/apache/spark/commit/3e58b81569ba3a6de7f39642effc45a67e73bfb8). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548690865 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113078/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548691723 Here's what I mean: test query: ``` SELECT TRANSFORM(key, abs(key)) USING 'cat' FROM (SELECT DISTINCT key FROM src); ``` The physical plan without this PR (on master): ``` ScriptTransformation [key#7, abs(key#7)], cat, [key#9, value#10], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim, )),List((field.delim, )),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),true) +- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) +- Exchange hashpartitioning(1#16, 1), true, [id=#27] +- *(1) HashAggregate(keys=[1 AS 1#16], functions=[], output=[1#16]) +- *(1) Scan OneRowRelation[] ``` The physical plan with this PR: ``` ScriptTransformation [key#0, abs(key)#9], cat, [key#2, value#3], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim, )),List((field.delim, )),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),true) +- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9]) +- Exchange hashpartitioning(1#10, 1), true, [id=#21] +- *(1) HashAggregate(keys=[1 AS 1#10], functions=[], output=[1#10]) +- *(1) Scan OneRowRelation[] ``` Difference: `+- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) `vs `+- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9]) ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548692408 Another issue: query: `MAP k / 10 USING 'cat' AS (aa) from (select 10 as k); ` Error in query: cannot resolve '`(k / 10)`' given input columns: [(CAST(k AS DOUBLE) / CAST(10 AS DOUBLE))];; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alfozan removed a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
alfozan removed a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548692408 Another issue: query: `MAP k / 10 USING 'cat' AS (aa) from (select 10 as k); ` Error in query: cannot resolve '`(k / 10)`' given input columns: [(CAST(k AS DOUBLE) / CAST(10 AS DOUBLE))];; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548693738 > Another issue: > > query: > `MAP k / 10 USING 'cat' AS (aa) from (select 10 as k); ` > > Error in query: cannot resolve '`(k / 10)`' given input columns: [(CAST(k AS DOUBLE) / CAST(10 AS DOUBLE))];; This can be solved by https://github.com/apache/spark/pull/25028#issuecomment-548675869 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix
SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix URL: https://github.com/apache/spark/pull/25287#issuecomment-548693798 **[Test build #113079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113079/testReport)** for PR 25287 at commit [`ac70da4`](https://github.com/apache/spark/commit/ac70da4f494624368aa1570100e2a00ec0e1c308). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548694667 > output=[key#7] This is right since we current tread transform's child logicalplan as an entire one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548694618 > > Another issue: > > query: > > `MAP k / 10 USING 'cat' AS (aa) from (select 10 as k); ` > > Error in query: cannot resolve '`(k / 10)`' given input columns: [(CAST(k AS DOUBLE) / CAST(10 AS DOUBLE))];; > > This can be solved by [#25028 (comment)](https://github.com/apache/spark/pull/25028#issuecomment-548675869) Yes we already discussed a solution in https://github.com/apache/spark/pull/25028#discussion_r339890074 which I can confirm it works. Currently the open issue is https://github.com/apache/spark/pull/25028#issuecomment-548691723 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh edited a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548694667 > Here's what I mean: > > test query: > > ``` > SELECT TRANSFORM(key, abs(key)) > USING 'cat' > FROM (SELECT DISTINCT key FROM src); > ``` > > The physical plan without this PR (on master): > > ``` > ScriptTransformation [key#7, abs(key#7)], cat, [key#9, value#10], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim, )),List((field.delim, )),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),true) > +- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) >+- Exchange hashpartitioning(1#16, 1), true, [id=#27] > +- *(1) HashAggregate(keys=[1 AS 1#16], functions=[], output=[1#16]) > +- *(1) Scan OneRowRelation[] > ``` > > The physical plan with this PR: > > ``` > ScriptTransformation [key#0, abs(key)#9], cat, [key#2, value#3], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim, )),List((field.delim, )),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),true) > +- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9]) >+- Exchange hashpartitioning(1#10, 1), true, [id=#21] > +- *(1) HashAggregate(keys=[1 AS 1#10], functions=[], output=[1#10]) > +- *(1) Scan OneRowRelation[] > ``` > > Difference: > `+- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) `vs > `+- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9]) ` This right since we now tread transform's child as an entire logical plan. It 's final output is two column, it's right. And transform use it 's out put as transform 's input. Reasonable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
SparkQA commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548696207 **[Test build #113067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113067/testReport)** for PR 26352 at commit [`6e733fe`](https://github.com/apache/spark/commit/6e733fe510a6f8b1b2aa53d9884865ca1418c560). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
SparkQA removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548657216 **[Test build #113067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113067/testReport)** for PR 26352 at commit [`6e733fe`](https://github.com/apache/spark/commit/6e733fe510a6f8b1b2aa53d9884865ca1418c560). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548696642 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548696640 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548696640 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548696642 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
SparkQA commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548698845 **[Test build #113068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113068/testReport)** for PR 26314 at commit [`b071e09`](https://github.com/apache/spark/commit/b071e09a2914dcfb566112cb77d8e187ab0d5a7b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
SparkQA removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548658539 **[Test build #113068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113068/testReport)** for PR 26314 at commit [`b071e09`](https://github.com/apache/spark/commit/b071e09a2914dcfb566112cb77d8e187ab0d5a7b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
AmplabJenkins commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548699258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
AmplabJenkins commented on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548699252 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
AmplabJenkins removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548699252 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts
AmplabJenkins removed a comment on issue #26314: [SPARK-29371][SQL] Support interval field values with fractional parts URL: https://github.com/apache/spark/pull/26314#issuecomment-548699258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya opened a new pull request #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
viirya opened a new pull request #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354 ### What changes were proposed in this pull request? If the resolved table is v1 table, AlterTableAlterColumnStatement fallbacks to v1 AlterTableChangeColumnCommand. ### Why are the changes needed? To make the catalog/table lookup logic consistent. ### Does this PR introduce any user-facing change? Yes, a ALTER TABLE ALTER COLUMN command previously fails on v1 tables. After this, it falls back to v1 AlterTableChangeColumnCommand. ### How was this patch tested? Unit test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
SparkQA commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354#issuecomment-548701092 **[Test build #113080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113080/testReport)** for PR 26354 at commit [`45ed688`](https://github.com/apache/spark/commit/45ed6887fb20e21f6e52d78c7b32ca67a6e97628). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24898: [SPARK-22340][PYTHON] Add a mode to pin Python thread into JVM's
HyukjinKwon commented on issue #24898: [SPARK-22340][PYTHON] Add a mode to pin Python thread into JVM's URL: https://github.com/apache/spark/pull/24898#issuecomment-548701197 reteat this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands
viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26338#discussion_r341480740 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -151,7 +151,7 @@ statement | ALTER TABLE multipartIdentifier (ALTER | CHANGE) COLUMN? qualifiedName (TYPE dataType)? (COMMENT comment=STRING)? colPosition? #alterTableColumn -| ALTER TABLE tableIdentifier partitionSpec? +| ALTER TABLE multipartIdentifier partitionSpec? Review comment: Seems we do not really support it, though we allow such syntax: https://github.com/apache/spark/blob/7aca0dd658b8bda05574b3df3254aaf66eb2a174/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala#L343-L351 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
AmplabJenkins commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354#issuecomment-548701525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
AmplabJenkins removed a comment on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354#issuecomment-548701525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands
viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26338#discussion_r341480870 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -151,7 +151,7 @@ statement | ALTER TABLE multipartIdentifier (ALTER | CHANGE) COLUMN? qualifiedName (TYPE dataType)? (COMMENT comment=STRING)? colPosition? #alterTableColumn -| ALTER TABLE tableIdentifier partitionSpec? +| ALTER TABLE multipartIdentifier partitionSpec? Review comment: Not sure what if the original plan was to support that in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
AmplabJenkins commented on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354#issuecomment-548701526 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17997/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands
viirya commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26338#discussion_r341480870 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -151,7 +151,7 @@ statement | ALTER TABLE multipartIdentifier (ALTER | CHANGE) COLUMN? qualifiedName (TYPE dataType)? (COMMENT comment=STRING)? colPosition? #alterTableColumn -| ALTER TABLE tableIdentifier partitionSpec? +| ALTER TABLE multipartIdentifier partitionSpec? Review comment: Not sure what if the original plan is to support that in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand
AmplabJenkins removed a comment on issue #26354: [SPARK-29353][SQL] Fallback AlterTableAlterColumnStatement to v1 AlterTableChangeColumnCommand URL: https://github.com/apache/spark/pull/26354#issuecomment-548701526 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17997/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
SparkQA commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548703401 **[Test build #113069 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113069/testReport)** for PR 22952 at commit [`01f5750`](https://github.com/apache/spark/commit/01f57506c9032d7a31bc383432ed09bf1801d402). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
SparkQA removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548658566 **[Test build #113069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113069/testReport)** for PR 22952 at commit [`01f5750`](https://github.com/apache/spark/commit/01f57506c9032d7a31bc383432ed09bf1801d402). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT
AngersZh commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT URL: https://github.com/apache/spark/pull/26340#issuecomment-548703769 @juliuszsompolski Current file tree ``` src/ ├── main │  ├── java │  │  └── org │  │  └── apache │  │  └── spark │  │  └── sql │  │  └── thriftserver │  │  ├── AbstractService.java │  │  ├── CompositeService.java │  │  ├── CookieSigner.java │  │  ├── Service.java │  │  ├── ServiceException.java │  │  ├── ServiceOperations.java │  │  ├── ServiceStateChangeListener.java │  │  ├── ServiceUtils.java │  │  ├── auth │  │  │  ├── AnonymousAuthenticationProviderImpl.java │  │  │  ├── AuthenticationProviderFactory.java │  │  │  ├── CustomAuthenticationProviderImpl.java │  │  │  ├── HiveAuthFactory.java │  │  │  ├── HttpAuthUtils.java │  │  │  ├── HttpAuthenticationException.java │  │  │  ├── KerberosSaslHelper.java │  │  │  ├── LdapAuthenticationProviderImpl.java │  │  │  ├── PamAuthenticationProviderImpl.java │  │  │  ├── PasswdAuthenticationProvider.java │  │  │  ├── PlainSaslHelper.java │  │  │  ├── PlainSaslServer.java │  │  │  ├── SaslQOP.java │  │  │  ├── TSetIpAddressProcessor.java │  │  │  ├── TSubjectAssumingTransport.java │  │  │  └── ThriftDelegationTokenManager.java │  │  ├── cli │  │  │  ├── CLIServiceClient.java │  │  │  ├── CLIServiceUtils.java │  │  │  ├── FetchOrientation.java │  │  │  ├── FetchType.java │  │  │  ├── GetInfoType.java │  │  │  ├── GetInfoValue.java │  │  │  ├── Handle.java │  │  │  ├── HandleIdentifier.java │  │  │  ├── ICLIService.java │  │  │  ├── OperationHandle.java │  │  │  ├── OperationState.java │  │  │  ├── OperationStatus.java │  │  │  ├── OperationType.java │  │  │  ├── PatternOrIdentifier.java │  │  │  ├── SessionHandle.java │  │  │  ├── SparkThriftServerSQLException.java │  │  │  ├── Type.java │  │  │  ├── operation │  │  │  │  └── LogDivertAppender.java │  │  │  ├── session │  │  │  │  ├── ThriftServerSessionBase.java │  │  │  │  └── ThriftServerSessionProxy.java │  │  │  ├── thrift │  │  │  │  ├── ThriftBinaryCLIService.java │  │  │  │  ├── ThriftCLIService.java │  │  │  │  ├── ThriftCLIServiceClient.java │  │  │  │  ├── ThriftHttpCLIService.java │  │  │  │  └── ThriftHttpServlet.java │  │  │  └── utils │  │  │  └── VariableSubstitution.java │  │  └── server │  │  ├── ThreadFactoryWithGarbageCleanup.java │  │  └── ThreadWithGarbageCleanup.java │  └── scala │  └── org │  └── apache │  └── spark │  └── sql │  └── thriftserver │  ├── ReflectionUtils.scala │  ├── SparkSQLEnv.scala │  ├── cli │  │  ├── CLIService.scala │  │  ├── ColumnBasedSet.scala │  │  ├── ColumnBuffer.scala │  │  ├── ColumnDescriptor.scala │  │  ├── RowBasedSet.scala │  │  ├── RowSet.scala │  │  ├── RowSetFactory.scala │  │  ├── SchemaMapper.scala │  │  ├── TypeDescriptor.scala │  │  ├── TypeQualifiers.scala │  │  ├── operation │ Â
[GitHub] [spark] AmplabJenkins commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548703841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113069/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins commented on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548703835 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548703841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113069/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-548703835 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT
AngersZh commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT URL: https://github.com/apache/spark/pull/26340#issuecomment-548705691 ``` ColumnBasedSet.scala ColumnBuffer.scala ColumnDescriptor.scala RowBasedSet.scala RowSet.scala RowSetFactory.scala SchemaMapper.scala TypeDescriptor.scala TypeQualifiers.scala ``` Changed a lot with `StructType` `Row` `DataType` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk opened a new pull request #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
MaxGekk opened a new pull request #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355 ### What changes were proposed in this pull request? In the PR, I propose to extract parsing of the seconds interval units to the private method `parseNanos` in `CalendarInterval` and modify the code to correctly parse the fractional part of the seconds unit of intervals in the cases: - When the fractional part has less than 9 digits - The seconds unit is negative This is a back port of the commit https://github.com/apache/spark/commit/3206a9987001d78cf2f48509a93d73af86f51cfe. ### Why are the changes needed? The changes are needed to fix the issues: ```sql spark-sql> select interval 10.123456 seconds; interval 10 seconds 123 microseconds ``` The correct result must be `interval 10 seconds 123 milliseconds 456 microseconds` ```sql spark-sql> select interval -10.123456789 seconds; interval -9 seconds -876 milliseconds -544 microseconds ``` but the whole interval should be negated, and the result must be `interval -10 seconds -123 milliseconds -456 microseconds`, taking into account the truncation to microseconds. ### Does this PR introduce any user-facing change? Yes. After changes: ```sql spark-sql> select interval 10.123456 seconds; interval 10 seconds 123 milliseconds 456 microseconds spark-sql> select interval -10.123456789 seconds; interval -10 seconds -123 milliseconds -456 microseconds ``` ### How was this patch tested? By existing test suite, `literals.sql` and new tests in `ExpressionParserSuite`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
SparkQA commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355#issuecomment-548706718 **[Test build #113081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113081/testReport)** for PR 26355 at commit [`56faeb0`](https://github.com/apache/spark/commit/56faeb0a637264fb6245196293afcfe96543be43). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
SparkQA commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548706524 **[Test build #113073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113073/testReport)** for PR 26134 at commit [`2f90189`](https://github.com/apache/spark/commit/2f901894eb24d7deec31947708b9afdaae2a4866). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
SparkQA removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548674854 **[Test build #113073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113073/testReport)** for PR 26134 at commit [`2f90189`](https://github.com/apache/spark/commit/2f901894eb24d7deec31947708b9afdaae2a4866). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
AmplabJenkins commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548707028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
AmplabJenkins commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548707020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
AmplabJenkins commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355#issuecomment-548707090 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
AmplabJenkins removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548707028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113073/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
AmplabJenkins commented on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355#issuecomment-548707087 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
AmplabJenkins removed a comment on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355#issuecomment-548707090 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17998/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT
AngersZh commented on issue #26340: [WIP][SPARK-29108][SQL] Add new module sql/thriftserver with all code and UT URL: https://github.com/apache/spark/pull/26340#issuecomment-548707215 @juliuszsompolski Move SparkThriftServer2 to SparkThriftServer, and Integrated startup code in main method. Remove unexecuted code in SparkThriftServer such as `startupShutdownMessage`, replace unexecuted `startHiveServer2()` to `startSparkThriftServer()` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
AmplabJenkins removed a comment on issue #26355: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction URL: https://github.com/apache/spark/pull/26355#issuecomment-548707087 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
AmplabJenkins removed a comment on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548707020 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LinhongLiu commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds
LinhongLiu commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-548707716 @cloud-fan tests are all pass, could you please check this PR again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548709822 @alfozan You can see my branch's newest change. and check the update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548710975 > Difference: > `+- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) `vs > `+- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9])` Seems after my pr. the output is right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh edited a comment on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548710975 > Difference: > `+- *(2) HashAggregate(keys=[1#16], functions=[], output=[key#7]) `vs > `+- *(2) HashAggregate(keys=[1#10], functions=[], output=[key#0, abs(key)#9])` Seems after my pr. the output is right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu opened a new pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
fuwhu opened a new pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356 ### What changes were proposed in this pull request? Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' to set the max string length of one metadata in DataSourceScanExec. ### Why are the changes needed? To make the max length of metadata string of `DataSourceScanExec` configurable, please see [SPARK-29707](https://issues.apache.org/jira/browse/SPARK-29707) for details. ### Does this PR introduce any user-facing change? Yes, user can configure the parameter to get more information about data source scan metadata. ### How was this patch tested? Manual test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548714602 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548714945 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
alfozan commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548715130 > @alfozan > You can see my branch's newest change. and check the update Could you share a link to the new branch/PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548714602 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
SparkQA commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548715516 **[Test build #113071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113071/testReport)** for PR 26352 at commit [`94b7e79`](https://github.com/apache/spark/commit/94b7e79a97ab303fb227dd2f256990efd3c93849). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
SparkQA removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548670945 **[Test build #113071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113071/testReport)** for PR 26352 at commit [`94b7e79`](https://github.com/apache/spark/commit/94b7e79a97ab303fb227dd2f256990efd3c93849). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548715955 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113071/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins commented on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548715950 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
wangyum commented on a change in pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#discussion_r341495574 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -58,7 +58,8 @@ trait DataSourceScanExec extends LeafExecNode { override def simpleString(maxFields: Int): String = { val metadataEntries = metadata.toSeq.sorted.map { case (key, value) => -key + ": " + StringUtils.abbreviate(redact(value), 100) +key + ": " + StringUtils.abbreviate(redact(value), + SQLConf.get.maxDataSourceScanMetadataStringLength) Review comment: `SQLConf.get.maxDataSourceScanMetadataStringLength)` -> `conf.maxDataSourceScanMetadataStringLength)`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548715955 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113071/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql
AmplabJenkins removed a comment on issue #26352: [SPARK-29696][SQL][TESTS] Port groupingsets.sql URL: https://github.com/apache/spark/pull/26352#issuecomment-548715950 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation.
AngersZh commented on issue #25028: [SPARK-28227][SQL] Support TRANSFORM with aggregation. URL: https://github.com/apache/spark/pull/25028#issuecomment-548716323 > > @alfozan > > You can see my branch's newest change. and check the update > > Could you share a link to the new branch/PR? https://github.com/AngersZh/spark/commit/f18a6f99043ec26ac19a8337f41194b6f972090c This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on a change in pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
fuwhu commented on a change in pull request #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#discussion_r341498093 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -58,7 +58,8 @@ trait DataSourceScanExec extends LeafExecNode { override def simpleString(maxFields: Int): String = { val metadataEntries = metadata.toSeq.sorted.map { case (key, value) => -key + ": " + StringUtils.abbreviate(redact(value), 100) +key + ": " + StringUtils.abbreviate(redact(value), + SQLConf.get.maxDataSourceScanMetadataStringLength) Review comment: thanks, udpated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
wangyum commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548718473 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548714945 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
SparkQA commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548718809 **[Test build #113082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113082/testReport)** for PR 26356 at commit [`f207bff`](https://github.com/apache/spark/commit/f207bffbda506d5919ec60357db6a62742c450f7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548719186 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins commented on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548719188 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548719186 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength'
AmplabJenkins removed a comment on issue #26356: [SPARK-29707][SQL] Add configuration 'spark.sql.maxDataSourceScanMetadataStringLength' URL: https://github.com/apache/spark/pull/26356#issuecomment-548719188 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17999/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands
cloud-fan commented on a change in pull request #26338: [SPARK-29680][SQL] ALTER TABLE CHANGE COLUMN should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26338#discussion_r341499950 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -151,7 +151,7 @@ statement | ALTER TABLE multipartIdentifier (ALTER | CHANGE) COLUMN? qualifiedName (TYPE dataType)? (COMMENT comment=STRING)? colPosition? #alterTableColumn -| ALTER TABLE tableIdentifier partitionSpec? +| ALTER TABLE multipartIdentifier partitionSpec? Review comment: Let's simply remove this syntax. We have ALTER TABLE CHANGE COLUMN and ALTER TABLE RENAME COLUMN, which is good enough to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] planga82 commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
planga82 commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-548720773 About your questions, yes, it add the relation to the graph always when appears InMemoryTableScan node. Looking for this kind of node it seems only appear when the chache method is called so there is no other uses of InMemorytableScan. There is no special indication that this is a cached dataframe, only the node type (InMemoryTableScan). We could try to put a diferent color to the node to clarify it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
SparkQA commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548723684 **[Test build #113077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113077/testReport)** for PR 26353 at commit [`e86b15a`](https://github.com/apache/spark/commit/e86b15a51b8af06844fcfd5fa75c1115145bd88e). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait HasRelativeError extends Params ` * `class _ImputerParams(HasInputCol, HasInputCols, HasOutputCol, HasOutputCols, HasRelativeError):` * `class _OneHotEncoderParams(HasInputCol, HasInputCols, HasOutputCol, HasOutputCols,` * `class _RobustScalerParams(HasInputCol, HasOutputCol, HasRelativeError):` * `class HasRelativeError(Params):` * ` class ResolveCoalesceHints(conf: SQLConf) extends Rule[LogicalPlan] ` * `case class AlterTableRecoverPartitionsStatement(` * `case class AlterTableRenamePartitionStatement(` * `case class DropNamespaceStatement(` * `case class LoadDataStatement(` * `case class ShowColumnsStatement(` * `case class DropNamespace(` * `case class DataSourceV2ScanRelation(` * `case class OptimizeLocalShuffleReader(conf: SQLConf) extends Rule[SparkPlan] ` * `case class LocalShuffleReaderExec(child: QueryStageExec) extends UnaryExecNode ` * `case class DropNamespaceExec(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
SparkQA removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548688871 **[Test build #113077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113077/testReport)** for PR 26353 at commit [`e86b15a`](https://github.com/apache/spark/commit/e86b15a51b8af06844fcfd5fa75c1115145bd88e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
AmplabJenkins removed a comment on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548724288 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113077/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs.
AmplabJenkins commented on issue #26353: [SPARK-29590][WEBUI]JDBC/ODBC tab in the spark UI support hide tables, to make it consistent with other tabs. URL: https://github.com/apache/spark/pull/26353#issuecomment-548724281 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org