[GitHub] [spark] AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-507012828 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-507012833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12242/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-507012833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12242/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-507012828 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
SparkQA commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-507012594 **[Test build #107049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107049/testReport)** for PR 24382 at commit [`3b53e88`](https://github.com/apache/spark/commit/3b53e884463c04d27246f93875b34c0fe7ec6898). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25016: [SPARK-28200][SQL] Decimal overflow handling in ExpressionEncoder
SparkQA commented on issue #25016: [SPARK-28200][SQL] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010913 **[Test build #107048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107048/testReport)** for PR 25016 at commit [`b3740ba`](https://github.com/apache/spark/commit/b3740ba153920d0dbee9b9d769c0e3b79029befa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mgaido91 commented on a change in pull request #25016: [SPARK-28200][SQL] Decimal overflow handling in ExpressionEncoder
mgaido91 commented on a change in pull request #25016: [SPARK-28200][SQL] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#discussion_r298819201 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ## @@ -379,6 +380,80 @@ class ExpressionEncoderSuite extends CodegenInterpretedPlanTest with AnalysisTes assert(e.getMessage.contains("tuple with more than 22 elements are not supported")) } + // Scala / Java big decimals -- + + encodeDecodeTest(BigDecimal(("9" * 20) + "." + "9" * 18), +"scala decimal within precision/scale limit") + encodeDecodeTest(new java.math.BigDecimal(("9" * 20) + "." + "9" * 18), +"java decimal within precision/scale limit") + + encodeDecodeTest(BigDecimal(("9" * 20) + "." + "9" * 18).unary_-, +"negative scala decimal within precision/scale limit") + encodeDecodeTest(new java.math.BigDecimal(("9" * 20) + "." + "9" * 18).negate, +"negative java decimal within precision/scale limit") + + testOverflowingBigNumeric(BigDecimal("1" * 21), "scala big decimal") + testOverflowingBigNumeric(new java.math.BigDecimal("1" * 21), "java big decimal") + + testOverflowingBigNumeric(-BigDecimal("1" * 21), "negative scala big decimal") + testOverflowingBigNumeric(new java.math.BigDecimal("1" * 21).negate, "negative java big decimal") + + testOverflowingBigNumeric(BigDecimal(("1" * 21) + ".123"), +"scala big decimal with fractional part") + testOverflowingBigNumeric(new java.math.BigDecimal(("1" * 21) + ".123"), +"java big decimal with fractional part") + + testOverflowingBigNumeric(BigDecimal(("1" * 21) + "." + "" * 100), +"scala big decimal with long fractional part") + testOverflowingBigNumeric(new java.math.BigDecimal(("1" * 21) + "." + "" * 100), +"java big decimal with long fractional part") + + // Scala / Java big integers -- + + encodeDecodeTest(BigInt("9" * 38), "scala big integer within precision limit") + encodeDecodeTest(new BigInteger("9" * 38), "java big integer within precision limit") + + encodeDecodeTest(-BigInt("9" * 38), +"negative scala big integer within precision limit") + encodeDecodeTest(new BigInteger("9" * 38).negate(), +"negative java big integer within precision limit") + + testOverflowingBigNumeric(BigInt("1" * 39), "scala big int") + testOverflowingBigNumeric(new BigInteger("1" * 39), "java big integer") + + testOverflowingBigNumeric(-BigInt("1" * 39), "negative scala big int") + testOverflowingBigNumeric(new BigInteger("1" * 39).negate, "negative java big integer") + + testOverflowingBigNumeric(BigInt("9" * 100), "scala very large big int") + testOverflowingBigNumeric(new BigInteger("9" * 100), "java very big int") + + private def testOverflowingBigNumeric[T: TypeTag](bigDecimal: T, testName: String): Unit = { +for { Review comment: `Seq("true", "false").foreach` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-507010834 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12241/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010829 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010830 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12240/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-507010834 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12241/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010830 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12240/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010829 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-507010832 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-507010832 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen edited a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
JoshRosen edited a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010617 @mickjermsurawong-stripe, I think we also had a test case for `RowEncoder`? Could you also submit that change (in `RowEncoderSuite.scala`) as part of this PR? Even though we're not actually changing `RowEncoder` behavior here, the old code was lacking explicit test coverage for that path, so it'd be good to pull in that change to strengthen the tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
JoshRosen commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010788 jenkins this is ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen edited a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
JoshRosen edited a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010617 jenkins this is ok to test @mickjermsurawong-stripe, I think we also had a test case for `RowEncoder`? Could you also submit that change (in `RowEncoderSuite.scala`) as part of this PR? Even though we're not actually changing `RowEncoder` behavior here, the old code was lacking explicit test coverage for that path, so it'd be good to pull in that change to strengthen the tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010257 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder
JoshRosen commented on issue #25016: [SPARK-28200] Decimal overflow handling in ExpressionEncoder URL: https://github.com/apache/spark/pull/25016#issuecomment-507010617 jenkins this is ok to test @mickjermsurawong-stripe, I think we also had a test case for `RowEncoder`? Could you also submit that change as part of this PR? Even though we're not actually changing `RowEncoder` behavior here, the old code was lacking explicit test coverage for that path, so it'd be good to pull in that change to strengthen the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-507010569 **[Test build #107047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107047/testReport)** for PR 24978 at commit [`b767977`](https://github.com/apache/spark/commit/b767977755179f2950e4a9b8ee3bcc2a58902d2e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling
AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016#issuecomment-507010257 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling
AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016#issuecomment-507010152 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling
AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016#issuecomment-507010152 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling
AmplabJenkins removed a comment on issue #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016#issuecomment-507010137 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling
AmplabJenkins commented on issue #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016#issuecomment-507010137 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mickjermsurawong-stripe opened a new pull request #25016: [SPARK-28200] Decimal overflow handling
mickjermsurawong-stripe opened a new pull request #25016: [SPARK-28200] Decimal overflow handling URL: https://github.com/apache/spark/pull/25016 ## What changes were proposed in this pull request? - In [SPARK-23179](https://github.com/apache/spark/pull/20350), an option to throw exception on decimal overflow was introduced. The option was applied only to sql Decimal operation and `RowEncoder` but not `ExpressionEncoder`. - The problem now is round-tripping overflowing java/scala BigDecimal/BigInteger currently return null results. - The serializer encode java/scala BigDecimal to to sql Decimal, which still has the underlying the former. - When writing out to UnsafeRow, `changePrecision` will be false and row has null value. https://github.com/apache/spark/blob/24e1e41648de58d3437e008b187b84828830e238/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java#L202-L206 - This PR adds the option to throw when detecting overflowing BigDecimal/BigInteger. This gives a consistent behavior between decimal arithmetic on sql expression (DecimalPrecision), and getting decimal from dataframe (RowEncoder) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mgaido91 commented on a change in pull request #25011: [SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation
mgaido91 commented on a change in pull request #25011: [SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation URL: https://github.com/apache/spark/pull/25011#discussion_r298818695 ## File path: python/pyspark/ml/linalg/__init__.py ## @@ -386,14 +386,14 @@ def squared_distance(self, other): def toArray(self): """ -Returns an numpy.ndarray +Returns the underlying numpy.ndarray Review comment: In case of `DenseVector` it means that there is a vector containing the values...and it think makes sense to report it to the users, in order that they know what happens if they modify it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
SparkQA commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507007256 **[Test build #107046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107046/testReport)** for PR 24972 at commit [`1d7e899`](https://github.com/apache/spark/commit/1d7e89926287ff6d4e017dbbbcc906c69e500b58). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507007272 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
SparkQA removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006436 **[Test build #107046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107046/testReport)** for PR 24972 at commit [`1d7e899`](https://github.com/apache/spark/commit/1d7e89926287ff6d4e017dbbbcc906c69e500b58). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507007273 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107046/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507007273 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107046/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507007272 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
SparkQA commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006436 **[Test build #107046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107046/testReport)** for PR 24972 at commit [`1d7e899`](https://github.com/apache/spark/commit/1d7e89926287ff6d4e017dbbbcc906c69e500b58). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12239/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins removed a comment on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006361 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006361 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool
AmplabJenkins commented on issue #24972: [SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#issuecomment-507006362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12239/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
SparkQA commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed URL: https://github.com/apache/spark/pull/24964#issuecomment-507005118 **[Test build #4813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4813/testReport)** for PR 24964 at commit [`d2330cc`](https://github.com/apache/spark/commit/d2330cc8b4a74378556f7ae5924b3bb9388219d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
SparkQA removed a comment on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed URL: https://github.com/apache/spark/pull/24964#issuecomment-506998921 **[Test build #4813 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4813/testReport)** for PR 24964 at commit [`d2330cc`](https://github.com/apache/spark/commit/d2330cc8b4a74378556f7ae5924b3bb9388219d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tonix517 commented on issue #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL
Tonix517 commented on issue #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL URL: https://github.com/apache/spark/pull/24994#issuecomment-507002652 Copied the link from the JIRA ticket: https://www.postgresql.org/docs/12/functions-math.html#FUNCTIONS-MATH-HYP-TABLE Just tried Hive and these functions were not supported indeed. > These are very niche functions. Does any other DB support it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-507001560 > @WangGuangxin Could you submit a follow-up PR for updating the document? See the example https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#schema-merging ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-507001494 > Thanks for your work @WangGuangxin ! > > What is your JIRA account? We need to assign the assignee field to your JIRA account. https://issues.apache.org/jira/browse/SPARK-11412 Thanks. My jira account is EdisonWang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #25012: [SPARK-28215][SQL][R] as_tibble was removed from Arrow R API
viirya commented on issue #25012: [SPARK-28215][SQL][R] as_tibble was removed from Arrow R API URL: https://github.com/apache/spark/pull/25012#issuecomment-50779 cc @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #24995: [SPARK-28196][SQL] Add a new `listTables` and `listLocalTempViews` APIs for SessionCatalog
gatorsmile closed pull request #24995: [SPARK-28196][SQL] Add a new `listTables` and `listLocalTempViews` APIs for SessionCatalog URL: https://github.com/apache/spark/pull/24995 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #24985: [SPARK-28184][SQL][TEST] Avoid creating new sessions in SparkMetadataOperationSuite
gatorsmile closed pull request #24985: [SPARK-28184][SQL][TEST] Avoid creating new sessions in SparkMetadataOperationSuite URL: https://github.com/apache/spark/pull/24985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24991: SPARK-28188 Materialize Dataframe API
srowen commented on issue #24991: SPARK-28188 Materialize Dataframe API URL: https://github.com/apache/spark/pull/24991#issuecomment-506999407 I don't think we should add this. It's already very common to `.count()` or `.mapPartitions` with a no-op to do this. I do think there are use cases for proactively materializing, but, it's overused too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #24992: [SPARK-28194][SQL] Refactor code to prevent None.get in EnsureRequirements
srowen commented on a change in pull request #24992: [SPARK-28194][SQL] Refactor code to prevent None.get in EnsureRequirements URL: https://github.com/apache/spark/pull/24992#discussion_r298815400 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala ## @@ -231,14 +231,15 @@ case class EnsureRequirements(conf: SQLConf) extends Rule[SparkPlan] { val keysAndIndexes = currentOrderOfKeys.zipWithIndex expectedOrderOfKeys.foreach(expression => { - val index = keysAndIndexes.find { case (e, idx) => + keysAndIndexes.find { case (e, idx) => // As we may have the same key used many times, we need to filter out its occurrence we // have already used. e.semanticEquals(expression) && !pickedIndexes.contains(idx) - }.map(_._2).get - pickedIndexes += index - leftKeysBuffer.append(leftKeys(index)) - rightKeysBuffer.append(rightKeys(index)) + }.map(_._2).map(index => { Review comment: This has to be `foreach`, not `map`, and can be `.foreach { index =>` However I wonder if there's some other bug if the code really doesn't expect to not find a value? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24996: [SPARK-28199][SS] Remove usage of deprecated ProcessingTime in Spark codebase
srowen commented on issue #24996: [SPARK-28199][SS] Remove usage of deprecated ProcessingTime in Spark codebase URL: https://github.com/apache/spark/pull/24996#issuecomment-506999280 Why introduce a new abstraction? This is what `Trigger.ProcessingTime` is meant to be. Just move the implementation to that class, and have the deprecated impl use it, rather than the other way around? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24998: [SPARK-28202] [Core] [Test] Avoid noises of system props in SparkConfSuite
srowen commented on issue #24998: [SPARK-28202] [Core] [Test] Avoid noises of system props in SparkConfSuite URL: https://github.com/apache/spark/pull/24998#issuecomment-506999082 This test doesn't fail in a normal test run -- why would you customize the test env? that seems like the problem? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25011: [SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation
srowen commented on a change in pull request #25011: [SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation URL: https://github.com/apache/spark/pull/25011#discussion_r298815289 ## File path: python/pyspark/ml/linalg/__init__.py ## @@ -386,14 +386,14 @@ def squared_distance(self, other): def toArray(self): """ -Returns an numpy.ndarray +Returns the underlying numpy.ndarray Review comment: I think this is OK, but just wondering how much to promise that it's the underlying array This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
AmplabJenkins removed a comment on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed URL: https://github.com/apache/spark/pull/24964#issuecomment-505377517 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
SparkQA commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed URL: https://github.com/apache/spark/pull/24964#issuecomment-506998921 **[Test build #4813 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4813/testReport)** for PR 24964 at commit [`d2330cc`](https://github.com/apache/spark/commit/d2330cc8b4a74378556f7ae5924b3bb9388219d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve
vanzin commented on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve URL: https://github.com/apache/spark/pull/24982#issuecomment-506998016 > The target is getting all tasks with particular status from all tasks. Right. That means a view with the "status" index, configured with both "first" and 'last" to match the desired status. That will return all the tasks with the status you're looking for. And doesn't require deserializing anything to find the desired tasks. Unless I'm not understanding what exactly it is you're trying to find. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
LantaoJin commented on issue #24964: [SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed URL: https://github.com/apache/spark/pull/24964#issuecomment-506997893 Gentle ping @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile edited a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile edited a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-506996308 Thanks for your work @WangGuangxin ! What is your JIRA account? We need to assign the assignee field to your JIRA account. https://issues.apache.org/jira/browse/SPARK-11412 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-506996308 Thanks for your work @WangGuangxin ! What is your JIRA account? We need to assign the author to your JIRA account. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile edited a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile edited a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-506996308 Thanks for your work @WangGuangxin ! What is your JIRA account? We need to assign the assignee field to your JIRA account. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile closed pull request #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-506996085 Thanks! Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
gatorsmile commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-506996039 @WangGuangxin Could you submit a follow-up PR for updating the document? See the example https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#schema-merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL
Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL URL: https://github.com/apache/spark/pull/24994#discussion_r298813645 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala ## @@ -557,6 +578,27 @@ case class Sin(child: Expression) extends UnaryMathExpression(math.sin, "SIN") """) case class Sinh(child: Expression) extends UnaryMathExpression(math.sinh, "SINH") +@ExpressionDescription( + usage = """ +_FUNC_(expr) - Returns inverse hyperbolic sine of `expr`. + """, + arguments = """ +Arguments: + * expr - hyperbolic angle + """, + examples = """ +Examples: + > SELECT _FUNC_(0); + 0.0 + """) +case class Asinh(child: Expression) + extends UnaryMathExpression((x: Double) => math.log(x + math.sqrt(x * x + 1.0)), "ASINH") { + override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = { +defineCodeGen(ctx, ev, c => + s"${ev.value} = java.lang.Math.log($c + java.lang.Math.sqrt($c * $c + 1.0));") Review comment: took a closer look at asinh\atanh implementation in FastMath. the negative handling inside is to apply calculation optimization on abs(double a) and then apply proper sign at the end of the function to return, which should produce the same value as the original formula i'm using here (https://en.wikipedia.org/wiki/Inverse_hyperbolic_functions#Definitions_in_terms_of_logarithms). But FastMath runs faster than the original functions though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24972: [WIP][SPARK-28167][SQL] Show global temporary view in database tool
gatorsmile commented on a change in pull request #24972: [WIP][SPARK-28167][SQL] Show global temporary view in database tool URL: https://github.com/apache/spark/pull/24972#discussion_r298813556 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -775,32 +775,61 @@ class SessionCatalog( * Note that, if the specified database is global temporary view database, we will list global * temporary views. */ - def listTables(db: String): Seq[TableIdentifier] = listTables(db, "*") Review comment: Yes. Definitely, we should not change the semantics of listTable API. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
AmplabJenkins removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506991600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
AmplabJenkins removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506991602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107044/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
AmplabJenkins commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506991602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107044/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
AmplabJenkins commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506991600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
SparkQA removed a comment on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506981375 **[Test build #107044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107044/testReport)** for PR 25010 at commit [`4928330`](https://github.com/apache/spark/commit/4928330c928e988f71002a5f5f17ae9283232e2a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
SparkQA commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506991499 **[Test build #107044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107044/testReport)** for PR 25010 at commit [`4928330`](https://github.com/apache/spark/commit/4928330c928e988f71002a5f5f17ae9283232e2a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506989224 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107045/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506989222 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506989222 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506989224 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107045/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506981759 **[Test build #107045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107045/testReport)** for PR 24978 at commit [`7f0397a`](https://github.com/apache/spark/commit/7f0397a11958fe5712352d58cc4938df4b926b3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506989182 **[Test build #107045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107045/testReport)** for PR 24978 at commit [`7f0397a`](https://github.com/apache/spark/commit/7f0397a11958fe5712352d58cc4938df4b926b3c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506988061 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107043/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506988059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506988061 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107043/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506980445 **[Test build #107043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107043/testReport)** for PR 24978 at commit [`228ed58`](https://github.com/apache/spark/commit/228ed58e0b79f1647247977982f0bd13f0fd0bc8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506988059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506988026 **[Test build #107043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107043/testReport)** for PR 24978 at commit [`228ed58`](https://github.com/apache/spark/commit/228ed58e0b79f1647247977982f0bd13f0fd0bc8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan.
AmplabJenkins removed a comment on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan. URL: https://github.com/apache/spark/pull/25015#issuecomment-506984834 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan.
AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan. URL: https://github.com/apache/spark/pull/25015#issuecomment-506984870 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan.
AmplabJenkins removed a comment on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan. URL: https://github.com/apache/spark/pull/25015#issuecomment-506984581 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan.
AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a pluggable statistics plan visitor for a logical plan. URL: https://github.com/apache/spark/pull/25015#issuecomment-506984834 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 opened a new pull request #25015: [SPARK-28217][SQL] Allow a custom statistics logical plan visitor to be plugged in.
imback82 opened a new pull request #25015: [SPARK-28217][SQL] Allow a custom statistics logical plan visitor to be plugged in. URL: https://github.com/apache/spark/pull/25015 ## What changes were proposed in this pull request? Spark currently has two built-in statistics plan visitor: SizeInBytesOnlyStatsPlanVisitor and BasicStatsPlanVisitor. However, this is a bit limited since there is no way to plug in a custom plan visitor - from which a custom query optimizer can benefit from. This PR allowers the user to specify a custom stat plan visitor via a Spark conf to override the built-in one: ```Scala // First create your custom stat plan visitor. class MyStatsPlanVisitor extends LogicalPlanVisitor[Statistics] { // Implement LogicalPlanVisitor[Statistics] trait } // Set the visitor via Spark conf. spark.conf.set("spark.sql.catalyst.statsPlanVisitorClass", "MyStatsPlanVisitor") // Now, stat() on a LogicalPlan object will use MyStatsPlanVisitor as a stat plan visitor. ``` ## How was this patch tested? Existing and new unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a custom statistics logical plan visitor to be plugged in.
AmplabJenkins commented on issue #25015: [SPARK-28217][SQL] Allow a custom statistics logical plan visitor to be plugged in. URL: https://github.com/apache/spark/pull/25015#issuecomment-506984581 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
AmplabJenkins removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506982534 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107040/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
AmplabJenkins removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506982533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
AmplabJenkins commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506982534 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107040/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
AmplabJenkins commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506982533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
SparkQA removed a comment on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506970649 **[Test build #107040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107040/testReport)** for PR 25014 at commit [`d4589be`](https://github.com/apache/spark/commit/d4589be4a8fe15b04f3779a301173b2a048480e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils
SparkQA commented on issue #25014: [SPARK-28216][SQl][TEST] Move getDataSize from StatisticsCollectionTestBase to SQLTestUtils URL: https://github.com/apache/spark/pull/25014#issuecomment-506982422 **[Test build #107040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107040/testReport)** for PR 25014 at commit [`d4589be`](https://github.com/apache/spark/commit/d4589be4a8fe15b04f3779a301173b2a048480e9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506982146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12238/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins removed a comment on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506982143 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506982146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12238/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
AmplabJenkins commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506982143 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
SparkQA commented on issue #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution URL: https://github.com/apache/spark/pull/24978#issuecomment-506981759 **[Test build #107045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107045/testReport)** for PR 24978 at commit [`7f0397a`](https://github.com/apache/spark/commit/7f0397a11958fe5712352d58cc4938df4b926b3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow
SparkQA commented on issue #25010: [SPARK-28201][SQL] Revisit MakeDecimal behavior on overflow URL: https://github.com/apache/spark/pull/25010#issuecomment-506981375 **[Test build #107044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107044/testReport)** for PR 25010 at commit [`4928330`](https://github.com/apache/spark/commit/4928330c928e988f71002a5f5f17ae9283232e2a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org