[jira] [Assigned] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37921: - Assignee: William Hyun > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Assignee: William Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37921. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35219 [https://github.com/apache/spark/pull/35219] > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37921: Assignee: (was: Apache Spark) > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476731#comment-17476731 ] Apache Spark commented on SPARK-37921: -- User 'williamhyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35219 > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476732#comment-17476732 ] Apache Spark commented on SPARK-37921: -- User 'williamhyun' has created a pull request for this issue: https://github.com/apache/spark/pull/35219 > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
[ https://issues.apache.org/jira/browse/SPARK-37921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37921: Assignee: Apache Spark > Update OrcReadBenchmark to use Hive ORC reader as the basis > --- > > Key: SPARK-37921 > URL: https://issues.apache.org/jira/browse/SPARK-37921 > Project: Spark > Issue Type: Test > Components: Tests >Affects Versions: 3.3.0 >Reporter: William Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37921) Update OrcReadBenchmark to use Hive ORC reader as the basis
William Hyun created SPARK-37921: Summary: Update OrcReadBenchmark to use Hive ORC reader as the basis Key: SPARK-37921 URL: https://issues.apache.org/jira/browse/SPARK-37921 Project: Spark Issue Type: Test Components: Tests Affects Versions: 3.3.0 Reporter: William Hyun -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37920) Change Code style for pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37920. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35218 [https://github.com/apache/spark/pull/35218] > Change Code style for pom.xml > - > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Assignee: Jackey Lee >Priority: Major > Fix For: 3.3.0 > > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37920) Remove tab character and trailing space in pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37920: -- Summary: Remove tab character and trailing space in pom.xml (was: Change Code style for pom.xml) > Remove tab character and trailing space in pom.xml > -- > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Assignee: Jackey Lee >Priority: Major > Fix For: 3.3.0 > > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37920) Change Code style for pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37920: - Assignee: Jackey Lee > Change Code style for pom.xml > - > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Assignee: Jackey Lee >Priority: Major > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476710#comment-17476710 ] Venkat Sambath commented on SPARK-20178: Is design doc referenced in this jira https://docs.google.com/document/d/1D3b_ishMfm5sXmRS494JrOJmL9V_TRVZUB4TgK1l1fY/edit?usp=sharing available anywhere else? > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler, Spark Core >Affects Versions: 2.1.0 >Reporter: Thomas Graves >Priority: Major > Labels: bulk-closed > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37897) Filter with subexpression elimination may cause query failed
[
https://issues.apache.org/jira/browse/SPARK-37897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476700#comment-17476700
]
L. C. Hsieh commented on SPARK-37897:
-
SQL is a declarative language, please don't think it in imperative style of
short-circuit evaluation. How to evaluate the predicate is
implementation-dependent.
> Filter with subexpression elimination may cause query failed
>
>
> Key: SPARK-37897
> URL: https://issues.apache.org/jira/browse/SPARK-37897
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: hujiahua
>Priority: Major
> Attachments: image-2022-01-13-20-22-09-055.png
>
>
>
> The following test results will fail, the root cause was that the execution
> order of filter predicates had changed after subexpression elimination. So I
> think we should keep predicates execution order after subexpression
> elimination.
> {code:java}
> test("filter with subexpression elimination may cause query failed.") {
> withSQLConf((SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key, "false")) {
> val df = Seq(-1, 1, 2).toDF("c1")
> //register `plusOne` udf, and the function will failed if input was not a
> positive number.
> spark.sqlContext.udf.register("plusOne",
> (n: Int) => { if (n >= 0) n + 1 else throw new SparkException("Must be
> positive number.") })
> val result = df.filter("c1 >= 0 and plusOne(c1) > 1 and plusOne(c1) <
> 3").collect()
> assert(result.size === 1)
> }
> }
> Caused by: org.apache.spark.SparkException: Must be positive number.
> at
> org.apache.spark.sql.DataFrameSuite.$anonfun$new$3(DataFrameSuite.scala:67)
> at
> scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
> ... 20 more{code}
>
> https://github.com/apache/spark/blob/0e186e8a19926f91810f3eaf174611b71e598de6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala#L63
> !image-2022-01-13-20-22-09-055.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37920) Change Code style for pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37920: Assignee: (was: Apache Spark) > Change Code style for pom.xml > - > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Priority: Major > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37920) Change Code style for pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37920: Assignee: Apache Spark > Change Code style for pom.xml > - > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Assignee: Apache Spark >Priority: Major > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37920) Change Code style for pom.xml
[ https://issues.apache.org/jira/browse/SPARK-37920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476672#comment-17476672 ] Apache Spark commented on SPARK-37920: -- User 'stczwd' has created a pull request for this issue: https://github.com/apache/spark/pull/35218 > Change Code style for pom.xml > - > > Key: SPARK-37920 > URL: https://issues.apache.org/jira/browse/SPARK-37920 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Jackey Lee >Priority: Major > > There are some tabs in pom.xml, which don't seem to be standardized. This pr > tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37920) Change Code style for pom.xml
Jackey Lee created SPARK-37920: -- Summary: Change Code style for pom.xml Key: SPARK-37920 URL: https://issues.apache.org/jira/browse/SPARK-37920 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.3.0 Reporter: Jackey Lee There are some tabs in pom.xml, which don't seem to be standardized. This pr tries to modify this problem. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37919) SQL UI shows accurate Metrics with stage retries
Jackey Lee created SPARK-37919: -- Summary: SQL UI shows accurate Metrics with stage retries Key: SPARK-37919 URL: https://issues.apache.org/jira/browse/SPARK-37919 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 3.3.0 Reporter: Jackey Lee At present, the SQL UI page can show us the metrics of sql execution. However the metrics are not accurate with task or stage retries. This PR is used to provide accurate Metrics in the UI page, mainly based on SPARK-37831 . -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37842) Use `multi-catch` to simplify duplicate exception handling behavior in Java code
[ https://issues.apache.org/jira/browse/SPARK-37842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37842. -- Resolution: Won't Fix > Use `multi-catch` to simplify duplicate exception handling behavior in Java > code > > > Key: SPARK-37842 > URL: https://issues.apache.org/jira/browse/SPARK-37842 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yang Jie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37876) Move SpecificParquetRecordReaderBase.listDirectory to TestUtils
[ https://issues.apache.org/jira/browse/SPARK-37876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37876: Assignee: Yang Jie > Move SpecificParquetRecordReaderBase.listDirectory to TestUtils > --- > > Key: SPARK-37876 > URL: https://issues.apache.org/jira/browse/SPARK-37876 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > > The `SpecificParquetRecordReaderBase.listDirectory` method is used to return > the list of files at `path` recursively and the result will skips files that > are ignored normally by MapReduce. > This method is only used by tests in Spark now and it also includes non > parquet related test cases, such as `OrcColumnarBatchReaderSuite`. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37876) Move SpecificParquetRecordReaderBase.listDirectory to TestUtils
[ https://issues.apache.org/jira/browse/SPARK-37876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37876. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35177 [https://github.com/apache/spark/pull/35177] > Move SpecificParquetRecordReaderBase.listDirectory to TestUtils > --- > > Key: SPARK-37876 > URL: https://issues.apache.org/jira/browse/SPARK-37876 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.3.0 > > > The `SpecificParquetRecordReaderBase.listDirectory` method is used to return > the list of files at `path` recursively and the result will skips files that > are ignored normally by MapReduce. > This method is only used by tests in Spark now and it also includes non > parquet related test cases, such as `OrcColumnarBatchReaderSuite`. > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37862) RecordBinaryComparator should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37862. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35161 [https://github.com/apache/spark/pull/35161] > RecordBinaryComparator should fast skip the check of aligning with unaligned > platform > - > > Key: SPARK-37862 > URL: https://issues.apache.org/jira/browse/SPARK-37862 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Priority: Minor > Fix For: 3.3.0 > > > Same with SPARK-37796 > It would be better to fast sikip the check of aligning if the platform is > unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37862) RecordBinaryComparator should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37862: Assignee: XiDuo You > RecordBinaryComparator should fast skip the check of aligning with unaligned > platform > - > > Key: SPARK-37862 > URL: https://issues.apache.org/jira/browse/SPARK-37862 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Assignee: XiDuo You >Priority: Minor > Fix For: 3.3.0 > > > Same with SPARK-37796 > It would be better to fast sikip the check of aligning if the platform is > unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37862) RecordBinaryComparator should fast skip the check of aligning with unaligned platform
[ https://issues.apache.org/jira/browse/SPARK-37862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37862: - Priority: Minor (was: Major) > RecordBinaryComparator should fast skip the check of aligning with unaligned > platform > - > > Key: SPARK-37862 > URL: https://issues.apache.org/jira/browse/SPARK-37862 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: XiDuo You >Priority: Minor > > Same with SPARK-37796 > It would be better to fast sikip the check of aligning if the platform is > unaligned. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37854) Use type match to simplify TestUtils#withHttpConnection
[
https://issues.apache.org/jira/browse/SPARK-37854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-37854:
Assignee: Yang Jie
> Use type match to simplify TestUtils#withHttpConnection
> ---
>
> Key: SPARK-37854
> URL: https://issues.apache.org/jira/browse/SPARK-37854
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Trivial
>
> {code:java}
> if (connection.isInstanceOf[HttpsURLConnection]) {
>
>
> connection.asInstanceOf[HttpsURLConnection].setSSLSocketFactory(sslCtx.getSocketFactory())
>
> connection.asInstanceOf[HttpsURLConnection].setHostnameVerifier(verifier)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37854) Use type match to simplify TestUtils#withHttpConnection
[
https://issues.apache.org/jira/browse/SPARK-37854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-37854.
--
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 35154
[https://github.com/apache/spark/pull/35154]
> Use type match to simplify TestUtils#withHttpConnection
> ---
>
> Key: SPARK-37854
> URL: https://issues.apache.org/jira/browse/SPARK-37854
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Trivial
> Fix For: 3.3.0
>
>
> {code:java}
> if (connection.isInstanceOf[HttpsURLConnection]) {
>
>
> connection.asInstanceOf[HttpsURLConnection].setSSLSocketFactory(sslCtx.getSocketFactory())
>
> connection.asInstanceOf[HttpsURLConnection].setHostnameVerifier(verifier)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf resolved SPARK-37918. -- Resolution: Invalid > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37918: Assignee: Apache Spark > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Assignee: Apache Spark >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476621#comment-17476621 ] Apache Spark commented on SPARK-37918: -- User 'yikf' has created a pull request for this issue: https://github.com/apache/spark/pull/35217 > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476622#comment-17476622 ] Apache Spark commented on SPARK-37918: -- User 'yikf' has created a pull request for this issue: https://github.com/apache/spark/pull/35217 > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37918: Assignee: (was: Apache Spark) > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37918) Specified construct when instance []SessionStateBuilder
yikf created SPARK-37918: Summary: Specified construct when instance []SessionStateBuilder Key: SPARK-37918 URL: https://issues.apache.org/jira/browse/SPARK-37918 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.2.0 Reporter: yikf Fix For: 3.3.0 Currently, we support that a developer can modify the builder by providing custom versions of components, but not instance custom stateBuilder with specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37918) Specified construct when instance SessionStateBuilder
[ https://issues.apache.org/jira/browse/SPARK-37918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37918: - Summary: Specified construct when instance SessionStateBuilder (was: Specified construct when instance []SessionStateBuilder) > Specified construct when instance SessionStateBuilder > - > > Key: SPARK-37918 > URL: https://issues.apache.org/jira/browse/SPARK-37918 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.2.0 >Reporter: yikf >Priority: Major > Fix For: 3.3.0 > > > Currently, we support that a developer can modify the builder by providing > custom versions of components, but not instance custom stateBuilder with > specified construct -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37917) Push down limit 1 for right side of left semi/anti join
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476608#comment-17476608 ] Apache Spark commented on SPARK-37917: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/35216 > Push down limit 1 for right side of left semi/anti join > --- > > Key: SPARK-37917 > URL: https://issues.apache.org/jira/browse/SPARK-37917 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37917) Push down limit 1 for right side of left semi/anti join
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37917: Assignee: (was: Apache Spark) > Push down limit 1 for right side of left semi/anti join > --- > > Key: SPARK-37917 > URL: https://issues.apache.org/jira/browse/SPARK-37917 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37917) Push down limit 1 for right side of left semi/anti join
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476607#comment-17476607 ] Apache Spark commented on SPARK-37917: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/35216 > Push down limit 1 for right side of left semi/anti join > --- > > Key: SPARK-37917 > URL: https://issues.apache.org/jira/browse/SPARK-37917 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37917) Push down limit 1 for right side of left semi/anti join
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37917: Assignee: Apache Spark > Push down limit 1 for right side of left semi/anti join > --- > > Key: SPARK-37917 > URL: https://issues.apache.org/jira/browse/SPARK-37917 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37917) Push down limit 1 for right side of left semi/anti join
[ https://issues.apache.org/jira/browse/SPARK-37917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37917: Summary: Push down limit 1 for right side of left semi/anti join (was: Push down limit for right side of left semi/anti join) > Push down limit 1 for right side of left semi/anti join > --- > > Key: SPARK-37917 > URL: https://issues.apache.org/jira/browse/SPARK-37917 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37917) Push down limit for right side of left semi/anti join
Yuming Wang created SPARK-37917: --- Summary: Push down limit for right side of left semi/anti join Key: SPARK-37917 URL: https://issues.apache.org/jira/browse/SPARK-37917 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.0 Reporter: Yuming Wang -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join
[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34514: --- Assignee: Cheng Su (was: Apache Spark) > Push down limit for LEFT SEMI and LEFT ANTI join > > > Key: SPARK-34514 > URL: https://issues.apache.org/jira/browse/SPARK-34514 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Cheng Su >Assignee: Cheng Su >Priority: Trivial > Fix For: 3.2.0 > > > I found out during code review of > [https://github.com/apache/spark/pull/31567|https://github.com/apache/spark/pull/31567,]( > [https://github.com/apache/spark/pull/31567#discussion_r577379572] ), where > we can push down limit to the left side of LEFT SEMI and LEFT ANTI join, if > the join condition is empty. > Why it's safe to push down limit: > The semantics of LEFT SEMI join without condition: > (1). if right side is non-empty, output all rows from left side. > (2). if right side is empty, output nothing. > > The semantics of LEFT ANTI join without condition: > (1). if right side is non-empty, output nothing. > (2). if right side is empty, output all rows from left side. > > With the semantics of output all rows from left side or nothing (all or > nothing), it's safe to push down limit to left side. > NOTE: LEFT SEMI / LEFT ANTI join with non-empty condition is not safe for > limit push down, because output can be a portion of left side rows. > > Physical operator for LEFT SEMI / LEFT ANTI join without condition - > [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala#L200-L204] > . -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37916) The ConfigMap is assigned to incorrect namespace
[ https://issues.apache.org/jira/browse/SPARK-37916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476586#comment-17476586 ] Apache Spark commented on SPARK-37916: -- User 'chia7712' has created a pull request for this issue: https://github.com/apache/spark/pull/35215 > The ConfigMap is assigned to incorrect namespace > > > Key: SPARK-37916 > URL: https://issues.apache.org/jira/browse/SPARK-37916 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Chia-Ping Tsai >Priority: Major > > SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does > not have namespace value, so it assigns the ConfigMap to "default" namespace. > When we submit spark application in non-default namespace, it causes error > since it can't find the configmap in non-default namespace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37916) The ConfigMap is assigned to incorrect namespace
[ https://issues.apache.org/jira/browse/SPARK-37916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37916: Assignee: (was: Apache Spark) > The ConfigMap is assigned to incorrect namespace > > > Key: SPARK-37916 > URL: https://issues.apache.org/jira/browse/SPARK-37916 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Chia-Ping Tsai >Priority: Major > > SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does > not have namespace value, so it assigns the ConfigMap to "default" namespace. > When we submit spark application in non-default namespace, it causes error > since it can't find the configmap in non-default namespace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-37916) The ConfigMap is assigned to incorrect namespace
[ https://issues.apache.org/jira/browse/SPARK-37916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37916: Assignee: Apache Spark > The ConfigMap is assigned to incorrect namespace > > > Key: SPARK-37916 > URL: https://issues.apache.org/jira/browse/SPARK-37916 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Chia-Ping Tsai >Assignee: Apache Spark >Priority: Major > > SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does > not have namespace value, so it assigns the ConfigMap to "default" namespace. > When we submit spark application in non-default namespace, it causes error > since it can't find the configmap in non-default namespace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37916) The ConfigMap is assigned to incorrect namespace
[ https://issues.apache.org/jira/browse/SPARK-37916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476585#comment-17476585 ] Apache Spark commented on SPARK-37916: -- User 'chia7712' has created a pull request for this issue: https://github.com/apache/spark/pull/35215 > The ConfigMap is assigned to incorrect namespace > > > Key: SPARK-37916 > URL: https://issues.apache.org/jira/browse/SPARK-37916 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Chia-Ping Tsai >Priority: Major > > SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does > not have namespace value, so it assigns the ConfigMap to "default" namespace. > When we submit spark application in non-default namespace, it causes error > since it can't find the configmap in non-default namespace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37916) The ConfigMap is assigned to incorrect namespace
Chia-Ping Tsai created SPARK-37916: -- Summary: The ConfigMap is assigned to incorrect namespace Key: SPARK-37916 URL: https://issues.apache.org/jira/browse/SPARK-37916 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.3.0 Reporter: Chia-Ping Tsai SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does not have namespace value, so it assigns the ConfigMap to "default" namespace. When we submit spark application in non-default namespace, it causes error since it can't find the configmap in non-default namespace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
