[jira] [Commented] (SPARK-34112) Upgrade ORC

[ https://issues.apache.org/jira/browse/SPARK-34112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414731#comment-17414731 ] Dongjoon Hyun commented on SPARK-34112: --- Yep. ORC 1.7 is developed to be align with this,

[jira] [Commented] (SPARK-36696) spark.read.parquet loads empty dataset

[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414721#comment-17414721 ] Micah Kornfield commented on SPARK-36696: - What [~gershinsky]  wrote seems to make sense from my

[jira] [Commented] (SPARK-36706) OverwriteByExpression conversion in DataSourceV2Strategy use wrong deleteExpr translation

[ https://issues.apache.org/jira/browse/SPARK-36706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414717#comment-17414717 ] Huaxin Gao commented on SPARK-36706: I will fix this. Thanks for pinging me [~hyukjin.kwon] >

[jira] [Assigned] (SPARK-36683) Support secant and cosecant

[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36683: Assignee: Apache Spark > Support secant and cosecant > --- > >

[jira] [Commented] (SPARK-36683) Support secant and cosecant

[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414712#comment-17414712 ] Apache Spark commented on SPARK-36683: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36683) Support secant and cosecant

[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36683: Assignee: (was: Apache Spark) > Support secant and cosecant >

[jira] [Commented] (SPARK-34208) Upgrade ORC to 1.6.7

[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414708#comment-17414708 ] Dongjoon Hyun commented on SPARK-34208: --- It's already reported, [~holden]. The fix landed the

[jira] [Comment Edited] (SPARK-34208) Upgrade ORC to 1.6.7

[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414708#comment-17414708 ] Dongjoon Hyun edited comment on SPARK-34208 at 9/14/21, 4:17 AM: - It's

[jira] [Updated] (SPARK-34208) Upgrade ORC to 1.6.7

[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34208: -- Attachment: Screen Shot 2021-09-13 at 9.15.01 PM.png > Upgrade ORC to 1.6.7 >

[jira] [Commented] (SPARK-36706) OverwriteByExpression conversion in DataSourceV2Strategy use wrong deleteExpr translation

[ https://issues.apache.org/jira/browse/SPARK-36706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414704#comment-17414704 ] Hyukjin Kwon commented on SPARK-36706: -- cc [~huaxingao] FYI > OverwriteByExpression conversion in

[jira] [Commented] (SPARK-36749) The count result of the dimension table filed changes as `exector.memory` changes.

[ https://issues.apache.org/jira/browse/SPARK-36749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414700#comment-17414700 ] Hyukjin Kwon commented on SPARK-36749: -- Is this bug still reproducible in Spark 3.x? > The count

[jira] [Commented] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

[ https://issues.apache.org/jira/browse/SPARK-36701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414701#comment-17414701 ] Hyukjin Kwon commented on SPARK-36701: -- Can you see if it works w Spark 3.x? > Structured

[jira] [Updated] (SPARK-36749) The count result of the dimension table filed changes as `exector.memory` changes.

[ https://issues.apache.org/jira/browse/SPARK-36749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LanYang updated SPARK-36749: Description: hi~, every one! Here‘s a very strange questions!!!  The meaning of this sql is count the

[jira] [Created] (SPARK-36749) The count result of the dimension table filed changes as `exector.memory` changes.

LanYang created SPARK-36749: --- Summary: The count result of the dimension table filed changes as `exector.memory` changes. Key: SPARK-36749 URL: https://issues.apache.org/jira/browse/SPARK-36749 Project:

[jira] [Updated] (SPARK-36749) The count result of the dimension table filed changes as `exector.memory` changes.

[ https://issues.apache.org/jira/browse/SPARK-36749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LanYang updated SPARK-36749: Attachment: wrong_result.log corrent_result.log > The count result of the dimension table

[jira] [Commented] (SPARK-36596) Review and fix issues in 3.2.0 Documents

[ https://issues.apache.org/jira/browse/SPARK-36596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414681#comment-17414681 ] Gengliang Wang commented on SPARK-36596: [~holden] Yes, marking this one as fixed. Thanks! >

[jira] [Resolved] (SPARK-36596) Review and fix issues in 3.2.0 Documents

[ https://issues.apache.org/jira/browse/SPARK-36596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36596. Resolution: Fixed > Review and fix issues in 3.2.0 Documents >

[jira] [Commented] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414678#comment-17414678 ] Apache Spark commented on SPARK-36705: -- User 'c21' has created a pull request for this issue:

[jira] [Commented] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414677#comment-17414677 ] Apache Spark commented on SPARK-36705: -- User 'c21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36705: --- Assignee: Minchu Yang > Disable push based shuffle when IO encryption is

[jira] [Resolved] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36748. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33982

[jira] [Assigned] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36748: Assignee: Xinrong Meng > Introduce the 'compute.isin_limit' option >

[jira] [Updated] (SPARK-36727) Support sql overwrite a path that is also being read from when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-36727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongwei updated SPARK-36727: External issue URL: https://github.com/apache/spark/pull/33986 > Support sql overwrite a path that is

[jira] [Updated] (SPARK-36727) Support sql overwrite a path that is also being read from when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-36727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongwei updated SPARK-36727: External issue URL: (was: https://github.com/apache/spark/pull/33986) > Support sql overwrite a path

[jira] [Commented] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414660#comment-17414660 ] Apache Spark commented on SPARK-36745: -- User 'YannisSismanis' has created a pull request for this

[jira] [Assigned] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36745: Assignee: (was: Apache Spark) > Cleanup pattern ExtractEquiJoinKeys >

[jira] [Assigned] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36745: Assignee: Apache Spark > Cleanup pattern ExtractEquiJoinKeys >

[jira] [Commented] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414658#comment-17414658 ] Apache Spark commented on SPARK-36705: -- User 'rmcyang' has created a pull request for this issue:

[jira] [Updated] (SPARK-35930) Upgrade kinesis-client to 1.14.4

[ https://issues.apache.org/jira/browse/SPARK-35930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-35930: --- Priority: Major (was: Minor) > Upgrade kinesis-client to 1.14.4 >

[jira] [Commented] (SPARK-35930) Upgrade kinesis-client to 1.14.4

[ https://issues.apache.org/jira/browse/SPARK-35930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414656#comment-17414656 ] Kousuke Saruta commented on SPARK-35930: [~holden] Yes, I didn't think it's a common case so I

[jira] [Resolved] (SPARK-36715) explode(UDF) throw an exception

[ https://issues.apache.org/jira/browse/SPARK-36715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36715. -- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-36739) Add Apache license header to makefiles of python documents

[ https://issues.apache.org/jira/browse/SPARK-36739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36739. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33979

[jira] [Assigned] (SPARK-36739) Add Apache license header to makefiles of python documents

[ https://issues.apache.org/jira/browse/SPARK-36739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36739: Assignee: Leona Yoda > Add Apache license header to makefiles of python documents >

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414645#comment-17414645 ] Hyukjin Kwon commented on SPARK-33782: -- Thanks [~holden]! > Place spark.files, spark.jars and

[jira] [Resolved] (SPARK-35834) Use the same cleanup logic as Py4J in inheritable thread API

[ https://issues.apache.org/jira/browse/SPARK-35834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35834. -- Fix Version/s: 3.2.0 Assignee: Hyukjin Kwon Resolution: Fixed Fixed in

[jira] [Commented] (SPARK-33152) Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414642#comment-17414642 ] Apache Spark commented on SPARK-33152: -- User 'ahshahid' has created a pull request for this issue:

[jira] [Commented] (SPARK-34943) Upgrade flake8 to 3.8.0 or above in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414644#comment-17414644 ] Hyukjin Kwon commented on SPARK-34943: -- Thx! > Upgrade flake8 to 3.8.0 or above in Jenkins >

[jira] [Resolved] (SPARK-36251) Cover GitHub Actions runs without SHA in testing script

[ https://issues.apache.org/jira/browse/SPARK-36251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36251. -- Fix Version/s: 3.2.0 Assignee: Hyukjin Kwon Resolution: Fixed This is

[jira] [Commented] (SPARK-35834) Use the same cleanup logic as Py4J in inheritable thread API

[ https://issues.apache.org/jira/browse/SPARK-35834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414641#comment-17414641 ] Hyukjin Kwon commented on SPARK-35834: -- This is actually fixed too. I wonder why it wasn't

[jira] [Commented] (SPARK-24943) Convert a SQL Struct to StructType

[ https://issues.apache.org/jira/browse/SPARK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414639#comment-17414639 ] Hyukjin Kwon commented on SPARK-24943: -- uniontype is not supported in Spark at all. For varchar and

[jira] [Updated] (SPARK-36743) Backporting SPARK-36327 changes into Spark 2.4 version

[ https://issues.apache.org/jira/browse/SPARK-36743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36743: - Fix Version/s: (was: 3.3.0) > Backporting SPARK-36327 changes into Spark 2.4 version >

[jira] [Commented] (SPARK-36743) Backporting SPARK-36327 changes into Spark 2.4 version

[ https://issues.apache.org/jira/browse/SPARK-36743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414638#comment-17414638 ] Hyukjin Kwon commented on SPARK-36743: -- Spark 2.x is EOL. so the backport won't likely happen. >

[jira] [Resolved] (SPARK-36743) Backporting SPARK-36327 changes into Spark 2.4 version

[ https://issues.apache.org/jira/browse/SPARK-36743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36743. -- Resolution: Incomplete > Backporting SPARK-36327 changes into Spark 2.4 version >

[jira] [Assigned] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36748: Assignee: (was: Apache Spark) > Introduce the 'compute.isin_limit' option >

[jira] [Assigned] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36748: Assignee: Apache Spark > Introduce the 'compute.isin_limit' option >

[jira] [Commented] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414632#comment-17414632 ] Apache Spark commented on SPARK-36748: -- User 'xinrong-databricks' has created a pull request for

[jira] [Created] (SPARK-36748) Introduce the 'compute.isin_limit' option

Xinrong Meng created SPARK-36748: Summary: Introduce the 'compute.isin_limit' option Key: SPARK-36748 URL: https://issues.apache.org/jira/browse/SPARK-36748 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36748) Introduce the 'compute.isin_limit' option

[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414629#comment-17414629 ] Xinrong Meng commented on SPARK-36748: -- I am working on that. > Introduce the 'compute.isin_limit'

[jira] [Updated] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-33782: - Target Version/s: 3.3.0 > Place spark.files, spark.jars and spark.files under the current

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414624#comment-17414624 ] Holden Karau commented on SPARK-33782: -- I think this missed the window for Spark 3.2, but I'm happy

[jira] [Resolved] (SPARK-33885) The position of unresolved identifier for DDL commands should be respected..

[ https://issues.apache.org/jira/browse/SPARK-33885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33885. -- Fix Version/s: 3.2.0 Assignee: Terry Kim Resolution: Fixed > The position of

[jira] [Commented] (SPARK-34019) Keep same quantiles of UI and restful API

[ https://issues.apache.org/jira/browse/SPARK-34019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414622#comment-17414622 ] Holden Karau commented on SPARK-34019: -- This is targeting 4 since it's backwards incompat change.

[jira] [Commented] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

[ https://issues.apache.org/jira/browse/SPARK-34064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414621#comment-17414621 ] Holden Karau commented on SPARK-34064: -- [~inetfuture]it's hard to say since the initial fix was

[jira] [Resolved] (SPARK-34156) Unify the output of DDL and pass output attributes properly

[ https://issues.apache.org/jira/browse/SPARK-34156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34156. -- Fix Version/s: 3.2.0 Resolution: Fixed > Unify the output of DDL and pass output

[jira] [Commented] (SPARK-34208) Upgrade ORC to 1.6.7

[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414618#comment-17414618 ] Holden Karau commented on SPARK-34208: -- Is ORC-965 a regression and if so should we switch this to

[jira] [Commented] (SPARK-34156) Unify the output of DDL and pass output attributes properly

[ https://issues.apache.org/jira/browse/SPARK-34156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414619#comment-17414619 ] Holden Karau commented on SPARK-34156: -- All of the sub issues are resolved so I'm going to go ahead

[jira] [Commented] (SPARK-34329) When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

[ https://issues.apache.org/jira/browse/SPARK-34329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414617#comment-17414617 ] Holden Karau commented on SPARK-34329: -- Is this a regresion or has this behaviour been around in

[jira] [Updated] (SPARK-34208) Upgrade ORC to 1.6.7

[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34208: - Description: Apache ORC 1.6.7 has the following fixes including ORC-711 Support

[jira] [Updated] (SPARK-34478) Ignore or reject wrong config when start sparksession

[ https://issues.apache.org/jira/browse/SPARK-34478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34478: - Priority: Minor (was: Trivial) > Ignore or reject wrong config when start sparksession >

[jira] [Updated] (SPARK-34478) Ignore or reject wrong config when start sparksession

[ https://issues.apache.org/jira/browse/SPARK-34478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34478: - Priority: Trivial (was: Major) > Ignore or reject wrong config when start sparksession >

[jira] [Updated] (SPARK-34478) Ignore or reject wrong config when start sparksession

[ https://issues.apache.org/jira/browse/SPARK-34478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34478: - Issue Type: Improvement (was: Bug) > Ignore or reject wrong config when start sparksession >

[jira] [Comment Edited] (SPARK-34943) Upgrade flake8 to 3.8.0 or above in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414616#comment-17414616 ] Shane Knapp edited comment on SPARK-34943 at 9/13/21, 10:15 PM: flake8

[jira] [Commented] (SPARK-34943) Upgrade flake8 to 3.8.0 or above in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414616#comment-17414616 ] Shane Knapp commented on SPARK-34943: - flake8 tests passing w/3.8.0! from

[jira] [Resolved] (SPARK-36653) Implement Series.__xor__

[ https://issues.apache.org/jira/browse/SPARK-36653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36653. --- Fix Version/s: 3.3.0 Assignee: dgd_contributor Resolution: Fixed Issue

[jira] [Updated] (SPARK-34530) logError for interrupting block migrations is too high

[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34530: - Affects Version/s: 3.3.0 > logError for interrupting block migrations is too high >

[jira] [Updated] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36462: - Affects Version/s: 3.3.0 > Allow Spark on Kube to operate without polling or watchers >

[jira] [Resolved] (SPARK-36581) Add back transformAllExpressions to AnalysisHelper

[ https://issues.apache.org/jira/browse/SPARK-36581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingyi Bu resolved SPARK-36581. --- Resolution: Not A Problem > Add back transformAllExpressions to AnalysisHelper >

[jira] [Commented] (SPARK-36581) Add back transformAllExpressions to AnalysisHelper

[ https://issues.apache.org/jira/browse/SPARK-36581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414593#comment-17414593 ] Yingyi Bu commented on SPARK-36581: --- No, we don't need to keep this interface anymore. We were worried

[jira] [Resolved] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36705. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-34943) Upgrade flake8 to 3.8.0 or above in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414585#comment-17414585 ] Shane Knapp commented on SPARK-34943: - done:   {noformat} parallel-ssh -h ubuntu_workers.txt -i

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414577#comment-17414577 ] L. C. Hsieh commented on SPARK-36681: - The possible workaround is to use pure java implementation

[jira] [Updated] (SPARK-36747) Do not collapse Project with Aggregate when correlated subqueries are present in the project list

[ https://issues.apache.org/jira/browse/SPARK-36747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-36747: - Description: Currently CollapseProject combines Project with Aggregate when the shared

[jira] [Updated] (SPARK-36747) Do not collapse Project with Aggregate when correlated subqueries are present in the project list

[ https://issues.apache.org/jira/browse/SPARK-36747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-36747: - Description: Currently CollapseProject combines Project with Aggregate when the shared

[jira] [Created] (SPARK-36747) Do not collapse Project with Aggregate when correlated subqueries are present in the project list

Allison Wang created SPARK-36747: Summary: Do not collapse Project with Aggregate when correlated subqueries are present in the project list Key: SPARK-36747 URL: https://issues.apache.org/jira/browse/SPARK-36747

[jira] [Commented] (SPARK-34530) logError for interrupting block migrations is too high

[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414565#comment-17414565 ] Holden Karau commented on SPARK-34530: -- My bad on not describing this enough, I've honestly

[jira] [Updated] (SPARK-34943) Upgrade flake8 to 3.8.0 or above in Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34943: - Issue Type: Improvement (was: Bug) > Upgrade flake8 to 3.8.0 or above in Jenkins >

[jira] [Commented] (SPARK-35531) Can not insert into hive bucket table if create table with upper case schema

[ https://issues.apache.org/jira/browse/SPARK-35531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414563#comment-17414563 ] Holden Karau commented on SPARK-35531: -- Did this use to work? > Can not insert into hive bucket

[jira] [Commented] (SPARK-35834) Use the same cleanup logic as Py4J in inheritable thread API

[ https://issues.apache.org/jira/browse/SPARK-35834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414562#comment-17414562 ] Holden Karau commented on SPARK-35834: -- Is this a test only issue or a regression for Python users?

[jira] [Commented] (SPARK-35930) Upgrade kinesis-client to 1.14.4

[ https://issues.apache.org/jira/browse/SPARK-35930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414560#comment-17414560 ] Holden Karau commented on SPARK-35930: -- So to be clear, is it minor because we don't normally

[jira] [Commented] (SPARK-36238) Spark UI load event timeline too slow for huge stage

[ https://issues.apache.org/jira/browse/SPARK-36238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414557#comment-17414557 ] Holden Karau commented on SPARK-36238: -- hows it going [~angerszhuuu]? > Spark UI load event

[jira] [Commented] (SPARK-36251) Cover GitHub Actions runs without SHA in testing script

[ https://issues.apache.org/jira/browse/SPARK-36251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414556#comment-17414556 ] Holden Karau commented on SPARK-36251: -- Is this a blocker for 3.2 since it might affect release

[jira] [Updated] (SPARK-36433) Logs should show correct URL of where HistoryServer is started

[ https://issues.apache.org/jira/browse/SPARK-36433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36433: - Priority: Blocker (was: Major) > Logs should show correct URL of where HistoryServer is

[jira] [Commented] (SPARK-36433) Logs should show correct URL of where HistoryServer is started

[ https://issues.apache.org/jira/browse/SPARK-36433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414555#comment-17414555 ] Holden Karau commented on SPARK-36433: -- I think if this is a regression we should make this a

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414554#comment-17414554 ] Holden Karau commented on SPARK-36462: -- I'll probably pick this up this week. > Allow Spark on

[jira] [Updated] (SPARK-36543) Decommission logs too frequent when waiting migration to finish

[ https://issues.apache.org/jira/browse/SPARK-36543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36543: - Shepherd: Holden Karau > Decommission logs too frequent when waiting migration to finish >

[jira] [Commented] (SPARK-36543) Decommission logs too frequent when waiting migration to finish

[ https://issues.apache.org/jira/browse/SPARK-36543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414553#comment-17414553 ] Holden Karau commented on SPARK-36543: -- We could make the logging less frequent I agree. Anyone

[jira] [Commented] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

[ https://issues.apache.org/jira/browse/SPARK-36746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414552#comment-17414552 ] Apache Spark commented on SPARK-36746: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

[ https://issues.apache.org/jira/browse/SPARK-36746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36746: Assignee: Apache Spark > Refactor _select_rows_by_iterable in iLocIndexer to use

[jira] [Assigned] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

[ https://issues.apache.org/jira/browse/SPARK-36746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36746: Assignee: (was: Apache Spark) > Refactor _select_rows_by_iterable in iLocIndexer to

[jira] [Updated] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

[ https://issues.apache.org/jira/browse/SPARK-36745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yannis Sismanis updated SPARK-36745: Description: The join condition returned from ExtractEquiJoinKeys does not correspond to

[jira] [Commented] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

[ https://issues.apache.org/jira/browse/SPARK-36746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414549#comment-17414549 ] Apache Spark commented on SPARK-36746: -- User 'xinrong-databricks' has created a pull request for

[jira] [Commented] (SPARK-36581) Add back transformAllExpressions to AnalysisHelper

[ https://issues.apache.org/jira/browse/SPARK-36581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414550#comment-17414550 ] Holden Karau commented on SPARK-36581: -- [~buyingyi] / [~gengliang] is there a particular reason you

[jira] [Commented] (SPARK-36596) Review and fix issues in 3.2.0 Documents

[ https://issues.apache.org/jira/browse/SPARK-36596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414548#comment-17414548 ] Holden Karau commented on SPARK-36596: -- All of the sub tasks are resolved, are we good to resolve

[jira] [Created] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

Xinrong Meng created SPARK-36746: Summary: Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin Key: SPARK-36746 URL: https://issues.apache.org/jira/browse/SPARK-36746 Project: Spark

[jira] [Updated] (SPARK-36664) Log time spent waiting for cluster resources

[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36664: - Shepherd: Holden Karau Target Version/s: 3.3.0 > Log time spent waiting for cluster

[jira] [Updated] (SPARK-36664) Log time spent waiting for cluster resources

[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36664: - Affects Version/s: 3.3.0 > Log time spent waiting for cluster resources >

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414544#comment-17414544 ] Holden Karau commented on SPARK-36681: -- Is there a known work around? Can we put the workaround in

[jira] [Comment Edited] (SPARK-24943) Convert a SQL Struct to StructType

[ https://issues.apache.org/jira/browse/SPARK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414535#comment-17414535 ] Varun Bharill edited comment on SPARK-24943 at 9/13/21, 7:26 PM: - Hi

[jira] [Commented] (SPARK-24943) Convert a SQL Struct to StructType

[ https://issues.apache.org/jira/browse/SPARK-24943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414535#comment-17414535 ] Varun Bharill commented on SPARK-24943: --- Hi [~hyukjin.kwon],  Thank you for your response. The

[jira] [Created] (SPARK-36745) Cleanup pattern ExtractEquiJoinKeys

Yannis Sismanis created SPARK-36745: --- Summary: Cleanup pattern ExtractEquiJoinKeys Key: SPARK-36745 URL: https://issues.apache.org/jira/browse/SPARK-36745 Project: Spark Issue Type:

  1   2   >