[jira] [Created] (SPARK-49919) SpecialLimits strategy doesn't work when return the content of the Dataset as a Dataset of JSON strings

2024-10-09 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-49919: -- Summary: SpecialLimits strategy doesn't work when return the content of the Dataset as a Dataset of JSON strings Key: SPARK-49919 URL: https://issues.apache.org/jira/browse/SPARK-4991

[jira] [Updated] (SPARK-49919) SpecialLimits strategy doesn't work when return the content of the Dataset as a Dataset of JSON strings

2024-10-09 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-49919: --- Description: `CollectLimitExec` is used when a logical `Limit` and/or `Offset` operation is the fin

[jira] [Created] (SPARK-49782) ResolveDataFrameDropColumns rule mistakenly handles UnresolvedAttribute

2024-09-25 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-49782: -- Summary: ResolveDataFrameDropColumns rule mistakenly handles UnresolvedAttribute Key: SPARK-49782 URL: https://issues.apache.org/jira/browse/SPARK-49782 Project: Spark

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-30 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17780929#comment-17780929 ] Lantao Jin commented on SPARK-44124: Thanks, I created three sub-tasks so far. [~Jun

[jira] [Created] (SPARK-45721) Upgrade AWS SDK to v2 for Hadoop dependency

2023-10-30 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-45721: -- Summary: Upgrade AWS SDK to v2 for Hadoop dependency Key: SPARK-45721 URL: https://issues.apache.org/jira/browse/SPARK-45721 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45719) Upgrade AWS SDK to v2 for Kubernetes integration tests

2023-10-30 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-45719: -- Summary: Upgrade AWS SDK to v2 for Kubernetes integration tests Key: SPARK-45719 URL: https://issues.apache.org/jira/browse/SPARK-45719 Project: Spark Issue Type

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779407#comment-17779407 ] Lantao Jin commented on SPARK-44124: Is it possible to convert this JIRA to an issue

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779405#comment-17779405 ] Lantao Jin commented on SPARK-44124: Added a design doc (more like a plan) in descri

[jira] [Updated] (SPARK-44124) Upgrade AWS SDK to v2

2023-10-25 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-44124: --- Description: Here is a design doc: https://docs.google.com/document/d/1nGWbGTqxuFBG2ftfYYXxzrkipINIL

[jira] [Commented] (SPARK-44124) Upgrade AWS SDK to v2

2023-06-29 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738865#comment-17738865 ] Lantao Jin commented on SPARK-44124: Hi [~dongjoon], this is Lantao from AWS, we are

[jira] [Created] (SPARK-34122) Remove duplicated branches in case when

2021-01-14 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34122: -- Summary: Remove duplicated branches in case when Key: SPARK-34122 URL: https://issues.apache.org/jira/browse/SPARK-34122 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-12 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-34082. Resolution: Invalid Close it due to {{cannot resolve 'b' given input columns}} seems a correct er

[jira] [Closed] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-12 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin closed SPARK-34082. -- > Window expression with alias inside WHERE and HAVING clauses fail with > non-descriptive exceptions > -

[jira] [Created] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34082: -- Summary: Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions Key: SPARK-34082 URL: https://issues.apache.org/jira/browse/SPARK-34082

[jira] [Updated] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

2021-01-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-34064: --- Description: SPARK-27036 introduced a runId for BroadcastExchangeExec to resolve the problem that a

[jira] [Updated] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

2021-01-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-34064: --- Description: SPARK-27036 introduced a runId for BroadcastExchangeExec to resolve the problem that a

[jira] [Updated] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

2021-01-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-34064: --- Attachment: Screen Shot 2021-01-11 at 12.03.13 PM.png > Broadcast job is not aborted even the SQL st

[jira] [Created] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

2021-01-10 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34064: -- Summary: Broadcast job is not aborted even the SQL statement canceled Key: SPARK-34064 URL: https://issues.apache.org/jira/browse/SPARK-34064 Project: Spark Iss

[jira] [Updated] (SPARK-34000) ExecutorAllocationListener threw an exception java.util.NoSuchElementException

2021-01-04 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-34000: --- Affects Version/s: 3.0.1 > ExecutorAllocationListener threw an exception java.util.NoSuchElementExce

[jira] [Updated] (SPARK-34000) ExecutorAllocationListener threw an exception java.util.NoSuchElementException

2021-01-04 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-34000: --- Affects Version/s: (was: 3.0.1) > ExecutorAllocationListener threw an exception java.util.NoSuch

[jira] [Created] (SPARK-34000) ExecutorAllocationListener threw an exception java.util.NoSuchElementException

2021-01-04 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34000: -- Summary: ExecutorAllocationListener threw an exception java.util.NoSuchElementException Key: SPARK-34000 URL: https://issues.apache.org/jira/browse/SPARK-34000 Project: S

[jira] [Created] (SPARK-33014) Multiple bucket column not works in DataSourceV2 table

2020-09-28 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-33014: -- Summary: Multiple bucket column not works in DataSourceV2 table Key: SPARK-33014 URL: https://issues.apache.org/jira/browse/SPARK-33014 Project: Spark Issue Type

[jira] [Updated] (SPARK-32994) Heavy external accumulators may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Summary: Heavy external accumulators may lead driver full GC problem (was: External accumulators (n

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.26 PM.png > External accumulators (not start with Interna

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-24 at 5.19.58 PM.png > External accumulators (not start with Interna

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.36.48 AM.png > External accumulators (not start with Intern

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.32.51 AM.png > External accumulators (not start with Intern

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Attachment: Screen Shot 2020-09-25 at 11.35.01 AM.png > External accumulators (not start with Intern

[jira] [Updated] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32994: --- Description: We use Spark + Delta Lake, recently we find our Spark driver faced full GC problem (ve

[jira] [Created] (SPARK-32994) External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem

2020-09-24 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32994: -- Summary: External accumulators (not start with InternalAccumulator.METRICS_PREFIX) may lead driver full GC problem Key: SPARK-32994 URL: https://issues.apache.org/jira/browse/SPARK-32

[jira] [Updated] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-31 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32715: --- Affects Version/s: 2.4.6 3.0.0 > Broadcast block pieces may memory leak > ---

[jira] [Updated] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32715: --- Description: We use Spark thrift-server as a long-running service. A bad query submitted a heavy Br

[jira] [Created] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32715: -- Summary: Broadcast block pieces may memory leak Key: SPARK-32715 URL: https://issues.apache.org/jira/browse/SPARK-32715 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32672) Data corruption in some cached compressed boolean columns

2020-08-20 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181553#comment-17181553 ] Lantao Jin commented on SPARK-32672: Changed to Critical, Blocker is reserved for co

[jira] [Updated] (SPARK-32672) Data corruption in some cached compressed boolean columns

2020-08-20 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32672: --- Priority: Critical (was: Blocker) > Data corruption in some cached compressed boolean columns > ---

[jira] [Commented] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180365#comment-17180365 ] Lantao Jin commented on SPARK-32638: Yes. This problem exists in 3.0 and master. Th

[jira] [Updated] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32638: --- Affects Version/s: 3.0.0 > WidenSetOperationTypes in subquery attribute missing >

[jira] [Commented] (SPARK-32598) Not able to see driver logs in spark history server in standalone mode

2020-08-13 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177429#comment-17177429 ] Lantao Jin commented on SPARK-32598: [~sriramgr] PullRequest is welcome. Please comm

[jira] [Commented] (SPARK-32598) See driver logs in Spark history server in standalone mode

2020-08-12 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176186#comment-17176186 ] Lantao Jin commented on SPARK-32598: Does this problem exist in Spark3.0? I think br

[jira] [Comment Edited] (SPARK-32582) Spark SQL Infer Schema Performance

2020-08-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175981#comment-17175981 ] Lantao Jin edited comment on SPARK-32582 at 8/12/20, 4:31 AM:

[jira] [Commented] (SPARK-32582) Spark SQL Infer Schema Performance

2020-08-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175981#comment-17175981 ] Lantao Jin commented on SPARK-32582: {quote} I remember I investigated this issue a

[jira] [Commented] (SPARK-32582) Spark SQL Infer Schema Performance

2020-08-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175203#comment-17175203 ] Lantao Jin commented on SPARK-32582: Maybe we could offer a new interface to break o

[jira] [Commented] (SPARK-32582) Spark SQL Infer Schema Performance

2020-08-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175197#comment-17175197 ] Lantao Jin commented on SPARK-32582: I see. The implementation of {{inferSchema}} me

[jira] [Commented] (SPARK-32582) Spark SQL Infer Schema Performance

2020-08-10 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175143#comment-17175143 ] Lantao Jin commented on SPARK-32582: {code} files.toIterator.map(file => readSchema(

[jira] [Commented] (SPARK-32536) deleted not existing hdfs locations when use spark sql to execute "insert overwrite" statement to dynamic partition

2020-08-09 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174060#comment-17174060 ] Lantao Jin commented on SPARK-32536: {quote} I found that I mistake the issue condit

[jira] [Commented] (SPARK-32536) deleted not existing hdfs locations when use spark sql to execute "insert overwrite" statement to dynamic partition

2020-08-05 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171393#comment-17171393 ] Lantao Jin commented on SPARK-32536: {{org.apache.hadoop.hive.ql.metadata.Hive.delet

[jira] [Commented] (SPARK-32536) deleted not existing hdfs locations when use spark sql to execute "insert overwrite" statement to dynamic partition

2020-08-05 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171385#comment-17171385 ] Lantao Jin commented on SPARK-32536: Thanks for reporting this. Which Hive version d

[jira] [Updated] (SPARK-32537) Add a hint-specific suite for CTE for test coverage

2020-08-05 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32537: --- Priority: Major (was: Minor) > Add a hint-specific suite for CTE for test coverage > --

[jira] [Created] (SPARK-32537) Add a hint-specific suite for CTE for test coverage

2020-08-05 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32537: -- Summary: Add a hint-specific suite for CTE for test coverage Key: SPARK-32537 URL: https://issues.apache.org/jira/browse/SPARK-32537 Project: Spark Issue Type: T

[jira] [Resolved] (SPARK-32535) Query with broadcast hints fail when query has a WITH clause

2020-08-05 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-32535. Resolution: Duplicate > Query with broadcast hints fail when query has a WITH clause > ---

[jira] [Commented] (SPARK-32535) Query with broadcast hints fail when query has a WITH clause

2020-08-05 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171300#comment-17171300 ] Lantao Jin commented on SPARK-32535: I think this issue has fixed by SPARK-32237. I

[jira] [Updated] (SPARK-32362) AdaptiveQueryExecSuite misses verifying AE results

2020-07-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32362: --- Description: {code} QueryTest.sameRows(result.toSeq, df.collect().toSeq) {code} Even the results are

[jira] [Updated] (SPARK-32362) AdaptiveQueryExecSuite misses verifying AE results

2020-07-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32362: --- Summary: AdaptiveQueryExecSuite misses verifying AE results (was: AdaptiveQueryExecSuite has proble

[jira] [Created] (SPARK-32362) AdaptiveQueryExecSuite has problem

2020-07-19 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32362: -- Summary: AdaptiveQueryExecSuite has problem Key: SPARK-32362 URL: https://issues.apache.org/jira/browse/SPARK-32362 Project: Spark Issue Type: Bug Comp

[jira] [Commented] (SPARK-32347) BROADCAST hint makes a weird message that "column can't be resolved" (it was OK in Spark 2.4)

2020-07-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160859#comment-17160859 ] Lantao Jin commented on SPARK-32347: Duplicates to SPARK-32237 > BROADCAST hint mak

[jira] [Commented] (SPARK-32283) Multiple Kryo registrators can't be used anymore

2020-07-15 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158018#comment-17158018 ] Lantao Jin commented on SPARK-32283: Thanks for reporting this. Will file a patch.

[jira] [Comment Edited] (SPARK-32237) Cannot resolve column when put hint in the views of common table expression

2020-07-09 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155085#comment-17155085 ] Lantao Jin edited comment on SPARK-32237 at 7/10/20, 4:21 AM:

[jira] [Commented] (SPARK-32237) Cannot resolve column when put hint in the views of common table expression

2020-07-09 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155085#comment-17155085 ] Lantao Jin commented on SPARK-32237: Thanks to report this. I am going to fix that.

[jira] [Comment Edited] (SPARK-29038) SPIP: Support Spark Materialized View

2020-07-07 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152738#comment-17152738 ] Lantao Jin edited comment on SPARK-29038 at 7/7/20, 1:14 PM: -

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2020-07-07 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152738#comment-17152738 ] Lantao Jin commented on SPARK-29038: Hi [~AidenZhang], my focusings of MV in recent

[jira] [Updated] (SPARK-32201) More general skew join pattern matching

2020-07-06 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32201: --- Summary: More general skew join pattern matching (was: More general skew Join pattern matching) >

[jira] [Created] (SPARK-32201) More general skew Join pattern matching

2020-07-06 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32201: -- Summary: More general skew Join pattern matching Key: SPARK-32201 URL: https://issues.apache.org/jira/browse/SPARK-32201 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149403#comment-17149403 ] Lantao Jin commented on SPARK-32147: set spark.sql.sources.partitionColumnTypeInfere

[jira] [Updated] (SPARK-32143) Fast fail when the AQE skew join produce too many splits

2020-06-30 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32143: --- Description: In handling skewed SortMergeJoin, when matching partitions from the left side and the

[jira] [Updated] (SPARK-32143) Fast fail when the AQE skew join produce too many splits

2020-06-30 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32143: --- Description: In handling skewed SortMergeJoin, when matching partitions from the left side and the

[jira] [Commented] (SPARK-32143) Fast fail when the AQE skew join produce too many splits

2020-06-30 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149084#comment-17149084 ] Lantao Jin commented on SPARK-32143: A PR will be submitted soon. > Fast fail when

[jira] [Updated] (SPARK-32143) Fast fail when the AQE skew join produce too many splits

2020-06-30 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32143: --- Description: In handling skewed SortMergeJoin, when matching partitions from the left side and the

[jira] [Created] (SPARK-32143) Fast fail when the AQE skew join produce too many splits

2020-06-30 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32143: -- Summary: Fast fail when the AQE skew join produce too many splits Key: SPARK-32143 URL: https://issues.apache.org/jira/browse/SPARK-32143 Project: Spark Issue Ty

[jira] [Updated] (SPARK-32129) Support AQE skew join with Union

2020-06-29 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32129: --- Description: Current, the AQE skew join only supports two tables join such as {code} SMJ :-Sort :

[jira] [Updated] (SPARK-32129) Support AQE skew join with Union

2020-06-29 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32129: --- Description: Current, the AQE skew join only supports two tables join such as {code} SMJ :-Sort :

[jira] [Created] (SPARK-32129) Support AQE skew join with Union

2020-06-29 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32129: -- Summary: Support AQE skew join with Union Key: SPARK-32129 URL: https://issues.apache.org/jira/browse/SPARK-32129 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-32118) Use fine-grained read write lock for each database in HiveExternalCatalog

2020-06-28 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32118: -- Summary: Use fine-grained read write lock for each database in HiveExternalCatalog Key: SPARK-32118 URL: https://issues.apache.org/jira/browse/SPARK-32118 Project: Spark

[jira] [Resolved] (SPARK-32117) Thread spark-listener-group-streams is cpu costing

2020-06-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin resolved SPARK-32117. Resolution: Won't Fix > Thread spark-listener-group-streams is cpu costing > -

[jira] [Commented] (SPARK-32117) Thread spark-listener-group-streams is cpu costing

2020-06-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147278#comment-17147278 ] Lantao Jin commented on SPARK-32117: I think it might be fixed by SPARK-29423 > Thr

[jira] [Created] (SPARK-32117) Thread spark-listener-group-streams is cpu costing

2020-06-28 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32117: -- Summary: Thread spark-listener-group-streams is cpu costing Key: SPARK-32117 URL: https://issues.apache.org/jira/browse/SPARK-32117 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-32108) Silent mode of spark-sql is broken

2020-06-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147264#comment-17147264 ] Lantao Jin commented on SPARK-32108: [~maxgekk] I think it works. The INFO logs only

[jira] [Comment Edited] (SPARK-32063) Spark native temporary table

2020-06-23 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143575#comment-17143575 ] Lantao Jin edited comment on SPARK-32063 at 6/24/20, 6:53 AM:

[jira] [Commented] (SPARK-32063) Spark native temporary table

2020-06-23 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143575#comment-17143575 ] Lantao Jin commented on SPARK-32063: For 1, even RDD cache or table cache can improv

[jira] [Created] (SPARK-32065) Supporting analyze temporary table

2020-06-22 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32065: -- Summary: Supporting analyze temporary table Key: SPARK-32065 URL: https://issues.apache.org/jira/browse/SPARK-32065 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32066) Supporting create temporary table LIKE

2020-06-22 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32066: -- Summary: Supporting create temporary table LIKE Key: SPARK-32066 URL: https://issues.apache.org/jira/browse/SPARK-32066 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32064) Supporting create temporary table

2020-06-22 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32064: -- Summary: Supporting create temporary table Key: SPARK-32064 URL: https://issues.apache.org/jira/browse/SPARK-32064 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32063) Spark native temporary table

2020-06-22 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32063: -- Summary: Spark native temporary table Key: SPARK-32063 URL: https://issues.apache.org/jira/browse/SPARK-32063 Project: Spark Issue Type: New Feature Co

[jira] [Issue Comment Deleted] (SPARK-31904) Char and varchar partition columns throw MetaException

2020-06-04 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-31904: --- Comment: was deleted (was: [https://github.com/apache/spark/pull/28724]) > Char and varchar partiti

[jira] [Updated] (SPARK-31904) Char and varchar partition columns throw MetaException

2020-06-04 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-31904: --- Description: {code} CREATE TABLE t1(a STRING, B VARCHAR(10), C CHAR(10)) STORED AS parquet; CREATE T

[jira] [Commented] (SPARK-31904) Char and varchar partition columns throw MetaException

2020-06-04 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125627#comment-17125627 ] Lantao Jin commented on SPARK-31904: [https://github.com/apache/spark/pull/28724] >

[jira] [Created] (SPARK-31904) Char and varchar partition columns throw MetaException

2020-06-04 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-31904: -- Summary: Char and varchar partition columns throw MetaException Key: SPARK-31904 URL: https://issues.apache.org/jira/browse/SPARK-31904 Project: Spark Issue Type

[jira] [Commented] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094215#comment-17094215 ] Lantao Jin commented on SPARK-31591: https://github.com/apache/spark/pull/28385 > n

[jira] [Commented] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-28 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094214#comment-17094214 ] Lantao Jin commented on SPARK-31591: [~Ankitraj] I have already filed a PR. > nameP

[jira] [Created] (SPARK-31591) namePrefix could be null in Utils.createDirectory

2020-04-27 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-31591: -- Summary: namePrefix could be null in Utils.createDirectory Key: SPARK-31591 URL: https://issues.apache.org/jira/browse/SPARK-31591 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-31154) Expose basic write metrics for InsertIntoDataSourceCommand

2020-03-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-31154: --- Description: Spark provides interface `InsertableRelation` and the `InsertIntoDataSourceCommand` to

[jira] [Updated] (SPARK-31154) Expose basic write metrics for InsertIntoDataSourceCommand

2020-03-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-31154: --- Description: Spark provides interface `InsertableRelation` and the `InsertIntoDataSourceCommand` to

[jira] [Created] (SPARK-31154) Expose basic write metrics for InsertIntoDataSourceCommand

2020-03-14 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-31154: -- Summary: Expose basic write metrics for InsertIntoDataSourceCommand Key: SPARK-31154 URL: https://issues.apache.org/jira/browse/SPARK-31154 Project: Spark Issue

[jira] [Created] (SPARK-31068) IllegalArgumentException in BroadcastExchangeExec

2020-03-05 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-31068: -- Summary: IllegalArgumentException in BroadcastExchangeExec Key: SPARK-31068 URL: https://issues.apache.org/jira/browse/SPARK-31068 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30785) Create table like should keep tracksPartitionsInCatalog same with source table

2020-02-10 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-30785: -- Summary: Create table like should keep tracksPartitionsInCatalog same with source table Key: SPARK-30785 URL: https://issues.apache.org/jira/browse/SPARK-30785 Project: S

  1   2   3   4   >