[jira] [Resolved] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35636. -- Fix Version/s: 3.1.3 3.2.0 Assignee: Karen Feng Resolution:

[jira] [Created] (SPARK-35649) Failed to read data from Db2 on the column of DECFLOAT data type

2021-06-03 Thread sunguor (Jira)
sunguor created SPARK-35649: --- Summary: Failed to read data from Db2 on the column of DECFLOAT data type Key: SPARK-35649 URL: https://issues.apache.org/jira/browse/SPARK-35649 Project: Spark Issue

[jira] [Resolved] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35648. -- Fix Version/s: 3.2.0 Assignee: Hyukjin Kwon Resolution: Fixed Fixed in https:/

[jira] [Assigned] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35316: --- Assignee: Fu Chen > UnwrapCastInBinaryComparison support In/InSet predicate > -

[jira] [Commented] (SPARK-35608) Support AQE optimizer side transformUpWithPruning

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357082#comment-17357082 ] Dongjoon Hyun commented on SPARK-35608: --- Thank you! I collected this to SPARK-3382

[jira] [Updated] (SPARK-35608) Support AQE optimizer side transformUpWithPruning

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35608: -- Parent: SPARK-33828 Issue Type: Sub-task (was: Improvement) > Support AQE optimizer s

[jira] [Assigned] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35648: Assignee: Apache Spark > Refine and add dependencies needed for dev in dev/requirement.tx

[jira] [Commented] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357081#comment-17357081 ] Apache Spark commented on SPARK-35648: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35648: Assignee: (was: Apache Spark) > Refine and add dependencies needed for dev in dev/req

[jira] [Commented] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357080#comment-17357080 ] Apache Spark commented on SPARK-35648: -- User 'HyukjinKwon' has created a pull reque

[jira] [Created] (SPARK-35648) Refine and add dependencies needed for dev in dev/requirement.txt

2021-06-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-35648: Summary: Refine and add dependencies needed for dev in dev/requirement.txt Key: SPARK-35648 URL: https://issues.apache.org/jira/browse/SPARK-35648 Project: Spark

[jira] [Updated] (SPARK-32975) Add config for driver readiness timeout before executors start

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32975: -- Summary: Add config for driver readiness timeout before executors start (was: [K8S] - executo

[jira] [Updated] (SPARK-32975) [K8S] - executor fails to be restarted after it goes to ERROR/Failure state

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32975: -- Affects Version/s: 3.2.0 3.0.2 3.1.2 > [K8S] - e

[jira] [Resolved] (SPARK-34491) Move data source options docs for Python and Scala into a single page

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34491. -- Fix Version/s: 3.2.0 Assignee: Haejoon Lee Resolution: Done > Move data source

[jira] [Resolved] (SPARK-35523) Fix the default value in Data Source Options page

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35523. -- Fix Version/s: 3.2.0 Assignee: Haejoon Lee Resolution: Fixed Fixed in https://

[jira] [Resolved] (SPARK-35396) Support to manual close/release entries in MemoryStore and InMemoryRelation instead of replying on GC

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35396. --- Resolution: Fixed Issue resolved by pull request 32743 [https://github.com/apache/spark/pull

[jira] [Assigned] (SPARK-35499) Apply black to pandas API on Spark codes.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35499: Assignee: (was: Apache Spark) > Apply black to pandas API on Spark codes. > -

[jira] [Assigned] (SPARK-35499) Apply black to pandas API on Spark codes.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35499: Assignee: Apache Spark > Apply black to pandas API on Spark codes. >

[jira] [Commented] (SPARK-35499) Apply black to pandas API on Spark codes.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357064#comment-17357064 ] Apache Spark commented on SPARK-35499: -- User 'itholic' has created a pull request f

[jira] [Commented] (SPARK-35499) Apply black to pandas API on Spark codes.

2021-06-03 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357045#comment-17357045 ] Haejoon Lee commented on SPARK-35499: - I'm working on this > Apply black to pandas

[jira] [Created] (SPARK-35647) Merge contents and remove obsolete pages in User Guide section

2021-06-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-35647: Summary: Merge contents and remove obsolete pages in User Guide section Key: SPARK-35647 URL: https://issues.apache.org/jira/browse/SPARK-35647 Project: Spark

[jira] [Created] (SPARK-35646) Merge contents and remove obsolete pages in API reference section

2021-06-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-35646: Summary: Merge contents and remove obsolete pages in API reference section Key: SPARK-35646 URL: https://issues.apache.org/jira/browse/SPARK-35646 Project: Spark

[jira] [Updated] (SPARK-35645) Merge contents and remove obsolete pages in Getting Started section

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35645: - Parent: SPARK-34885 Issue Type: Sub-task (was: Bug) > Merge contents and remove obsolet

[jira] [Updated] (SPARK-35644) Merge contents and remove obsolete pages in Development section

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35644: - Summary: Merge contents and remove obsolete pages in Development section (was: Merge contents a

[jira] [Created] (SPARK-35645) Merge contents and remove obsolete pages in Getting Started section

2021-06-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-35645: Summary: Merge contents and remove obsolete pages in Getting Started section Key: SPARK-35645 URL: https://issues.apache.org/jira/browse/SPARK-35645 Project: Spark

[jira] [Updated] (SPARK-35644) Merge contents and remove obsolete pages in development section

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35644: - Description: Now Koalas documentation is in PySpark documentations. We should probably now remov

[jira] [Updated] (SPARK-35644) Merge contents and remove obsolete pages in development section

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35644: - Summary: Merge contents and remove obsolete pages in development section (was: Remove obsolete

[jira] [Updated] (SPARK-35588) Merge Binder integration and quickstart notebook

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35588: - Summary: Merge Binder integration and quickstart notebook (was: Merge Binder integration and qu

[jira] [Created] (SPARK-35644) Remove obsolete pages such as blog post and talks

2021-06-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-35644: Summary: Remove obsolete pages such as blog post and talks Key: SPARK-35644 URL: https://issues.apache.org/jira/browse/SPARK-35644 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35642: Assignee: Takuya Ueshin > Split pyspark-pandas tests. > --- > >

[jira] [Resolved] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35642. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32778 [https://gi

[jira] [Resolved] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33863. -- Resolution: Cannot Reproduce > Pyspark UDF wrongly changes timestamps to UTC > ---

[jira] [Commented] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-06-03 Thread dc-heros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357029#comment-17357029 ] dc-heros commented on SPARK-33863: -- Have this issues resolved, I couldn't reproduce it

[jira] [Created] (SPARK-35643) Ambiguous variable reference in functions.py column()

2021-06-03 Thread Keerthan Harish Vasist (Jira)
Keerthan Harish Vasist created SPARK-35643: -- Summary: Ambiguous variable reference in functions.py column() Key: SPARK-35643 URL: https://issues.apache.org/jira/browse/SPARK-35643 Project: Spa

[jira] [Assigned] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35642: Assignee: Apache Spark > Split pyspark-pandas tests. > --- > >

[jira] [Assigned] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35642: Assignee: (was: Apache Spark) > Split pyspark-pandas tests. > ---

[jira] [Commented] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357023#comment-17357023 ] Apache Spark commented on SPARK-35642: -- User 'ueshin' has created a pull request fo

[jira] [Created] (SPARK-35642) Split pyspark-pandas tests.

2021-06-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-35642: - Summary: Split pyspark-pandas tests. Key: SPARK-35642 URL: https://issues.apache.org/jira/browse/SPARK-35642 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-35587) Initial porting of Koalas documentation

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35587: Assignee: Hyukjin Kwon > Initial porting of Koalas documentation > --

[jira] [Resolved] (SPARK-35587) Initial porting of Koalas documentation

2021-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35587. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32726 [https://gi

[jira] [Updated] (SPARK-35641) TPCDS Q32 running failed

2021-06-03 Thread YuanGuanhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuanGuanhu updated SPARK-35641: --- Description: TPCDS Q32 sql running failed. 1、reproduce step: create table catalog_sales (     cs_s

[jira] [Updated] (SPARK-35641) TPCDS Q32 running failed

2021-06-03 Thread YuanGuanhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuanGuanhu updated SPARK-35641: --- Description: TPCDS Q32 sql running failed. {code:java} // code placeholder {code} SELECT 1 AS `exces

[jira] [Created] (SPARK-35641) TPCDS Q32 running failed

2021-06-03 Thread YuanGuanhu (Jira)
YuanGuanhu created SPARK-35641: -- Summary: TPCDS Q32 running failed Key: SPARK-35641 URL: https://issues.apache.org/jira/browse/SPARK-35641 Project: Spark Issue Type: Bug Components: SQ

[jira] [Commented] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356995#comment-17356995 ] Apache Spark commented on SPARK-35640: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35640: Assignee: (was: Apache Spark) > Refactor Parquet vectorized reader to remove duplicat

[jira] [Assigned] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35640: Assignee: Apache Spark > Refactor Parquet vectorized reader to remove duplicated code pat

[jira] [Commented] (SPARK-35130) Construct day-time interval column from integral fields

2021-06-03 Thread Hector Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356990#comment-17356990 ] Hector Zhang commented on SPARK-35130: -- Hi, [~mrpowers], I'm sorry for creating a P

[jira] [Created] (SPARK-35640) Refactor Parquet vectorized reader to remove duplicated code paths

2021-06-03 Thread Chao Sun (Jira)
Chao Sun created SPARK-35640: Summary: Refactor Parquet vectorized reader to remove duplicated code paths Key: SPARK-35640 URL: https://issues.apache.org/jira/browse/SPARK-35640 Project: Spark I

[jira] [Commented] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356919#comment-17356919 ] Apache Spark commented on SPARK-35639: -- User 'ekoifman' has created a pull request

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: (was: Apache Spark) > Add metrics about coalesced partitions to CustomShuff

[jira] [Commented] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356918#comment-17356918 ] Apache Spark commented on SPARK-35639: -- User 'ekoifman' has created a pull request

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: Apache Spark > Add metrics about coalesced partitions to CustomShuffleReader in

[jira] [Created] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Eugene Koifman (Jira)
Eugene Koifman created SPARK-35639: -- Summary: Add metrics about coalesced partitions to CustomShuffleReader in AQE Key: SPARK-35639 URL: https://issues.apache.org/jira/browse/SPARK-35639 Project: Spa

[jira] [Commented] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356912#comment-17356912 ] Apache Spark commented on SPARK-35638: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35638: Assignee: Apache Spark > Introduce Field to manage dtypes and StructFields. > ---

[jira] [Assigned] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35638: Assignee: (was: Apache Spark) > Introduce Field to manage dtypes and StructFields. >

[jira] [Updated] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-35638: -- Summary: Introduce Field to manage dtypes and StructFields. (was: Introduce Field to manage d

[jira] [Created] (SPARK-35638) Introduce Field to manage dtypes and StructField.

2021-06-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-35638: - Summary: Introduce Field to manage dtypes and StructField. Key: SPARK-35638 URL: https://issues.apache.org/jira/browse/SPARK-35638 Project: Spark Issue Typ

[jira] [Assigned] (SPARK-35606) List Python 3.9 installed libraries in build_and_test workflow

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35606: - Assignee: Xinrong Meng > List Python 3.9 installed libraries in build_and_test workflow

[jira] [Resolved] (SPARK-35606) List Python 3.9 installed libraries in build_and_test workflow

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35606. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32737 [https://

[jira] [Commented] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356734#comment-17356734 ] Apache Spark commented on SPARK-35589: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356733#comment-17356733 ] Apache Spark commented on SPARK-35589: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356731#comment-17356731 ] Apache Spark commented on SPARK-35636: -- User 'karenfeng' has created a pull request

[jira] [Assigned] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35636: Assignee: (was: Apache Spark) > Do not push down extract value in higher order functi

[jira] [Assigned] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35636: Assignee: Apache Spark > Do not push down extract value in higher order function that ref

[jira] [Commented] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356728#comment-17356728 ] Apache Spark commented on SPARK-35636: -- User 'karenfeng' has created a pull request

[jira] [Resolved] (SPARK-35612) Support LZ4 compression in ORC data source

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35612. --- Resolution: Fixed Issue resolved by pull request 32751 [https://github.com/apache/spark/pull

[jira] [Assigned] (SPARK-35612) Support LZ4 compression in ORC data source

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35612: - Assignee: Han > Support LZ4 compression in ORC data source > --

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Affects Version/s: (was: 3.1.2) 3.2.0 > Support reading inputbytes

[jira] [Commented] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356703#comment-17356703 ] Dongjoon Hyun commented on SPARK-35624: --- Thank you for filing JIRA, [~allendang].

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Target Version/s: (was: 3.1.2) > Support reading inputbytes and inputrecords of the stage in

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Fix Version/s: (was: 3.1.2) > Support reading inputbytes and inputrecords of the stage int

[jira] [Commented] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356701#comment-17356701 ] Apache Spark commented on SPARK-35599: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35599: Assignee: (was: Apache Spark) > Introduce a way to compare series of array for older

[jira] [Commented] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356699#comment-17356699 ] Apache Spark commented on SPARK-35599: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35599: Assignee: Apache Spark > Introduce a way to compare series of array for older pandas > --

[jira] [Commented] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356697#comment-17356697 ] Dongjoon Hyun commented on SPARK-35625: --- Thank you for reporting, [~lbuanos]. BTW,

[jira] [Created] (SPARK-35637) 2 active tasks shown in Spark UI but executor core is 1 and task cpu is 1

2021-06-03 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-35637: --- Summary: 2 active tasks shown in Spark UI but executor core is 1 and task cpu is 1 Key: SPARK-35637 URL: https://issues.apache.org/jira/browse/SPARK-35637 Project: Spar

[jira] [Updated] (SPARK-35533) Do not drop cached RDD blocks to accommodate blocks from decommissioned block manager if enough memory is not available

2021-06-03 Thread abhishek kumar tiwari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] abhishek kumar tiwari updated SPARK-35533: -- Fix Version/s: (was: 3.2.0) > Do not drop cached RDD blocks to accommodate

[jira] [Updated] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Summary: Do not push down extract value in higher order function that references both sides of a joi

[jira] [Updated] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Description: Currently, lambda keys can be referenced outside of the lambda function: {quote}Projec

[jira] [Created] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
Karen Feng created SPARK-35636: -- Summary: Do not push lambda variables out of lambda functions in NestedColumnAliasing Key: SPARK-35636 URL: https://issues.apache.org/jira/browse/SPARK-35636 Project: Spa

[jira] [Updated] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Description: Currently, lambda keys can be referenced outside of the lambda function: Project [tran

[jira] [Commented] (SPARK-34298) SaveMode.Overwrite not usable when using s3a root paths

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356625#comment-17356625 ] Steve Loughran commented on SPARK-34298: Actually, you've just found a bug in Pa

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

2021-06-03 Thread Saurabh Chawla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356619#comment-17356619 ] Saurabh Chawla commented on SPARK-35406: [~ste...@apache.org]- This backport of 

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3A does not delete old files with S3 object-put to table path/

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-35299: --- Summary: Dataframe overwrite on S3A does not delete old files with S3 object-put to table pa

[jira] [Commented] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356615#comment-17356615 ] Steve Loughran commented on SPARK-35299: Actually if you are doing a PUT of test

[jira] [Commented] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

2021-06-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356614#comment-17356614 ] Kousuke Saruta commented on SPARK-35602: [~dej...@gmail.com] Thank you for repor

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356607#comment-17356607 ] Steve Loughran commented on SPARK-35406: This is fixed by HADOOP-17338 and by th

[jira] [Commented] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-06-03 Thread Fu Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356574#comment-17356574 ] Fu Chen commented on SPARK-35316: - Hi, [~cloud_fan], this is my Jira account, what shoul

[jira] [Updated] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35589: -- Fix Version/s: 3.1.3 > BlockManagerMasterEndpoint should not ignore index-only shuffle file du

[jira] [Closed] (SPARK-35631) java.lang.ArithmeticException: integer overflow when SELECT 2147483647 + 1 executed with set spark.sql.ansi.enabled=true

2021-06-03 Thread Chetan Bhat (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat closed SPARK-35631. ---  this is not a issue. Closed without handling. > java.lang.ArithmeticException: integer overflow when S

[jira] [Resolved] (SPARK-35580) Support subexpression elimination for higher order functions

2021-06-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-35580. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32735 [https://gith

[jira] [Assigned] (SPARK-35580) Support subexpression elimination for higher order functions

2021-06-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-35580: --- Assignee: L. C. Hsieh > Support subexpression elimination for higher order functions >

[jira] [Commented] (SPARK-34435) ArrayIndexOutOfBoundsException when select in different case

2021-06-03 Thread Enver Osmanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356550#comment-17356550 ] Enver Osmanov commented on SPARK-34435: --- I see it was fixed in [this PR|https://gi

[jira] [Created] (SPARK-35635) concurrent insert statements from multiple beeline fail with job aborted exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35635: --- Summary: concurrent insert statements from multiple beeline fail with job aborted exception Key: SPARK-35635 URL: https://issues.apache.org/jira/browse/SPARK-35635 Proj

[jira] [Created] (SPARK-35634) Select queries with union throw parsing exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35634: --- Summary: Select queries with union throw parsing exception Key: SPARK-35634 URL: https://issues.apache.org/jira/browse/SPARK-35634 Project: Spark Issue Type: B

[jira] [Created] (SPARK-35633) Select nvl query throws parse exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35633: --- Summary: Select nvl query throws parse exception Key: SPARK-35633 URL: https://issues.apache.org/jira/browse/SPARK-35633 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-35632) SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35632: --- Summary: SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5 Key: SPARK-35632 URL: https://issues.apache.org/jira/browse/SPARK-35

[jira] [Resolved] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35316. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32488 [https://gith

  1   2   >