[jira] [Commented] (SPARK-37073) Pass all UTs in `external/avro` with Java 17

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432815#comment-17432815 ] Apache Spark commented on SPARK-37073: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-37073) Pass all UTs in `external/avro` with Java 17

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37073: Assignee: Apache Spark > Pass all UTs in `external/avro` with Java 17 > -

[jira] [Assigned] (SPARK-37073) Pass all UTs in `external/avro` with Java 17

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37073: Assignee: (was: Apache Spark) > Pass all UTs in `external/avro` with Java 17 > --

[jira] [Commented] (SPARK-30537) toPandas gets wrong dtypes when applied on empty DF when Arrow enabled

2021-10-21 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432812#comment-17432812 ] pralabhkumar commented on SPARK-30537: -- Thx [~hyukjin.kwon] , working on this  > t

[jira] [Commented] (SPARK-30537) toPandas gets wrong dtypes when applied on empty DF when Arrow enabled

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432810#comment-17432810 ] Hyukjin Kwon commented on SPARK-30537: -- Please go ahead! > toPandas gets wrong dty

[jira] [Commented] (SPARK-30537) toPandas gets wrong dtypes when applied on empty DF when Arrow enabled

2021-10-21 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432805#comment-17432805 ] pralabhkumar commented on SPARK-30537: -- [~hyukjin.kwon]   I would like to work on

[jira] [Commented] (SPARK-37095) Inline type hints for files in python/pyspark/broadcast.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432804#comment-17432804 ] dch nguyen commented on SPARK-37095: working on this > Inline type hints for files

[jira] [Created] (SPARK-37095) Inline type hints for files in python/pyspark/broadcast.py

2021-10-21 Thread dch nguyen (Jira)
dch nguyen created SPARK-37095: -- Summary: Inline type hints for files in python/pyspark/broadcast.py Key: SPARK-37095 URL: https://issues.apache.org/jira/browse/SPARK-37095 Project: Spark Issue

[jira] [Updated] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37013: - Priority: Minor (was: Major) > `select format_string('%0$s', 'Hello')` has different behavior w

[jira] [Assigned] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37013: Assignee: Yang Jie > `select format_string('%0$s', 'Hello')` has different behavior when

[jira] [Resolved] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37013. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34313 [https://gi

[jira] [Assigned] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37083: Assignee: Apache Spark > Inline type hints for python/pyspark/accumulators.py > -

[jira] [Commented] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432801#comment-17432801 ] Apache Spark commented on SPARK-37083: -- User 'dchvn' has created a pull request for

[jira] [Assigned] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37083: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/accumulators.py

[jira] [Resolved] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37086. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34357 [https://gi

[jira] [Resolved] (SPARK-37050) Update conda installation instructions

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37050. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34315 [https://gi

[jira] [Assigned] (SPARK-37050) Update conda installation instructions

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37050: Assignee: Kousuke Saruta > Update conda installation instructions > -

[jira] [Assigned] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37069: Assignee: Chao Sun > HiveClientImpl throws NoSuchMethodError: > org.apache.hadoop.hive.q

[jira] [Resolved] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37069. -- Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37083: --- Parent: SPARK-37094 Issue Type: Sub-task (was: Bug) > Inline type hints for python/pyspark/

[jira] [Updated] (SPARK-36969) Inline type hints for SparkContext

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-36969: --- Parent: SPARK-37094 Issue Type: Sub-task (was: Bug) > Inline type hints for SparkContext >

[jira] [Updated] (SPARK-36969) Inline type hints for SparkContext

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-36969: --- Parent: (was: SPARK-36845) Issue Type: Bug (was: Sub-task) > Inline type hints for Spar

[jira] [Created] (SPARK-37094) Inline type hints for files in python/pyspark

2021-10-21 Thread dch nguyen (Jira)
dch nguyen created SPARK-37094: -- Summary: Inline type hints for files in python/pyspark Key: SPARK-37094 URL: https://issues.apache.org/jira/browse/SPARK-37094 Project: Spark Issue Type: Umbrell

[jira] [Updated] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37083: --- Parent: (was: SPARK-36845) Issue Type: Bug (was: Sub-task) > Inline type hints for pyth

[jira] [Updated] (SPARK-37093) Inline type hints python/pyspark/streaming

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37093: --- Issue Type: Umbrella (was: Bug) > Inline type hints python/pyspark/streaming >

[jira] [Updated] (SPARK-37042) Inline type hints for kinesis.py and listener.py in python/pyspark/streaming

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37042: --- Parent: SPARK-37093 Issue Type: Sub-task (was: Bug) > Inline type hints for kinesis.py and

[jira] [Updated] (SPARK-37042) Inline type hints for kinesis.py and listener.py in python/pyspark/streaming

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37042: --- Parent: (was: SPARK-36845) Issue Type: Bug (was: Sub-task) > Inline type hints for kine

[jira] [Updated] (SPARK-37015) Inline type hints for python/pyspark/streaming/dstream.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37015: --- Parent: SPARK-37093 Issue Type: Sub-task (was: Bug) > Inline type hints for python/pyspark/

[jira] [Updated] (SPARK-37015) Inline type hints for python/pyspark/streaming/dstream.py

2021-10-21 Thread dch nguyen (Jira)

[jira] [Updated] (SPARK-37014) Inline type hints for python/pyspark/streaming/context.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37014: --- Parent: SPARK-37093 Issue Type: Sub-task (was: Bug) > Inline type hints for python/pyspark/

[jira] [Updated] (SPARK-37014) Inline type hints for python/pyspark/streaming/context.py

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-37014: --- Parent: (was: SPARK-36845) Issue Type: Bug (was: Sub-task) > Inline type hints for pyth

[jira] [Created] (SPARK-37093) Inline type hints python/pyspark/streaming

2021-10-21 Thread dch nguyen (Jira)
dch nguyen created SPARK-37093: -- Summary: Inline type hints python/pyspark/streaming Key: SPARK-37093 URL: https://issues.apache.org/jira/browse/SPARK-37093 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432761#comment-17432761 ] Apache Spark commented on SPARK-37090: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-36845) Inline type hint files

2021-10-21 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432755#comment-17432755 ] dch nguyen commented on SPARK-36845: [~ueshin], yes, i will > Inline type hint file

[jira] [Assigned] (SPARK-37092) Add Spark error classes to error message and enforce test coverage

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37092: Assignee: Apache Spark > Add Spark error classes to error message and enforce test covera

[jira] [Assigned] (SPARK-37092) Add Spark error classes to error message and enforce test coverage

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37092: Assignee: (was: Apache Spark) > Add Spark error classes to error message and enforce

[jira] [Commented] (SPARK-37092) Add Spark error classes to error message and enforce test coverage

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432752#comment-17432752 ] Apache Spark commented on SPARK-37092: -- User 'karenfeng' has created a pull request

[jira] [Created] (SPARK-37092) Add Spark error classes to error message and enforce test coverage

2021-10-21 Thread Karen Feng (Jira)
Karen Feng created SPARK-37092: -- Summary: Add Spark error classes to error message and enforce test coverage Key: SPARK-37092 URL: https://issues.apache.org/jira/browse/SPARK-37092 Project: Spark

[jira] [Comment Edited] (SPARK-32423) class 'DataFrame' returns instance of type(self) instead of DataFrame

2021-10-21 Thread Jacob Duenke (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432744#comment-17432744 ] Jacob Duenke edited comment on SPARK-32423 at 10/21/21, 11:40 PM:

[jira] [Commented] (SPARK-32423) class 'DataFrame' returns instance of type(self) instead of DataFrame

2021-10-21 Thread Jacob Duenke (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432744#comment-17432744 ] Jacob Duenke commented on SPARK-32423: -- I'm looking for this same thing a year late

[jira] [Commented] (SPARK-36654) Drop type ignores from numpy imports

2021-10-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432740#comment-17432740 ] Maciej Szymkiewicz commented on SPARK-36654: Issue resolved by pull request

[jira] [Updated] (SPARK-36654) Drop type ignores from numpy imports

2021-10-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-36654: --- Fix Version/s: 3.3.0 > Drop type ignores from numpy imports > --

[jira] [Resolved] (SPARK-36654) Drop type ignores from numpy imports

2021-10-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-36654. Resolution: Fixed > Drop type ignores from numpy imports > ---

[jira] [Resolved] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski resolved SPARK-37090. --- Resolution: Duplicate > Upgrade libthrift to resolve security vulnerabilities >

[jira] [Commented] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432731#comment-17432731 ] Juliusz Sompolski commented on SPARK-37090: --- Duplicate of https://issues.apach

[jira] [Commented] (SPARK-36913) Implement createIndex and IndexExists in JDBC (MySQL dialect)

2021-10-21 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432727#comment-17432727 ] L. C. Hsieh commented on SPARK-36913: - Sounds a good idea. +1 with [~dongjoon]. > I

[jira] [Comment Edited] (SPARK-36913) Implement createIndex and IndexExists in JDBC (MySQL dialect)

2021-10-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432725#comment-17432725 ] Dongjoon Hyun edited comment on SPARK-36913 at 10/21/21, 9:27 PM:

[jira] [Commented] (SPARK-36913) Implement createIndex and IndexExists in JDBC (MySQL dialect)

2021-10-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432725#comment-17432725 ] Dongjoon Hyun commented on SPARK-36913: --- I agree with [~rxin] that we cannot do th

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Lekshmi Ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432714#comment-17432714 ] Lekshmi Ramachandran commented on SPARK-36554: -- @Nicolas Azrak So how do i

[jira] [Created] (SPARK-37091) Bump SystemRequirements to use Java > 11

2021-10-21 Thread Darek (Jira)
Darek created SPARK-37091: - Summary: Bump SystemRequirements to use Java > 11 Key: SPARK-37091 URL: https://issues.apache.org/jira/browse/SPARK-37091 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37090: Assignee: (was: Apache Spark) > Upgrade libthrift to resolve security vulnerabilities

[jira] [Commented] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432706#comment-17432706 ] Apache Spark commented on SPARK-37090: -- User 'juliuszsompolski' has created a pull

[jira] [Assigned] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37090: Assignee: Apache Spark > Upgrade libthrift to resolve security vulnerabilities >

[jira] [Created] (SPARK-37090) Upgrade libthrift to resolve security vulnerabilities

2021-10-21 Thread Juliusz Sompolski (Jira)
Juliusz Sompolski created SPARK-37090: - Summary: Upgrade libthrift to resolve security vulnerabilities Key: SPARK-37090 URL: https://issues.apache.org/jira/browse/SPARK-37090 Project: Spark

[jira] [Assigned] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37069: Assignee: Apache Spark > HiveClientImpl throws NoSuchMethodError: > org.apache.hadoop.hi

[jira] [Assigned] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37069: Assignee: (was: Apache Spark) > HiveClientImpl throws NoSuchMethodError: > org.apach

[jira] [Commented] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432691#comment-17432691 ] Apache Spark commented on SPARK-37069: -- User 'sunchao' has created a pull request f

[jira] [Updated] (SPARK-37089) ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37089: --- Description: The task completion listener that closes the vectorized reader is registered lazily in

[jira] [Updated] (SPARK-37089) ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37089: --- Summary: ParquetFileFormat registers task completion listeners lazily, causing Python writer thread

[jira] [Updated] (SPARK-37089) ParquetFileFormat/OrcFileFormat register task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37089: --- Description: The task completion listener that closes the vectorized reader is registered lazily in

[jira] [Updated] (SPARK-37089) ParquetFileFormat/OrcFileFormat register task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37089: --- Summary: ParquetFileFormat/OrcFileFormat register task completion listeners lazily, causing Python w

[jira] [Commented] (SPARK-36845) Inline type hint files

2021-10-21 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432684#comment-17432684 ] Takuya Ueshin commented on SPARK-36845: --- Hi [~dchvn], shall we file separate umbre

[jira] [Updated] (SPARK-37089) ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37089: --- Description: The task completion listener that closes the vectorized reader is registered lazily in

[jira] [Created] (SPARK-37089) ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled

2021-10-21 Thread Ankur Dave (Jira)
Ankur Dave created SPARK-37089: -- Summary: ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled Key: SPARK-37089 URL: https

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Nicolas Azrak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432647#comment-17432647 ] Nicolas Azrak commented on SPARK-36554: --- Depends on when it gets merged. I don't k

[jira] [Assigned] (SPARK-36986) Improving external schema management flexibility

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36986: Assignee: Apache Spark > Improving external schema management flexibility > -

[jira] [Assigned] (SPARK-36986) Improving external schema management flexibility

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36986: Assignee: (was: Apache Spark) > Improving external schema management flexibility > --

[jira] [Commented] (SPARK-36986) Improving external schema management flexibility

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432642#comment-17432642 ] Apache Spark commented on SPARK-36986: -- User 'risinga' has created a pull request f

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Lekshmi Ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432640#comment-17432640 ] Lekshmi Ramachandran commented on SPARK-36554: -- [~nicolasazrak] what is the

[jira] [Commented] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-21 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432624#comment-17432624 ] Chao Sun commented on SPARK-37069: -- Thanks for the ping [~zhouyifan279]! yes this is a

[jira] [Updated] (SPARK-36986) Improving external schema management flexibility

2021-10-21 Thread Rodrigo Boavida (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Boavida updated SPARK-36986: Description: Our spark usage, requires us to build an external schema and pass it on while

[jira] [Commented] (SPARK-37072) Pass all UTs in `repl` with Java 17

2021-10-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432575#comment-17432575 ] Dongjoon Hyun commented on SPARK-37072: --- Thank you, [~LuciferYang]. > Pass all UT

[jira] [Commented] (SPARK-37088) Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432562#comment-17432562 ] Apache Spark commented on SPARK-37088: -- User 'ankurdave' has created a pull request

[jira] [Resolved] (SPARK-37070) Pass all UTs in `mllib-local` and `mllib` with Java 17

2021-10-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37070. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34355 [https://

[jira] [Assigned] (SPARK-37070) Pass all UTs in `mllib-local` and `mllib` with Java 17

2021-10-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37070: - Assignee: Yang Jie > Pass all UTs in `mllib-local` and `mllib` with Java 17 > -

[jira] [Resolved] (SPARK-37088) Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread

2021-10-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37088. - Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed > Python UDF after off-he

[jira] [Updated] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37013: - Docs Text: Since Spark 3.3, the `strfmt` in `format_string(strfmt, obj, ...)` and `printf(strfmt

[jira] [Updated] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37013: - Labels: release-notes (was: ) > `select format_string('%0$s', 'Hello')` has different behavior

[jira] [Commented] (SPARK-37088) Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432540#comment-17432540 ] Ankur Dave commented on SPARK-37088: https://github.com/apache/spark/pull/34245 > P

[jira] [Updated] (SPARK-37088) Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread

2021-10-21 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-37088: --- Description: Python UDFs in Spark SQL are run in a separate Python process. The Python process is f

[jira] [Created] (SPARK-37088) Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread

2021-10-21 Thread Ankur Dave (Jira)
Ankur Dave created SPARK-37088: -- Summary: Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread Key: SPARK-37088 URL: https://issues.apache.org/jira/browse/SPARK-37088

[jira] [Commented] (SPARK-37087) merge three relation resolutions into one

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432535#comment-17432535 ] Apache Spark commented on SPARK-37087: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-37087) merge three relation resolutions into one

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37087: Assignee: Apache Spark > merge three relation resolutions into one >

[jira] [Assigned] (SPARK-37087) merge three relation resolutions into one

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37087: Assignee: (was: Apache Spark) > merge three relation resolutions into one > -

[jira] [Commented] (SPARK-37087) merge three relation resolutions into one

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432534#comment-17432534 ] Apache Spark commented on SPARK-37087: -- User 'cloud-fan' has created a pull request

[jira] [Created] (SPARK-37087) merge three relation resolutions into one

2021-10-21 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-37087: --- Summary: merge three relation resolutions into one Key: SPARK-37087 URL: https://issues.apache.org/jira/browse/SPARK-37087 Project: Spark Issue Type: Improveme

[jira] [Commented] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432525#comment-17432525 ] Apache Spark commented on SPARK-37086: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432524#comment-17432524 ] Apache Spark commented on SPARK-37086: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37086: Assignee: Kousuke Saruta (was: Apache Spark) > Fix the R test of FPGrowthModel for Scala

[jira] [Assigned] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37086: Assignee: Apache Spark (was: Kousuke Saruta) > Fix the R test of FPGrowthModel for Scala

[jira] [Updated] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37086: --- Description: Similar to the issue filed in SPARK-37059, the R test of FPGrowthModel assumes

[jira] [Created] (SPARK-37086) Fix the R test of FPGrowthModel for Scala 2.13

2021-10-21 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37086: -- Summary: Fix the R test of FPGrowthModel for Scala 2.13 Key: SPARK-37086 URL: https://issues.apache.org/jira/browse/SPARK-37086 Project: Spark Issue Type

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Nicolas Azrak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432494#comment-17432494 ] Nicolas Azrak commented on SPARK-36554: --- [~lekshmiii] yes, that will work without

[jira] [Comment Edited] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Lekshmi Ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432488#comment-17432488 ] Lekshmi Ramachandran edited comment on SPARK-36554 at 10/21/21, 1:48 PM: -

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Lekshmi Ramachandran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432488#comment-17432488 ] Lekshmi Ramachandran commented on SPARK-36554: -- from pyspark.sql.functions

[jira] [Assigned] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36554: Assignee: (was: Apache Spark) > Error message while trying to use spark sql functions

[jira] [Commented] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432482#comment-17432482 ] Apache Spark commented on SPARK-36554: -- User 'nicolasazrak' has created a pull requ

[jira] [Assigned] (SPARK-36554) Error message while trying to use spark sql functions directly on dataframe columns without using select expression

2021-10-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36554: Assignee: Apache Spark > Error message while trying to use spark sql functions directly o

[jira] [Assigned] (SPARK-37047) Add overloads for lpad and rpad for BINARY strings

2021-10-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37047: --- Assignee: Menelaos Karavelas > Add overloads for lpad and rpad for BINARY strings > ---

[jira] [Resolved] (SPARK-37047) Add overloads for lpad and rpad for BINARY strings

2021-10-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37047. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34154 [https://gith

  1   2   >