[jira] [Assigned] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37016: Assignee: Apache Spark > Publicise UpperCaseCharStream > - >

[jira] [Assigned] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37016: Assignee: (was: Apache Spark) > Publicise UpperCaseCharStream >

[jira] [Commented] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429117#comment-17429117 ] Apache Spark commented on SPARK-37016: -- User 'dohongdayi' has created a pull request for this

[jira] [Commented] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-14 Thread dohongdayi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429114#comment-17429114 ] dohongdayi commented on SPARK-37016: I have submitted a PR

[jira] [Created] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-14 Thread dohongdayi (Jira)
dohongdayi created SPARK-37016: -- Summary: Publicise UpperCaseCharStream Key: SPARK-37016 URL: https://issues.apache.org/jira/browse/SPARK-37016 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37014) Inline type hints for python/pyspark/streaming/context.py

2021-10-14 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429112#comment-17429112 ] dch nguyen commented on SPARK-37014: working on this > Inline type hints for

[jira] [Commented] (SPARK-37015) Inline type hints for python/pyspark/streaming/dstream.py

2021-10-14 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429111#comment-17429111 ] dch nguyen commented on SPARK-37015: working on this > Inline type hints for

[jira] [Created] (SPARK-37015) Inline type hints for python/pyspark/streaming/dstream.py

2021-10-14 Thread dch nguyen (Jira)
dch nguyen created SPARK-37015: -- Summary: Inline type hints for python/pyspark/streaming/dstream.py Key: SPARK-37015 URL: https://issues.apache.org/jira/browse/SPARK-37015 Project: Spark Issue

[jira] [Created] (SPARK-37014) Inline type hints for python/pyspark/streaming/context.py

2021-10-14 Thread dch nguyen (Jira)
dch nguyen created SPARK-37014: -- Summary: Inline type hints for python/pyspark/streaming/context.py Key: SPARK-37014 URL: https://issues.apache.org/jira/browse/SPARK-37014 Project: Spark Issue

[jira] [Created] (SPARK-37013) `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17

2021-10-14 Thread Yang Jie (Jira)
Yang Jie created SPARK-37013: Summary: `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17 Key: SPARK-37013 URL: https://issues.apache.org/jira/browse/SPARK-37013

[jira] [Updated] (SPARK-36964) Reuse CachedDNSToSwitchMapping for yarn container requests

2021-10-14 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-36964: --- Description: Similar to SPARK-13704​, In some cases, YarnAllocator add container requests with

[jira] [Updated] (SPARK-36964) Reuse CachedDNSToSwitchMapping for yarn container requests

2021-10-14 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-36964: --- Description: Similar to SPARK-13704​, In some cases, YarnAllocator add or remove container

[jira] [Updated] (SPARK-36964) Reuse CachedDNSToSwitchMapping for yarn container requests

2021-10-14 Thread gaoyajun02 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaoyajun02 updated SPARK-36964: --- Description: Similar to SPARK-13704​, In some cases, YarnAllocator add or remove container

[jira] [Assigned] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36945: Assignee: Apache Spark > Inline type hints for python/pyspark/sql/udf.py >

[jira] [Assigned] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36945: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/sql/udf.py >

[jira] [Commented] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429100#comment-17429100 ] Apache Spark commented on SPARK-36945: -- User 'dchvn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36337: Assignee: Yikun Jiang > decimal('Nan') is unsupported in net.razorvine.pickle >

[jira] [Resolved] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36337. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34285

[jira] [Assigned] (SPARK-37012) Disable pinned thread mode by default

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37012: Assignee: Apache Spark > Disable pinned thread mode by default >

[jira] [Commented] (SPARK-37012) Disable pinned thread mode by default

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429086#comment-17429086 ] Apache Spark commented on SPARK-37012: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37012) Disable pinned thread mode by default

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37012: Assignee: (was: Apache Spark) > Disable pinned thread mode by default >

[jira] [Resolved] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37010. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-37005) pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job path

2021-10-14 Thread WeiNan Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WeiNan Zhao updated SPARK-37005: Description: when  i submit a spark job , use spark-submit and set option --files file1,  then

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429079#comment-17429079 ] jinhai commented on SPARK-37006: Or whether we can generate localDirs based on appId and execId, just

[jira] [Commented] (SPARK-36525) DS V2 Index Support

2021-10-14 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429077#comment-17429077 ] dch nguyen commented on SPARK-36525: [~huaxingao] yes, i'd like to > DS V2 Index Support >

[jira] [Updated] (SPARK-37012) Disable pinned thread mode by default

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37012: - Priority: Blocker (was: Major) > Disable pinned thread mode by default >

[jira] [Created] (SPARK-37012) Disable pinned thread mode by default

2021-10-14 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37012: Summary: Disable pinned thread mode by default Key: SPARK-37012 URL: https://issues.apache.org/jira/browse/SPARK-37012 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Description: In flake8 < 3.9.0, F401 error occurs for imports when the imported identities

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Description: In flake8 < 3.9.0, F401 error occurs for imports when the imported identities

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Description: In flake8 < 3.9.0, F401 error occurs for imports when the impo Since this

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Fix Version/s: (was: 3.3.0) > Upgrade flake8 to 3.9.0 or above in Jenkins >

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Affects Version/s: (was: 3.2.0) 3.3.0 > Upgrade flake8 to 3.9.0 or

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Description: In flake8 < 3.8.0, F401 error occurs for imports in *if* statements when

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Reporter: Takuya Ueshin (was: Haejoon Lee) > Upgrade flake8 to 3.9.0 or above in Jenkins >

[jira] [Assigned] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-37011: - Assignee: (was: Shane Knapp) > Upgrade flake8 to 3.9.0 or above in Jenkins >

[jira] [Created] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37011: - Summary: Upgrade flake8 to 3.9.0 or above in Jenkins Key: SPARK-37011 URL: https://issues.apache.org/jira/browse/SPARK-37011 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36942. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Commented] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429041#comment-17429041 ] Apache Spark commented on SPARK-37010: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429040#comment-17429040 ] Apache Spark commented on SPARK-37010: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37010: Assignee: (was: Apache Spark) > Remove unnecessary "noqa: F401" comments in

[jira] [Assigned] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37010: Assignee: Apache Spark > Remove unnecessary "noqa: F401" comments in pandas-on-Spark >

[jira] [Created] (SPARK-37010) Remove unnecessary "noqa: F401" comments in pandas-on-Spark

2021-10-14 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37010: - Summary: Remove unnecessary "noqa: F401" comments in pandas-on-Spark Key: SPARK-37010 URL: https://issues.apache.org/jira/browse/SPARK-37010 Project: Spark

[jira] [Resolved] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

2021-10-14 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-23626. Fix Version/s: 3.3.0 3.2.1 3.0.4 3.1.3

[jira] [Assigned] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

2021-10-14 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-23626: -- Assignee: Josh Rosen > DAGScheduler blocked due to JobSubmitted event >

[jira] [Created] (SPARK-37009) Add checks to DebugFilesystem to ensure that FS operations are not performed in the DAGScheduler event loop

2021-10-14 Thread Josh Rosen (Jira)
Josh Rosen created SPARK-37009: -- Summary: Add checks to DebugFilesystem to ensure that FS operations are not performed in the DAGScheduler event loop Key: SPARK-37009 URL:

[jira] [Resolved] (SPARK-37000) Add type hints to python/pyspark/sql/util.py

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37000. --- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin (was: Apache Spark)

[jira] [Resolved] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36938. --- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36905: Affects Version/s: (was: 3.2.0) 3.3.0 > Reading Hive view without

[jira] [Updated] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36905: Affects Version/s: (was: 3.3.0) 3.2.0 > Reading Hive view without

[jira] [Assigned] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36905: --- Assignee: Linhong Liu > Reading Hive view without explicit column names fails in Spark >

[jira] [Resolved] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36905. - Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed > Reading Hive view

[jira] [Resolved] (SPARK-37003) Merge INSERT related docs

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37003. - Resolution: Fixed Issue resolved by pull request 34282

[jira] [Assigned] (SPARK-37003) Merge INSERT related docs

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37003: --- Assignee: angerszhu > Merge INSERT related docs > - > >

[jira] [Commented] (SPARK-36525) DS V2 Index Support

2021-10-14 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428834#comment-17428834 ] Huaxin Gao commented on SPARK-36525: Yes, it would be great if you can please help [~dchvn] > DS V2

[jira] [Commented] (SPARK-37007) ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout

2021-10-14 Thread Nandini (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428806#comment-17428806 ] Nandini commented on SPARK-37007: - Hello, I would like to work on this jira. >

[jira] [Issue Comment Deleted] (SPARK-37007) ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout

2021-10-14 Thread Nandini (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandini updated SPARK-37007: Comment: was deleted (was: Hello Team, I would like to work on this jira.) > ExecutorAllocationManager

[jira] [Commented] (SPARK-37007) ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout

2021-10-14 Thread Nandini (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428805#comment-17428805 ] Nandini commented on SPARK-37007: - Hello Team, I would like to work on this jira. >

[jira] [Updated] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2021-10-14 Thread Michelle m Hovington (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michelle m Hovington updated SPARK-21187: - Attachment: 0--1172099527-254246775-1412485878 > Complete support for remaining

[jira] [Assigned] (SPARK-37008) WholeStageCodegenSparkSubmitSuite Failed with Java 17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37008: Assignee: (was: Apache Spark) > WholeStageCodegenSparkSubmitSuite Failed with Java

[jira] [Assigned] (SPARK-37008) WholeStageCodegenSparkSubmitSuite Failed with Java 17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37008: Assignee: Apache Spark > WholeStageCodegenSparkSubmitSuite Failed with Java 17 >

[jira] [Commented] (SPARK-37008) WholeStageCodegenSparkSubmitSuite Failed with Java 17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428783#comment-17428783 ] Apache Spark commented on SPARK-37008: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-37008) WholeStageCodegenSparkSubmitSuite Failed with Java 17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37008: Assignee: (was: Apache Spark) > WholeStageCodegenSparkSubmitSuite Failed with Java

[jira] [Assigned] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36337: Assignee: (was: Apache Spark) > decimal('Nan') is unsupported in

[jira] [Assigned] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36337: Assignee: Apache Spark > decimal('Nan') is unsupported in net.razorvine.pickle >

[jira] [Commented] (SPARK-36337) decimal('Nan') is unsupported in net.razorvine.pickle

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428773#comment-17428773 ] Apache Spark commented on SPARK-36337: -- User 'Yikun' has created a pull request for this issue:

[jira] [Updated] (SPARK-37008) WholeStageCodegenSparkSubmitSuite Failed with Java 17

2021-10-14 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37008: - Summary: WholeStageCodegenSparkSubmitSuite Failed with Java 17 (was: Use UseCompressedClassPointers

[jira] [Created] (SPARK-37008) Use UseCompressedClassPointers instead of UseCompressedOops to pass WholeStageCodegenSparkSubmitSuite with Java 17

2021-10-14 Thread Yang Jie (Jira)
Yang Jie created SPARK-37008: Summary: Use UseCompressedClassPointers instead of UseCompressedOops to pass WholeStageCodegenSparkSubmitSuite with Java 17 Key: SPARK-37008 URL:

[jira] [Created] (SPARK-37007) ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout

2021-10-14 Thread Nandini (Jira)
Nandini created SPARK-37007: --- Summary: ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout Key: SPARK-37007 URL: https://issues.apache.org/jira/browse/SPARK-37007

[jira] [Assigned] (SPARK-37001) Disable two level of map for final hash aggregation by default

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37001: --- Assignee: Cheng Su > Disable two level of map for final hash aggregation by default >

[jira] [Resolved] (SPARK-37001) Disable two level of map for final hash aggregation by default

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37001. - Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-12567) Add aes_encrypt and aes_decrypt UDFs

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-12567: --- Assignee: Kousuke Saruta > Add aes_encrypt and aes_decrypt UDFs >

[jira] [Resolved] (SPARK-12567) Add aes_encrypt and aes_decrypt UDFs

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-12567. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 32801

[jira] [Commented] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428714#comment-17428714 ] jinhai commented on SPARK-37006: hi [~cloud_fan]，can you review this issue for me? > MapStatus adds

[jira] [Created] (SPARK-37006) MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading

2021-10-14 Thread jinhai (Jira)
jinhai created SPARK-37006: -- Summary: MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading Key: SPARK-37006 URL: https://issues.apache.org/jira/browse/SPARK-37006

[jira] [Resolved] (SPARK-36632) DivideYMInterval and DivideDTInterval should throw the same exception when divide by zero.

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36632. - Fix Version/s: 3.2.1 Resolution: Fixed Issue resolved by pull request 33889

[jira] [Assigned] (SPARK-36632) DivideYMInterval and DivideDTInterval should throw the same exception when divide by zero.

2021-10-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36632: --- Assignee: jiaan.geng > DivideYMInterval and DivideDTInterval should throw the same

[jira] [Assigned] (SPARK-36571) Optimized FileOutputCommitter with StagingDir

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36571: Assignee: (was: Apache Spark) > Optimized FileOutputCommitter with StagingDir >

[jira] [Commented] (SPARK-36571) Optimized FileOutputCommitter with StagingDir

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428693#comment-17428693 ] Apache Spark commented on SPARK-36571: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36571) Optimized FileOutputCommitter with StagingDir

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36571: Assignee: Apache Spark > Optimized FileOutputCommitter with StagingDir >

[jira] [Commented] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428689#comment-17428689 ] Apache Spark commented on SPARK-36464: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428688#comment-17428688 ] Apache Spark commented on SPARK-36464: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36464) Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428686#comment-17428686 ] Apache Spark commented on SPARK-36464: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428684#comment-17428684 ] Apache Spark commented on SPARK-36900: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428685#comment-17428685 ] Apache Spark commented on SPARK-36900: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-36525) DS V2 Index Support

2021-10-14 Thread dch nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428676#comment-17428676 ] dch nguyen commented on SPARK-36525: [~huaxingao], Should we do these functions for supportsIndex in

[jira] [Assigned] (SPARK-36946) Support time for ps.to_datetime

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36946: Assignee: dgd_contributor > Support time for ps.to_datetime >

[jira] [Resolved] (SPARK-36946) Support time for ps.to_datetime

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36946. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34211

[jira] [Commented] (SPARK-37004) Job cancellation causes py4j errors on Jupyter due to pinned thread mode

2021-10-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428663#comment-17428663 ] Hyukjin Kwon commented on SPARK-37004: -- It was found that it's a Py4J issue. I made a fix

[jira] [Updated] (SPARK-37005) pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job path

2021-10-14 Thread WeiNan Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WeiNan Zhao updated SPARK-37005: Labels: python spark-core (was: ) > pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job

[jira] [Created] (SPARK-37005) pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job path

2021-10-14 Thread WeiNan Zhao (Jira)
WeiNan Zhao created SPARK-37005: --- Summary: pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job path Key: SPARK-37005 URL: https://issues.apache.org/jira/browse/SPARK-37005 Project: Spark

[jira] [Commented] (SPARK-36871) Migrate CreateViewStatement to v2 command

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428651#comment-17428651 ] Apache Spark commented on SPARK-36871: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-36871) Migrate CreateViewStatement to v2 command

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428649#comment-17428649 ] Apache Spark commented on SPARK-36871: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Updated] (SPARK-36993) Fix json_tuple throw NPE if fields exist no foldable null value

2021-10-14 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-36993: - Fix Version/s: 3.0.4 > Fix json_tuple throw NPE if fields exist no foldable null value >

[jira] [Resolved] (SPARK-36059) Add the ability to specify a scheduler & queue

2021-10-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36059. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34239

[jira] [Resolved] (SPARK-36922) The SIGN/SIGNUM functions should support ANSI intervals

2021-10-14 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36922. Fix Version/s: 3.3.0 Assignee: PengLei Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-36967) Report accurate shuffle block size if its skewed

2021-10-14 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-36967: Summary: Report accurate shuffle block size if its skewed (was: Report accurate block size threshold per

[jira] [Commented] (SPARK-37003) Merge INSERT related docs

2021-10-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428601#comment-17428601 ] Apache Spark commented on SPARK-37003: -- User 'AngersZh' has created a pull request for this