[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Comment Edited] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444972#comment-17444972 ] liu edited comment on SPARK-37325 at 11/17/21, 7:19 AM: [~hyukjin.kwon]  I don't

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Comment Edited] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444972#comment-17444972 ] liu edited comment on SPARK-37325 at 11/17/21, 7:18 AM: [~hyukjin.kwon]  I don't

[jira] [Comment Edited] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444972#comment-17444972 ] liu edited comment on SPARK-37325 at 11/17/21, 7:16 AM: [~hyukjin.kwon]  I don't

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Comment Edited] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444972#comment-17444972 ] liu edited comment on SPARK-37325 at 11/17/21, 7:14 AM: [~hyukjin.kwon]  I don't

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Commented] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444972#comment-17444972 ] liu commented on SPARK-37325: - [~hyukjin.kwon]  I don't think it will work. I have to make the questions

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description:   {code:java} schema = StructType([ StructField("node", StringType()) ]) rdd =

[jira] [Created] (SPARK-37353) Resolved attribute(s) region_id#7993 missing from lm_tam_code#2666,ld_month#2697,geo_sub_district_id#2677 when doing left join

2021-11-16 Thread Anubhav Tarar (Jira)
Anubhav Tarar created SPARK-37353: - Summary: Resolved attribute(s) region_id#7993 missing from lm_tam_code#2666,ld_month#2697,geo_sub_district_id#2677 when doing left join Key: SPARK-37353 URL:

[jira] [Commented] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444956#comment-17444956 ] Apache Spark commented on SPARK-37270: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37352) Silence the `index_col` advice in `to_spark()` for internal usage

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37352: Assignee: Apache Spark > Silence the `index_col` advice in `to_spark()` for internal

[jira] [Assigned] (SPARK-37352) Silence the `index_col` advice in `to_spark()` for internal usage

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37352: Assignee: (was: Apache Spark) > Silence the `index_col` advice in `to_spark()` for

[jira] [Commented] (SPARK-37352) Silence the `index_col` advice in `to_spark()` for internal usage

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444953#comment-17444953 ] Apache Spark commented on SPARK-37352: -- User 'itholic' has created a pull request for this issue:

[jira] [Updated] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu updated SPARK-37325: Description: schema = StructType([ StructField("node", StringType()) ]) {{rdd =

[jira] [Updated] (SPARK-37352) Silence the `index_col` advice in `to_spark()` for internal usage

2021-11-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37352: Description: The advice warning for `to_spark()` - when the `index_col` parameter is not

[jira] [Created] (SPARK-37352) Silence the `index_col` advice in `to_spark()` for internal usage

2021-11-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-37352: --- Summary: Silence the `index_col` advice in `to_spark()` for internal usage Key: SPARK-37352 URL: https://issues.apache.org/jira/browse/SPARK-37352 Project: Spark

[jira] [Assigned] (SPARK-37279) Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37279: Assignee: Apache Spark > Support DayTimeIntervalType in createDataFrame/toPandas and

[jira] [Assigned] (SPARK-37279) Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37279: Assignee: (was: Apache Spark) > Support DayTimeIntervalType in

[jira] [Commented] (SPARK-37279) Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444944#comment-17444944 ] Apache Spark commented on SPARK-37279: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37281) Support DayTimeIntervalType in Py4J

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37281: Assignee: (was: Apache Spark) > Support DayTimeIntervalType in Py4J >

[jira] [Commented] (SPARK-37281) Support DayTimeIntervalType in Py4J

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444943#comment-17444943 ] Apache Spark commented on SPARK-37281: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37281) Support DayTimeIntervalType in Py4J

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37281: Assignee: Apache Spark > Support DayTimeIntervalType in Py4J >

[jira] (SPARK-37277) Support DayTimeIntervalType in Arrow

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37277 ] Hyukjin Kwon deleted comment on SPARK-37277: -- was (Author: apachespark): User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/34614 > Support

[jira] [Reopened] (SPARK-37277) Support DayTimeIntervalType in Arrow

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-37277: -- > Support DayTimeIntervalType in Arrow > > >

[jira] [Resolved] (SPARK-37277) Support DayTimeIntervalType in Arrow

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37277. -- Resolution: Fixed > Support DayTimeIntervalType in Arrow >

[jira] [Resolved] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37325. -- Resolution: Invalid > Result vector from pandas_udf was not the required length >

[jira] [Commented] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444938#comment-17444938 ] Hyukjin Kwon commented on SPARK-37348: -- The question should be how often it is used in Python

[jira] [Commented] (SPARK-37325) Result vector from pandas_udf was not the required length

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444939#comment-17444939 ] Hyukjin Kwon commented on SPARK-37325: -- Again, please consider using DataFrame.mapInPandas.

[jira] [Commented] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444937#comment-17444937 ] Hyukjin Kwon commented on SPARK-37350: -- Spark 2.4.x is EOL. can you check if the same issue

[jira] [Updated] (SPARK-37334) pandas `convert_dtypes` method support

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37334: - Parent: SPARK-37197 Issue Type: Sub-task (was: New Feature) > pandas `convert_dtypes`

[jira] [Updated] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37348: - Parent: (was: SPARK-37197) Issue Type: Bug (was: Sub-task) > PySpark pmod function

[jira] [Updated] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37348: - Labels: (was: newbie) > PySpark pmod function > - > >

[jira] [Updated] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37348: - Parent: SPARK-37197 Issue Type: Sub-task (was: New Feature) > PySpark pmod function >

[jira] [Updated] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37348: - Issue Type: Improvement (was: Bug) > PySpark pmod function > - > >

[jira] [Commented] (SPARK-37230) Document DataFrame.mapInArrow

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444934#comment-17444934 ] Apache Spark commented on SPARK-37230: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-37230) Document DataFrame.mapInArrow

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444933#comment-17444933 ] Apache Spark commented on SPARK-37230: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37230) Document DataFrame.mapInArrow

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37230: Assignee: Apache Spark > Document DataFrame.mapInArrow > - >

[jira] [Assigned] (SPARK-37230) Document DataFrame.mapInArrow

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37230: Assignee: (was: Apache Spark) > Document DataFrame.mapInArrow >

[jira] [Commented] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444919#comment-17444919 ] Apache Spark commented on SPARK-37347: -- User 'lk246' has created a pull request for this issue:

[jira] [Updated] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shefron Yudy updated SPARK-37350: - Description: I saw the error in SparkThriftServer process's log when I restart all datanodes

[jira] [Updated] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shefron Yudy updated SPARK-37350: - Description: I saw the same error in SparkThriftServer process's log when I restart all

[jira] [Updated] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shefron Yudy updated SPARK-37350: - Description: I saw the same error in SparkThriftServer process's log when I restart all

[jira] [Created] (SPARK-37351) Supports write data flow control

2021-11-16 Thread melin (Jira)
melin created SPARK-37351: - Summary: Supports write data flow control Key: SPARK-37351 URL: https://issues.apache.org/jira/browse/SPARK-37351 Project: Spark Issue Type: Sub-task

[jira] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-16 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907 ] melin deleted comment on SPARK-24907: --- was (Author: melin): The hive table data is written to a relational database, generally an online production database. If the writing speed has no traffic

[jira] [Updated] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shefron Yudy updated SPARK-37350: - Environment: Spark-2.4.0、Hadoop-3.0.0、Hive-2.1.1 (was: Spark-2.4.0 Hadoop-3.0.0 Hive-2.1.1) >

[jira] [Updated] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shefron Yudy updated SPARK-37350: - Description: I saw the same error in SparkThriftServer process's log when I restart all

[jira] [Commented] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-16 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444916#comment-17444916 ] melin commented on SPARK-24907: --- The hive table data is written to a relational database, generally an

[jira] [Created] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

2021-11-16 Thread Shefron Yudy (Jira)
Shefron Yudy created SPARK-37350: Summary: EventLoggingListener keep logging errors after hdfs restart all datanodes Key: SPARK-37350 URL: https://issues.apache.org/jira/browse/SPARK-37350 Project:

[jira] [Resolved] (SPARK-35345) Add BloomFilter Benchmark test for Parquet

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35345. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34594

[jira] [Assigned] (SPARK-35345) Add BloomFilter Benchmark test for Parquet

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35345: Assignee: Huaxin Gao > Add BloomFilter Benchmark test for Parquet >

[jira] [Assigned] (SPARK-37341) Avoid unnecessary buffer and copy in full outer sort merge join

2021-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37341: --- Assignee: Cheng Su > Avoid unnecessary buffer and copy in full outer sort merge join >

[jira] [Resolved] (SPARK-37341) Avoid unnecessary buffer and copy in full outer sort merge join

2021-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37341. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34612

[jira] [Commented] (SPARK-28242) DataStreamer keeps logging errors even after fixing writeStream output sink

2021-11-16 Thread Shefron Yudy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444900#comment-17444900 ] Shefron Yudy commented on SPARK-28242: -- I saw the same error in SparkThriftServer process's log

[jira] [Resolved] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36231. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34314

[jira] [Resolved] (SPARK-36000) Support creation and operations of ps.Series/Index with Decimal('NaN')

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36000. -- Fix Version/s: 3.3.0 Resolution: Done > Support creation and operations of

[jira] [Assigned] (SPARK-36000) Support creation and operations of ps.Series/Index with Decimal('NaN')

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36000: Assignee: Yikun Jiang > Support creation and operations of ps.Series/Index with

[jira] [Assigned] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36231: Assignee: Yikun Jiang > Support arithmetic operations of Series containing

[jira] [Assigned] (SPARK-37345) Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS

2021-11-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37345: --- Assignee: Yuming Wang > Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS

[jira] [Resolved] (SPARK-37345) Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS

2021-11-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37345. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34615

[jira] [Commented] (SPARK-37329) File system delegation tokens are leaked

2021-11-16 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444890#comment-17444890 ] Wei-Chiu Chuang commented on SPARK-37329: - I should also note that this affects not just KMS,

[jira] [Commented] (SPARK-37349) Improve SQL Rest API Parsing

2021-11-16 Thread Yian Liou (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444891#comment-17444891 ] Yian Liou commented on SPARK-37349: --- Will be working on PR > Improve SQL Rest API Parsing >

[jira] [Created] (SPARK-37349) Improve SQL Rest API Parsing

2021-11-16 Thread Yian Liou (Jira)
Yian Liou created SPARK-37349: - Summary: Improve SQL Rest API Parsing Key: SPARK-37349 URL: https://issues.apache.org/jira/browse/SPARK-37349 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-37340) Display StageIds in Operators for SQL UI

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37340: Assignee: (was: Apache Spark) > Display StageIds in Operators for SQL UI >

[jira] [Assigned] (SPARK-37340) Display StageIds in Operators for SQL UI

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37340: Assignee: Apache Spark > Display StageIds in Operators for SQL UI >

[jira] [Commented] (SPARK-37340) Display StageIds in Operators for SQL UI

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444764#comment-17444764 ] Apache Spark commented on SPARK-37340: -- User 'yliou' has created a pull request for this issue:

[jira] [Commented] (SPARK-37219) support AS OF syntax

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444669#comment-17444669 ] Apache Spark commented on SPARK-37219: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-11-16 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35672. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Tim Schwab (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Schwab updated SPARK-37348: --- Description: Because Spark is built on the JVM, in PySpark, F.lit(-1) % F.lit(2) returns -1.

[jira] [Commented] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Tim Schwab (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444636#comment-17444636 ] Tim Schwab commented on SPARK-37348: Also, this has been the case for over 6 years:

[jira] [Created] (SPARK-37348) PySpark pmod function

2021-11-16 Thread Tim Schwab (Jira)
Tim Schwab created SPARK-37348: -- Summary: PySpark pmod function Key: SPARK-37348 URL: https://issues.apache.org/jira/browse/SPARK-37348 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-37181) pyspark.pandas.read_csv() should support latin-1 encoding

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37181: Assignee: Apache Spark > pyspark.pandas.read_csv() should support latin-1 encoding >

[jira] [Assigned] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37209: Assignee: (was: Apache Spark) > YarnShuffleIntegrationSuite and other two similar

[jira] [Commented] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444603#comment-17444603 ] Apache Spark commented on SPARK-37209: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-37181) pyspark.pandas.read_csv() should support latin-1 encoding

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37181: Assignee: (was: Apache Spark) > pyspark.pandas.read_csv() should support latin-1

[jira] [Commented] (SPARK-37181) pyspark.pandas.read_csv() should support latin-1 encoding

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444604#comment-17444604 ] Apache Spark commented on SPARK-37181: -- User 'pralabhkumar' has created a pull request for this

[jira] [Assigned] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37209: Assignee: Apache Spark > YarnShuffleIntegrationSuite and other two similar cases in >

[jira] [Assigned] (SPARK-37219) support AS OF syntax

2021-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37219: --- Assignee: Huaxin Gao > support AS OF syntax > > >

[jira] [Resolved] (SPARK-37219) support AS OF syntax

2021-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37219. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34497

[jira] [Commented] (SPARK-37346) Link migration guide for structured stream.

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444585#comment-17444585 ] Apache Spark commented on SPARK-37346: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-37346) Link migration guide for structured stream.

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37346: Assignee: (was: Apache Spark) > Link migration guide for structured stream. >

[jira] [Assigned] (SPARK-37346) Link migration guide for structured stream.

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37346: Assignee: Apache Spark > Link migration guide for structured stream. >

[jira] [Updated] (SPARK-37346) Link migration guide for struct stream.

2021-11-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37346: -- Summary: Link migration guide for struct stream. (was: Link migration guide to each project.) >

[jira] [Updated] (SPARK-37346) Link migration guide for structured stream.

2021-11-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37346: -- Summary: Link migration guide for structured stream. (was: Link migration guide for struct stream.)

[jira] [Comment Edited] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444502#comment-17444502 ] Yang Jie edited comment on SPARK-37209 at 11/16/21, 1:31 PM: - After some

[jira] [Comment Edited] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-11-16 Thread Wei Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444386#comment-17444386 ] Wei Zhang edited comment on SPARK-18105 at 11/16/21, 1:16 PM: -- In our case,

[jira] [Assigned] (SPARK-37342) Upgrade Apache Arrow to 6.0.0

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37342: Assignee: Chao Sun > Upgrade Apache Arrow to 6.0.0 > - > >

[jira] [Resolved] (SPARK-37342) Upgrade Apache Arrow to 6.0.0

2021-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37342. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34613

[jira] [Assigned] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37347: Assignee: (was: Apache Spark) > Spark Thrift Server (STS) driver fullFC becourse of

[jira] [Updated] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37347: Target Version/s: (was: 3.1.2, 3.2.0) > Spark Thrift Server (STS) driver fullFC becourse of

[jira] [Updated] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-37347: Fix Version/s: (was: 3.1.3) (was: 3.2.1) > Spark Thrift Server (STS)

[jira] [Commented] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444507#comment-17444507 ] Apache Spark commented on SPARK-37347: -- User 'lk246' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37347: Assignee: Apache Spark > Spark Thrift Server (STS) driver fullFC becourse of

[jira] [Commented] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444502#comment-17444502 ] Yang Jie commented on SPARK-37209: -- After some investigation, I found that this issue maybe related to

[jira] [Comment Edited] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

2021-11-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444502#comment-17444502 ] Yang Jie edited comment on SPARK-37209 at 11/16/21, 12:10 PM: -- After some

[jira] [Created] (SPARK-37347) Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly

2021-11-16 Thread liukai (Jira)
liukai created SPARK-37347: -- Summary: Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly Key: SPARK-37347 URL: https://issues.apache.org/jira/browse/SPARK-37347

[jira] [Commented] (SPARK-37346) Link migration guide to each project.

2021-11-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1775#comment-1775 ] angerszhu commented on SPARK-37346: --- Raise a pr soon > Link migration guide to each project. >

  1   2   >