[jira] [Created] (SPARK-32081) facing Invalid UTF-32 character v2.4.5 running pyspark

2020-06-23 Thread Yaniv Kempler (Jira)
Yaniv Kempler created SPARK-32081: - Summary: facing Invalid UTF-32 character v2.4.5 running pyspark Key: SPARK-32081 URL: https://issues.apache.org/jira/browse/SPARK-32081 Project: Spark

[jira] [Resolved] (SPARK-32062) Reset listenerRegistered in SparkSession

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32062. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28899

[jira] [Assigned] (SPARK-32062) Reset listenerRegistered in SparkSession

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32062: --- Assignee: ulysses you > Reset listenerRegistered in SparkSession >

[jira] [Assigned] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32072: --- Assignee: Maxim Gekk > Unaligned benchmark results > > >

[jira] [Resolved] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32072. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28906

[jira] [Resolved] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32075. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Fixed in

[jira] [Assigned] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32075: Assignee: Zuo Dao > Fix a few issues in parameters table >

[jira] [Updated] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-23 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-32080: - Priority: Trivial (was: Major) > Simplify ArrowColumnVector ListArray accessor >

[jira] [Commented] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143479#comment-17143479 ] Apache Spark commented on SPARK-32080: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32080: Assignee: Apache Spark > Simplify ArrowColumnVector ListArray accessor >

[jira] [Assigned] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32080: Assignee: (was: Apache Spark) > Simplify ArrowColumnVector ListArray accessor >

[jira] [Created] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-23 Thread Bryan Cutler (Jira)
Bryan Cutler created SPARK-32080: Summary: Simplify ArrowColumnVector ListArray accessor Key: SPARK-32080 URL: https://issues.apache.org/jira/browse/SPARK-32080 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-32028. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-32028: Assignee: Zhen Li > App id link in history summary page point to wrong application

[jira] [Resolved] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32073. -- Fix Version/s: 3.1.0 2.4.7 3.0.1 Resolution:

[jira] [Assigned] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32073: Assignee: Hyukjin Kwon > Drop R < 3.5 support > > >

[jira] [Assigned] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31918: Assignee: Hyukjin Kwon > SparkR CRAN check gives a warning with R 4.0.0 on OSX >

[jira] [Resolved] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31918. -- Fix Version/s: 3.1.0 2.4.7 3.0.1 Resolution:

[jira] [Updated] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32073: - Labels: releasenotes (was: ) > Drop R < 3.5 support > > >

[jira] [Created] (SPARK-32079) PySpark <> Beam pickling issues for collections.namedtuple

2020-06-23 Thread Gerard Casas Saez (Jira)
Gerard Casas Saez created SPARK-32079: - Summary: PySpark <> Beam pickling issues for collections.namedtuple Key: SPARK-32079 URL: https://issues.apache.org/jira/browse/SPARK-32079 Project: Spark

[jira] [Commented] (SPARK-32078) Add a redirect to sql-ref from sql-reference

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143429#comment-17143429 ] Apache Spark commented on SPARK-32078: -- User 'gatorsmile' has created a pull request for this

[jira] [Assigned] (SPARK-32078) Add a redirect to sql-ref from sql-reference

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32078: Assignee: Xiao Li (was: Apache Spark) > Add a redirect to sql-ref from sql-reference >

[jira] [Assigned] (SPARK-32078) Add a redirect to sql-ref from sql-reference

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32078: Assignee: Apache Spark (was: Xiao Li) > Add a redirect to sql-ref from sql-reference >

[jira] [Created] (SPARK-32078) Add a redirect to sql-ref from sql-reference

2020-06-23 Thread Xiao Li (Jira)
Xiao Li created SPARK-32078: --- Summary: Add a redirect to sql-ref from sql-reference Key: SPARK-32078 URL: https://issues.apache.org/jira/browse/SPARK-32078 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-23631) Add summary to RandomForestClassificationModel

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143392#comment-17143392 ] Apache Spark commented on SPARK-23631: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Updated] (SPARK-32067) [K8S] Executor pod template of subsequent submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8S] Executor pod template of subsequent submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8S] Executor pod template of subsequent submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Summary: [K8S] Executor pod template of subsequent submission inadvertently applies to ongoing

[jira] [Updated] (SPARK-32067) [K8s] Executor pod template of subsequent submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Summary: [K8s] Executor pod template of subsequent submission inadvertently applies to ongoing

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequent submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Summary: [K8s] Pod template from subsequent submission inadvertently applies to ongoing submission

[jira] [Commented] (SPARK-31998) Change package references for ArrowBuf

2020-06-23 Thread Kouhei Sutou (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143306#comment-17143306 ] Kouhei Sutou commented on SPARK-31998: -- Yes. This change will be included in Apache Arrow 1.0.0.

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-23 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143271#comment-17143271 ] Jungtaek Lim commented on SPARK-32051: -- [~frankivo] Could you put full source code for

[jira] [Commented] (SPARK-32057) SparkExecuteStatementOperation does not set CANCELED state correctly

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143231#comment-17143231 ] Apache Spark commented on SPARK-32057: -- User 'alismess-db' has created a pull request for this

[jira] [Commented] (SPARK-32063) Spark native temporary table

2020-06-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143233#comment-17143233 ] L. C. Hsieh commented on SPARK-32063: - For 1 and 2, it seems all related to performance. In Spark,

[jira] [Commented] (SPARK-32057) SparkExecuteStatementOperation does not set CANCELED state correctly

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143230#comment-17143230 ] Apache Spark commented on SPARK-32057: -- User 'alismess-db' has created a pull request for this

[jira] [Assigned] (SPARK-32057) SparkExecuteStatementOperation does not set CANCELED state correctly

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32057: Assignee: Apache Spark > SparkExecuteStatementOperation does not set CANCELED state

[jira] [Assigned] (SPARK-32057) SparkExecuteStatementOperation does not set CANCELED state correctly

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32057: Assignee: (was: Apache Spark) > SparkExecuteStatementOperation does not set CANCELED

[jira] [Updated] (SPARK-32057) SparkExecuteStatementOperation does not set CANCELED state correctly

2020-06-23 Thread Ali Smesseim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Smesseim updated SPARK-32057: - Summary: SparkExecuteStatementOperation does not set CANCELED state correctly (was:

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Updated] (SPARK-32067) [K8s] Pod template from subsequently submission inadvertently applies to ongoing submission

2020-06-23 Thread James Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Yu updated SPARK-32067: - Description: THE BUG: The bug is reproducible by spark-submit two different apps (app1 and app2) with

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-06-23 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143151#comment-17143151 ] Thomas Graves commented on SPARK-32037: --- I agree healthy/unhealty could mean other things then the

[jira] [Assigned] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32077: Assignee: (was: Apache Spark) > Support host-local shuffle data reading with

[jira] [Commented] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143021#comment-17143021 ] Apache Spark commented on SPARK-32077: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143020#comment-17143020 ] Apache Spark commented on SPARK-32077: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32077: Assignee: Apache Spark > Support host-local shuffle data reading with external shuffle

[jira] [Comment Edited] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not

2020-06-23 Thread Jim Huang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142990#comment-17142990 ] Jim Huang edited comment on SPARK-31995 at 6/23/20, 3:00 PM: - Thanks Gabor

[jira] [Comment Edited] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143007#comment-17143007 ] Gabor Somogyi edited comment on SPARK-32001 at 6/23/20, 2:54 PM: - With

[jira] [Created] (SPARK-32077) Support host-local shuffle data reading with external shuffle service disabled

2020-06-23 Thread wuyi (Jira)
wuyi created SPARK-32077: Summary: Support host-local shuffle data reading with external shuffle service disabled Key: SPARK-32077 URL: https://issues.apache.org/jira/browse/SPARK-32077 Project: Spark

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143007#comment-17143007 ] Gabor Somogyi commented on SPARK-32001: --- With the service loader no registration is needed and

[jira] [Updated] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-23 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Oosterhuis updated SPARK-32051: - Fix Version/s: (was: 3.0.1) (was: 3.1.0) >

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-23 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143001#comment-17143001 ] Frank Oosterhuis commented on SPARK-32051: -- Looks like a similar conflict happens with

[jira] [Assigned] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32075: Assignee: Apache Spark > Fix a few issues in parameters table >

[jira] [Commented] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142994#comment-17142994 ] Apache Spark commented on SPARK-32075: -- User 'sidedoorleftroad' has created a pull request for this

[jira] [Assigned] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32075: Assignee: (was: Apache Spark) > Fix a few issues in parameters table >

[jira] [Commented] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not have

2020-06-23 Thread Jim Huang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142990#comment-17142990 ] Jim Huang commented on SPARK-31995: --- Thanks Gabor for triaging this issue.  SPARK-32076 has been

[jira] [Created] (SPARK-32076) Structured Streaming application continuity when encountering streaming query task level error

2020-06-23 Thread Jim Huang (Jira)
Jim Huang created SPARK-32076: - Summary: Structured Streaming application continuity when encountering streaming query task level error Key: SPARK-32076 URL: https://issues.apache.org/jira/browse/SPARK-32076

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142969#comment-17142969 ] Takeshi Yamamuro commented on SPARK-32001: -- I just want to know the approach

[jira] [Created] (SPARK-32075) Fix a few issues in parameters table

2020-06-23 Thread Zuo Dao (Jira)
Zuo Dao created SPARK-32075: --- Summary: Fix a few issues in parameters table Key: SPARK-32075 URL: https://issues.apache.org/jira/browse/SPARK-32075 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not have

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-31995. --- Resolution: Information Provided The issue should disappear w/ Spark 3.0. Please re-open it

[jira] [Closed] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not have en

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-31995. - > Spark Structure Streaming checkpiontFileManager ERROR when > HDFS.DFSOutputStream.completeFile

[jira] [Commented] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not have

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142957#comment-17142957 ] Gabor Somogyi commented on SPARK-31995: --- W/o super deep consideration I would say streaming query

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142937#comment-17142937 ] Gabor Somogyi commented on SPARK-32001: --- Here is the DT provider API:

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142934#comment-17142934 ] Gabor Somogyi commented on SPARK-32001: --- Which system you mean? A provider can be just added as an

[jira] [Commented] (SPARK-31995) Spark Structure Streaming checkpiontFileManager ERROR when HDFS.DFSOutputStream.completeFile with IOException unable to close file because the last block does not have

2020-06-23 Thread Jim Huang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142932#comment-17142932 ] Jim Huang commented on SPARK-31995: --- Thank you for providing a helpful perspective.  I was able to

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142916#comment-17142916 ] Takeshi Yamamuro commented on SPARK-32001: -- Do you know any other systems supporting that kind

[jira] [Assigned] (SPARK-32074) Update AppVeyor R to 4.0.2

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32074: Assignee: Apache Spark > Update AppVeyor R to 4.0.2 > -- > >

[jira] [Commented] (SPARK-32074) Update AppVeyor R to 4.0.2

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142915#comment-17142915 ] Apache Spark commented on SPARK-32074: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-32074) Update AppVeyor R to 4.0.2

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142914#comment-17142914 ] Apache Spark commented on SPARK-32074: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32074) Update AppVeyor R to 4.0.2

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32074: Assignee: (was: Apache Spark) > Update AppVeyor R to 4.0.2 >

[jira] [Updated] (SPARK-32074) Update AppVeyor R to 4.0.2

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32074: - Summary: Update AppVeyor R to 4.0.2 (was: Update AppVeyor R to 4.0.1) > Update AppVeyor R to

[jira] [Updated] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-23 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Oosterhuis updated SPARK-32051: - Fix Version/s: 3.1.0 > Dataset.foreachPartition returns object >

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142911#comment-17142911 ] Gabor Somogyi commented on SPARK-32001: --- Yeah, develper's API for sure. My plan is to load

[jira] [Updated] (SPARK-32074) Update AppVeyor R to 4.0.1

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32074: - Summary: Update AppVeyor R to 4.0.1 (was: Update AppVeyor R and Rtools to 4.0.1) > Update

[jira] [Updated] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-23 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Oosterhuis updated SPARK-32051: - Fix Version/s: 3.0.1 > Dataset.foreachPartition returns object >

[jira] [Updated] (SPARK-32074) Update AppVeyor R and Rtools to 4.0.1

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32074: - Summary: Update AppVeyor R and Rtools to 4.0.1 (was: Update AppVeyor R to 4.0.0) > Update

[jira] [Created] (SPARK-32074) Update AppVeyor R to 4.0.0

2020-06-23 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32074: Summary: Update AppVeyor R to 4.0.0 Key: SPARK-32074 URL: https://issues.apache.org/jira/browse/SPARK-32074 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142907#comment-17142907 ] Apache Spark commented on SPARK-32073: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32073: Assignee: (was: Apache Spark) > Drop R < 3.5 support > > >

[jira] [Assigned] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32073: Assignee: Apache Spark > Drop R < 3.5 support > > >

[jira] [Commented] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142906#comment-17142906 ] Apache Spark commented on SPARK-32073: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2020-06-23 Thread Itamar Turner-Trauring (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142901#comment-17142901 ] Itamar Turner-Trauring commented on SPARK-23206: Just checking in again, any chance

[jira] [Created] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32073: Summary: Drop R < 3.5 support Key: SPARK-32073 URL: https://issues.apache.org/jira/browse/SPARK-32073 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32073) Drop R < 3.5 support

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32073: - Description: Spark 3.0.0 is built by R 3.6.3 which does not support R < 3.5: {code} Error in

[jira] [Commented] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142862#comment-17142862 ] Apache Spark commented on SPARK-31918: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32043) Replace decimal by int op in `make_interval` and `make_timestamp`

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32043: --- Assignee: Maxim Gekk > Replace decimal by int op in `make_interval` and `make_timestamp` >

[jira] [Assigned] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31918: Assignee: (was: Apache Spark) > SparkR CRAN check gives a warning with R 4.0.0 on

[jira] [Resolved] (SPARK-32043) Replace decimal by int op in `make_interval` and `make_timestamp`

2020-06-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32043. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28886

[jira] [Assigned] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31918: Assignee: Apache Spark > SparkR CRAN check gives a warning with R 4.0.0 on OSX >

[jira] [Commented] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142859#comment-17142859 ] Apache Spark commented on SPARK-31918: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32072: Assignee: (was: Apache Spark) > Unaligned benchmark results >

[jira] [Assigned] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32072: Assignee: Apache Spark > Unaligned benchmark results > > >

[jira] [Commented] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142851#comment-17142851 ] Apache Spark commented on SPARK-32072: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Created] (SPARK-32072) Unaligned benchmark results

2020-06-23 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-32072: -- Summary: Unaligned benchmark results Key: SPARK-32072 URL: https://issues.apache.org/jira/browse/SPARK-32072 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-32001) Create Kerberos authentication provider API in JDBC connector

2020-06-23 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142838#comment-17142838 ] Takeshi Yamamuro commented on SPARK-32001: -- Do you propose to provide a develper's API for

[jira] [Commented] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142837#comment-17142837 ] Hyukjin Kwon commented on SPARK-31918: -- Ok.. I finally made all tests being passed. I will make a

[jira] [Updated] (SPARK-32049) Upgrade Oracle JDBC Driver 8

2020-06-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32049: -- Summary: Upgrade Oracle JDBC Driver 8 (was: Upgrade oracle JDBC driver to the latest) >

  1   2   >