[jira] [Commented] (SPARK-17025) Cannot persist PySpark ML Pipeline model that includes custom Transformer

[ https://issues.apache.org/jira/browse/SPARK-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836905#comment-16836905 ] Yogesh Agrawal commented on SPARK-17025: Hello all, [~josephkb], [~Hadar],[~nchammas] , i have

[jira] [Commented] (SPARK-21172) EOFException reached end of stream in UnsafeRowSerializer

[ https://issues.apache.org/jira/browse/SPARK-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836901#comment-16836901 ] Lasantha Fernando commented on SPARK-21172: --- Encountered similar kind of issues in 2.2.2 as

[jira] [Commented] (SPARK-26437) Decimal data becomes bigint to query, unable to query

[ https://issues.apache.org/jira/browse/SPARK-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836878#comment-16836878 ] Xiao Li commented on SPARK-26437: - Even if we do not use our native ORC reader, Spark 3.0 will be able

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836871#comment-16836871 ] bupt_ljy commented on SPARK-26182: -- [~mgaido] asNondetermistic is added since 2.3. It works fine after

[jira] [Resolved] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bupt_ljy resolved SPARK-26182. -- Resolution: Invalid > Cost increases when optimizing scalaUDF >

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836828#comment-16836828 ] tommy duan commented on SPARK-27648: Hi [~gsomogyi] & [~kabhwan] , Please check the log file

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: houragg(1).out > In Spark2.4 Structured Streaming：The executor storage memory

[jira] [Assigned] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27669: Assignee: (was: Apache Spark) > Refactor DataFrameWriter to resolve datasources in a

[jira] [Assigned] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27669: Assignee: Apache Spark > Refactor DataFrameWriter to resolve datasources in a command >

[jira] [Updated] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-27669: --- Summary: Refactor DataFrameWriter to resolve datasources in a command (was: Refactor

[jira] [Created] (SPARK-27669) Refactor DataFrameWriter to always go through Catalyst for analysis

Eric Liang created SPARK-27669: -- Summary: Refactor DataFrameWriter to always go through Catalyst for analysis Key: SPARK-27669 URL: https://issues.apache.org/jira/browse/SPARK-27669 Project: Spark

[jira] [Reopened] (SPARK-27600) Unable to start Spark Hive Thrift Server when multiple hive server server share the same metastore

[ https://issues.apache.org/jira/browse/SPARK-27600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pin_zhang reopened SPARK-27600: --- The issue is not resolved > Unable to start Spark Hive Thrift Server when multiple hive server server

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836750#comment-16836750 ] Xingbo Jiang commented on SPARK-27520: -- The major problem of `SparkContext.hadoopConfiguration` is

[jira] [Resolved] (SPARK-27271) Migrate Text to File Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-27271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-27271. Resolution: Fixed Resolved in https://github.com/apache/spark/pull/24207 > Migrate Text

[jira] [Updated] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27668: --- Description: In File source V1, the statistics of `HadoopFsRelation` is `compressionFactor

[jira] [Updated] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27668: --- Environment: (was: In File source V1, the statistics of `HadoopFsRelation` is

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836730#comment-16836730 ] Thomas Graves commented on SPARK-27520: --- Can we add more to the description to explain why we are

[jira] [Assigned] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27668: Assignee: (was: Apache Spark) > File source V2: support reporting statistics >

[jira] [Assigned] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27668: Assignee: Apache Spark > File source V2: support reporting statistics >

[jira] [Created] (SPARK-27668) File source V2: support reporting statistics

Gengliang Wang created SPARK-27668: -- Summary: File source V2: support reporting statistics Key: SPARK-27668 URL: https://issues.apache.org/jira/browse/SPARK-27668 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836658#comment-16836658 ] Siddharth Dangi commented on SPARK-23986: - [~pedromorfeu] I tried the workaround you mentioned

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836616#comment-16836616 ] Gabor Somogyi commented on SPARK-27648: --- [~yy3b2007com] [~kabhwan] pointed you to the right

[jira] [Commented] (SPARK-23098) Migrate Kafka batch source to v2

[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836598#comment-16836598 ] Gabor Somogyi commented on SPARK-23098: --- Guys, thanks for the confirmation. There are minor things

[jira] [Assigned] (SPARK-23191) Workers registration failes in case of network drop

[ https://issues.apache.org/jira/browse/SPARK-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23191: Assignee: Apache Spark > Workers registration failes in case of network drop >

[jira] [Assigned] (SPARK-23191) Workers registration failes in case of network drop

[ https://issues.apache.org/jira/browse/SPARK-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23191: Assignee: (was: Apache Spark) > Workers registration failes in case of network drop

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836486#comment-16836486 ] Marco Gaido commented on SPARK-26182: - Actually you just need to mark it {{asNondetermistic}} to

[jira] [Resolved] (SPARK-27089) Loss of precision during decimal division

[ https://issues.apache.org/jira/browse/SPARK-27089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-27089. - Resolution: Information Provided > Loss of precision during decimal division >

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836480#comment-16836480 ] Weichen Xu commented on SPARK-26412: Discuss with [~mengxr] , discard proposal (2), this should be

[jira] [Comment Edited] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836377#comment-16836377 ] Weichen Xu edited comment on SPARK-26412 at 5/9/19 3:19 PM: [~mengxr]  

[jira] [Comment Edited] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836377#comment-16836377 ] Weichen Xu edited comment on SPARK-26412 at 5/9/19 3:18 PM: [~mengxr]  

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836435#comment-16836435 ] Hyukjin Kwon commented on SPARK-27663: -- +1 should better be narrowed down and check if the issue

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Resolved] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27636. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24531

[jira] [Assigned] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27636: - Assignee: shahid > Remove cached RDD blocks after PIC execution >

[jira] [Updated] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27636: -- Priority: Minor (was: Major) > Remove cached RDD blocks after PIC execution >

[jira] [Resolved] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27540. --- Resolution: Fixed Fix Version/s: 3.1.0 Issue resolved by pull request 24543

[jira] [Assigned] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27540: - Assignee: Tarush Grover > Add 'meanAveragePrecision_at_k' metric to RankingMetrics >

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836392#comment-16836392 ] Sean Owen commented on SPARK-27663: --- Also, you'd need to reproduce on master. 2.1.0 is very old. >

[jira] [Updated] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27540: -- Fix Version/s: (was: 3.1.0) 3.0.0 > Add 'meanAveragePrecision_at_k' metric to

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836391#comment-16836391 ] Sean Owen commented on SPARK-27663: --- No idea, I don't think this narrows it down enough to say. You

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836377#comment-16836377 ] Weichen Xu commented on SPARK-26412: [~mengxr]   There's one issue:   There're 2 proposals in the

[jira] [Commented] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836370#comment-16836370 ] Yuming Wang commented on SPARK-27631: - {code:java} **Related code path**:

[jira] [Assigned] (SPARK-27638) date format yyyy-M-dd string comparison not handled properly

[ https://issues.apache.org/jira/browse/SPARK-27638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27638: Assignee: Apache Spark > date format -M-dd string comparison not handled properly >

[jira] [Assigned] (SPARK-27638) date format yyyy-M-dd string comparison not handled properly

[ https://issues.apache.org/jira/browse/SPARK-27638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27638: Assignee: (was: Apache Spark) > date format -M-dd string comparison not handled

[jira] [Assigned] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27667: Assignee: (was: Apache Spark) > when hive.cli.print.current.db is set, spark cli is

[jira] [Assigned] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27667: Assignee: Apache Spark > when hive.cli.print.current.db is set, spark cli is not working

[jira] [Assigned] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27665: Assignee: (was: Apache Spark) > Split fetch shuffle blocks protocol from OpenBlocks

[jira] [Assigned] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27665: Assignee: Apache Spark > Split fetch shuffle blocks protocol from OpenBlocks >

[jira] [Created] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

Sandeep Katta created SPARK-27667: - Summary: when hive.cli.print.current.db is set, spark cli is not working as expected Key: SPARK-27667 URL: https://issues.apache.org/jira/browse/SPARK-27667

[jira] [Updated] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-27667: -- Attachment: before.png > when hive.cli.print.current.db is set, spark cli is not working as

[jira] [Created] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

Yuanjian Li created SPARK-27665: --- Summary: Split fetch shuffle blocks protocol from OpenBlocks Key: SPARK-27665 URL: https://issues.apache.org/jira/browse/SPARK-27665 Project: Spark Issue

[jira] [Created] (SPARK-27666) Stop python runner threads when task finishes

Wenchen Fan created SPARK-27666: --- Summary: Stop python runner threads when task finishes Key: SPARK-27666 URL: https://issues.apache.org/jira/browse/SPARK-27666 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-27665: Description: As the current approach in OneForOneBlockFetcher, we reuse the OpenBlocks protocol

[jira] [Updated] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-27664: Description: In short, This issue(i.e. degraded performance ) surfaces when the number

[jira] [Created] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

Prashant Sharma created SPARK-27664: --- Summary: Performance issue with FileStatusCache, while reading from object stores. Key: SPARK-27664 URL: https://issues.apache.org/jira/browse/SPARK-27664

[jira] [Assigned] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27625: Assignee: Apache Spark > ScalaReflection.serializerFor fails for annotated types >

[jira] [Assigned] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27625: Assignee: (was: Apache Spark) > ScalaReflection.serializerFor fails for annotated

[jira] [Assigned] (SPARK-27617) Not able to specify LOCATION for internal table

[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27617: Assignee: Apache Spark > Not able to specify LOCATION for internal table >

[jira] [Assigned] (SPARK-27617) Not able to specify LOCATION for internal table

[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27617: Assignee: (was: Apache Spark) > Not able to specify LOCATION for internal table >

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836244#comment-16836244 ] tommy duan edited comment on SPARK-27648 at 5/9/19 9:55 AM: [~kabhwan]  the

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836244#comment-16836244 ] tommy duan commented on SPARK-27648: [~kabhwan]  the print the progress information，as bellow：

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836241#comment-16836241 ] tommy duan commented on SPARK-27648: [~gsomogyi] yes,The agg opreation is used in my program  

[jira] [Updated] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27631: Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-23710) > Avoid repeating