[jira] [Comment Edited] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836950#comment-16836950 ] Pedro Fernandes edited comment on SPARK-23986 at 5/10/19 6:52 AM:

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836950#comment-16836950 ] Pedro Fernandes commented on SPARK-23986: - [~sdangi1], how many operations do yo

[jira] [Commented] (SPARK-17025) Cannot persist PySpark ML Pipeline model that includes custom Transformer

[ https://issues.apache.org/jira/browse/SPARK-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836905#comment-16836905 ] Yogesh Agrawal commented on SPARK-17025: Hello all, [~josephkb], [~Hadar],[~nch

[jira] [Commented] (SPARK-21172) EOFException reached end of stream in UnsafeRowSerializer

[ https://issues.apache.org/jira/browse/SPARK-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836901#comment-16836901 ] Lasantha Fernando commented on SPARK-21172: --- Encountered similar kind of issue

[jira] [Commented] (SPARK-26437) Decimal data becomes bigint to query, unable to query

[ https://issues.apache.org/jira/browse/SPARK-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836878#comment-16836878 ] Xiao Li commented on SPARK-26437: - Even if we do not use our native ORC reader, Spark 3.

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836871#comment-16836871 ] bupt_ljy commented on SPARK-26182: -- [~mgaido] asNondetermistic is added since 2.3. It w

[jira] [Resolved] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bupt_ljy resolved SPARK-26182. -- Resolution: Invalid > Cost increases when optimizing scalaUDF > --

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836828#comment-16836828 ] tommy duan commented on SPARK-27648: Hi [~gsomogyi] & [~kabhwan] , Please check the

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: houragg(1).out > In Spark2.4 Structured Streaming：The executor storage memory increasing

[jira] [Assigned] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27669: Assignee: (was: Apache Spark) > Refactor DataFrameWriter to resolve datasources in a

[jira] [Assigned] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27669: Assignee: Apache Spark > Refactor DataFrameWriter to resolve datasources in a command > -

[jira] [Updated] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-27669: --- Summary: Refactor DataFrameWriter to resolve datasources in a command (was: Refactor DataFrameWrite

[jira] [Created] (SPARK-27669) Refactor DataFrameWriter to always go through Catalyst for analysis

Eric Liang created SPARK-27669: -- Summary: Refactor DataFrameWriter to always go through Catalyst for analysis Key: SPARK-27669 URL: https://issues.apache.org/jira/browse/SPARK-27669 Project: Spark

[jira] [Reopened] (SPARK-27600) Unable to start Spark Hive Thrift Server when multiple hive server server share the same metastore

[ https://issues.apache.org/jira/browse/SPARK-27600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pin_zhang reopened SPARK-27600: --- The issue is not resolved > Unable to start Spark Hive Thrift Server when multiple hive server server

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836750#comment-16836750 ] Xingbo Jiang commented on SPARK-27520: -- The major problem of `SparkContext.hadoopCo

[jira] [Resolved] (SPARK-27271) Migrate Text to File Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-27271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-27271. Resolution: Fixed Resolved in https://github.com/apache/spark/pull/24207 > Migrate Text

[jira] [Updated] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27668: --- Description: In File source V1, the statistics of `HadoopFsRelation` is `compressionFactor *

[jira] [Updated] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27668: --- Environment: (was: In File source V1, the statistics of `HadoopFsRelation` is `compressi

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836730#comment-16836730 ] Thomas Graves commented on SPARK-27520: --- Can we add more to the description to exp

[jira] [Assigned] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27668: Assignee: (was: Apache Spark) > File source V2: support reporting statistics > --

[jira] [Assigned] (SPARK-27668) File source V2: support reporting statistics

[ https://issues.apache.org/jira/browse/SPARK-27668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27668: Assignee: Apache Spark > File source V2: support reporting statistics > -

[jira] [Created] (SPARK-27668) File source V2: support reporting statistics

Gengliang Wang created SPARK-27668: -- Summary: File source V2: support reporting statistics Key: SPARK-27668 URL: https://issues.apache.org/jira/browse/SPARK-27668 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836658#comment-16836658 ] Siddharth Dangi commented on SPARK-23986: - [~pedromorfeu] I tried the workaround

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836616#comment-16836616 ] Gabor Somogyi commented on SPARK-27648: --- [~yy3b2007com] [~kabhwan] pointed you to

[jira] [Commented] (SPARK-23098) Migrate Kafka batch source to v2

[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836598#comment-16836598 ] Gabor Somogyi commented on SPARK-23098: --- Guys, thanks for the confirmation. There

[jira] [Assigned] (SPARK-23191) Workers registration failes in case of network drop

[ https://issues.apache.org/jira/browse/SPARK-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23191: Assignee: Apache Spark > Workers registration failes in case of network drop > --

[jira] [Assigned] (SPARK-23191) Workers registration failes in case of network drop

[ https://issues.apache.org/jira/browse/SPARK-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23191: Assignee: (was: Apache Spark) > Workers registration failes in case of network drop >

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836486#comment-16836486 ] Marco Gaido commented on SPARK-26182: - Actually you just need to mark it {{asNondete

[jira] [Resolved] (SPARK-27089) Loss of precision during decimal division

[ https://issues.apache.org/jira/browse/SPARK-27089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-27089. - Resolution: Information Provided > Loss of precision during decimal division > -

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836480#comment-16836480 ] Weichen Xu commented on SPARK-26412: Discuss with [~mengxr] , discard proposal (2),

[jira] [Comment Edited] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836377#comment-16836377 ] Weichen Xu edited comment on SPARK-26412 at 5/9/19 3:19 PM:

[jira] [Comment Edited] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836377#comment-16836377 ] Weichen Xu edited comment on SPARK-26412 at 5/9/19 3:18 PM:

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836435#comment-16836435 ] Hyukjin Kwon commented on SPARK-27663: -- +1 should better be narrowed down and check

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark,  https://issues.apach

[jira] [Resolved] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27636. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24531 [https://github.c

[jira] [Assigned] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27636: - Assignee: shahid > Remove cached RDD blocks after PIC execution > -

[jira] [Updated] (SPARK-27636) Remove cached RDD blocks after PIC execution

[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27636: -- Priority: Minor (was: Major) > Remove cached RDD blocks after PIC execution > ---

[jira] [Resolved] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27540. --- Resolution: Fixed Fix Version/s: 3.1.0 Issue resolved by pull request 24543 [https://github.c

[jira] [Assigned] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27540: - Assignee: Tarush Grover > Add 'meanAveragePrecision_at_k' metric to RankingMetrics > --

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836392#comment-16836392 ] Sean Owen commented on SPARK-27663: --- Also, you'd need to reproduce on master. 2.1.0 is

[jira] [Updated] (SPARK-27540) Add 'meanAveragePrecision_at_k' metric to RankingMetrics

[ https://issues.apache.org/jira/browse/SPARK-27540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27540: -- Fix Version/s: (was: 3.1.0) 3.0.0 > Add 'meanAveragePrecision_at_k' metric to R

[jira] [Commented] (SPARK-27663) Task accomplished incompletely but marked as success

[ https://issues.apache.org/jira/browse/SPARK-27663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836391#comment-16836391 ] Sean Owen commented on SPARK-27663: --- No idea, I don't think this narrows it down enoug

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames or Arrow batches

[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836377#comment-16836377 ] Weichen Xu commented on SPARK-26412: [~mengxr]   There's one issue:   There're 2 p

[jira] [Commented] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836370#comment-16836370 ] Yuming Wang commented on SPARK-27631: - {code:java} **Related code path**: https://gi

[jira] [Assigned] (SPARK-27638) date format yyyy-M-dd string comparison not handled properly

[ https://issues.apache.org/jira/browse/SPARK-27638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27638: Assignee: Apache Spark > date format -M-dd string comparison not handled properly >

[jira] [Assigned] (SPARK-27638) date format yyyy-M-dd string comparison not handled properly

[ https://issues.apache.org/jira/browse/SPARK-27638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27638: Assignee: (was: Apache Spark) > date format -M-dd string comparison not handled p

[jira] [Assigned] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27667: Assignee: (was: Apache Spark) > when hive.cli.print.current.db is set, spark cli is n

[jira] [Assigned] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27667: Assignee: Apache Spark > when hive.cli.print.current.db is set, spark cli is not working

[jira] [Assigned] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27665: Assignee: (was: Apache Spark) > Split fetch shuffle blocks protocol from OpenBlocks >

[jira] [Assigned] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27665: Assignee: Apache Spark > Split fetch shuffle blocks protocol from OpenBlocks > --

[jira] [Created] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

Sandeep Katta created SPARK-27667: - Summary: when hive.cli.print.current.db is set, spark cli is not working as expected Key: SPARK-27667 URL: https://issues.apache.org/jira/browse/SPARK-27667 Project

[jira] [Updated] (SPARK-27667) when hive.cli.print.current.db is set, spark cli is not working as expected

[ https://issues.apache.org/jira/browse/SPARK-27667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-27667: -- Attachment: before.png > when hive.cli.print.current.db is set, spark cli is not working as ex

[jira] [Created] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

Yuanjian Li created SPARK-27665: --- Summary: Split fetch shuffle blocks protocol from OpenBlocks Key: SPARK-27665 URL: https://issues.apache.org/jira/browse/SPARK-27665 Project: Spark Issue Type:

[jira] [Created] (SPARK-27666) Stop python runner threads when task finishes

Wenchen Fan created SPARK-27666: --- Summary: Stop python runner threads when task finishes Key: SPARK-27666 URL: https://issues.apache.org/jira/browse/SPARK-27666 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-27665: Description: As the current approach in OneForOneBlockFetcher, we reuse the OpenBlocks protocol to

[jira] [Updated] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-27664: Description: In short, This issue(i.e. degraded performance ) surfaces when the number of

[jira] [Created] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

Prashant Sharma created SPARK-27664: --- Summary: Performance issue with FileStatusCache, while reading from object stores. Key: SPARK-27664 URL: https://issues.apache.org/jira/browse/SPARK-27664 Proje

[jira] [Assigned] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27625: Assignee: Apache Spark > ScalaReflection.serializerFor fails for annotated types > --

[jira] [Assigned] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27625: Assignee: (was: Apache Spark) > ScalaReflection.serializerFor fails for annotated typ

[jira] [Assigned] (SPARK-27617) Not able to specify LOCATION for internal table

[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27617: Assignee: Apache Spark > Not able to specify LOCATION for internal table > --

[jira] [Assigned] (SPARK-27617) Not able to specify LOCATION for internal table

[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27617: Assignee: (was: Apache Spark) > Not able to specify LOCATION for internal table > ---

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836244#comment-16836244 ] tommy duan edited comment on SPARK-27648 at 5/9/19 9:55 AM:

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836244#comment-16836244 ] tommy duan commented on SPARK-27648: [~kabhwan]  the print the progress information

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836241#comment-16836241 ] tommy duan commented on SPARK-27648: [~gsomogyi] yes,The agg opreation is used in my

[jira] [Updated] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27631: Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-23710) > Avoid repeating