[jira] [Commented] (SPARK-25313) Fix regression in FileFormatWriter output schema

[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605335#comment-16605335 ] Apache Spark commented on SPARK-25313: -- User 'gengliangwang' has created a pull req

[jira] [Commented] (SPARK-12321) JSON format for logical/physical execution plans

[ https://issues.apache.org/jira/browse/SPARK-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605317#comment-16605317 ] Apache Spark commented on SPARK-12321: -- User 'gatorsmile' has created a pull reques

[jira] [Commented] (SPARK-12321) JSON format for logical/physical execution plans

[ https://issues.apache.org/jira/browse/SPARK-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605316#comment-16605316 ] Apache Spark commented on SPARK-12321: -- User 'gatorsmile' has created a pull reques

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605308#comment-16605308 ] Gengliang Wang commented on SPARK-24771: [~vanzin] I am OK with either way. Shad

[jira] [Commented] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605263#comment-16605263 ] Apache Spark commented on SPARK-25352: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25352: Assignee: Apache Spark > Perform ordered global limit when limit number is bigger than >

[jira] [Assigned] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25352: Assignee: (was: Apache Spark) > Perform ordered global limit when limit number is big

[jira] [Created] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

Liang-Chi Hsieh created SPARK-25352: --- Summary: Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold Key: SPARK-25352 URL: https://issues.apache.org/jira/browse/SPARK-25352

[jira] [Resolved] (SPARK-25252) Support arrays of any types in to_json

[ https://issues.apache.org/jira/browse/SPARK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25252. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 6 [https://gi

[jira] [Assigned] (SPARK-25252) Support arrays of any types in to_json

[ https://issues.apache.org/jira/browse/SPARK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25252: Assignee: Maxim Gekk > Support arrays of any types in to_json > -

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605250#comment-16605250 ] Imran Rashid commented on SPARK-25344: -- kinda related, maybe this should get its ow

[jira] [Assigned] (SPARK-25337) HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasour

[ https://issues.apache.org/jira/browse/SPARK-25337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25337: - Assignee: Dongjoon Hyun > HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: >

[jira] [Resolved] (SPARK-25337) HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasour

[ https://issues.apache.org/jira/browse/SPARK-25337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25337. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22340 [https://github.c

[jira] [Updated] (SPARK-20918) Use FunctionIdentifier as function identifiers in FunctionRegistry

[ https://issues.apache.org/jira/browse/SPARK-20918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-20918: Labels: release-notes (was: ) > Use FunctionIdentifier as function identifiers in FunctionRegistr

[jira] [Resolved] (SPARK-25313) Fix regression in FileFormatWriter output schema

[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25313. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22320 [https://gith

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25313: --- Assignee: Gengliang Wang > Fix regression in FileFormatWriter output schema > -

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605186#comment-16605186 ] Hyukjin Kwon commented on SPARK-18112: -- We need the metastore jar if I understood c

[jira] [Commented] (SPARK-25346) Document Spark builtin data sources

[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605141#comment-16605141 ] Hyukjin Kwon commented on SPARK-25346: -- Avro - documentation was added in SPARK-251

[jira] [Commented] (SPARK-25346) Document Spark builtin data sources

[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605140#comment-16605140 ] Hyukjin Kwon commented on SPARK-25346: -- [~mengxr], actually there are documentation

[jira] [Comment Edited] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605131#comment-16605131 ] Yuming Wang edited comment on SPARK-25330 at 9/6/18 1:09 AM: -

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605131#comment-16605131 ] Yuming Wang commented on SPARK-25330: - I try to build Hadoop 2.7.7 with[{{Configura

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605129#comment-16605129 ] Yuming Wang commented on SPARK-25330: - No. The issue occurred in this commit:  [apac

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605114#comment-16605114 ] Eric Yang commented on SPARK-25330: --- [~yumwang] Does Hadoop 2.7.5 works? It might hel

[jira] [Assigned] (SPARK-25268) runParallelPersonalizedPageRank throws serialization Exception

[ https://issues.apache.org/jira/browse/SPARK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-25268: - Assignee: shahid > runParallelPersonalizedPageRank throws serialization Excepti

[jira] [Updated] (SPARK-20901) Feature parity for ORC with Parquet

[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20901: -- Affects Version/s: 2.4.0 > Feature parity for ORC with Parquet > -

[jira] [Resolved] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-23774. --- Resolution: Won't Do Per review comments, we will revisit this when we can support CHAR/VARC

[jira] [Assigned] (SPARK-23131) Kryo raises StackOverflow during serializing GLR model

[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23131: - Assignee: Yuming Wang > Kryo raises StackOverflow during serializing GLR model > --

[jira] [Assigned] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25176: - Assignee: Yuming Wang > Kryo fails to serialize a parametrised type hierarchy > ---

[jira] [Assigned] (SPARK-25258) Upgrade kryo package to version 4.0.2

[ https://issues.apache.org/jira/browse/SPARK-25258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25258: - Assignee: Yuming Wang > Upgrade kryo package to version 4.0.2 > ---

[jira] [Resolved] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25176. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22179 [https://github.c

[jira] [Resolved] (SPARK-23131) Kryo raises StackOverflow during serializing GLR model

[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23131. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22179 [https://github.c

[jira] [Resolved] (SPARK-25258) Upgrade kryo package to version 4.0.2

[ https://issues.apache.org/jira/browse/SPARK-25258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25258. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22179 [https://github.c

[jira] [Resolved] (SPARK-25335) Skip Zinc downloading if it's installed in the system

[ https://issues.apache.org/jira/browse/SPARK-25335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25335. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22333 [https://github.c

[jira] [Assigned] (SPARK-25335) Skip Zinc downloading if it's installed in the system

[ https://issues.apache.org/jira/browse/SPARK-25335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25335: - Assignee: Dongjoon Hyun > Skip Zinc downloading if it's installed in the system > -

[jira] [Resolved] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23243. - Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.4.0 > Shuffle+Repartition on an

[jira] [Updated] (SPARK-25268) runParallelPersonalizedPageRank throws serialization Exception

[ https://issues.apache.org/jira/browse/SPARK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25268: -- Shepherd: Joseph K. Bradley > runParallelPersonalizedPageRank throws serialization Exc

[jira] [Resolved] (SPARK-25231) Running a Large Job with Speculation On Causes Executor Heartbeats to Time Out on Driver

[ https://issues.apache.org/jira/browse/SPARK-25231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25231. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.4.0

[jira] [Created] (SPARK-25351) Handle Pandas category type when converting from Python with Arrow

Bryan Cutler created SPARK-25351: Summary: Handle Pandas category type when converting from Python with Arrow Key: SPARK-25351 URL: https://issues.apache.org/jira/browse/SPARK-25351 Project: Spark

[jira] [Updated] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21187: - Description: This is to track adding the remaining type support in Arrow Converters. Currently,

[jira] [Commented] (SPARK-19809) NullPointerException on zero-size ORC file

[ https://issues.apache.org/jira/browse/SPARK-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604926#comment-16604926 ] Shirish Tatikonda commented on SPARK-19809: --- Thank you [~dongjoon] > NullPoin

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604877#comment-16604877 ] Marcelo Vanzin commented on SPARK-24771: I ran a couple of our tests that exerci

[jira] [Commented] (SPARK-25350) Spark Serving

[ https://issues.apache.org/jira/browse/SPARK-25350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604842#comment-16604842 ] Mark Hamilton commented on SPARK-25350: --- Hey, [~rxin], we had talked about this co

[jira] [Updated] (SPARK-25350) Spark Serving

[ https://issues.apache.org/jira/browse/SPARK-25350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamilton updated SPARK-25350: -- Description: Microsoft has created a new system to turn Structured Streaming jobs into RESTfu

[jira] [Updated] (SPARK-25350) Spark Serving

[ https://issues.apache.org/jira/browse/SPARK-25350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamilton updated SPARK-25350: -- Description: Microsoft has created a new system to turn Structured Streaming jobs into RESTfu

[jira] [Created] (SPARK-25350) Spark Serving

Mark Hamilton created SPARK-25350: - Summary: Spark Serving Key: SPARK-25350 URL: https://issues.apache.org/jira/browse/SPARK-25350 Project: Spark Issue Type: New Feature Components:

[jira] [Updated] (SPARK-25347) Document image data source in doc site

[ https://issues.apache.org/jira/browse/SPARK-25347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25347: -- Summary: Document image data source in doc site (was: Document image data sources in doc site

[jira] [Updated] (SPARK-25345) Deprecate public APIs from ImageSchema

[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25345: -- Description: After SPARK-22328, we can deprecate the public APIs in ImageSchema (Scala/Python)

[jira] [Created] (SPARK-25349) Support sample pushdown in Data Source V2

Xiangrui Meng created SPARK-25349: - Summary: Support sample pushdown in Data Source V2 Key: SPARK-25349 URL: https://issues.apache.org/jira/browse/SPARK-25349 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-25348) Data source for binary files

Xiangrui Meng created SPARK-25348: - Summary: Data source for binary files Key: SPARK-25348 URL: https://issues.apache.org/jira/browse/SPARK-25348 Project: Spark Issue Type: Story Co

[jira] [Created] (SPARK-25347) Document image data sources in doc site

Xiangrui Meng created SPARK-25347: - Summary: Document image data sources in doc site Key: SPARK-25347 URL: https://issues.apache.org/jira/browse/SPARK-25347 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-25346) Document Spark builtin data sources

[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25346: -- Summary: Document Spark builtin data sources (was: Document Spark built-in data sources) > D

[jira] [Updated] (SPARK-25346) Document Spark built-in data sources

[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25346: -- Summary: Document Spark built-in data sources (was: Document Spark buit-in data sources) > D

[jira] [Created] (SPARK-25346) Document Spark buit-in data sources

Xiangrui Meng created SPARK-25346: - Summary: Document Spark buit-in data sources Key: SPARK-25346 URL: https://issues.apache.org/jira/browse/SPARK-25346 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-25345) Deprecate public APIs from ImageSchema

Xiangrui Meng created SPARK-25345: - Summary: Deprecate public APIs from ImageSchema Key: SPARK-25345 URL: https://issues.apache.org/jira/browse/SPARK-25345 Project: Spark Issue Type: Story

[jira] [Resolved] (SPARK-22666) Spark datasource for image format

[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-22666. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22328 [https://

[jira] [Created] (SPARK-25344) Break large tests.py files into smaller files

Imran Rashid created SPARK-25344: Summary: Break large tests.py files into smaller files Key: SPARK-25344 URL: https://issues.apache.org/jira/browse/SPARK-25344 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-24360) Support Hive 3.0 metastore

[ https://issues.apache.org/jira/browse/SPARK-24360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604748#comment-16604748 ] Dongjoon Hyun commented on SPARK-24360: --- [~toopt4]. Yep. We should support Hive 3.

[jira] [Updated] (SPARK-24360) Support Hive 3.1 metastore

[ https://issues.apache.org/jira/browse/SPARK-24360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24360: -- Summary: Support Hive 3.1 metastore (was: Support Hive 3.0 metastore) > Support Hive 3.1 meta

[jira] [Updated] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Kemmer updated SPARK-25343: - Description: With the cvs() method it is currenty possible to create a Dataframe from Dataset[S

[jira] [Updated] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Kemmer updated SPARK-25343: - Description: With the cvs() method it is currenty possible to create a Dataframe from Dataset[S

[jira] [Updated] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Kemmer updated SPARK-25343: - Description: With the cvs() method it is currenty possible to create a Dataframe from Dataset[S

[jira] [Commented] (SPARK-25339) Refactor FilterPushdownBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604738#comment-16604738 ] Dongjoon Hyun commented on SPARK-25339: --- Thank you for filing this in order not to

[jira] [Created] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

Frank Kemmer created SPARK-25343: Summary: Extend CSV parsing to Dataset[List[String]] Key: SPARK-25343 URL: https://issues.apache.org/jira/browse/SPARK-25343 Project: Spark Issue Type: Impro

[jira] [Created] (SPARK-25342) Support rolling back a result stage

Wenchen Fan created SPARK-25342: --- Summary: Support rolling back a result stage Key: SPARK-25342 URL: https://issues.apache.org/jira/browse/SPARK-25342 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-25341) Support rolling back a shuffle map stage and re-generate the shuffle files

Wenchen Fan created SPARK-25341: --- Summary: Support rolling back a shuffle map stage and re-generate the shuffle files Key: SPARK-25341 URL: https://issues.apache.org/jira/browse/SPARK-25341 Project: Spa

[jira] [Assigned] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24748: Assignee: (was: Apache Spark) > Support for reporting custom metrics via Streaming Qu

[jira] [Assigned] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24748: Assignee: Apache Spark > Support for reporting custom metrics via Streaming Query Progres

[jira] [Resolved] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Gupta resolved SPARK-24539. - Resolution: Duplicate Resolving this as it has been fixed by SPARK-24415. > HistoryServer does

[jira] [Assigned] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24415: -- Assignee: Ankur Gupta > Stage page aggregated executor metrics wrong when failures >

[jira] [Resolved] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24415. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22209 [https:

[jira] [Commented] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604618#comment-16604618 ] Apache Spark commented on SPARK-14922: -- User 'mgaido91' has created a pull request

[jira] [Resolved] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichao Zhang resolved SPARK-25279. Resolution: Won't Fix > Throw exception: zzcclp java.io.NotSerializableException: > org

[jira] [Closed] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichao Zhang closed SPARK-25279. -- > Throw exception: zzcclp java.io.NotSerializableException: > org.apache.spark.sql.TypedColumn

[jira] [Commented] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604604#comment-16604604 ] Zhichao Zhang commented on SPARK-25279: [~viirya], Thanks. I closed this issue.

[jira] [Commented] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604532#comment-16604532 ] Apache Spark commented on SPARK-25132: -- User 'seancxmao' has created a pull request

[jira] [Commented] (SPARK-25132) Case-insensitive field resolution when reading from Parquet

[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604529#comment-16604529 ] Apache Spark commented on SPARK-25132: -- User 'seancxmao' has created a pull request

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604441#comment-16604441 ] Ameen Tayyebi commented on SPARK-23443: --- I've been sidetracked with lots of other

[jira] [Resolved] (SPARK-25228) Add executor CPU Time metric

[ https://issues.apache.org/jira/browse/SPARK-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25228. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22218 [https://github.c

[jira] [Assigned] (SPARK-25228) Add executor CPU Time metric

[ https://issues.apache.org/jira/browse/SPARK-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25228: - Assignee: Luca Canali > Add executor CPU Time metric > - > >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604391#comment-16604391 ] Sean Owen commented on SPARK-18112: --- I don't know much about this part, but do we need

[jira] [Updated] (SPARK-25340) Pushes down Sample beneath deterministic Project

[ https://issues.apache.org/jira/browse/SPARK-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25340: - Description: If computations in Project are heavy (e.g., UDFs), it is useful to push dow

[jira] [Commented] (SPARK-25340) Pushes down Sample beneath deterministic Project

[ https://issues.apache.org/jira/browse/SPARK-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604390#comment-16604390 ] Takeshi Yamamuro commented on SPARK-25340: -- Is this feasible? [~smilegator] >

[jira] [Created] (SPARK-25340) Pushes down Sample beneath deterministic Project

Takeshi Yamamuro created SPARK-25340: Summary: Pushes down Sample beneath deterministic Project Key: SPARK-25340 URL: https://issues.apache.org/jira/browse/SPARK-25340 Project: Spark Issu

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604331#comment-16604331 ] t oo commented on SPARK-23443: -- [~ameen.tayy...@gmail.com] any luck with the first PR? > S

[jira] [Commented] (SPARK-24632) Allow 3rd-party libraries to use pyspark.ml abstractions for Java wrappers for persistence

[ https://issues.apache.org/jira/browse/SPARK-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604298#comment-16604298 ] Mathew commented on SPARK-24632: [~bryanc] that line is only there because we use the ja

[jira] [Commented] (SPARK-24360) Support Hive 3.0 metastore

[ https://issues.apache.org/jira/browse/SPARK-24360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604292#comment-16604292 ] t oo commented on SPARK-24360: -- [~dongjoon] Can this be merged to master? Also, can hive3.1

[jira] [Comment Edited] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604254#comment-16604254 ] Liang-Chi Hsieh edited comment on SPARK-25279 at 9/5/18 10:34 AM:

[jira] [Commented] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604254#comment-16604254 ] Liang-Chi Hsieh commented on SPARK-25279: - The paste mode in REPL wraps pasted c

[jira] [Assigned] (SPARK-24889) dataset.unpersist() doesn't update storage memory stats

[ https://issues.apache.org/jira/browse/SPARK-24889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24889: Assignee: Apache Spark > dataset.unpersist() doesn't update storage memory stats > --

[jira] [Assigned] (SPARK-24889) dataset.unpersist() doesn't update storage memory stats

[ https://issues.apache.org/jira/browse/SPARK-24889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24889: Assignee: (was: Apache Spark) > dataset.unpersist() doesn't update storage memory sta

[jira] [Commented] (SPARK-24889) dataset.unpersist() doesn't update storage memory stats

[ https://issues.apache.org/jira/browse/SPARK-24889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604225#comment-16604225 ] Apache Spark commented on SPARK-24889: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604176#comment-16604176 ] Hyukjin Kwon commented on SPARK-18112: -- Can you post reproducer step by step? did y

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604167#comment-16604167 ] t oo commented on SPARK-18112: -- [~hyukjin.kwon] [~srowen] can this ticket be re-opened? Thi

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604162#comment-16604162 ] t oo commented on SPARK-13446: -- [~cloud_fan] I am hitting same issue as [~elgalu] :( > Spa

[jira] [Commented] (SPARK-17159) Improve FileInputDStream.findNewFiles list performance

[ https://issues.apache.org/jira/browse/SPARK-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604155#comment-16604155 ] Apache Spark commented on SPARK-17159: -- User 'ScrapCodes' has created a pull reques

[jira] [Commented] (SPARK-25337) HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasou

[ https://issues.apache.org/jira/browse/SPARK-25337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604154#comment-16604154 ] Apache Spark commented on SPARK-25337: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-25337) HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasour

[ https://issues.apache.org/jira/browse/SPARK-25337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25337: Assignee: Apache Spark > HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodErro

[jira] [Assigned] (SPARK-25337) HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileFormat.$init$(Lorg/apache/spark/sql/execution/datasour

[ https://issues.apache.org/jira/browse/SPARK-25337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25337: Assignee: (was: Apache Spark) > HiveExternalCatalogVersionsSuite + Scala 2.12 = NoSuc

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604146#comment-16604146 ] Apache Spark commented on SPARK-25317: -- User 'mgaido91' has created a pull request

[jira] [Assigned] (SPARK-25317) MemoryBlock performance regression

[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25317: Assignee: Apache Spark > MemoryBlock performance regression > ---

  1   2   >