[jira] [Created] (SPARK-25187) Revisit the life cycle of ReadSupport instances.

Wenchen Fan created SPARK-25187: --- Summary: Revisit the life cycle of ReadSupport instances. Key: SPARK-25187 URL: https://issues.apache.org/jira/browse/SPARK-25187 Project: Spark Issue Type:

[jira] [Created] (SPARK-25186) Stabilize Data Source V2 API

Wenchen Fan created SPARK-25186: --- Summary: Stabilize Data Source V2 API Key: SPARK-25186 URL: https://issues.apache.org/jira/browse/SPARK-25186 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25132) Case-insensitive field resolution when reading from Parquet/ORC

[ https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588414#comment-16588414 ] Apache Spark commented on SPARK-25132: -- User 'seancxmao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25155) Streaming from storage doesn't work when no directories exists

[ https://issues.apache.org/jira/browse/SPARK-25155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gil Vernik resolved SPARK-25155. Resolution: Cannot Reproduce > Streaming from storage doesn't work when no directories exists >

[jira] [Commented] (SPARK-25155) Streaming from storage doesn't work when no directories exists

[ https://issues.apache.org/jira/browse/SPARK-25155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588412#comment-16588412 ] Gil Vernik commented on SPARK-25155: [~ste...@apache.org] thanks for the input. This one

[jira] [Updated] (SPARK-25159) json schema inference should only trigger one job

[ https://issues.apache.org/jira/browse/SPARK-25159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25159: Fix Version/s: 2.4.0 > json schema inference should only trigger one job >

[jira] [Resolved] (SPARK-25159) json schema inference should only trigger one job

[ https://issues.apache.org/jira/browse/SPARK-25159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25159. - Resolution: Fixed Target Version/s: 2.4.0 > json schema inference should only trigger one job

[jira] [Resolved] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25140. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.4.0 > Add optional

[jira] [Created] (SPARK-25185) CBO rowcount statistics doesn't work for partitioned parquet external table

Amit created SPARK-25185: Summary: CBO rowcount statistics doesn't work for partitioned parquet external table Key: SPARK-25185 URL: https://issues.apache.org/jira/browse/SPARK-25185 Project: Spark

[jira] [Commented] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588389#comment-16588389 ] Jungtaek Lim commented on SPARK-24763: -- [~tdas] Got it. Thanks for the input. > Remove redundant

[jira] [Assigned] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25184: Assignee: (was: Apache Spark) > Flaky test: FlatMapGroupsWithState "streaming with

[jira] [Assigned] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25184: Assignee: Apache Spark > Flaky test: FlatMapGroupsWithState "streaming with processing

[jira] [Commented] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

[ https://issues.apache.org/jira/browse/SPARK-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588385#comment-16588385 ] Apache Spark commented on SPARK-25184: -- User 'tdas' has created a pull request for this issue:

[jira] [Created] (SPARK-25184) Flaky test: FlatMapGroupsWithState "streaming with processing time timeout"

Tathagata Das created SPARK-25184: - Summary: Flaky test: FlatMapGroupsWithState "streaming with processing time timeout" Key: SPARK-25184 URL: https://issues.apache.org/jira/browse/SPARK-25184

[jira] [Assigned] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

[ https://issues.apache.org/jira/browse/SPARK-25163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25163: Assignee: Apache Spark > Flaky test:

[jira] [Assigned] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

[ https://issues.apache.org/jira/browse/SPARK-25163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25163: Assignee: (was: Apache Spark) > Flaky test:

[jira] [Commented] (SPARK-25163) Flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuite.spilling with compression

[ https://issues.apache.org/jira/browse/SPARK-25163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588296#comment-16588296 ] Apache Spark commented on SPARK-25163: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25174: Assignee: Apache Spark > ApplicationMaster suspends when unregistering itself from RM

[jira] [Commented] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588283#comment-16588283 ] Apache Spark commented on SPARK-25174: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25174) ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

[ https://issues.apache.org/jira/browse/SPARK-25174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25174: Assignee: (was: Apache Spark) > ApplicationMaster suspends when unregistering itself

[jira] [Commented] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588234#comment-16588234 ] Tathagata Das commented on SPARK-24763: --- They will be. The merge script always puts the major

[jira] [Commented] (SPARK-23131) Stackoverflow using ML and Kryo serializer

[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588232#comment-16588232 ] Apache Spark commented on SPARK-23131: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25127) DataSourceV2: Remove SupportsPushDownCatalystFilters

[ https://issues.apache.org/jira/browse/SPARK-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reassigned SPARK-25127: --- Assignee: Reynold Xin > DataSourceV2: Remove SupportsPushDownCatalystFilters >

[jira] [Created] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

Steve Loughran created SPARK-25183: -- Summary: Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise Key: SPARK-25183 URL:

[jira] [Commented] (SPARK-25164) Parquet reader builds entire list of columns once for each column

[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588168#comment-16588168 ] Bruce Robbins commented on SPARK-25164: --- [~viirya] Sure. I will try to get something up by tonight

[jira] [Commented] (SPARK-25119) stages in wrong order within job page DAG chart

[ https://issues.apache.org/jira/browse/SPARK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588160#comment-16588160 ] Yunjian Zhang commented on SPARK-25119: --- create PR as below

[jira] [Comment Edited] (SPARK-25164) Parquet reader builds entire list of columns once for each column

[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588159#comment-16588159 ] Liang-Chi Hsieh edited comment on SPARK-25164 at 8/21/18 11:30 PM: ---

[jira] [Commented] (SPARK-25164) Parquet reader builds entire list of columns once for each column

[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588159#comment-16588159 ] Liang-Chi Hsieh commented on SPARK-25164: - This is easy and looks good to have. [~bersprockets]

[jira] [Assigned] (SPARK-25181) Block Manager master and slave thread pools are unbounded

[ https://issues.apache.org/jira/browse/SPARK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25181: Assignee: (was: Apache Spark) > Block Manager master and slave thread pools are

[jira] [Assigned] (SPARK-25181) Block Manager master and slave thread pools are unbounded

[ https://issues.apache.org/jira/browse/SPARK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25181: Assignee: Apache Spark > Block Manager master and slave thread pools are unbounded >

[jira] [Commented] (SPARK-25181) Block Manager master and slave thread pools are unbounded

[ https://issues.apache.org/jira/browse/SPARK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588154#comment-16588154 ] Apache Spark commented on SPARK-25181: -- User 'mukulmurthy' has created a pull request for this

[jira] [Resolved] (SPARK-25182) Block Manager master and slave thread pools are unbounded

[ https://issues.apache.org/jira/browse/SPARK-25182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Murthy resolved SPARK-25182. -- Resolution: Duplicate Target Version/s: (was: 2.4.0) > Block Manager master and

[jira] [Created] (SPARK-25182) Block Manager master and slave thread pools are unbounded

Mukul Murthy created SPARK-25182: Summary: Block Manager master and slave thread pools are unbounded Key: SPARK-25182 URL: https://issues.apache.org/jira/browse/SPARK-25182 Project: Spark

[jira] [Created] (SPARK-25181) Block Manager master and slave thread pools are unbounded

Mukul Murthy created SPARK-25181: Summary: Block Manager master and slave thread pools are unbounded Key: SPARK-25181 URL: https://issues.apache.org/jira/browse/SPARK-25181 Project: Spark

[jira] [Commented] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588121#comment-16588121 ] Xiao Li commented on SPARK-25114: - Let us update the fix version after the fix of 2.2 is merged >

[jira] [Resolved] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25114. - Resolution: Fixed Assignee: Jiang Xingbo Fix Version/s: 2.4.0 2.3.2

[jira] [Resolved] (SPARK-25095) Python support for BarrierTaskContext

[ https://issues.apache.org/jira/browse/SPARK-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25095. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22085

[jira] [Assigned] (SPARK-25095) Python support for BarrierTaskContext

[ https://issues.apache.org/jira/browse/SPARK-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25095: - Assignee: Jiang Xingbo > Python support for BarrierTaskContext >

[jira] [Commented] (SPARK-24564) Add test suite for RecordBinaryComparator

[ https://issues.apache.org/jira/browse/SPARK-24564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588114#comment-16588114 ] Apache Spark commented on SPARK-24564: -- User 'bersprockets' has created a pull request for this

[jira] [Commented] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588115#comment-16588115 ] Apache Spark commented on SPARK-25114: -- User 'bersprockets' has created a pull request for this

[jira] [Comment Edited] (SPARK-25168) PlanTest.comparePlans may make a supplied resolved plan unresolved.

[ https://issues.apache.org/jira/browse/SPARK-25168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587956#comment-16587956 ] Dilip Biswal edited comment on SPARK-25168 at 8/21/18 10:47 PM:

[jira] [Commented] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588101#comment-16588101 ] Jungtaek Lim commented on SPARK-24763: -- [~tdas] One question regarding fix version: I guess we

[jira] [Assigned] (SPARK-24441) Expose total estimated size of states in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24441: - Assignee: Jungtaek Lim > Expose total estimated size of states in

[jira] [Resolved] (SPARK-24441) Expose total estimated size of states in HDFSBackedStateStoreProvider

[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24441. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21469

[jira] [Assigned] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24763: - Assignee: Jungtaek Lim (was: Tathagata Das) > Remove redundant key data from value in

[jira] [Assigned] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24763: - Assignee: Tathagata Das > Remove redundant key data from value in streaming

[jira] [Resolved] (SPARK-24763) Remove redundant key data from value in streaming aggregation

[ https://issues.apache.org/jira/browse/SPARK-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24763. --- Resolution: Done Fix Version/s: 3.0.0 2.4.0 > Remove redundant

[jira] [Resolved] (SPARK-25149) Personalized PageRank raises an error if vertexIDs are > MaxInt

[ https://issues.apache.org/jira/browse/SPARK-25149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-25149. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22139

[jira] [Assigned] (SPARK-25149) Personalized PageRank raises an error if vertexIDs are > MaxInt

[ https://issues.apache.org/jira/browse/SPARK-25149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-25149: - Assignee: Bago Amirbekian > Personalized PageRank raises an error if vertexIDs

[jira] [Updated] (SPARK-25149) Personalized PageRank raises an error if vertexIDs are > MaxInt

[ https://issues.apache.org/jira/browse/SPARK-25149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25149: -- Summary: Personalized PageRank raises an error if vertexIDs are > MaxInt (was:

[jira] [Updated] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24307: Labels: release-notes (was: releasenotes) > Support sending messages over 2GB from memory >

[jira] [Updated] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24307: Labels: releasenotes (was: ) > Support sending messages over 2GB from memory >

[jira] [Commented] (SPARK-25050) Handle more than two types in avro union types when writing avro files

[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588068#comment-16588068 ] DB Tsai commented on SPARK-25050: - We handle more than two types in avro when reading into spark, but

[jira] [Updated] (SPARK-25050) Handle more than two types in avro union types when writing avro files

[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25050: Summary: Handle more than two types in avro union types when writing avro files (was: Handle more than

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588053#comment-16588053 ] Steve Loughran commented on SPARK-6305: --- 1. exclusion of log4j 1.x you can only safely exclude it

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588035#comment-16588035 ] Steve Loughran commented on SPARK-25180: FWIW, there was no in-progress data at the dest store,

[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host

[ https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588014#comment-16588014 ] Yinan Li commented on SPARK-25162: -- We actually moved away from using the IP address of the driver pod

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588012#comment-16588012 ] Steve Loughran commented on SPARK-25180: Netty converts the UnknownHostException into an IOE in

[jira] [Updated] (SPARK-25149) Personalized Page Rank raises an error if vertexIDs are > MaxInt

[ https://issues.apache.org/jira/browse/SPARK-25149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-25149: -- Shepherd: Joseph K. Bradley > Personalized Page Rank raises an error if vertexIDs are

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588005#comment-16588005 ] Steve Loughran commented on SPARK-25180: Stack {code} scala> text("hello all!") res10: String =

[jira] [Commented] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

[ https://issues.apache.org/jira/browse/SPARK-25180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588001#comment-16588001 ] Steve Loughran commented on SPARK-25180: code snippet was some trivial CSV => ORC with both src

[jira] [Created] (SPARK-25180) Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails

Steve Loughran created SPARK-25180: -- Summary: Spark standalone failure in Utils.doFetchFile() if nslookup of local hostname fails Key: SPARK-25180 URL: https://issues.apache.org/jira/browse/SPARK-25180

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587962#comment-16587962 ] Sean Owen commented on SPARK-6305: -- It was a mess. Excluding dependencies is an ongoing issue because

[jira] [Commented] (SPARK-25168) PlanTest.comparePlans may make a supplied resolved plan unresolved.

[ https://issues.apache.org/jira/browse/SPARK-25168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587956#comment-16587956 ] Dilip Biswal commented on SPARK-25168: -- [~cloud_fan] OK.. Let me close this then since we are

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587928#comment-16587928 ] Alexander commented on SPARK-7768: -- It's been a while since this had any activity. What is the

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587897#comment-16587897 ] Kazuaki Ishizaki commented on SPARK-25178: -- [~rednaxelafx] Thank you for opening a JIRA entry

[jira] [Assigned] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25179: --- Assignee: Bryan Cutler > Document the features that require Pyarrow 0.10 >

[jira] [Updated] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25179: Description: binary type support requires pyarrow 0.10.0. (was: binary type support requires pyarrow

[jira] [Commented] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587884#comment-16587884 ] Xiao Li commented on SPARK-25179: - Thanks! It is not urgent as long as it is documented before we

[jira] [Commented] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587882#comment-16587882 ] Bryan Cutler commented on SPARK-25179: -- I can work on this, probably can't get to it right away tho

[jira] [Updated] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-25179: - Description: binary type support requires pyarrow 0.10.0 > Document the features that require

[jira] [Commented] (SPARK-22779) ConfigEntry's default value should actually be a value

[ https://issues.apache.org/jira/browse/SPARK-22779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587869#comment-16587869 ] Apache Spark commented on SPARK-22779: -- User 'GregOwen' has created a pull request for this issue:

[jira] [Resolved] (SPARK-6236) Support caching blocks larger than 2G

[ https://issues.apache.org/jira/browse/SPARK-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6236. - Resolution: Fixed Fix Version/s: 2.4.0 As mentioned above, this was covered elsewhere,

[jira] [Commented] (SPARK-24961) sort operation causes out of memory

[ https://issues.apache.org/jira/browse/SPARK-24961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587850#comment-16587850 ] Markus Breuer commented on SPARK-24961: --- Some notes to your last comment: Reducing number of

[jira] [Commented] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587847#comment-16587847 ] Xiao Li commented on SPARK-25179: - cc [~bryanc] > Document the features that require Pyarrow 0.10 >

[jira] [Updated] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25179: Issue Type: Documentation (was: Improvement) > Document the features that require Pyarrow 0.10 >

[jira] [Created] (SPARK-25179) Document the features that require Pyarrow 0.10

Xiao Li created SPARK-25179: --- Summary: Document the features that require Pyarrow 0.10 Key: SPARK-25179 URL: https://issues.apache.org/jira/browse/SPARK-25179 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587841#comment-16587841 ] Xiao Li commented on SPARK-25178: - cc [~kiszk] Do you have a bandwidth to take this? > Use dummy name

[jira] [Resolved] (SPARK-24296) Support replicating blocks larger than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24296. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21451

[jira] [Assigned] (SPARK-24296) Support replicating blocks larger than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24296: -- Assignee: Imran Rashid > Support replicating blocks larger than 2 GB >

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587820#comment-16587820 ] Kris Mok commented on SPARK-25178: -- FYI: I've just come across this behavior and raised a ticket so

[jira] [Created] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

Kris Mok created SPARK-25178: Summary: Use dummy name for xxxHashMapGenerator key/value schema field Key: SPARK-25178 URL: https://issues.apache.org/jira/browse/SPARK-25178 Project: Spark Issue

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587814#comment-16587814 ] Chris Martin commented on SPARK-6305: - [~srowen] I've taken a look at this and I'd like to sync up

[jira] [Commented] (SPARK-24335) Dataset.map schema not applied in some cases

[ https://issues.apache.org/jira/browse/SPARK-24335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587785#comment-16587785 ] Apache Spark commented on SPARK-24335: -- User 'redsanket' has created a pull request for this issue:

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587770#comment-16587770 ] Bryan Cutler commented on SPARK-23874: -- Yes, I still would recommend users upgrade pyarrow if

[jira] [Resolved] (SPARK-25173) fail to pass kerberos authentification on executers

[ https://issues.apache.org/jira/browse/SPARK-25173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25173. Resolution: Duplicate > fail to pass kerberos authentification on executers >

[jira] [Resolved] (SPARK-25172) kererbos issue when use jdbc to connect hive server2 on executors

[ https://issues.apache.org/jira/browse/SPARK-25172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25172. Resolution: Invalid Please use the mailing list for questions:

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587702#comment-16587702 ] Xiao Li commented on SPARK-23874: - Great! Then, users can keep using pyarrow 0.8 and get the fixes we

[jira] [Assigned] (SPARK-25161) Fix several bugs in failure handling of barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-25161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25161: - Assignee: Jiang Xingbo > Fix several bugs in failure handling of barrier execution

[jira] [Resolved] (SPARK-25161) Fix several bugs in failure handling of barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-25161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25161. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22158

[jira] [Assigned] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25177: Assignee: Apache Spark > When dataframe decimal type column having scale higher than 6,

[jira] [Assigned] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25177: Assignee: (was: Apache Spark) > When dataframe decimal type column having scale

[jira] [Commented] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587578#comment-16587578 ] Apache Spark commented on SPARK-25177: -- User 'vinodkc' has created a pull request for this issue:

[jira] [Updated] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-25177: - Description: If scale of decimal type is > 6 , 0 value will be shown in scientific notation and hence,

[jira] [Updated] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-25177: - Description: If scale of decimal type is > 6 , 0 value will be shown in scientific notation and hence,

[jira] [Updated] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-25177: - Description: If scale of decimal type is > 6 , 0 value will be shown in scientific notation and hence,

[jira] [Updated] (SPARK-25177) When dataframe decimal type column having scale higher than 6, 0 values are shown in scientific notation

[ https://issues.apache.org/jira/browse/SPARK-25177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-25177: - Summary: When dataframe decimal type column having scale higher than 6, 0 values are shown in

[jira] [Created] (SPARK-25177) When dataframe decimal type column having a scale higher than 6, 0 values are shown in scientific notation

Vinod KC created SPARK-25177: Summary: When dataframe decimal type column having a scale higher than 6, 0 values are shown in scientific notation Key: SPARK-25177 URL:

[jira] [Updated] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pryakhin updated SPARK-25176: - Description: I'm using the latest spark version spark-core_2.11:2.3.1 which

[jira] [Updated] (SPARK-25176) Kryo fails to serialize a parametrised type hierarchy

[ https://issues.apache.org/jira/browse/SPARK-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Pryakhin updated SPARK-25176: - Description: I'm using the latest spark version spark-core_2.11:2.3.1 which

  1   2   >