[jira] [Updated] (SPARK-21513) SQL to_json should support all column types

2017-07-23 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-21513: --- Labels: Starter (was: ) > SQL to_json should support all column types >

[jira] [Updated] (SPARK-21513) SQL to_json should support all column types

2017-07-23 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-21513: --- Description: The built-in SQL UDF "to_json" currently supports serializing StructType

[jira] [Created] (SPARK-21513) SQL to_json should support all column types

2017-07-23 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-21513: -- Summary: SQL to_json should support all column types Key: SPARK-21513 URL: https://issues.apache.org/jira/browse/SPARK-21513 Project: Spark Issue Type:

[jira] [Created] (SPARK-18497) ForeachSink fails with "assertion failed: No plan for EventTimeWatermark"

2016-11-17 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-18497: -- Summary: ForeachSink fails with "assertion failed: No plan for EventTimeWatermark" Key: SPARK-18497 URL: https://issues.apache.org/jira/browse/SPARK-18497

[jira] [Created] (SPARK-9397) DataFrame should provide an API to find source data files if applicable

2015-07-27 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-9397: - Summary: DataFrame should provide an API to find source data files if applicable Key: SPARK-9397 URL: https://issues.apache.org/jira/browse/SPARK-9397 Project:

[jira] [Commented] (SPARK-8644) SparkException thrown due to Executor exceptions should include caller site in stack trace

2015-07-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630645#comment-14630645 ] Aaron Davidson commented on SPARK-8644: --- These are actually complementary.

[jira] [Updated] (SPARK-8707) RDD#toDebugString fails if any cached RDD has invalid partitions

2015-06-29 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-8707: -- Summary: RDD#toDebugString fails if any cached RDD has invalid partitions (was:

[jira] [Updated] (SPARK-8707) RDD#toDebugString fails if any cached RDD has invalid partitions

2015-06-29 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-8707: -- Description: Repro: {code} sc.textFile(/ThisFileDoesNotExist).cache() sc.parallelize(0 until

[jira] [Created] (SPARK-8707) RDD#toDebugString fails if any cached RDD is invalid

2015-06-29 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-8707: - Summary: RDD#toDebugString fails if any cached RDD is invalid Key: SPARK-8707 URL: https://issues.apache.org/jira/browse/SPARK-8707 Project: Spark Issue

[jira] [Created] (SPARK-8644) SparkException thrown due to Executor exceptions should include caller site in stack trace

2015-06-25 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-8644: - Summary: SparkException thrown due to Executor exceptions should include caller site in stack trace Key: SPARK-8644 URL: https://issues.apache.org/jira/browse/SPARK-8644

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-05-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543226#comment-14543226 ] Aaron Davidson commented on SPARK-7183: --- Sorry for delay, this should be fine to

[jira] [Resolved] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-05-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-7183. --- Resolution: Fixed Fix Version/s: 1.5.0 Memory leak in netty shuffle with spark

[jira] [Resolved] (SPARK-7003) Improve reliability of connection failure detection between Netty block transfer service endpoints

2015-04-20 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-7003. --- Resolution: Fixed Fix Version/s: 1.4.0 Improve reliability of connection failure

[jira] [Created] (SPARK-7003) Improve reliability of connection failure between Netty block transfer service endpoints

2015-04-19 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-7003: - Summary: Improve reliability of connection failure between Netty block transfer service endpoints Key: SPARK-7003 URL: https://issues.apache.org/jira/browse/SPARK-7003

[jira] [Updated] (SPARK-7003) Improve reliability of connection failure detection between Netty block transfer service endpoints

2015-04-19 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-7003: -- Summary: Improve reliability of connection failure detection between Netty block transfer

[jira] [Commented] (SPARK-6962) Netty BlockTransferService hangs in the middle of SQL query

2015-04-19 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502270#comment-14502270 ] Aaron Davidson commented on SPARK-6962: --- I created SPARK-7003 to track a fix to the

[jira] [Commented] (SPARK-6962) Netty BlockTransferService hangs in the middle of SQL query

2015-04-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499308#comment-14499308 ] Aaron Davidson commented on SPARK-6962: --- Executor logs in particular. Are all

[jira] [Commented] (SPARK-6962) Netty BlockTransferService hangs in the middle of SQL query

2015-04-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501054#comment-14501054 ] Aaron Davidson commented on SPARK-6962: --- Thanks for those log excerpts. It is likely

[jira] [Commented] (SPARK-6962) Netty BlockTransferService hangs in the middle of SQL query

2015-04-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501063#comment-14501063 ] Aaron Davidson commented on SPARK-6962: --- I have a hypothesis that the above is

[jira] [Updated] (SPARK-6962) Netty BlockTransferService hangs in the middle of SQL query

2015-04-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-6962: -- Summary: Netty BlockTransferService hangs in the middle of SQL query (was: Spark gets stuck on

[jira] [Resolved] (SPARK-6729) DriverQuirks get can get OutOfBounds exception is some cases

2015-04-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-6729. --- Resolution: Fixed Fix Version/s: 1.4.0 DriverQuirks get can get OutOfBounds exception

[jira] [Updated] (SPARK-6729) DriverQuirks get can get OutOfBounds exception is some cases

2015-04-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-6729: -- Assignee: Volodymyr Lyubinets DriverQuirks get can get OutOfBounds exception is some cases

[jira] [Commented] (SPARK-6646) Spark 2.0: Rearchitecting Spark for Mobile Platforms

2015-04-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390205#comment-14390205 ] Aaron Davidson commented on SPARK-6646: --- Please help, I tried putting spark on

[jira] [Commented] (SPARK-6373) Add SSL/TLS for the Netty based BlockTransferService

2015-03-24 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378084#comment-14378084 ] Aaron Davidson commented on SPARK-6373: --- From my brief look at your work, it seemed

[jira] [Resolved] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-6122. --- Resolution: Fixed Upgrade Tachyon dependency to 0.6.0 ---

[jira] [Commented] (SPARK-6373) Add SSL/TLS for the Netty based BlockTransferService

2015-03-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375186#comment-14375186 ] Aaron Davidson commented on SPARK-6373: --- Thanks for taking a look at this. The API

[jira] [Commented] (SPARK-6229) Support SASL encryption in network/common module

2015-03-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375197#comment-14375197 ] Aaron Davidson commented on SPARK-6229: --- Two points: 1. network/common as written

[jira] [Resolved] (SPARK-4012) Uncaught OOM in ContextCleaner

2015-03-19 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4012. --- Resolution: Fixed Fix Version/s: 1.4.0 Uncaught OOM in ContextCleaner

[jira] [Resolved] (SPARK-6330) newParquetRelation gets incorrect FileSystem

2015-03-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-6330. --- Resolution: Fixed Fix Version/s: 1.3.1 Assignee: Volodymyr Lyubinets

[jira] [Updated] (SPARK-6330) newParquetRelation gets incorrect FileSystem

2015-03-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-6330: -- Fix Version/s: 1.4.0 newParquetRelation gets incorrect FileSystem

[jira] [Resolved] (SPARK-6351) ParquetRelation2 does not support paths for different file systems

2015-03-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-6351. --- Resolution: Duplicate Fix Version/s: 1.3.1 Assignee: Volodymyr Lyubinets

[jira] [Commented] (SPARK-6229) Support encryption in network/common module

2015-03-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360808#comment-14360808 ] Aaron Davidson commented on SPARK-6229: --- The reason we did not originally put the

[jira] [Commented] (SPARK-2545) Add a diagnosis mode for closures to figure out what they're bringing in

2015-03-02 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344502#comment-14344502 ] Aaron Davidson commented on SPARK-2545: --- Probably especially useful there, yes.

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2015-02-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341962#comment-14341962 ] Aaron Davidson commented on SPARK-3889: --- This may be a new issue. I would open a new

[jira] [Commented] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container

2015-02-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341970#comment-14341970 ] Aaron Davidson commented on SPARK-6056: --- It's possible that it's actually the

[jira] [Issue Comment Deleted] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2015-02-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-3889: -- Comment: was deleted (was: The only place we memory map in 1.1 is this method:

[jira] [Commented] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container

2015-02-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341344#comment-14341344 ] Aaron Davidson commented on SPARK-6056: --- This should already be the case:

[jira] [Commented] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container

2015-02-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340834#comment-14340834 ] Aaron Davidson commented on SPARK-6056: --- cc [~rxin] Unlimit offHeap memory use

[jira] [Commented] (SPARK-6056) Unlimit offHeap memory use cause RM killing the container

2015-02-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340858#comment-14340858 ] Aaron Davidson commented on SPARK-6056: --- Actually, that line just calls

[jira] [Commented] (SPARK-5124) Standardize internal RPC interface

2015-02-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336974#comment-14336974 ] Aaron Davidson commented on SPARK-5124: --- I tend to prefer having explicit

[jira] [Commented] (SPARK-5124) Standardize internal RPC interface

2015-02-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337745#comment-14337745 ] Aaron Davidson commented on SPARK-5124: --- [~zsxwing] For receiveAndReply, I think the

[jira] [Commented] (SPARK-6026) Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code

2015-02-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337870#comment-14337870 ] Aaron Davidson commented on SPARK-6026: --- What is the main motivation for this

[jira] [Commented] (SPARK-6026) Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code

2015-02-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338002#comment-14338002 ] Aaron Davidson commented on SPARK-6026: --- Interesting -- what was the setup in which

[jira] [Commented] (SPARK-5124) Standardize internal RPC interface

2015-02-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337992#comment-14337992 ] Aaron Davidson commented on SPARK-5124: --- As Reynold mentioned me mentioning, I am

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2015-02-10 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314651#comment-14314651 ] Aaron Davidson commented on SPARK-3889: --- The only place we memory map in 1.1 is this

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2015-02-10 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314650#comment-14314650 ] Aaron Davidson commented on SPARK-3889: --- The only place we memory map in 1.1 is this

[jira] [Closed] (SPARK-2958) FileClientHandler should not be shared in the pipeline

2015-02-08 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson closed SPARK-2958. - Resolution: Done FileClientHandler should not be shared in the pipeline

[jira] [Commented] (SPARK-2958) FileClientHandler should not be shared in the pipeline

2015-02-08 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311796#comment-14311796 ] Aaron Davidson commented on SPARK-2958: --- This class no longer exists (I think we

[jira] [Resolved] (SPARK-1180) Allow to provide a custom persistence engine

2015-02-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1180. --- Resolution: Duplicate Fix Version/s: 1.3.0 Assignee: Prashant Sharma Allow

[jira] [Resolved] (SPARK-5073) spark.storage.memoryMapThreshold has two default values

2015-01-11 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-5073. --- Resolution: Fixed spark.storage.memoryMapThreshold has two default values

[jira] [Created] (SPARK-4864) Add documentation to Netty-based configs

2014-12-16 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4864: - Summary: Add documentation to Netty-based configs Key: SPARK-4864 URL: https://issues.apache.org/jira/browse/SPARK-4864 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4837) NettyBlockTransferService does not abide by spark.blockManager.port config optino

2014-12-12 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4837: - Summary: NettyBlockTransferService does not abide by spark.blockManager.port config optino Key: SPARK-4837 URL: https://issues.apache.org/jira/browse/SPARK-4837

[jira] [Updated] (SPARK-4837) NettyBlockTransferService does not abide by spark.blockManager.port config option

2014-12-12 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-4837: -- Summary: NettyBlockTransferService does not abide by spark.blockManager.port config option

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-10 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241463#comment-14241463 ] Aaron Davidson commented on SPARK-4740: --- Clarification: The merged version of

[jira] [Resolved] (SPARK-4805) BlockTransferMessage.toByteArray() trips assertion

2014-12-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4805. --- Resolution: Fixed Assignee: Sean Owen BlockTransferMessage.toByteArray() trips

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240579#comment-14240579 ] Aaron Davidson commented on SPARK-4740: --- I spent another day trying to reproduce the

[jira] [Comment Edited] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240579#comment-14240579 ] Aaron Davidson edited comment on SPARK-4740 at 12/10/14 3:22 AM:

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240704#comment-14240704 ] Aaron Davidson commented on SPARK-4740: --- We had 4 i2.8xlarge executors each with 32

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-08 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14239102#comment-14239102 ] Aaron Davidson commented on SPARK-4740: --- I tried to reproduce this on an EC2 cluster

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236898#comment-14236898 ] Aaron Davidson commented on SPARK-4740: --- Thanks for testing out the patch. Could we

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237018#comment-14237018 ] Aaron Davidson commented on SPARK-4740: --- Very interesting -- good executor is using

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237026#comment-14237026 ] Aaron Davidson commented on SPARK-4740: --- Could we get logs from the good/bad

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14237034#comment-14237034 ] Aaron Davidson commented on SPARK-4740: --- Do you have speculation enabled, by the

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-05 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235813#comment-14235813 ] Aaron Davidson commented on SPARK-4740: --- I think we have ourselves a winner. NIO is

[jira] [Commented] (SPARK-4740) Netty's network bandwidth is much lower than NIO in spark-perf and Netty takes longer running time

2014-12-04 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234758#comment-14234758 ] Aaron Davidson commented on SPARK-4740: --- Could you try to set

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-04 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235015#comment-14235015 ] Aaron Davidson commented on SPARK-4740: --- To clarify, we have two hypotheses

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

2014-12-04 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235241#comment-14235241 ] Aaron Davidson commented on SPARK-4740: --- I could believe that the result is no

[jira] [Commented] (SPARK-4189) FileSegmentManagedBuffer should have a configurable memory map threshold

2014-12-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231035#comment-14231035 ] Aaron Davidson commented on SPARK-4189: --- This was actually filed before

[jira] [Resolved] (SPARK-4189) FileSegmentManagedBuffer should have a configurable memory map threshold

2014-12-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4189. --- Resolution: Fixed Assignee: Reynold Xin FileSegmentManagedBuffer should have a

[jira] [Commented] (SPARK-4644) Implement skewed join

2014-11-29 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228839#comment-14228839 ] Aaron Davidson commented on SPARK-4644: --- [~zsxwing] I believe that this problem is

[jira] [Resolved] (SPARK-4170) Closure problems when running Scala app that extends App

2014-11-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4170. --- Resolution: Fixed Assignee: Sean Owen Closure problems when running Scala app that

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225681#comment-14225681 ] Aaron Davidson commented on SPARK-2468: --- Hey guys, I finally got a chance to run a

[jira] [Commented] (SPARK-4516) Netty off-heap memory use causes executors to be killed by OS

2014-11-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225700#comment-14225700 ] Aaron Davidson commented on SPARK-4516: --- It turns out there was a real bug which

[jira] [Commented] (SPARK-4516) Netty off-heap memory use causes executors to be killed by OS

2014-11-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225743#comment-14225743 ] Aaron Davidson commented on SPARK-4516: --- About my last point, [~rxin], [~pwendell],

[jira] [Commented] (SPARK-4516) Race condition in netty

2014-11-20 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220106#comment-14220106 ] Aaron Davidson commented on SPARK-4516: --- The code in question, despite the log

[jira] [Commented] (SPARK-4516) Race condition in netty

2014-11-20 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220114#comment-14220114 ] Aaron Davidson commented on SPARK-4516: --- Also ideally there would be some other

[jira] [Commented] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1

2014-11-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214166#comment-14214166 ] Aaron Davidson commented on SPARK-4434: --- Side note: the error message about file://,

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210074#comment-14210074 ] Aaron Davidson commented on SPARK-2468: --- Here is my spark configuration (note 32

[jira] [Comment Edited] (SPARK-2468) Netty-based block server / client module

2014-11-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210074#comment-14210074 ] Aaron Davidson edited comment on SPARK-2468 at 11/13/14 5:48 PM:

[jira] [Resolved] (SPARK-4326) unidoc is broken on master

2014-11-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4326. --- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Xiangrui Meng unidoc is

[jira] [Created] (SPARK-4370) Limit cores used by Netty transfer service based on executor size

2014-11-12 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4370: - Summary: Limit cores used by Netty transfer service based on executor size Key: SPARK-4370 URL: https://issues.apache.org/jira/browse/SPARK-4370 Project: Spark

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-12 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209407#comment-14209407 ] Aaron Davidson commented on SPARK-2468: --- My test was significantly less strict in

[jira] [Resolved] (SPARK-1830) Deploy failover, Make Persistence engine and LeaderAgent Pluggable.

2014-11-11 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1830. --- Resolution: Fixed Fix Version/s: (was: 1.2.0) (was: 1.0.1)

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-11 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207774#comment-14207774 ] Aaron Davidson commented on SPARK-2468: --- [~zzcclp] Ah, man, that's not good. Just to

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201766#comment-14201766 ] Aaron Davidson commented on SPARK-2468: --- [~zzcclp] Yes, please do. What's the memory

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201774#comment-14201774 ] Aaron Davidson commented on SPARK-2468: --- Yup, that would work. Netty-based block

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201783#comment-14201783 ] Aaron Davidson commented on SPARK-2468: --- Thanks a lot for those diagnostics. Can you

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201812#comment-14201812 ] Aaron Davidson commented on SPARK-2468: --- Looking at the netty code a bit more, it

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201832#comment-14201832 ] Aaron Davidson commented on SPARK-2468: --- [~lianhuiwang] I have created

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201849#comment-14201849 ] Aaron Davidson commented on SPARK-2468: --- [~zzcclp] Thank you for the writeup. Is it

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-07 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202490#comment-14202490 ] Aaron Davidson commented on SPARK-2468: --- [~lianhuiwang] Can you try again with

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200519#comment-14200519 ] Aaron Davidson commented on SPARK-2468: --- [~zzcclp] Use of epoll mode is highly

[jira] [Closed] (SPARK-4238) Perform network-level retry of shuffle file fetches

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson closed SPARK-4238. - Resolution: Duplicate Perform network-level retry of shuffle file fetches

[jira] [Updated] (SPARK-4188) Shuffle fetches should be retried at a lower level

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-4188: -- Description: During periods of high network (or GC) load, it is not uncommon that IOExceptions

[jira] [Commented] (SPARK-4238) Perform network-level retry of shuffle file fetches

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200526#comment-14200526 ] Aaron Davidson commented on SPARK-4238: --- Oh, whoops. Perform network-level retry

[jira] [Created] (SPARK-4277) Support external shuffle service on Worker

2014-11-06 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4277: - Summary: Support external shuffle service on Worker Key: SPARK-4277 URL: https://issues.apache.org/jira/browse/SPARK-4277 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-4264) SQL HashJoin induces refCnt = 0 error in ShuffleBlockFetcherIterator

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4264. --- Resolution: Fixed SQL HashJoin induces refCnt = 0 error in ShuffleBlockFetcherIterator

[jira] [Commented] (SPARK-4287) Add TTL-based cleanup in external shuffle service

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201428#comment-14201428 ] Aaron Davidson commented on SPARK-4287: --- I think the right solution here might be to

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201432#comment-14201432 ] Aaron Davidson commented on SPARK-2468: --- [~zzcclp] I believe it is close to merging.

[jira] [Resolved] (SPARK-4236) External shuffle service must cleanup its shuffle files

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4236. --- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Aaron Davidson External

  1   2   3   >