[jira] [Commented] (SPARK-15487) Spark Master UI to reverse proxy Application and Workers UI

[ https://issues.apache.org/jira/browse/SPARK-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601794#comment-15601794 ] Matthew Farrellee commented on SPARK-15487: --- well, unless you're putting anothe

[jira] [Commented] (SPARK-15487) Spark Master UI to reverse proxy Application and Workers UI

[ https://issues.apache.org/jira/browse/SPARK-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15600710#comment-15600710 ] Matthew Farrellee commented on SPARK-15487: --- try just setting the proxy url to

[jira] [Commented] (SPARK-5368) Spark should support NAT (via akka improvements)

[ https://issues.apache.org/jira/browse/SPARK-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377096#comment-14377096 ] Matthew Farrellee commented on SPARK-5368: -- [~jayunit100] the relevant config is

[jira] [Comment Edited] (SPARK-5368) Spark should support NAT (via akka improvements)

[ https://issues.apache.org/jira/browse/SPARK-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377096#comment-14377096 ] Matthew Farrellee edited comment on SPARK-5368 at 3/24/15 1:58 AM: -

[jira] [Commented] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

[ https://issues.apache.org/jira/browse/SPARK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375158#comment-14375158 ] Matthew Farrellee commented on SPARK-5113: -- [~pwendell] would SPARK_INTERNAL_HOST

[jira] [Commented] (SPARK-5368) Spark should support NAT (via akka improvements)

[ https://issues.apache.org/jira/browse/SPARK-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375157#comment-14375157 ] Matthew Farrellee commented on SPARK-5368: -- [~srowen] i was able to workaround my

[jira] [Commented] (SPARK-6245) jsonRDD() of empty RDD results in exception

[ https://issues.apache.org/jira/browse/SPARK-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363290#comment-14363290 ] Matthew Farrellee commented on SPARK-6245: -- [~srowen] thanks for fixing this. it'

[jira] [Commented] (SPARK-6245) jsonRDD() of empty RDD results in exception

[ https://issues.apache.org/jira/browse/SPARK-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355843#comment-14355843 ] Matthew Farrellee commented on SPARK-6245: -- this is an issue for the scala interf

[jira] [Created] (SPARK-6245) jsonRDD() of empty RDD results in exception

Matthew Farrellee created SPARK-6245: Summary: jsonRDD() of empty RDD results in exception Key: SPARK-6245 URL: https://issues.apache.org/jira/browse/SPARK-6245 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5368) Spark should support NAT (via akka improvements)

[ https://issues.apache.org/jira/browse/SPARK-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353534#comment-14353534 ] Matthew Farrellee commented on SPARK-5368: -- [~srowen] will you take a look at thi

[jira] [Commented] (SPARK-2313) PySpark should accept port via a command line argument rather than STDIN

[ https://issues.apache.org/jira/browse/SPARK-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318221#comment-14318221 ] Matthew Farrellee commented on SPARK-2313: -- that'd work, also requires a py4j cha

[jira] [Resolved] (SPARK-927) PySpark sample() doesn't work if numpy is installed on master but not on workers

[ https://issues.apache.org/jira/browse/SPARK-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee resolved SPARK-927. - Resolution: Fixed Fix Version/s: 1.2.0 > PySpark sample() doesn't work if numpy is i

[jira] [Commented] (SPARK-927) PySpark sample() doesn't work if numpy is installed on master but not on workers

[ https://issues.apache.org/jira/browse/SPARK-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265290#comment-14265290 ] Matthew Farrellee commented on SPARK-927: - PR #2313 was subsumed by PR #3351, which

[jira] [Closed] (SPARK-2256) pyspark: .take doesn't work ... sometimes ...

[ https://issues.apache.org/jira/browse/SPARK-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-2256. Resolution: Fixed Fix Version/s: 1.1.0 > pyspark: .take doesn't work ... sometimes ..

[jira] [Commented] (SPARK-3733) Support for programmatically submitting Spark jobs

[ https://issues.apache.org/jira/browse/SPARK-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153142#comment-14153142 ] Matthew Farrellee commented on SPARK-3733: -- will you describe what you mean by "s

[jira] [Commented] (SPARK-3685) Spark's local dir should accept only local paths

[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152526#comment-14152526 ] Matthew Farrellee commented on SPARK-3685: -- if you're going to go down this path

[jira] [Commented] (SPARK-3685) Spark's local dir should accept only local paths

[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152152#comment-14152152 ] Matthew Farrellee commented on SPARK-3685: -- the root of the resource problem is h

[jira] [Commented] (SPARK-3685) Spark's local dir should accept only local paths

[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151691#comment-14151691 ] Matthew Farrellee commented on SPARK-3685: -- [~andrewor] thanks for the info. afai

[jira] [Commented] (SPARK-3685) Spark's local dir scheme is not configurable

[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151148#comment-14151148 ] Matthew Farrellee commented on SPARK-3685: -- i'm skeptical. what would be the bene

[jira] [Commented] (SPARK-3639) Kinesis examples set master as local

[ https://issues.apache.org/jira/browse/SPARK-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146245#comment-14146245 ] Matthew Farrellee commented on SPARK-3639: -- seems reasonable to me > Kinesis exa

[jira] [Updated] (SPARK-542) Cache Miss when machine have multiple hostname

[ https://issues.apache.org/jira/browse/SPARK-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-542: Priority: Minor (was: Blocker) > Cache Miss when machine have multiple hostname > --

[jira] [Commented] (SPARK-578) Fix interpreter code generation to only capture needed dependencies

[ https://issues.apache.org/jira/browse/SPARK-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142558#comment-14142558 ] Matthew Farrellee commented on SPARK-578: - [~matei] is this related to slimming dow

[jira] [Closed] (SPARK-575) Maintain a cache of JARs on each node to avoid unnecessary copying

[ https://issues.apache.org/jira/browse/SPARK-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-575. --- Resolution: Incomplete > Maintain a cache of JARs on each node to avoid unnecessary copying > -

[jira] [Commented] (SPARK-575) Maintain a cache of JARs on each node to avoid unnecessary copying

[ https://issues.apache.org/jira/browse/SPARK-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142553#comment-14142553 ] Matthew Farrellee commented on SPARK-575: - [~joshrosen] is quite correct. this iss

[jira] [Commented] (SPARK-584) Pass slave ip address when starting a cluster

[ https://issues.apache.org/jira/browse/SPARK-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142545#comment-14142545 ] Matthew Farrellee commented on SPARK-584: - what's the use case for this? > Pass sl

[jira] [Updated] (SPARK-604) reconnect if mesos slaves dies

[ https://issues.apache.org/jira/browse/SPARK-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-604: Component/s: Mesos > reconnect if mesos slaves dies > -- > >

[jira] [Commented] (SPARK-610) Support master failover in standalone mode

[ https://issues.apache.org/jira/browse/SPARK-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142528#comment-14142528 ] Matthew Farrellee commented on SPARK-610: - [~matei] given YARN and Mesos implementa

[jira] [Closed] (SPARK-690) Stack overflow when running pagerank more than 10000 iterators

[ https://issues.apache.org/jira/browse/SPARK-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-690. --- Resolution: Unresolved > Stack overflow when running pagerank more than 1 iterators > -

[jira] [Commented] (SPARK-690) Stack overflow when running pagerank more than 10000 iterators

[ https://issues.apache.org/jira/browse/SPARK-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142511#comment-14142511 ] Matthew Farrellee commented on SPARK-690: - [~andrew xia] this is reported against a

[jira] [Closed] (SPARK-709) Dropping a block reports 0 bytes

[ https://issues.apache.org/jira/browse/SPARK-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-709. --- Resolution: Incomplete [~rxin] there isn't enough information to make progress on this, but fee

[jira] [Commented] (SPARK-718) NPE when performing action during transformation

[ https://issues.apache.org/jira/browse/SPARK-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142506#comment-14142506 ] Matthew Farrellee commented on SPARK-718: - Spark simply does not support nesting RD

[jira] [Closed] (SPARK-718) NPE when performing action during transformation

[ https://issues.apache.org/jira/browse/SPARK-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-718. --- Resolution: Done > NPE when performing action during transformation > -

[jira] [Closed] (SPARK-567) Unified directory structure for temporary data

[ https://issues.apache.org/jira/browse/SPARK-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-567. --- Resolution: Incomplete please re-open with additional details for how this could be implemented

[jira] [Commented] (SPARK-559) Automatically register all classes used in fields of a class with Kryo

[ https://issues.apache.org/jira/browse/SPARK-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142479#comment-14142479 ] Matthew Farrellee commented on SPARK-559: - the last comment on this, from 2 years a

[jira] [Closed] (SPARK-559) Automatically register all classes used in fields of a class with Kryo

[ https://issues.apache.org/jira/browse/SPARK-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-559. --- Resolution: Done > Automatically register all classes used in fields of a class with Kryo > ---

[jira] [Commented] (SPARK-550) Hiding the default spark context in the spark shell creates serialization issues

[ https://issues.apache.org/jira/browse/SPARK-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142477#comment-14142477 ] Matthew Farrellee commented on SPARK-550: - a lot of code has changed in this space

[jira] [Closed] (SPARK-550) Hiding the default spark context in the spark shell creates serialization issues

[ https://issues.apache.org/jira/browse/SPARK-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-550. --- Resolution: Done > Hiding the default spark context in the spark shell creates serialization >

[jira] [Updated] (SPARK-542) Cache Miss when machine have multiple hostname

[ https://issues.apache.org/jira/browse/SPARK-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-542: Component/s: Mesos Priority: Blocker > Cache Miss when machine have multiple hostname

[jira] [Closed] (SPARK-538) INFO spark.MesosScheduler: Ignoring update from TID 9 because its job is gone

[ https://issues.apache.org/jira/browse/SPARK-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-538. --- Resolution: Done > INFO spark.MesosScheduler: Ignoring update from TID 9 because its job is gon

[jira] [Commented] (SPARK-538) INFO spark.MesosScheduler: Ignoring update from TID 9 because its job is gone

[ https://issues.apache.org/jira/browse/SPARK-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142475#comment-14142475 ] Matthew Farrellee commented on SPARK-538: - this is a reasonable question for the us

[jira] [Resolved] (SPARK-537) driver.run() returned with code DRIVER_ABORTED

[ https://issues.apache.org/jira/browse/SPARK-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee resolved SPARK-537. - Resolution: Fixed Fix Version/s: 1.0.0 > driver.run() returned with code DRIVER_ABOR

[jira] [Commented] (SPARK-537) driver.run() returned with code DRIVER_ABORTED

[ https://issues.apache.org/jira/browse/SPARK-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142474#comment-14142474 ] Matthew Farrellee commented on SPARK-537: - this should be resolved by a number of f

[jira] [Commented] (SPARK-3593) Support Sorting of Binary Type Data

[ https://issues.apache.org/jira/browse/SPARK-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142468#comment-14142468 ] Matthew Farrellee commented on SPARK-3593: -- [~pmagid] will you provide some examp

[jira] [Commented] (SPARK-637) Create troubleshooting checklist

[ https://issues.apache.org/jira/browse/SPARK-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142463#comment-14142463 ] Matthew Farrellee commented on SPARK-637: - this is a good idea, and it will take a

[jira] [Closed] (SPARK-637) Create troubleshooting checklist

[ https://issues.apache.org/jira/browse/SPARK-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-637. --- Resolution: Later > Create troubleshooting checklist > > >

[jira] [Commented] (SPARK-719) Add FAQ page to documentation or webpage

[ https://issues.apache.org/jira/browse/SPARK-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142459#comment-14142459 ] Matthew Farrellee commented on SPARK-719: - it looks like this has some good content

[jira] [Closed] (SPARK-719) Add FAQ page to documentation or webpage

[ https://issues.apache.org/jira/browse/SPARK-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-719. --- Resolution: Done Fix Version/s: (was: 0.7.1) > Add FAQ page to documentation or webp

[jira] [Commented] (SPARK-614) Make last 4 digits of framework id in standalone mode logging monotonically increasing

[ https://issues.apache.org/jira/browse/SPARK-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142456#comment-14142456 ] Matthew Farrellee commented on SPARK-614: - it looks like nothing has happened with

[jira] [Closed] (SPARK-614) Make last 4 digits of framework id in standalone mode logging monotonically increasing

[ https://issues.apache.org/jira/browse/SPARK-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-614. --- Resolution: Unresolved Fix Version/s: (was: 0.7.1) > Make last 4 digits of framework

[jira] [Commented] (SPARK-1748) I installed the spark_standalone,but I did not know how to use sbt to compile the programme of spark?

[ https://issues.apache.org/jira/browse/SPARK-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142455#comment-14142455 ] Matthew Farrellee commented on SPARK-1748: -- thanks for the question. you'll get a

[jira] [Closed] (SPARK-1748) I installed the spark_standalone,but I did not know how to use sbt to compile the programme of spark?

[ https://issues.apache.org/jira/browse/SPARK-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-1748. Resolution: Done Fix Version/s: (was: 0.8.1) > I installed the spark_standalone,b

[jira] [Resolved] (SPARK-1176) Adding port configuration for HttpBroadcast

[ https://issues.apache.org/jira/browse/SPARK-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee resolved SPARK-1176. -- Resolution: Fixed Fix Version/s: (was: 0.9.0) 1.1.0 > Addi

[jira] [Commented] (SPARK-1176) Adding port configuration for HttpBroadcast

[ https://issues.apache.org/jira/browse/SPARK-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142454#comment-14142454 ] Matthew Farrellee commented on SPARK-1176: -- [~epakhomov] it looks like this was r

[jira] [Closed] (SPARK-1177) Allow SPARK_JAR to be set in system properties

[ https://issues.apache.org/jira/browse/SPARK-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-1177. Resolution: Fixed Fix Version/s: (was: 0.9.0) > Allow SPARK_JAR to be set in syst

[jira] [Commented] (SPARK-1177) Allow SPARK_JAR to be set in system properties

[ https://issues.apache.org/jira/browse/SPARK-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142447#comment-14142447 ] Matthew Farrellee commented on SPARK-1177: -- [~epakhomov] it looks like this has b

[jira] [Commented] (SPARK-1443) Unable to Access MongoDB GridFS data with Spark using mongo-hadoop API

[ https://issues.apache.org/jira/browse/SPARK-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142446#comment-14142446 ] Matthew Farrellee commented on SPARK-1443: -- [~PavanKumarVarma] i hope you've been

[jira] [Resolved] (SPARK-1443) Unable to Access MongoDB GridFS data with Spark using mongo-hadoop API

[ https://issues.apache.org/jira/browse/SPARK-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee resolved SPARK-1443. -- Resolution: Done Fix Version/s: (was: 0.9.0) > Unable to Access MongoDB GridF

[jira] [Closed] (SPARK-3321) Defining a class within python main script

[ https://issues.apache.org/jira/browse/SPARK-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-3321. Resolution: Not a Problem > Defining a class within python main script > ---

[jira] [Closed] (SPARK-3581) RDD API(distinct/subtract) does not work for RDD of Dictionaries

[ https://issues.apache.org/jira/browse/SPARK-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-3581. Resolution: Not a Problem > RDD API(distinct/subtract) does not work for RDD of Dictionaries

[jira] [Commented] (SPARK-3562) Periodic cleanup event logs

[ https://issues.apache.org/jira/browse/SPARK-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139786#comment-14139786 ] Matthew Farrellee commented on SPARK-3562: -- is logrotate an option for you? > Pe

[jira] [Commented] (SPARK-3580) Add Consistent Method To Get Number of RDD Partitions Across Different Languages

[ https://issues.apache.org/jira/browse/SPARK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139043#comment-14139043 ] Matthew Farrellee commented on SPARK-3580: -- what do you think about going the oth

[jira] [Commented] (SPARK-3581) RDD API(distinct/subtract) does not work for RDD of Dictionaries

[ https://issues.apache.org/jira/browse/SPARK-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139000#comment-14139000 ] Matthew Farrellee commented on SPARK-3581: -- a dict is mutable and therefore doesn

[jira] [Commented] (SPARK-3321) Defining a class within python main script

[ https://issues.apache.org/jira/browse/SPARK-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138867#comment-14138867 ] Matthew Farrellee commented on SPARK-3321: -- [~guoxu1231] i think so too. ok if i

[jira] [Closed] (SPARK-2022) Spark 1.0.0 is failing if mesos.coarse set to true

[ https://issues.apache.org/jira/browse/SPARK-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-2022. Resolution: Fixed > Spark 1.0.0 is failing if mesos.coarse set to true > ---

[jira] [Closed] (SPARK-1807) Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos.

[ https://issues.apache.org/jira/browse/SPARK-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee closed SPARK-1807. Resolution: Won't Fix > Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos. >

[jira] [Commented] (SPARK-3508) annotate the Spark configs to indicate which ones are meant for the end user

[ https://issues.apache.org/jira/browse/SPARK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135631#comment-14135631 ] Matthew Farrellee commented on SPARK-3508: -- documented == public is a good metric

[jira] [Created] (SPARK-3538) Provide way for workers to log messages to driver's out/err

Matthew Farrellee created SPARK-3538: Summary: Provide way for workers to log messages to driver's out/err Key: SPARK-3538 URL: https://issues.apache.org/jira/browse/SPARK-3538 Project: Spark

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134456#comment-14134456 ] Matthew Farrellee commented on SPARK-3215: -- i'd like to echo [~shivaram]'s point

[jira] [Commented] (SPARK-2377) Create a Python API for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134225#comment-14134225 ] Matthew Farrellee commented on SPARK-2377: -- it's a little tricky. you need to clo

[jira] [Commented] (SPARK-3470) Have JavaSparkContext implement Closeable/AutoCloseable

[ https://issues.apache.org/jira/browse/SPARK-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128778#comment-14128778 ] Matthew Farrellee commented on SPARK-3470: -- i stand corrected. http://spark.apac

[jira] [Commented] (SPARK-3470) Have JavaSparkContext implement Closeable/AutoCloseable

[ https://issues.apache.org/jira/browse/SPARK-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128751#comment-14128751 ] Matthew Farrellee commented on SPARK-3470: -- while you can implement Closeable in

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127187#comment-14127187 ] Matthew Farrellee commented on SPARK-2972: -- +1 close this and open 2 feature requ

[jira] [Updated] (SPARK-3458) enable use of python's "with" statements for SparkContext management

[ https://issues.apache.org/jira/browse/SPARK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-3458: - Description: best practice for managing SparkContexts involves exception handling, e.g. {

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127137#comment-14127137 ] Matthew Farrellee commented on SPARK-2972: -- fyi - i've created https://issues.apa

[jira] [Created] (SPARK-3458) enable use of python's "with" statements for SparkContext management

Matthew Farrellee created SPARK-3458: Summary: enable use of python's "with" statements for SparkContext management Key: SPARK-3458 URL: https://issues.apache.org/jira/browse/SPARK-3458 Project: S

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127016#comment-14127016 ] Matthew Farrellee commented on SPARK-2972: -- > I suggest having context implement

[jira] [Commented] (SPARK-1087) Separate file for traceback and callsite related functions

[ https://issues.apache.org/jira/browse/SPARK-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125957#comment-14125957 ] Matthew Farrellee commented on SPARK-1087: -- [~jyotiska] please do! > Separate fi

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125938#comment-14125938 ] Matthew Farrellee commented on SPARK-2972: -- > Thanks for answering. I guess it's

[jira] [Commented] (SPARK-927) PySpark sample() doesn't work if numpy is installed on master but not on workers

[ https://issues.apache.org/jira/browse/SPARK-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124924#comment-14124924 ] Matthew Farrellee commented on SPARK-927: - it looks like the issue is rddsampler ch

[jira] [Commented] (SPARK-675) Gateway JVM should ask for less than SPARK_MEM memory

[ https://issues.apache.org/jira/browse/SPARK-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124918#comment-14124918 ] Matthew Farrellee commented on SPARK-675: - [~joshrosen] it looks like SPARK-674 was

[jira] [Commented] (SPARK-2023) PySpark reduce does a map side reduce and then sends the results to the driver for final reduce, instead do this more like Scala Spark.

[ https://issues.apache.org/jira/browse/SPARK-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124879#comment-14124879 ] Matthew Farrellee commented on SPARK-2023: -- [~holdenk] are you still concerned ab

[jira] [Commented] (SPARK-1087) Separate file for traceback and callsite related functions

[ https://issues.apache.org/jira/browse/SPARK-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124877#comment-14124877 ] Matthew Farrellee commented on SPARK-1087: -- [~jyotiska] PR 581 was merged (though

[jira] [Updated] (SPARK-2256) pyspark: .take doesn't work ... sometimes ...

[ https://issues.apache.org/jira/browse/SPARK-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-2256: - Labels: RDD pyspark take windows (was: RDD pyspark take) > pyspark: .take doesn't work ..

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124872#comment-14124872 ] Matthew Farrellee commented on SPARK-2972: -- [~roji] this was addressed for a pysp

[jira] [Resolved] (SPARK-3401) Wrong usage of tee command in python/run-tests

[ https://issues.apache.org/jira/browse/SPARK-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee resolved SPARK-3401. -- Resolution: Fixed > Wrong usage of tee command in python/run-tests > ---

[jira] [Updated] (SPARK-3401) Wrong usage of tee command in python/run-tests

[ https://issues.apache.org/jira/browse/SPARK-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-3401: - Fix Version/s: 1.1.1 > Wrong usage of tee command in python/run-tests > --

[jira] [Commented] (SPARK-3401) Wrong usage of tee command in python/run-tests

[ https://issues.apache.org/jira/browse/SPARK-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124705#comment-14124705 ] Matthew Farrellee commented on SPARK-3401: -- nice catch > Wrong usage of tee comm

[jira] [Commented] (SPARK-3321) Defining a class within python main script

[ https://issues.apache.org/jira/browse/SPARK-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124704#comment-14124704 ] Matthew Farrellee commented on SPARK-3321: -- this has come up a few times. it's no

[jira] [Issue Comment Deleted] (SPARK-1701) Inconsistent naming: "slice" or "partition"

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-1701: - Comment: was deleted (was: ok, and one more https://github.com/apache/spark/pull/2304 to

[jira] [Issue Comment Deleted] (SPARK-1701) Inconsistent naming: "slice" or "partition"

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Farrellee updated SPARK-1701: - Comment: was deleted (was: ok, i also created 2 other PRs https://github.com/apache/spark

[jira] [Commented] (SPARK-1701) Inconsistent naming: "slice" or "partition"

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124510#comment-14124510 ] Matthew Farrellee commented on SPARK-1701: -- ok, and one more https://github.com/

[jira] [Commented] (SPARK-1701) Inconsistent naming: "slice" or "partition"

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124506#comment-14124506 ] Matthew Farrellee commented on SPARK-1701: -- ok, i also created 2 other PRs https

[jira] [Commented] (SPARK-3425) OpenJDK - when run with jvm 1.8, should not set MaxPermSize

[ https://issues.apache.org/jira/browse/SPARK-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124467#comment-14124467 ] Matthew Farrellee commented on SPARK-3425: -- this is still an issue for openjdk

[jira] [Created] (SPARK-3425) OpenJDK - when run with jvm 1.8, should not set MaxPermSize

Matthew Farrellee created SPARK-3425: Summary: OpenJDK - when run with jvm 1.8, should not set MaxPermSize Key: SPARK-3425 URL: https://issues.apache.org/jira/browse/SPARK-3425 Project: Spark

[jira] [Commented] (SPARK-1701) Inconsistent naming: "slice" or "partition"

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124458#comment-14124458 ] Matthew Farrellee commented on SPARK-1701: -- slice vs partition has also come up o

[jira] [Commented] (SPARK-3181) Add Robust Regression Algorithm with Huber Estimator

[ https://issues.apache.org/jira/browse/SPARK-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118566#comment-14118566 ] Matthew Farrellee commented on SPARK-3181: -- pls excuse my changes to this issue,

[jira] [Commented] (SPARK-2435) Add shutdown hook to bin/pyspark

[ https://issues.apache.org/jira/browse/SPARK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113816#comment-14113816 ] Matthew Farrellee commented on SPARK-2435: -- i couldn't find a PR for this, and it

[jira] [Commented] (SPARK-2855) pyspark test cases crashed for no reason

[ https://issues.apache.org/jira/browse/SPARK-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113769#comment-14113769 ] Matthew Farrellee commented on SPARK-2855: -- [~zhunansjtu] the link you supplied n

[jira] [Commented] (SPARK-2377) Create a Python API for Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111304#comment-14111304 ] Matthew Farrellee commented on SPARK-2377: -- thanks, i'll take a look > Create a

[jira] [Commented] (SPARK-1284) pyspark hangs after IOError on Executor

[ https://issues.apache.org/jira/browse/SPARK-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1412#comment-1412 ] Matthew Farrellee commented on SPARK-1284: -- [~jblomo] master should be buildable

  1   2   >