[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218610#comment-15218610 ] Juliet Hougland commented on SPARK-13587: - Being able to ship around pex files like we do .py and

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212191#comment-15212191 ] Juliet Hougland commented on SPARK-13587: - I really do think spark and pyspark needs to stay out

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179253#comment-15179253 ] Juliet Hougland commented on SPARK-13587: - That is wonderful. Let me know if you'd like me to

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179250#comment-15179250 ] Juliet Hougland commented on SPARK-13587: - I made a comment related to this below. TLDR I think

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179247#comment-15179247 ] Juliet Hougland commented on SPARK-13587: - Currently the way users specify the workers' python

[jira] [Comment Edited] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178536#comment-15178536 ] Juliet Hougland edited comment on SPARK-13587 at 3/3/16 9:21 PM: - If

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178536#comment-15178536 ] Juliet Hougland commented on SPARK-13587: - If pyspark allows users to create virtual

[jira] [Created] (SPARK-13303) Spark fails with pandas import error when pandas is not explicitly imported by user

Juliet Hougland created SPARK-13303: --- Summary: Spark fails with pandas import error when pandas is not explicitly imported by user Key: SPARK-13303 URL: https://issues.apache.org/jira/browse/SPARK-13303

[jira] [Commented] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

[ https://issues.apache.org/jira/browse/SPARK-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111607#comment-15111607 ] Juliet Hougland commented on SPARK-4073: For those playing along at home-- the solution for me was

[jira] [Commented] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

[ https://issues.apache.org/jira/browse/SPARK-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1548#comment-1548 ] Juliet Hougland commented on SPARK-4073: I have run in to a related problem. I am reading a snappy

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628947#comment-14628947 ] Juliet Hougland commented on SPARK-8646: The failure happens at the point that I

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628971#comment-14628971 ] Juliet Hougland commented on SPARK-8646: Yea, it works fine if I add that arg.

[jira] [Comment Edited] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628947#comment-14628947 ] Juliet Hougland edited comment on SPARK-8646 at 7/15/15 11:46 PM:

[jira] [Comment Edited] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628971#comment-14628971 ] Juliet Hougland edited comment on SPARK-8646 at 7/16/15 12:03 AM:

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625245#comment-14625245 ] Juliet Hougland commented on SPARK-8646: [~lianhuiwang] in $SPARK_HOME/conf I only

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8646: --- Attachment: executor.log PySpark does not run on YARN

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8646: --- Attachment: spark1.4-verbose.log verbose-executor.log PySpark does not run

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623008#comment-14623008 ] Juliet Hougland commented on SPARK-8646: [~lianhuiwang] I just uploaded the log

[jira] [Comment Edited] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623008#comment-14623008 ] Juliet Hougland edited comment on SPARK-8646 at 7/10/15 10:40 PM:

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614856#comment-14614856 ] Juliet Hougland commented on SPARK-8646: [~sowen] The pandas error came when I

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615809#comment-14615809 ] Juliet Hougland commented on SPARK-8646: [~davies] Please look at the logs I have

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604014#comment-14604014 ] Juliet Hougland commented on SPARK-8646: When I configure spark to use my

[jira] [Comment Edited] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603263#comment-14603263 ] Juliet Hougland edited comment on SPARK-8646 at 6/26/15 5:35 PM:

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8646: --- Attachment: pi-test.log Results from pu-test.log PySpark does not run on YARN

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8646: --- Attachment: spark1.4-SPARK_HOME-set-inline-HADOOP_CONF_DIR.log I ran the same line that gave

[jira] [Updated] (SPARK-8642) Ungraceful failure when yarn client is not configured.

[ https://issues.apache.org/jira/browse/SPARK-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8642: --- Attachment: yarnretries.log Log file from failed bc of misconfiguration spakr job. counting

[jira] [Created] (SPARK-8642) Ungraceful failure when yarn client is not configured.

Juliet Hougland created SPARK-8642: -- Summary: Ungraceful failure when yarn client is not configured. Key: SPARK-8642 URL: https://issues.apache.org/jira/browse/SPARK-8642 Project: Spark

[jira] [Updated] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-8646: --- Attachment: spark1.4-SPARK_HOME-set-PYTHONPATH-set.log

[jira] [Created] (SPARK-8646) PySpark does not run on YARN

Juliet Hougland created SPARK-8646: -- Summary: PySpark does not run on YARN Key: SPARK-8646 URL: https://issues.apache.org/jira/browse/SPARK-8646 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-8612) Yarn application status is misreported for failed PySpark apps.

Juliet Hougland created SPARK-8612: -- Summary: Yarn application status is misreported for failed PySpark apps. Key: SPARK-8612 URL: https://issues.apache.org/jira/browse/SPARK-8612 Project: Spark

[jira] [Created] (SPARK-7194) Vectors factors method for sparse vectors should accept the output of zipWithIndex

Juliet Hougland created SPARK-7194: -- Summary: Vectors factors method for sparse vectors should accept the output of zipWithIndex Key: SPARK-7194 URL: https://issues.apache.org/jira/browse/SPARK-7194

[jira] [Updated] (SPARK-7194) Vectors factors method for sparse vectors should accept the output of zipWithIndex

[ https://issues.apache.org/jira/browse/SPARK-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland updated SPARK-7194: --- Description: Let's say we have an RDD of Array[Double] where zero values are explictly

[jira] [Created] (SPARK-6938) Add informative error messages to require statements.

Juliet Hougland created SPARK-6938: -- Summary: Add informative error messages to require statements. Key: SPARK-6938 URL: https://issues.apache.org/jira/browse/SPARK-6938 Project: Spark

[jira] [Closed] (SPARK-5459) The reference of combineByKey in the programming guide should be replaced by aggregateByKey

[ https://issues.apache.org/jira/browse/SPARK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliet Hougland closed SPARK-5459. -- Resolution: Duplicate The reference of combineByKey in the programming guide should be

[jira] [Created] (SPARK-5459) The reference of combineByKey in the programming guide should be replaced by aggregateByKey

Juliet Hougland created SPARK-5459: -- Summary: The reference of combineByKey in the programming guide should be replaced by aggregateByKey Key: SPARK-5459 URL: https://issues.apache.org/jira/browse/SPARK-5459

[jira] [Created] (SPARK-5442) Docs claim users must explicitly depend on a hadoop client, but it is not actually required.

Juliet Hougland created SPARK-5442: -- Summary: Docs claim users must explicitly depend on a hadoop client, but it is not actually required. Key: SPARK-5442 URL: https://issues.apache.org/jira/browse/SPARK-5442

[jira] [Commented] (SPARK-3369) Java mapPartitions Iterator-Iterable is inconsistent with Scala's Iterator-Iterator

[ https://issues.apache.org/jira/browse/SPARK-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183301#comment-14183301 ] Juliet Hougland commented on SPARK-3369: The guaruntee of semantic versioning is