[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944564#comment-16944564
]
Furcy Pin commented on SPARK-13587:
---
Hello,
I don't know where to ask this, but we have been using
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659374#comment-16659374
]
Ruslan Dautkhanov commented on SPARK-13587:
---
We're using conda environments shared across
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512433#comment-16512433
]
Matt Mould commented on SPARK-13587:
What is the current status of this ticket please? This
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217221#comment-16217221
]
Semet commented on SPARK-13587:
---
Hello. For me this solution is equivalent with my "Wheelhouse"
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217115#comment-16217115
]
Nicholas Chammas commented on SPARK-13587:
--
To follow-up on my [earlier
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923683#comment-15923683
]
Jeff Zhang commented on SPARK-13587:
I linked a detailed document about how to use it in both batch
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747407#comment-15747407
]
Jeff Zhang commented on SPARK-13587:
If it is pretty large cluster, then I would suggest to set up a
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747379#comment-15747379
]
Prasanna Santhanam commented on SPARK-13587:
[~zjffdu] In case of Anaconda Python the
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747381#comment-15747381
]
Prasanna Santhanam commented on SPARK-13587:
[~zjffdu] In case of Anaconda Python the
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747338#comment-15747338
]
Jeff Zhang commented on SPARK-13587:
[~prasanna.santha...@icloud.com] I don't understand how this can
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747291#comment-15747291
]
Prasanna Santhanam commented on SPARK-13587:
[~nchammas] sorry, this got buried in several
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740419#comment-15740419
]
Nicholas Chammas commented on SPARK-13587:
--
Thanks to a lot of help from [~quasi...@gmail.com]
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714622#comment-15714622
]
Semet commented on SPARK-13587:
---
For myself, I share a NFS folder with all the executors. It works because
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713636#comment-15713636
]
Nicholas Chammas commented on SPARK-13587:
--
[~tsp]:
{quote}
Previously, I have had reasonable
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647331#comment-15647331
]
Prasanna Santhanam commented on SPARK-13587:
Thanks for this JIRA and the work related to it
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362294#comment-15362294
]
Semet commented on SPARK-13587:
---
Full proposal is here: https://issues.apache.org/jira/browse/SPARK-16367
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362014#comment-15362014
]
Takao Magoori commented on SPARK-13587:
---
Sorry. It seems there is no isolated site-package
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361940#comment-15361940
]
Takao Magoori commented on SPARK-13587:
---
What is the reason why virtualenv is required ?
I feel
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355172#comment-15355172
]
Semet commented on SPARK-13587:
---
yes it looks cool!
Here is what I have in mind, tell me if it is the wrong
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351512#comment-15351512
]
Jeff Zhang commented on SPARK-13587:
Thanks [~gae...@xeberon.net] Have you take a look at my PR ?
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351074#comment-15351074
]
Semet commented on SPARK-13587:
---
I back this proposal and willing to work on it.
> Support virtualenv in
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324323#comment-15324323
]
Jeff Zhang commented on SPARK-13587:
Sorry, guys, I am busy on other stuff recently and late for
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324318#comment-15324318
]
Apache Spark commented on SPARK-13587:
--
User 'zjffdu' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324316#comment-15324316
]
Apache Spark commented on SPARK-13587:
--
User 'zjffdu' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311649#comment-15311649
]
Jeff Zhang commented on SPARK-13587:
Yes, I focus on yarn mode, I did some test on local mode, but
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311646#comment-15311646
]
Jeff Zhang commented on SPARK-13587:
Thanks [~gbow...@fastmail.co.uk] In my POC, I implemented it
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310881#comment-15310881
]
Greg Bowyer commented on SPARK-13587:
-
There is also a small wrapper for this
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310872#comment-15310872
]
Greg Bowyer commented on SPARK-13587:
-
Does this work for people ?
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310782#comment-15310782
]
Greg Bowyer commented on SPARK-13587:
-
I have been out of this world for a long time.
The Spex
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234264#comment-15234264
]
John Berryman commented on SPARK-13587:
---
At my work we're using devpi and a homecooked proxy server
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218659#comment-15218659
]
Mike Sukmanowsky commented on SPARK-13587:
--
That's the (hopefully) beautiful thing about pex.
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218610#comment-15218610
]
Juliet Hougland commented on SPARK-13587:
-
Being able to ship around pex files like we do .py and
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218109#comment-15218109
]
Mike Sukmanowsky commented on SPARK-13587:
--
The NFS is a good idea for a current workaround.
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214038#comment-15214038
]
Dripple commented on SPARK-13587:
-
[~juliet] can you give details/pointer on the solution you describe ?
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212848#comment-15212848
]
Jeff Zhang commented on SPARK-13587:
bq. Have you considered using NFS or Amazon EFS to allow users
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212191#comment-15212191
]
Juliet Hougland commented on SPARK-13587:
-
I really do think spark and pyspark needs to stay out
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201763#comment-15201763
]
Mike Sukmanowsky commented on SPARK-13587:
--
Sorry to bug [~juliet] - any thoughts? We're
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184210#comment-15184210
]
Mike Sukmanowsky commented on SPARK-13587:
--
[~juliet] I get the concerns relating to Spark
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179254#comment-15179254
]
Jeff Zhang commented on SPARK-13587:
SPARK-13081 is almost done. But for this ticket, I definitely
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179253#comment-15179253
]
Juliet Hougland commented on SPARK-13587:
-
That is wonderful. Let me know if you'd like me to
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179251#comment-15179251
]
Jeff Zhang commented on SPARK-13587:
Actually I have created this ticket several weeks ago
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179250#comment-15179250
]
Juliet Hougland commented on SPARK-13587:
-
I made a comment related to this below. TLDR I think
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179247#comment-15179247
]
Juliet Hougland commented on SPARK-13587:
-
Currently the way users specify the workers' python
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179012#comment-15179012
]
Jeff Zhang commented on SPARK-13587:
Thanks for the feedback [~j_houg]. Yes, that's why I make the
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178883#comment-15178883
]
Jeff Zhang commented on SPARK-13587:
Thanks for letting me know this.
> Support virtualenv in
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178536#comment-15178536
]
Juliet Hougland commented on SPARK-13587:
-
If pyspark allows users to create virtual
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175776#comment-15175776
]
Dan Blanchard commented on SPARK-13587:
---
`conda list --export` will omit all pip-installed
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175715#comment-15175715
]
Jeff Zhang commented on SPARK-13587:
Thanks for the feedback [~dan.blanchard], In my POC, I use
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175672#comment-15175672
]
Dan Blanchard commented on SPARK-13587:
---
One thing to note is that conda doesn't use the same
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175646#comment-15175646
]
Mike Sukmanowsky commented on SPARK-13587:
--
Perfect and understood about not wanting to promote
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175045#comment-15175045
]
Jeff Zhang commented on SPARK-13587:
spark.pyspark.virtualenv.requirements is a local file (which
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175018#comment-15175018
]
Mike Sukmanowsky commented on SPARK-13587:
--
One thought that just occurred to me, does
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175010#comment-15175010
]
Mike Sukmanowsky commented on SPARK-13587:
--
Gotcha. I might suggest
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175003#comment-15175003
]
Jeff Zhang commented on SPARK-13587:
Thanks for your feedback [~msukmanowsky].
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174996#comment-15174996
]
Mike Sukmanowsky commented on SPARK-13587:
--
Thanks for letting me know about this [~jeffzhang].
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173590#comment-15173590
]
Sean Owen commented on SPARK-13587:
---
CC [~j_houg] for interest, comment
> Support virtualenv in
[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173228#comment-15173228
]
Jeff Zhang commented on SPARK-13587:
I have implemented POC for this features. Here's oen simple
57 matches
Mail list logo
