[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248377#comment-16248377 ] Felix Cheung commented on SPARK-18136: -- thanks [~hyukjin.kwon] > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 2.1.2, 2.2.0, 2.3.0 >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248368#comment-16248368 ] Hyukjin Kwon commented on SPARK-18136: -- Fixed in https://github.com/apache/spark/pull/19310 > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248367#comment-16248367 ] Hyukjin Kwon commented on SPARK-18136: -- Let me resolve this and separately open up another JIRA for finding Spark's home. pip installation itself works find I believe. > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248361#comment-16248361 ] Felix Cheung commented on SPARK-18136: -- is this still being targeted for 2.2.1? > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225081#comment-16225081 ] Hyukjin Kwon commented on SPARK-18136: -- Oh, no. Yes, there is something to be done to detect Spark home for pip installation on Windows. This is partly resolved at the first PR and second followup is in review. Will take an action for this soon e.g., taking over the second PR if it goes inactive. > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224845#comment-16224845 ] Sean Owen commented on SPARK-18136: --- Is this open because it's meant to be backported further (I imagine not)? or just that something else needs to be done? If it's substantially working, let's call this one resolved. > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.3, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[
https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183077#comment-16183077
]
Jakub Nowacki commented on SPARK-18136:
---
PR 19370 (https://github.com/apache/spark/pull/19370) fixes {{SPARK_HOME}}
issue using {{find_spark_home.py}} script. It's not maybe the most elegant way,
but it is simple.
I think in a long run it would be better to move to some Python packaging
mechanisms like {{console_scripts}} or related, as it will provide better
multiplatform support; see
https://packaging.python.org/tutorials/distributing-packages/#scripts and
https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation.
I'll create a separate issue with improvement proposal.
> Make PySpark pip install works on windows
> -
>
> Key: SPARK-18136
> URL: https://issues.apache.org/jira/browse/SPARK-18136
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Reporter: holdenk
> Fix For: 2.2.1, 2.3.0, 2.1.3
>
>
> Make sure that pip installer for PySpark works on windows
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182868#comment-16182868 ] Apache Spark commented on SPARK-18136: -- User 'jsnowacki' has created a pull request for this issue: https://github.com/apache/spark/pull/19370 > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.2.1, 2.3.0, 2.1.3 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[
https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179746#comment-16179746
]
Jakub Nowacki commented on SPARK-18136:
---
I can come back to this issue this Wednesday I think. I did some preliminary
tests with {{find_spark_home.py}} but I won't have time to sit to it until
Wednesday.
> Make PySpark pip install works on windows
> -
>
> Key: SPARK-18136
> URL: https://issues.apache.org/jira/browse/SPARK-18136
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Reporter: holdenk
> Fix For: 2.2.1, 2.3.0, 2.1.3
>
>
> Make sure that pip installer for PySpark works on windows
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179172#comment-16179172 ] holdenk commented on SPARK-18136: - [~fobofindia09] So currently we're working on a 2.1.2 release, and I believe the general consensus is to kick of 2.2.1 RC process after 2.1.2 is wrapped up, so it all depends on how quickly we can get 2.1.2 out the door. It's difficult to predict on the 2.1.2 side since we're doing it outside of the previous set of release managers so as to figure out what unwritten knowledge needs to get documented and we're running into a few unexpected issues as we go. > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.2.1, 2.3.0, 2.1.3 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179166#comment-16179166 ] Neel Shah commented on SPARK-18136: --- [~hyukjin.kwon] [~jsnowacki] is this the only thing holding 2.2.1 back? Is there an ETA for that release? > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.2.1, 2.3.0, 2.1.3 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177818#comment-16177818 ] Hyukjin Kwon commented on SPARK-18136: -- I haven't looked into the way you said but let's make the change minimised as possible as we can. > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > Fix For: 2.1.2, 2.2.1, 2.3.0 > > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[
https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177795#comment-16177795
]
Jakub Nowacki commented on SPARK-18136:
---
I've looked into it again and noticed the Bash script {{find_spark_home}},
which is used in Bash version of {{pyspark}} command. The Python script
{{find_spark_home.py}} seems to return the correct SPARK_HOME path on Windows,
so the all the cmd-files should be alter somehow to use it instead of
{{%~dp0}}. I'll look into it when I have time, maybe next week, and propose
something similar to the {{find_spark_home}} script approach.
> Make PySpark pip install works on windows
> -
>
> Key: SPARK-18136
> URL: https://issues.apache.org/jira/browse/SPARK-18136
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Reporter: holdenk
> Fix For: 2.1.2, 2.2.1, 2.3.0
>
>
> Make sure that pip installer for PySpark works on windows
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[
https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175303#comment-16175303
]
Jakub Nowacki commented on SPARK-18136:
---
I've tried using Windows command {{mklink}} to create symbolic links, but it
seems to resolve the folder in {{%~dp0}} to the Scripts folder
{{C:\Tools\Anaconda3\Scripts\}}.
> Make PySpark pip install works on windows
> -
>
> Key: SPARK-18136
> URL: https://issues.apache.org/jira/browse/SPARK-18136
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Reporter: holdenk
>
> Make sure that pip installer for PySpark works on windows
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[
https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175276#comment-16175276
]
Jakub Nowacki commented on SPARK-18136:
---
[PR|https://github.com/apache/spark/pull/19310] fixes how {{spark-class2.cmd}}
looks for jars directory on Windows. It fails to find jars and start JVM as the
condition for the env variable {{SPARK_JARS_DIR}} looks for
{{%SPARK_HOME%\RELEASE}}, which is not included in the {{pip/conda}} build.
Instead, it should look for {{%SPARK_HOME%\jars}}, which it is later referring
to.
The above fixes the errors while importing {{pyspark}} into Python and creating
SparkSession, but there is still an issue calling {{pyspark.cmd}}. Namely,
normal command call on command line, without path specification fails with
{{System cannot find the path specified.}}. It is likely due to the script link
being resolved to Script folder in Anaconda, e.g.
{{C:\Tools\Anaconda3\Scripts\pyspark.cmd}}. If the script is run via the full
path to the PySpark package, e.g.
{{\Tools\Anaconda3\Lib\site-packages\pyspark\bin\pyspark.cmd}} it works fine.
It is likely due to the fact that {{SPARK_HOME}} is resolved as follows {{set
SPARK_HOME=%~dp0..}}, which in case of the system call resolves (likely) to
{{\Tools\Anaconda3\}} and should resolve to
{{\Tools\Anaconda3\Lib\site-packages\pyspark\}}. Since I dion't know CMD
scripting that well, I haven't found solution to this issue yet, apart from the
workaround, i.e. calling it with full (direct) path.
> Make PySpark pip install works on windows
> -
>
> Key: SPARK-18136
> URL: https://issues.apache.org/jira/browse/SPARK-18136
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Reporter: holdenk
>
> Make sure that pip installer for PySpark works on windows
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18136) Make PySpark pip install works on windows
[ https://issues.apache.org/jira/browse/SPARK-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175258#comment-16175258 ] Apache Spark commented on SPARK-18136: -- User 'jsnowacki' has created a pull request for this issue: https://github.com/apache/spark/pull/19310 > Make PySpark pip install works on windows > - > > Key: SPARK-18136 > URL: https://issues.apache.org/jira/browse/SPARK-18136 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: holdenk > > Make sure that pip installer for PySpark works on windows -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
