[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701185#comment-14701185 ] Piotr Migdał commented on SPARK-9705: - Thanks, now it works perfectly! (Before I tried to solve it that was but I missed that my `python` links to a different version of python than my `ipython`.) As a side note, right now my preferred way to lunch Spark in IPython Notebook is with `findspark` (https://github.com/minrk/findspark) - as IPython Notebook can be lunched in a normal way (`ipython notebook`) and `pyspark` can be used as a normal library. However, as I see it does not support setting Python version for Spark workers (at least - not right now). Again, I thought it was a problem with Spark/PySpark, but it was with the `findspark` library. outdated Python 3 and IPython information - Key: SPARK-9705 URL: https://issues.apache.org/jira/browse/SPARK-9705 Project: Spark Issue Type: Documentation Components: Documentation, PySpark Affects Versions: 1.4.0, 1.4.1, 1.5.0 Reporter: Piotr Migdał Priority: Blocker Labels: documentation Original Estimate: 0.25h Remaining Estimate: 0.25h https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0) says explicitly: Spark 1.4.1 works with Python 2.6 or higher (but not Python 3). Affected: https://spark.apache.org/docs/1.4.0/programming-guide.html https://spark.apache.org/docs/1.4.1/programming-guide.html There are some other Python-related things, which are outdated, e.g. this line: For example, to launch the IPython Notebook with PyLab plot support: (At least since IPython 3.0 PyLab/Matplotlib support happens inside a notebook; and the line --pylab inline is already removed.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699983#comment-14699983 ] Davies Liu commented on SPARK-9705: --- The PyLab thing is already fixed by https://github.com/apache/spark/pull/5111, I will send out a PR to fix others. outdated Python 3 and IPython information - Key: SPARK-9705 URL: https://issues.apache.org/jira/browse/SPARK-9705 Project: Spark Issue Type: Documentation Components: Documentation, PySpark Affects Versions: 1.4.0, 1.4.1, 1.5.0 Reporter: Piotr Migdał Priority: Blocker Labels: documentation Original Estimate: 0.25h Remaining Estimate: 0.25h https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0) says explicitly: Spark 1.4.1 works with Python 2.6 or higher (but not Python 3). Affected: https://spark.apache.org/docs/1.4.0/programming-guide.html https://spark.apache.org/docs/1.4.1/programming-guide.html There are some other Python-related things, which are outdated, e.g. this line: For example, to launch the IPython Notebook with PyLab plot support: (At least since IPython 3.0 PyLab/Matplotlib support happens inside a notebook; and the line --pylab inline is already removed.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[
https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699973#comment-14699973
]
Davies Liu commented on SPARK-9705:
---
As the exception said, PySpark cannot run with different minor versions, you
need to use the same minor version of Python, for example, use both Python 3.4
on driver and workers.
Be default, PySpark use the default `python` found in PATH, if you have
different default python in driver and worker, then you need to specify
PYSPARK_PYTHON to tell pyspark which version of Python you want to use. For
example:
{code}
PYSPARK_PYTHON=the_path_of_python_3.4 bin/spark-submit xxx
{code}
outdated Python 3 and IPython information
-
Key: SPARK-9705
URL: https://issues.apache.org/jira/browse/SPARK-9705
Project: Spark
Issue Type: Documentation
Components: Documentation, PySpark
Affects Versions: 1.4.0, 1.4.1, 1.5.0
Reporter: Piotr Migdał
Priority: Blocker
Labels: documentation
Original Estimate: 0.25h
Remaining Estimate: 0.25h
https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to
1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0)
says explicitly:
Spark 1.4.1 works with Python 2.6 or higher (but not Python 3).
Affected:
https://spark.apache.org/docs/1.4.0/programming-guide.html
https://spark.apache.org/docs/1.4.1/programming-guide.html
There are some other Python-related things, which are outdated, e.g. this
line:
For example, to launch the IPython Notebook with PyLab plot support:
(At least since IPython 3.0 PyLab/Matplotlib support happens inside a
notebook; and the line --pylab inline is already removed.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700021#comment-14700021 ] Apache Spark commented on SPARK-9705: - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/8245 outdated Python 3 and IPython information - Key: SPARK-9705 URL: https://issues.apache.org/jira/browse/SPARK-9705 Project: Spark Issue Type: Documentation Components: Documentation, PySpark Affects Versions: 1.4.0, 1.4.1, 1.5.0 Reporter: Piotr Migdał Priority: Blocker Labels: documentation Original Estimate: 0.25h Remaining Estimate: 0.25h https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0) says explicitly: Spark 1.4.1 works with Python 2.6 or higher (but not Python 3). Affected: https://spark.apache.org/docs/1.4.0/programming-guide.html https://spark.apache.org/docs/1.4.1/programming-guide.html There are some other Python-related things, which are outdated, e.g. this line: For example, to launch the IPython Notebook with PyLab plot support: (At least since IPython 3.0 PyLab/Matplotlib support happens inside a notebook; and the line --pylab inline is already removed.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694836#comment-14694836 ] Reynold Xin commented on SPARK-9705: [~pmigdal] do you mind submitting a pull request for this? outdated Python 3 and IPython information - Key: SPARK-9705 URL: https://issues.apache.org/jira/browse/SPARK-9705 Project: Spark Issue Type: Documentation Components: Documentation, PySpark Affects Versions: 1.4.0, 1.4.1, 1.5.0 Reporter: Piotr Migdał Priority: Blocker Labels: documentation Original Estimate: 0.25h Remaining Estimate: 0.25h https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0) says explicitly: Spark 1.4.1 works with Python 2.6 or higher (but not Python 3). Affected: https://spark.apache.org/docs/1.4.0/programming-guide.html https://spark.apache.org/docs/1.4.1/programming-guide.html There are some other Python-related things, which are outdated, e.g. this line: For example, to launch the IPython Notebook with PyLab plot support: (At least since IPython 3.0 PyLab/Matplotlib support happens inside a notebook; and the line --pylab inline is already removed.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information
[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695273#comment-14695273 ] Piotr Migdał commented on SPARK-9705: - I added it as I am having problems with running it on Python 3.4 (and it runs smoothly on 2.7). So I was not sure whether it's a problem with my installation/configuration or it isn't supported. (Exception: Python in worker has different version 2.7 than that in driver 3.4, PySpark cannot run with different minor versions) I was told by Josh Rosen that 3.4 is supported. However, it makes to sense for me to write docs if I cannot run it on mine computer (maybe there are some options needed?). The error is the same no matter if I run it from command line (specifying paths to Python 3 libraries) or through a preferred option for me - findspark in IPython Notebook. outdated Python 3 and IPython information - Key: SPARK-9705 URL: https://issues.apache.org/jira/browse/SPARK-9705 Project: Spark Issue Type: Documentation Components: Documentation, PySpark Affects Versions: 1.4.0, 1.4.1, 1.5.0 Reporter: Piotr Migdał Priority: Blocker Labels: documentation Original Estimate: 0.25h Remaining Estimate: 0.25h https://issues.apache.org/jira/browse/SPARK-4897 adds Python 3.4 support to 1.4.0 and above, but the official docs (1.4.1, but the same is for 1.4.0) says explicitly: Spark 1.4.1 works with Python 2.6 or higher (but not Python 3). Affected: https://spark.apache.org/docs/1.4.0/programming-guide.html https://spark.apache.org/docs/1.4.1/programming-guide.html There are some other Python-related things, which are outdated, e.g. this line: For example, to launch the IPython Notebook with PyLab plot support: (At least since IPython 3.0 PyLab/Matplotlib support happens inside a notebook; and the line --pylab inline is already removed.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
