Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-210276089
I think it's a good thing to do if we enable "pip install spark" for local
modes. As you said, minimizing overhead would be great.
---
If your project is set up for it,
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-210265477
@jhlch - I think this will be a tough feature to get in, honestly, but if
you want to take a fresh stab at it then I'm interested in helping with review
and testing.
Github user jhlch commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-209955734
Pyspark being pip installable would be useful to many users. I've packaged
jars in pip installable python modules before, and will take a stab at
here. I'm going to
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-209125293
No worries. I just wanted to make sure that the idea was sound since there
were concerns early on about whether we should even try to package PySpark
independently.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-209116066
I'm do not have bandwidth to work on this (also don't think this is high
priority), someone could take over this to move forward.
---
If your project is set up for it,
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-209114815
@davies - Does this PR have a realistic chance of making it in for Spark
2.0? If not, are we held up by implementation details or by a more fundamental
problem with
Github user gracew commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r49144128
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,53 @@
Finer-grained cache persistence levels.
"""
+import os
+import re
Github user mnazbro commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r46585482
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,53 @@
Finer-grained cache persistence levels.
"""
+import os
+import re
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r44564442
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,53 @@
Finer-grained cache persistence levels.
"""
+import os
+import re
Github user alope107 commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-155524260
@holdenk @gracew
Thanks, I bumped py4j version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user gracew commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-155518149
@alope107 , could you update the py4j dependency? I would really like to
see this merged as well =)
---
If your project is set up for it, you can reply to this email
Github user jhlch commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r44198571
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r43605044
--- Diff: python/setup.py ---
@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-147128537
Added check for version number in assembly jar if pom.xml is not present.
@davies is this what you had in mind?
---
If your project is set up for it, you can reply to
Github user rgbkrk commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-146693064
Looks like @alope107 added fixes for the last few reviews. How does this
stand now?
---
If your project is set up for it, you can reply to this email and have your
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41572924
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
"""
+import os
+import sys
Github user mnazbro commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41315226
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user shoyer commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41329564
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41334652
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41334727
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41175012
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
"""
+import os
+import sys
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41173642
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
"""
+import os
+import sys
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41175819
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41176383
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41212537
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41212659
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
"""
+import os
+import sys
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41213092
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41220284
--- Diff: python/setup.py ---
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r41221064
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user justinuang commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-140871937
Thanks! Sorry for being demanding, was just hoping to get this into 1.6.0!
---
If your project is set up for it, you can reply to this email and have your
reply
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-140871407
@justinuang We can work on this now, will review it this week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user justinuang commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-140866466
What is this blocking on?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37657387
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37657884
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user ellisonbg commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37554224
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user Carreau commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37557958
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user Carreau commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37557915
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user alope107 commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-133104217
@rgbkrk Yes, as the pyspark and Spark versions must match each other
exactly, it makes sense for deployments to pin both.
---
If your project is set up for it, you
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37558783
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37559580
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37561130
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37563567
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user rgbkrk commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-133097872
:+1: very excited about this
I'm assuming for deployments one will pin both the pyspark version from
PyPI as well as the Spark version they're using?
---
If
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37564759
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37524370
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37524804
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user justinuang commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-132998065
@holdenk , thanks for working on this! Do we have plans to set up PyPI
publishing?
---
If your project is set up for it, you can reply to this email and have your
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37580512
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37569488
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37570377
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37569058
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37570006
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,31 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37570574
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37570383
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37459103
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,33 @@
Finer-grained cache persistence levels.
+import os
+import sys
Github user alope107 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37464406
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37457141
--- Diff: python/pyspark/pyspark_version.py ---
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37458924
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user justinuang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37459213
--- Diff: python/pyspark/__init__.py ---
@@ -36,6 +36,33 @@
Finer-grained cache persistence levels.
+import os
+import sys
GitHub user alope107 opened a pull request:
https://github.com/apache/spark/pull/8318
[SPARK-1267][PYSPARK] Adds pip installer for pyspark
Adds a setup.py so that pyspark can be installed and packaged for pip.
This allows for easier setup, and declaration of dependencies. Please
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-132749766
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37457415
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
Github user alope107 commented on the pull request:
https://github.com/apache/spark/pull/8318#issuecomment-132773416
@justinuang and @nchammas thanks for the feedback; I've made the suggested
changes.
---
If your project is set up for it, you can reply to this email and have your
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/8318#discussion_r37485263
--- Diff: python/setup.py ---
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+from setuptools import setup
+
64 matches
Mail list logo
