Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Thanks @1ambda for the fix. Thanks @zjffdu for reviewing and verifying.
It looks good to me, while it provides more consistent behavior between
`PysparkInterpreter` and `SPARK_HOME/bin/
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
**For reviewers**
Fixed to use `spark.jars` instead of `classpath`.
- classpath doesn't include submitted jars at this moment (i could get 7
days ago, but not now)
- it enable
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Thanks, @1ambda Do you mind to create a spark ticket as well ? The behavior
inconsistency between different modes seems an issue of spark, we need to
clarify it with spark community.
---
If your
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu I'v just fixed not to extend PYTHONPATH using submitted packages
only in yarn-client
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Sorry, I miss your last reply. Do you mean yarn-client mode works for you
in spark ?
I use the following command to launch pyspark and get the error as
following:
Launch pyspark ( I am
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
1. I read and replied before.
> Q. README shows that user can use --py-files
> A. Users cannot benefit from --packages. They need to download, find
location of all transitive deps and p
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
As I said before why not using `--py-files`, I check the repository of
pyspark-cassandra.
https://github.com/TargetHolding/pyspark-cassandra
README shows that user can use `--py-files`
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu
> since it doesn't resolve the yarn-client mode
1. PySpark also doens't support extending PYTHONPATH in yarn-client.
2. You are saying this is not right fix repeatedl
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
I still think this is not a correct fix since it doesn't resolve the
yarn-client mode which I believe most of users use this mode.
---
If your project is set up for it, you can reply to this emai
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
Any update on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
I tested on yarn-client, mesos-client and found that
- **mesos-client mode copy pyspark-cassandra submitted by `--packages`** as
you can see [here]
(https://gist.github.com/1ambda/e332610
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
hmm, it works in local mode but doesn't work in yarn-client mode. Could you
try yarn-client mode ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu
I'v just created gist to show `--packages` option download
pyspark-cassandra.
https://gist.github.com/1ambda/5caf92753ea2f95ada11b1c13945d261
```
downloading
https:/
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@1ambda Actually, pyspark-cassandra doesn't work for me in pyspark shell. I
guess it works because you have installed it locally.
```
>>> import pyspark_cassandra
Traceback (most recent
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@zjffdu Thanks for review :)
Then, How can I load
[pyspark-cassandra](https://github.com/TargetHolding/pyspark-cassandra#with-spark-packages)
for pyspark?
---
If your project is set up
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1831
right, I'm a bit concern if this would be the right fix for the issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@1ambda Spark doesn't support specifying python packages throught
`--packages`, the correct usage is to use `--py-files`. Although this PR could
resolve your issue, but the issue here is not due to
Github user 1ambda commented on the issue:
https://github.com/apache/zeppelin/pull/1831
@astroshim Thanks for review!
It's the expected behavior. If spark submit doens't be properly loaded,
spark interpreter will die without errors.
---
If your project is set up for it, you
Github user astroshim commented on the issue:
https://github.com/apache/zeppelin/pull/1831
In my test, I got a ` INFO [2017-01-02 09:08:12,358] ({Exec Default
Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:164) -
Interpreter process exited 0` error when i try to run
19 matches
Mail list logo
