Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2329
Do you mean there would be 2 yarn apps for launching one spark interpreter
? That looks a little weird to me. In this approach, we still launch remote
interpreter process in the zeppelin host, that doesn't solve much memory issue
of zeppelin host. And besides it would waste yarn resources as it require to
launch 2 yarn apps. Why not leverage spark-submit to support yarn-cluster mode
? And I look at all the interpreters of zeppelin. Most of them are just client
role. That means the computation happens in the backend not in the interpreter
side. so I think it would be fine to launch these interpreters in shared/scoped
mode for all users. Spark Interpreter would be a special case, we could use its
native yarn-cluster support. That would be the simplest way I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
