Lee moon soo created ZEPPELIN-533:
-------------------------------------
Summary: Helium
Key: ZEPPELIN-533
URL: https://issues.apache.org/jira/browse/ZEPPELIN-533
Project: Zeppelin
Issue Type: New Feature
Reporter: Lee moon soo
Helium proposal
https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal was about
pluggable application on top of Zeppelin. I'd like to extend the concept and
make Helium covers interpreters, notebook storages, too.
i.e. Interpreter and notebook storage are able to loaded from maven repository
on the fly.
*Interpreter*
Interpreter is already pluggable. User can just drop necessary binary under
interpreter/\[interpretername], add configuration into conf/zeppelin-site.xml
and restart Zeppelin will make interpreter work.
But it'll be much easier if Zeppelin can load interpreter binary from maven
repository and load them without restart.
*Notebook storage*
Notebook storage is also already pluggable. User can drop necessary binary into
ZeppelinServer's classpath (/lib), add configuraion into conf/zeppelin-site.xml
and restart Zeppelin will make notebook storage work.
Current implementation may bring library conflict while all notebook storage is
loaded from the same classloader. There should be dedicated classloader per
implementation like Interpreter. And then it can be loaded from maven
repository on the fly, just like an Interpreter.
*Application*
https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal tried to
define it. Application can be loaded and launched just like Interpreter and
Notebook storage, but not in ZeppelinServer process but in interpreter process
because of some application may want to use Interpreter provided resource (e.g.
SparkContext). The resource will be acceseed through Resource pool. Once
application is being loaded, application will able to use outputstream provided
by Zeppelin as well as InterpreterContext (provides api for angularobject, etc).
In short, not only the code provided by user but also the packaged code will be
run on the notebook.
*Package registry*
Package registry central repository for package metadata. Package metadata is
informations such as name, type, description, maven repository, version, etc.
One simplest way of making this package repository is create an separate git
repository and use it as a package repository. We will need automated
registration / update process of metadata from 3rd party developers.
* 'Helium' menu*
'Helium' menu on GUI will get metadata from package registry and list them to
user select. It'll need to save user's selection into conf/helium.json
Following sub tasks can be created
- \[Interpreter] Make interpreter loadable from maven repository and REST api
for it
- \[Notebook storage] Make each notebook storage uses separate classloader
- \[Notebook storage] Make notebook storage loadable from maven repository and
REST api for it
- \[Application] Provide output stream to application
- \[Application] Shared resource pool that user application can exchange data
between interpreter processes
- \[Application] Make each user application loaded from separate classloader
- \[Application] Add 'paragraph' scope for angular object in addition to
'notebook', 'global'
- \[Application] Application launch API
- \[GUI] Create Helium menu and provide user interface for REST api
- \[GUI] Display package from package registry
- \[GUI] Save / Load user selection into conf/helium.conf
- \[Registry] Helium package registry
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
