GitHub user doanduyhai opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/678
[Zeppelin-] AngularJS utilities functions and Hide/Show ParagraphId feature
### What is this PR for?
This is a PR to add **utilities functions for the AngularJS system** and a
new **Hide/Show Paragraph Id** feature.
> See screenshots below to have a feeling of how it works
#### 1) AngularJS Utility Functions
* **runParagraph('paragraphId')** : run a paragraph given its id
* **runParagraphs(['paragraphId1', 'paragraphId2'])** : run a list of
paragraphs given their id
* **pushToServer('varName', value, parameters)** : push a variable
_varName_ with its _value_ to the server/interpreter **angular object
repository** using current **noteId** and provided interpreter(s) (and
optionally paragraph id(s)). _parameters_ is a map with the following
properties:
Ex:
```javascript
parameters = {
interpreter: 'spark',
interpreters: ['md', 'sh'],
paragraph: '20160126-153136_1060166247',
paragraphs: ['20160126-171914_843040190',
'20160126-181556_1915782845'],
scope: 'note',
runParagraphs: false
}
```
* **interpreter**: name of the interpreter to be used for angular
object repository lookup
* **interpreters**: name of the interpreters to be used for angular
object repository lookup
* **paragraph**: paragraph id used to update the angular object
repository
* **paragraphs**: paragraph ids used to update the angular object
repository
* **scope**: scope of the angular object, can be _note_ or _paragraph_.
Default = _paragraph_
* **runParagraphs**: if _true_, run the paragraphs listed by the
properties **paragraph** or **paragraphs**. Default = _true_
<br/>
Remarks:
> 1. if both **interpreter** and **interpreters** are provided, the
single **interpreter** will be added to the **interpreters** list. Same
remark for **paragraph** and **paragraphs**
> 2. if a list of **interpreters** and **paragraphs** are provided,
**for each interpreter the object will be pushed only to the paragraphs which
matches this interpreter**
> 3. if one or several **paragraph** is provided, the object will be
pushed to the registry using the _(current note id, paragraph id)_ scope. This
correspond to the default _paragraph_ value for the **scope** variable.
However, it is also possible to save the same object to the _current note id_
scope. In this case set the **scope** variable to _note_. In this case the
object will be saved using paragraph scope and note scope
> 4. if **scope** = _paragraph_ and no paragraph id is provided, the
object will be saved using the _current note id_ scope
> 5. if **runParagraphs** = _true_ and no paragraph id is provided, it
will be ignored
> 6. set **runParagraphs** to _false_ if you want to push the variable
to the back-end without refreshing paragraph
* **removeFromServer('varName', parameters)**: remove the variable
_varName_ from server/interpreter **angular object repository** using current
**noteId**.
Ex:
```javascript
parameters = {
interpreter: 'spark',
interpreters: ['md', 'sh'],
paragraph: '20160126-153136_1060166247',
paragraphs: ['20160126-171914_843040190',
'20160126-181556_1915782845']
}
```
* **interpreter**: name of the interpreter to be used for angular
object repository lookup
* **interpreters**: name of the interpreters to be used for angular
object repository lookup
* **paragraph**: paragraph id used to remove object from the angular
object repository
* **paragraphs**: paragraph ids used to remove object from the angular
object repository
Remarks:
> 1. if both **interpreter** and **interpreters** are provided, the
single **interpreter** will be added to the **interpreters** list. Same
remark for **paragraph** and **paragraphs**
> 2. if a list of **interpreters** and **paragraphs** are provided,
**for each interpreter the object will be pushed only to the paragraphs which
matches this interpreter**
> 3. if one or several **paragraph** is provided, the object will be
remove from the registry using the _(current note id, paragraph id)_ scope
> 4. if no **paragraph** is provided, the object will be remove from the
registry using the _current note id_ scope
#### 2) Hide/Show Paragraph Id
This feature is necessary for the above AngularJS functions to work. One
can always retrieve the current paragraph Id using the code
```scala
z.getInterpreterContext().getParagraphId()
```
But it is far from being user-friendly. Therefore, now you can show or hide
paragraph Id using a toggle. This is placed at the same place as Show/Hide line
numbers.
The paragraph id is displayed at the top left corner, above the Title. I'm
completely open to any suggestion with regard to the location of the display
#### 3) Technical Impl
1. First I have added a new **Thrift** method _angularRegistryPush()_. This
method is used to push the current angular object registry from the Zeppelin
server to the remote interpreter. According to @leemoonsoo, when a note is
loaded from the JSON file, the interpreter **may not be initialized** yet so we
need to store all the angular variables in the local registry. As soon as the
remote interpreter is started, we can call the _angularRegistryPush()_ method
to push the local registry to the remote interpreter
2. A new public constructor has been added to the `AngularObject` class.
The reason is because when we push the whole `Map<String, Map<String,
AngularObject>>` registry to the remote interpreter using
_angularRegistryPush()_, the registry is serialized using **Gson**. At the
remote interpreter side, the payload is deserialized back into a `Map<String,
Map<String, AngularObject>>`. If a class **do not have a public constructor**
then **Gson** will use some `Unsafe` tricks to create it (see **[here]**).
Consequently, the `private transient List<AngularObjectWatcher> watchers = new
LinkedList<AngularObjectWatcher>();` property will NOT be initialized and will
throw `NullPointerException` later when the code access the `watchers`
3. The `runParagraph()` and `runParagraphs()` Angular functions are
injected into the **isolated scope** create for each paragraph so that people
can access them. Currently I'm using the following trick to retrieve the
paragraph details for a given paragraphId:
```javascript
var paragraphDiv = angular.element('#' + paragraphId +
'_paragraphColumn_main[ng-controller="ParagraphCtrl"]');
var paragraph = paragraphDiv.scope().paragraph;
```
It's not the cleanest way. Ideally, we should inject into the isolated
scope the `note.paragraphs` array from the Note scope but I'm not sure if it is
recommended to leak such information into an isolated scope. Any
idea/suggestion is welcomed
4. New message operations **`ANGULAR_OBJECT_CLIENT_UPDATE`** and
**`ANGULAR_OBJECT_CLIENT_REMOVE`** have been added to support `pushToServer()`
and `removeFromServer()` Angular functions. The core functionalities are
implemented in `NotebookServer.angularObjectClientUpdate()` and
`NotebookServer.angularObjectClientDelete()`. I also add 8 unit tests for those
2 methods.
> A remark on the unit tests. I had to mock a bunch of classes for the
tests, which suggest that the code is somehow entangled. I try to make the
method as modular as possible but there are a lot of dependencies on objects
like Notebook, Note, InterpreterGroup etc ... Maybe a general refactoring later
would be nice
5. The `ZeppelinContext.getAngularObject()` private method has been
updated. Right now it look for variable using only the note scope. I change the
code so that now, we look for a variable using:
* first _(noteId, current paragraphId)_ as key
* if no object found, then use _noteId_ as key
* if no object found, then look at the **global scope**
6. Last but not least, **I have changed the current behavior of dynamic
form**. In the method `Paragraph.jobRun()`, when we encounter a variable
definition block ( ${var = ...}) while parsing, we first look into the
**angular object registry** using _(noteId, paragraphId)_ scope, then
_(noteId)_ scope. If we find a matching variable we'll use it and **we do not
create any form input for the paragraph**. At worst, we can just remove the
last commit, it is independent from the other features.
@Leemoonsoo @jongyoul @bzz @felixcheung @corneadoug I really need your
remarks/comments for the above 6 technical points. Thanks
### What type of PR is it?
[Feature]
### Todos
* [ ] - Code review by committers/contributors/community
* [ ] - Responses and suggestions for the technical questions raised in
chapter **3) Technical Impl**
* [ ] - Features tested following the below step
### Is there a relevant Jira issue?
**[ZEPPELIN-635]**
### How should this be tested?
**See below 12 screenshots for all possible test scenarios**
### Screenshots (if appropriate)
**Hide/Show ParagraphId**
**Run Single Paragraph**
**Run Multiple Paragraphs**
**Push To Server Scenario 1**: single interpreter, **note** scope
**Push To Server Scenario 2**: multiple interpreters, **note** scope
**Push To Server Scenario 3**: single interpreter, single paragraph
**Push To Server Scenario 4**: single interpreter, multiples paragraphs
**Push To Server Scenario 5**: single interpreter, multiples paragraphs AND
note scope
**Push To Server Scenario 6**: single interpreter, multiples paragraphs AND
run paragraphs
**Push To Server Scenario 7**: multiple interpreters, multiples paragraphs,
**overriding dynamic forms**
**Remove From Server Scenario 1**: single interpreter, single paragraph
**Remove From Server Scenario 2**: single interpreter, **note** scope
### Questions:
* Does the licenses files need update? --> **No**
* Is there breaking changes for older versions? --> **No**
* Does this needs documentation? --> **Yes**
[here]:
http://stackoverflow.com/questions/18645050/is-default-no-args-constructor-mandatory-for-gson
[ZEPPELIN-635]: https://issues.apache.org/jira/browse/ZEPPELIN-635
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/doanduyhai/incubator-zeppelin
PushToServer_Angular_Function
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/678.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #678
----
commit 7c1e284b0509e9cd8419fe83485643d1a7ac7e66
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-24T18:55:30Z
Add Thrift RPC method angularRegistryPush()
commit ecb3a219478cc257d5069faccb00227c497569a0
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-26T15:50:10Z
Make AngularObject constructor public because of serialization issue
commit 65985cab69f337c1e31c7d667defd79b94b334ed
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-26T19:30:36Z
Implement runParagraph() and runParagraphs() Angular functions
commit 6eeaa4bd02ca78df9259342f9917c4f62b024c37
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-26T12:41:12Z
Implement pushToServer() Angular function
commit 96c30798b80a43ff6518c39138cb7a19361f5843
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-26T15:51:11Z
ZeppelinContext angular() method should look for variable using the
paragraph scope, then note scope and then global scope, in this order
commit 21a12ddd2d3ca2b81ef20e5fa3db6a3ab4793227
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-24T20:31:53Z
Add Hide/Show paragraph id feature
commit b544b0c47bd3af05e7a85a1d9d12b260e142988e
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-25T20:13:53Z
Replace dynamic form with angular object from registry if exists
commit bf4b29317c6e0aa09e48a47cb8a0849016362fa8
Author: DuyHai DOAN <[email protected]>
Date: 2016-01-26T19:29:20Z
Implement removeFromServer() Angular function
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
