Repository: zeppelin Updated Branches: refs/heads/master d1293c6bc -> 2be8f3506
[ZEPPELIN-3092] GitHub Integration ### What is this PR for? GitHub integration as a storage for notebooks. ### What type of PR is it? Feature ### What is the Jira issue? [ZEPPELIN-3092](https://issues.apache.org/jira/browse/ZEPPELIN-3092) ### How should this be tested? 1. Change the configuration in `zeppelin-site.xml` to enable GitHub integration (add GitHub url, username, access token and origin) as described in https://github.com/apache/zeppelin/compare/master...mohamagdy:zeppelin-3092-remote-github-integration?expand=1#diff-89104d48f0358450399a6f679bba9c4f 2. Start the Zeppelin server 3. Open an existing notebook or create a new notebook 4. Do some changes to the notebook, for example add a new paragraph 5. Click on the versioning button on the top menu to commit and save changes 6. Checkout the changes in the GitHub repository. The changes should be reflected ### Questions: * **Does the licenses files need update?** No * **Is there breaking changes for older versions?** No * **Does this needs documentation?** Yes. Documentation is updated as part of the pull request. Author: Mohamed Magdy <mohamed.ma...@fyber.com> Author: Mohamed Magdy <mohamed.ma...@smartfrog.com> Author: Mohamed Magdy <eng.mohama...@gmail.com> Closes #2700 from mohamagdy/zeppelin-3092-remote-github-integration and squashes the following commits: b445960 [Mohamed Magdy] [ZEPPELIN-3092] Optimize imports for `Notebook` class afa5de1 [Mohamed Magdy] Merge branch 'master' of github.com:apache/zeppelin into zeppelin-3092-remote-github-integration 548c423 [Mohamed Magdy] [ZEPPELIN-3092] Add `zeppelin-site.xml` to `.gitignore` e98d1b0 [Mohamed Magdy] [ZEPPELIN-3092] Remove `zeppelin-site.xml` from Zeppelin Server resources 7a02855 [Mohamed Magdy] [ZEPPELIN-3092] Add Apache Software Foundation header 9101e58 [Mohamed Magdy] [ZEPPELIN-3092] Replace `printStackTrace()` with error logging db94d55 [Mohamed Magdy] [ZEPPELIN-3092] Remove loading notebook from repository when requested af952a0 [Mohamed Magdy] [ZEPPELIN-3029] Change authentication to `anonymous` instead of `empty` b5fbc1e [Mohamed Magdy] [ZEPPELIN-3092] Break long line to smaller ones 4d6cc76 [Mohamed Magdy] [ZEPPELIN-3092] Load notebook from repository when requested d1d43eb [Mohamed Magdy] Merge branch 'zeppelin-3092-remote-github-integration' of github.com:mohamagdy/zeppelin into zeppelin-3092-remote-github-integration 579bd6f [Mohamed Magdy] [ZEPPELIN-3092] Load note from memory when reloading 2f1b8bc [Mohamed Magdy] [ZEPPELIN-3092] Load note from memory when reloading d545e81 [Mohamed Magdy] [ZEPPELIN-3092] Remove duplicated dependency from `pom.xml` fc13fa6 [Mohamed Magdy] Revert "[ZEPPELIN-3029] Increase Paragraph and Browser timeouts" be2c278 [Mohamed Magdy] Revert "[ZEPPELIN-3092] Set browser timeout to 180 seconds" f362dcb [Mohamed Magdy] [ZEPPELIN-3029] Use jGit version 4.5.4 instead of 4.3.1 8bd23d0 [Mohamed Magdy] [ZEPPELIN-3092] Set browser timeout to 180 seconds 30f2ab4 [Mohamed Magdy] [ZEPPELIN-3029] Increase Paragraph and Browser timeouts 13a0014 [Mohamed Magdy] [ZEPPELIN-3092] Disable GitHub configuration for Zeppelin server 14cb024 [Mohamed Magdy] [ZEPPELIN-3092] Fix notebook path for Git and GitHub tests 0e9db3f [Mohamed Magdy] [ZEPPELIN-3092] Remove test GitHub repository URL and access token 90de14c [Mohamed Magdy] Merge branch 'master' into zeppelin-3092-remote-github-integration 2c1cf74 [Mohamed Magdy] [ZEPPELIN-3029] Fix remote origin key name 6ba67ca [Mohamed Magdy] [ZEPPELIN-3092] Add Javadoc to `GitHubNotebookRepo` and fix line length to 100 264565b [Mohamed Magdy] [ZEPPELIN-3092] Fix line length to be 100 0174bbd [Mohamed Magdy] [ZEPPELIN-3092] Add documentation how to enabled `GitHubNotebookRepo` 81969e1 [Mohamed Magdy] [ZEPPELIN-3092] Add documentation for loading notebooks from repo 3009abd [Mohamed Magdy] [ZEPPELIN-3092] Reset `GitNotebookRepo` to `master` 6aa4ba7 [Mohamed Magdy] [ZEPPELIN-3092] Revert back `GitNotebookRepo` to `master` b77a2d3 [Mohamed Magdy] [ZEPPELIN-3092] Fix identation in `pom.xml` aadd9b5 [Mohamed Magdy] [ZEPPELIN-3092] Revert back ZeppelinServer changes 0dacbf1 [Mohamed Magdy] [ZEPPELIN-3092] Fix encoding in the documenation 2b093b2 [Mohamed Magdy] [ZEPPELIN-3092] Add documentation about GitHub integration 843e42a [Mohamed Magdy] [ZEPPELIN-3092] Cleanup GitHub repository tests 5236176 [Mohamed Magdy] [ZEPPELIN-3092] Move GitHub notebook repostiory to separte file 2dbf116 [Mohamed Magdy] [ZEPPELIN-3092] Add GitHub configuration to `zeppelin-site.xml` template bb0afe2 [Mohamed Magdy] [ZEPPELIN-3092] Add GitHub remote to configurations 33ae24a [Mohamed Magdy] [ZEPPELIN-3092] Add remote Github repository synchronzing 32f6764 [Mohamed Magdy] [ZEPPELIN-3092] Fix GitNotebook test eeb485a [Mohamed Magdy] [ZEPPELIN-3092] Add Github configuration reader 0bde310 [Mohamed Magdy] [ZEPPELIN-3092] Add `zeppelin-site.xml` to `zeppelin-server` resources 9467503 [Mohamed Magdy] [ZEPPELIN-3092] Add `zepplein-server/local-repo` to `.gitignore` Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/2be8f350 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/2be8f350 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/2be8f350 Branch: refs/heads/master Commit: 2be8f350658076c33d9d905b9e9907aa3d3a8792 Parents: d1293c6 Author: Mohamed Magdy <mohamed.ma...@fyber.com> Authored: Wed Jan 24 10:11:15 2018 +0100 Committer: Jeff Zhang <zjf...@apache.org> Committed: Tue Feb 13 11:28:19 2018 +0800 ---------------------------------------------------------------------- .gitignore | 3 + conf/zeppelin-site.xml.template | 24 ++ .../contribution/how_to_contribute_code.md | 12 + docs/setup/operation/configuration.md | 30 +- docs/setup/storage/storage.md | 42 +++ .../zeppelin/conf/ZeppelinConfiguration.java | 22 +- .../apache/zeppelin/socket/NotebookServer.java | 2 +- .../src/test/resources/2A94M5J1Z/note.json | 376 +++++++++++++++++++ .../src/test/resources/2A94M5J2Z/note.json | 376 +++++++++++++++++++ .../org/apache/zeppelin/notebook/Notebook.java | 38 +- .../notebook/repo/GitHubNotebookRepo.java | 126 +++++++ .../zeppelin/notebook/repo/GitNotebookRepo.java | 5 +- .../notebook/repo/GitHubNotebookRepoTest.java | 207 ++++++++++ .../notebook/repo/GitNotebookRepoTest.java | 16 +- 14 files changed, 1243 insertions(+), 36 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/.gitignore ---------------------------------------------------------------------- diff --git a/.gitignore b/.gitignore index 773edc8..4086a4b 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,9 @@ spark-1.*-bin-hadoop* lens/lens-cli-hist.log +# Zeppelin server +zeppelin-server/local-repo +zeppelin-server/src/main/resources/zeppelin-site.xml # conf file conf/zeppelin-env.sh http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/conf/zeppelin-site.xml.template ---------------------------------------------------------------------- diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template index 33aa8ac..9e9898b 100755 --- a/conf/zeppelin-site.xml.template +++ b/conf/zeppelin-site.xml.template @@ -499,5 +499,29 @@ </property> --> +<!-- GitHub configurations +<property> + <name>zeppelin.notebook.git.remote.url</name> + <value></value> + <description>remote Git repository URL</description> +</property> +<property> + <name>zeppelin.notebook.git.remote.username</name> + <value>token</value> + <description>remote Git repository username</description> +</property> + +<property> + <name>zeppelin.notebook.git.remote.access-token</name> + <value></value> + <description>remote Git repository password</description> +</property> + +<property> + <name>zeppelin.notebook.git.remote.origin</name> + <value>origin</value> + <description>Git repository remote</description> +</property> +--> </configuration> http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/docs/development/contribution/how_to_contribute_code.md ---------------------------------------------------------------------- diff --git a/docs/development/contribution/how_to_contribute_code.md b/docs/development/contribution/how_to_contribute_code.md index b172aa1..92b69b5 100644 --- a/docs/development/contribution/how_to_contribute_code.md +++ b/docs/development/contribution/how_to_contribute_code.md @@ -89,11 +89,17 @@ For the further ### Run Zeppelin server in development mode +#### Option 1 - Command Line + +1. Copy the `conf/zeppelin-site.xml.template` to `zeppelin-server/src/main/resources/zeppelin-site.xml` and change the configurations in this file if required +2. Run the following command ``` cd zeppelin-server HADOOP_HOME=YOUR_HADOOP_HOME JAVA_HOME=YOUR_JAVA_HOME mvn exec:java -Dexec.mainClass="org.apache.zeppelin.server.ZeppelinServer" -Dexec.args="" ``` +#### Option 2 - Daemon Script + > **Note:** Make sure you first run ```mvn clean install -DskipTests``` on > your zeppelin root directory, otherwise your server build will fail to find > the required dependencies in the local repro. or use daemon script @@ -104,6 +110,12 @@ bin/zeppelin-daemon start Server will be run on [http://localhost:8080](http://localhost:8080). +#### Option 3 - IDE + +1. Copy the `conf/zeppelin-site.xml.template` to `zeppelin-server/src/main/resources/zeppelin-site.xml` and change the configurations in this file if required +2. `ZeppelinServer.java` Main class + + ### Generating Thrift Code Some portions of the Zeppelin code are generated by [Thrift](http://thrift.apache.org). For most Zeppelin changes, you don't need to worry about this. But if you modify any of the Thrift IDL files (e.g. zeppelin-interpreter/src/main/thrift/*.thrift), then you also need to regenerate these files and submit their updated version as part of your patch. http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/docs/setup/operation/configuration.md ---------------------------------------------------------------------- diff --git a/docs/setup/operation/configuration.md b/docs/setup/operation/configuration.md index 1f4c6a2..ed4e1f2 100644 --- a/docs/setup/operation/configuration.md +++ b/docs/setup/operation/configuration.md @@ -329,6 +329,30 @@ If both are defined, then the **environment variables** will take priority. <td>false</td> <td>Enable directory listings on server.</td> </tr> + <tr> + <td><h6 class="properties">ZEPPELIN_NOTEBOOK_GIT_REMOTE_URL</h6></td> + <td><h6 class="properties">zeppelin.notebook.git.remote.url</h6></td> + <td></td> + <td>GitHub's repository URL. It could be either the HTTP URL or the SSH URL. For example g...@github.com:apache/zeppelin.git</td> + </tr> + <tr> + <td><h6 class="properties">ZEPPELIN_NOTEBOOK_GIT_REMOTE_USERNAME</h6></td> + <td><h6 class="properties">zeppelin.notebook.git.remote.username</h6></td> + <td>token</td> + <td>GitHub username. By default it is `token` to use GitHub's API</td> + </tr> + <tr> + <td><h6 class="properties">ZEPPELIN_NOTEBOOK_GIT_REMOTE_ACCESS_TOKEN</h6></td> + <td><h6 class="properties">zeppelin.notebook.git.remote.access-token</h6></td> + <td>token</td> + <td>GitHub access token to use GitHub's API. If username/password combination is used and not GitHub API, then this value is the password</td> + </tr> + <tr> + <td><h6 class="properties">ZEPPELIN_NOTEBOOK_GIT_REMOTE_ORIGIN</h6></td> + <td><h6 class="properties">zeppelin.notebook.git.remote.origin</h6></td> + <td>token</td> + <td>GitHub remote name. Default is `origin`</td> + </tr> </table> @@ -431,7 +455,7 @@ The following properties needs to be updated in the `zeppelin-site.xml` in order ### Storing user credentials -In order to avoid having to re-enter credentials every time you restart/redeploy Zeppelin, you can store the user credentials. Zeppelin supports this via the ZEPPELIN_CREDENTIALS_PERSIST configuration. +In order to avoid having to re-enter credentials every time you restart/redeploy Zeppelin, you can store the user credentials. Zeppelin supports this via the ZEPPELIN_CREDENTIALS_PERSIST configuration. Please notice that passwords will be stored in *plain text* by default. To encrypt the passwords, use the ZEPPELIN_CREDENTIALS_ENCRYPT_KEY config variable. This will encrypt passwords using the AES-128 algorithm. @@ -473,5 +497,9 @@ update your configuration with the obfuscated password : </property> ``` +### Create GitHub Access Token + +When using GitHub to track notebooks, one can use GitHub's API for authentication. To create an access token, please use the following link https://github.com/settings/tokens. +The value of the access token generated is set in the `zeppelin.notebook.git.remote.access-token` property. **Note:** After updating these configurations, Zeppelin server needs to be restarted. http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/docs/setup/storage/storage.md ---------------------------------------------------------------------- diff --git a/docs/setup/storage/storage.md b/docs/setup/storage/storage.md index f6b8b5c..f34fc2c 100644 --- a/docs/setup/storage/storage.md +++ b/docs/setup/storage/storage.md @@ -34,6 +34,7 @@ There are few notebook storage systems available for a use out of the box: * storage using Amazon S3 service - `S3NotebookRepo` * storage using Azure service - `AzureNotebookRepo` * storage using MongoDB - `MongoNotebookRepo` + * storage using GitHub - `GitHubNotebookRepo` Multiple storage systems can be used at the same time by providing a comma-separated list of the class-names in the configuration. By default, only first two of them will be automatically kept in sync by Zeppelin. @@ -361,3 +362,44 @@ export ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT=true #### Import your local notes automatically By setting `ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT` as `true` (default `false`), you can import your local notes automatically when Zeppelin daemon starts up. This feature is for easy migration from local file system storage to MongoDB storage. A note with ID already existing in the collection will not be imported. + +## Notebook Storage in GitHub + +To enable GitHub tracking, uncomment the following properties in `zeppelin-site.xml` + +```sh +<property> + <name>zeppelin.notebook.git.remote.url</name> + <value></value> + <description>remote Git repository URL</description> +</property> + +<property> + <name>zeppelin.notebook.git.remote.username</name> + <value>token</value> + <description>remote Git repository username</description> +</property> + +<property> + <name>zeppelin.notebook.git.remote.access-token</name> + <value></value> + <description>remote Git repository password</description> +</property> + +<property> + <name>zeppelin.notebook.git.remote.origin</name> + <value>origin</value> + <description>Git repository remote</description> +</property> +``` + +And set the `zeppelin.notebook.storage` propery to `org.apache.zeppelin.notebook.repo.GitHubNotebookRepo` + +```sh +<property> + <name>zeppelin.notebook.storage</name> + <value>org.apache.zeppelin.notebook.repo.GitHubNotebookRepo</value> +</property> +``` + +The access token could be obtained by following the steps on this link https://github.com/settings/tokens. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java ---------------------------------------------------------------------- diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java index 6bce468..f7b3d7b 100644 --- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java +++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java @@ -561,6 +561,22 @@ public class ZeppelinConfiguration extends XMLConfiguration { return getString(ConfVars.ZEPPELIN_INTERPRETER_LIFECYCLE_MANAGER_CLASS); } + public String getZeppelinNotebookGitURL() { + return getString(ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_URL); + } + + public String getZeppelinNotebookGitUsername() { + return getString(ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_USERNAME); + } + + public String getZeppelinNotebookGitAccessToken() { + return getString(ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_ACCESS_TOKEN); + } + + public String getZeppelinNotebookGitRemoteOrigin() { + return getString(ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_ORIGIN); + } + public Map<String, String> dumpConfigurations(ZeppelinConfiguration conf, ConfigurationKeyPredicate predicate) { Map<String, String> configurations = new HashMap<>(); @@ -745,8 +761,12 @@ public class ZeppelinConfiguration extends XMLConfiguration { ZEPPELIN_INTERPRETER_LIFECYCLE_MANAGER_TIMEOUT_THRESHOLD( "zeppelin.interpreter.lifecyclemanager.timeout.threshold", 3600000L), - ZEPPELIN_OWNER_ROLE("zeppelin.notebook.default.owner.username", ""); + ZEPPELIN_OWNER_ROLE("zeppelin.notebook.default.owner.username", ""), + ZEPPELIN_NOTEBOOK_GIT_REMOTE_URL("zeppelin.notebook.git.remote.url", ""), + ZEPPELIN_NOTEBOOK_GIT_REMOTE_USERNAME("zeppelin.notebook.git.remote.username", "token"), + ZEPPELIN_NOTEBOOK_GIT_REMOTE_ACCESS_TOKEN("zeppelin.notebook.git.remote.access-token", ""), + ZEPPELIN_NOTEBOOK_GIT_REMOTE_ORIGIN("zeppelin.notebook.git.remote.origin", "origin"); private String varName; @SuppressWarnings("rawtypes") http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java ---------------------------------------------------------------------- diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java index 56aa50a..20d5ba9 100644 --- a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java +++ b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java @@ -824,8 +824,8 @@ public class NotebookServer extends WebSocketServlet String user = fromMessage.principal; Note note = notebook.getNote(noteId); - if (note != null) { + if (note != null) { if (!hasParagraphReaderPermission(conn, notebook, noteId, userAndRoles, fromMessage.principal, "read")) { return; http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-server/src/test/resources/2A94M5J1Z/note.json ---------------------------------------------------------------------- diff --git a/zeppelin-server/src/test/resources/2A94M5J1Z/note.json b/zeppelin-server/src/test/resources/2A94M5J1Z/note.json new file mode 100644 index 0000000..6e8e06f --- /dev/null +++ b/zeppelin-server/src/test/resources/2A94M5J1Z/note.json @@ -0,0 +1,376 @@ +{ + "paragraphs": [ + { + "text": "%md\n## Welcome to Zeppelin.\n##### This is a live tutorial, you can run the code yourself. (Shift-Enter to Run)", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:32:15 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false, + "keys": [], + "values": [], + "groups": [], + "scatter": {} + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eWelcome to Zeppelin.\u003c/h2\u003e\n\u003ch5\u003eThis is a live tutorial, you can run the code yourself. (Shift-Enter to Run)\u003c/h5\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836981412_-1007008116", + "id": "20150213-231621_168813393", + "dateCreated": "Feb 13, 2015 11:16:21 PM", + "dateStarted": "Dec 17, 2016 3:32:15 PM", + "dateFinished": "Dec 17, 2016 3:32:18 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "title": "Load data into table", + "text": "import org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\n\n// Zeppelin creates and injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)\n// So you don\u0027t need create them manually\n\n// load bank data\nval bankText \u003d sc.parallelize(\n IOUtils.toString(\n new URL(\"https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\"),\n Charset.forName(\"utf8\")).split(\"\\n\"))\n\ncase class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)\n\nval bank \u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e Bank(s(0).toInt, \n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n).toDF()\nbank.registerTempTable(\"bank\")", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:09 PM", + "config": { + "colWidth": 12.0, + "title": true, + "enabled": true, + "editorMode": "ace/mode/scala", + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "editorSetting": { + "language": "scala", + "editOnDblClick": false + } + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TEXT", + "data": "import org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\nbankText: org.apache.spark.rdd.RDD[String] \u003d ParallelCollectionRDD[36] at parallelize at \u003cconsole\u003e:43\ndefined class Bank\nbank: org.apache.spark.sql.DataFrame \u003d [age: int, job: string ... 3 more fields]\nwarning: there were 1 deprecation warning(s); re-run with -deprecation for details\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423500779206_-1502780787", + "id": "20150210-015259_1403135953", + "dateCreated": "Feb 10, 2015 1:52:59 AM", + "dateStarted": "Dec 17, 2016 3:30:09 PM", + "dateFinished": "Dec 17, 2016 3:30:58 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value\nfrom bank \nwhere age \u003c 30 \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:18:02 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "multiBarChart", + "height": 366.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423500782552_-1439281894", + "id": "20150210-015302_1492795503", + "dateCreated": "Feb 10, 2015 1:53:02 AM", + "dateStarted": "Dec 17, 2016 3:30:13 PM", + "dateFinished": "Dec 17, 2016 3:31:04 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere age \u003c ${maxAge\u003d30} \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:17:39 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "multiBarChart", + "height": 294.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": { + "maxAge": "35" + }, + "forms": { + "maxAge": { + "name": "maxAge", + "defaultValue": "30", + "hidden": false + } + } + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423720444030_-1424110477", + "id": "20150212-145404_867439529", + "dateCreated": "Feb 12, 2015 2:54:04 PM", + "dateStarted": "Dec 17, 2016 3:30:58 PM", + "dateFinished": "Dec 17, 2016 3:31:07 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere marital\u003d\"${marital\u003dsingle,single|divorced|married}\" \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:18:18 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "stackedAreaChart", + "height": 280.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": { + "marital": "single" + }, + "forms": { + "marital": { + "name": "marital", + "defaultValue": "single", + "options": [ + { + "value": "single" + }, + { + "value": "divorced" + }, + { + "value": "married" + } + ], + "hidden": false + } + } + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t17\n24\t13\n25\t33\n26\t56\n27\t64\n28\t78\n29\t56\n30\t92\n31\t86\n32\t105\n33\t61\n34\t75\n35\t46\n36\t50\n37\t43\n38\t44\n39\t30\n40\t25\n41\t19\n42\t23\n43\t21\n44\t20\n45\t15\n46\t14\n47\t12\n48\t12\n49\t11\n50\t8\n51\t6\n52\t9\n53\t4\n55\t3\n56\t3\n57\t2\n58\t7\n59\t2\n60\t5\n66\t2\n69\t1\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836262027_-210588283", + "id": "20150213-230422_1600658137", + "dateCreated": "Feb 13, 2015 11:04:22 PM", + "dateStarted": "Dec 17, 2016 3:31:05 PM", + "dateFinished": "Dec 17, 2016 3:31:09 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%md\n## Congratulations, it\u0027s done.\n##### You can create your own notebook in \u0027Notebook\u0027 menu. Good luck!", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:24 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eCongratulations, it\u0026rsquo;s done.\u003c/h2\u003e\n\u003ch5\u003eYou can create your own notebook in \u0026lsquo;Notebook\u0026rsquo; menu. Good luck!\u003c/h5\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836268492_216498320", + "id": "20150213-230428_1231780373", + "dateCreated": "Feb 13, 2015 11:04:28 PM", + "dateStarted": "Dec 17, 2016 3:30:24 PM", + "dateFinished": "Dec 17, 2016 3:30:29 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%md\n\nAbout bank data\n\n```\nCitation Request:\n This dataset is public available for research. The details are described in [Moro et al., 2011]. \n Please include this citation if you plan to use this database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European Simulation and Modelling Conference - ESM\u00272011, pp. 117-121, Guimarães, Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf] http://hdl.handle.net/1822/14838\n [bib] http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n```", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:34 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eAbout bank data\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003eCitation Request:\n This dataset is public available for research. The details are described in [Moro et al., 2011]. \n Please include this citation if you plan to use this database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European Simulation and Modelling Conference - ESM\u0026#39;2011, pp. 117-121, Guimarães, Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf] http://hdl.handle.net/1822/14838\n [bib] http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1427420818407_872443482", + "id": "20150326-214658_12335843", + "dateCreated": "Mar 26, 2015 9:46:58 PM", + "dateStarted": "Dec 17, 2016 3:30:34 PM", + "dateFinished": "Dec 17, 2016 3:30:34 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "config": {}, + "settings": { + "params": {}, + "forms": {} + }, + "apps": [], + "jobName": "paragraph_1435955447812_-158639899", + "id": "20150703-133047_853701097", + "dateCreated": "Jul 3, 2015 1:30:47 PM", + "status": "READY", + "progressUpdateIntervalMs": 500 + } + ], + "name": "Zeppelin Tutorial/Basic Features (Spark)", + "id": "2A94M5J1Z", + "angularObjects": { + "2C73DY9P9:shared_process": [] + }, + "config": { + "looknfeel": "default" + }, + "info": {} +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-server/src/test/resources/2A94M5J2Z/note.json ---------------------------------------------------------------------- diff --git a/zeppelin-server/src/test/resources/2A94M5J2Z/note.json b/zeppelin-server/src/test/resources/2A94M5J2Z/note.json new file mode 100644 index 0000000..dd9a74d --- /dev/null +++ b/zeppelin-server/src/test/resources/2A94M5J2Z/note.json @@ -0,0 +1,376 @@ +{ + "paragraphs": [ + { + "text": "%md\n## Welcome to Zeppelin.\n##### This is a live tutorial, you can run the code yourself. (Shift-Enter to Run)", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:32:15 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false, + "keys": [], + "values": [], + "groups": [], + "scatter": {} + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eWelcome to Zeppelin.\u003c/h2\u003e\n\u003ch5\u003eThis is a live tutorial, you can run the code yourself. (Shift-Enter to Run)\u003c/h5\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836981412_-1007008116", + "id": "20150213-231621_168813393", + "dateCreated": "Feb 13, 2015 11:16:21 PM", + "dateStarted": "Dec 17, 2016 3:32:15 PM", + "dateFinished": "Dec 17, 2016 3:32:18 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "title": "Load data into table", + "text": "import org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\n\n// Zeppelin creates and injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)\n// So you don\u0027t need create them manually\n\n// load bank data\nval bankText \u003d sc.parallelize(\n IOUtils.toString(\n new URL(\"https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\"),\n Charset.forName(\"utf8\")).split(\"\\n\"))\n\ncase class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)\n\nval bank \u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e Bank(s(0).toInt, \n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n).toDF()\nbank.registerTempTable(\"bank\")", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:09 PM", + "config": { + "colWidth": 12.0, + "title": true, + "enabled": true, + "editorMode": "ace/mode/scala", + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "editorSetting": { + "language": "scala", + "editOnDblClick": false + } + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TEXT", + "data": "import org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\nbankText: org.apache.spark.rdd.RDD[String] \u003d ParallelCollectionRDD[36] at parallelize at \u003cconsole\u003e:43\ndefined class Bank\nbank: org.apache.spark.sql.DataFrame \u003d [age: int, job: string ... 3 more fields]\nwarning: there were 1 deprecation warning(s); re-run with -deprecation for details\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423500779206_-1502780787", + "id": "20150210-015259_1403135953", + "dateCreated": "Feb 10, 2015 1:52:59 AM", + "dateStarted": "Dec 17, 2016 3:30:09 PM", + "dateFinished": "Dec 17, 2016 3:30:58 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value\nfrom bank \nwhere age \u003c 30 \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:18:02 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "multiBarChart", + "height": 366.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423500782552_-1439281894", + "id": "20150210-015302_1492795503", + "dateCreated": "Feb 10, 2015 1:53:02 AM", + "dateStarted": "Dec 17, 2016 3:30:13 PM", + "dateFinished": "Dec 17, 2016 3:31:04 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere age \u003c ${maxAge\u003d30} \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:17:39 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "multiBarChart", + "height": 294.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": { + "maxAge": "35" + }, + "forms": { + "maxAge": { + "name": "maxAge", + "defaultValue": "30", + "hidden": false + } + } + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423720444030_-1424110477", + "id": "20150212-145404_867439529", + "dateCreated": "Feb 12, 2015 2:54:04 PM", + "dateStarted": "Dec 17, 2016 3:30:58 PM", + "dateFinished": "Dec 17, 2016 3:31:07 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%sql \nselect age, count(1) value \nfrom bank \nwhere marital\u003d\"${marital\u003dsingle,single|divorced|married}\" \ngroup by age \norder by age", + "user": "anonymous", + "dateUpdated": "Mar 17, 2017 12:18:18 PM", + "config": { + "colWidth": 4.0, + "results": [ + { + "graph": { + "mode": "stackedAreaChart", + "height": 280.0, + "optionOpen": false + }, + "helium": {} + } + ], + "enabled": true, + "editorSetting": { + "language": "sql", + "editOnDblClick": false + }, + "editorMode": "ace/mode/sql" + }, + "settings": { + "params": { + "marital": "single" + }, + "forms": { + "marital": { + "name": "marital", + "defaultValue": "single", + "options": [ + { + "value": "single" + }, + { + "value": "divorced" + }, + { + "value": "married" + } + ], + "hidden": false + } + } + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "TABLE", + "data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t17\n24\t13\n25\t33\n26\t56\n27\t64\n28\t78\n29\t56\n30\t92\n31\t86\n32\t105\n33\t61\n34\t75\n35\t46\n36\t50\n37\t43\n38\t44\n39\t30\n40\t25\n41\t19\n42\t23\n43\t21\n44\t20\n45\t15\n46\t14\n47\t12\n48\t12\n49\t11\n50\t8\n51\t6\n52\t9\n53\t4\n55\t3\n56\t3\n57\t2\n58\t7\n59\t2\n60\t5\n66\t2\n69\t1\n" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836262027_-210588283", + "id": "20150213-230422_1600658137", + "dateCreated": "Feb 13, 2015 11:04:22 PM", + "dateStarted": "Dec 17, 2016 3:31:05 PM", + "dateFinished": "Dec 17, 2016 3:31:09 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%md\n## Congratulations, it\u0027s done.\n##### You can create your own notebook in \u0027Notebook\u0027 menu. Good luck!", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:24 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch2\u003eCongratulations, it\u0026rsquo;s done.\u003c/h2\u003e\n\u003ch5\u003eYou can create your own notebook in \u0026lsquo;Notebook\u0026rsquo; menu. Good luck!\u003c/h5\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1423836268492_216498320", + "id": "20150213-230428_1231780373", + "dateCreated": "Feb 13, 2015 11:04:28 PM", + "dateStarted": "Dec 17, 2016 3:30:24 PM", + "dateFinished": "Dec 17, 2016 3:30:29 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "text": "%md\n\nAbout bank data\n\n```\nCitation Request:\n This dataset is public available for research. The details are described in [Moro et al., 2011]. \n Please include this citation if you plan to use this database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European Simulation and Modelling Conference - ESM\u00272011, pp. 117-121, Guimarães, Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf] http://hdl.handle.net/1822/14838\n [bib] http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n```", + "user": "anonymous", + "dateUpdated": "Dec 17, 2016 3:30:34 PM", + "config": { + "colWidth": 12.0, + "editorHide": true, + "results": [ + { + "graph": { + "mode": "table", + "height": 300.0, + "optionOpen": false + } + } + ], + "enabled": true, + "editorSetting": { + "language": "markdown", + "editOnDblClick": true + }, + "editorMode": "ace/mode/markdown", + "tableHide": false + }, + "settings": { + "params": {}, + "forms": {} + }, + "results": { + "code": "SUCCESS", + "msg": [ + { + "type": "HTML", + "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eAbout bank data\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003eCitation Request:\n This dataset is public available for research. The details are described in [Moro et al., 2011]. \n Please include this citation if you plan to use this database:\n\n [Moro et al., 2011] S. Moro, R. Laureano and P. Cortez. Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology. \n In P. Novais et al. (Eds.), Proceedings of the European Simulation and Modelling Conference - ESM\u0026#39;2011, pp. 117-121, Guimarães, Portugal, October, 2011. EUROSIS.\n\n Available at: [pdf] http://hdl.handle.net/1822/14838\n [bib] http://www3.dsi.uminho.pt/pcortez/bib/2011-esm-1.txt\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/div\u003e" + } + ] + }, + "apps": [], + "jobName": "paragraph_1427420818407_872443482", + "id": "20150326-214658_12335843", + "dateCreated": "Mar 26, 2015 9:46:58 PM", + "dateStarted": "Dec 17, 2016 3:30:34 PM", + "dateFinished": "Dec 17, 2016 3:30:34 PM", + "status": "FINISHED", + "progressUpdateIntervalMs": 500 + }, + { + "config": {}, + "settings": { + "params": {}, + "forms": {} + }, + "apps": [], + "jobName": "paragraph_1435955447812_-158639899", + "id": "20150703-133047_853701097", + "dateCreated": "Jul 3, 2015 1:30:47 PM", + "status": "READY", + "progressUpdateIntervalMs": 500 + } + ], + "name": "Zeppelin Tutorial/Basic Features (Spark)", + "id": "2A94M5J2Z", + "angularObjects": { + "2C73DY9P9:shared_process": [] + }, + "config": { + "looknfeel": "default" + }, + "info": {} +} http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java index ff0ac62..72ea2ac 100644 --- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java @@ -17,42 +17,16 @@ package org.apache.zeppelin.notebook; -import java.io.IOException; -import java.util.ArrayList; -import java.util.Collections; -import java.util.Comparator; -import java.util.Date; -import java.util.HashMap; -import java.util.LinkedHashMap; -import java.util.LinkedList; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.TimeUnit; - import com.google.common.base.Preconditions; import com.google.common.base.Predicate; import com.google.common.collect.FluentIterable; import com.google.common.collect.Sets; -import org.apache.zeppelin.interpreter.*; -import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry; -import org.quartz.CronScheduleBuilder; -import org.quartz.CronTrigger; -import org.quartz.JobBuilder; -import org.quartz.JobDetail; -import org.quartz.JobExecutionContext; -import org.quartz.JobExecutionException; -import org.quartz.JobKey; -import org.quartz.SchedulerException; -import org.quartz.TriggerBuilder; -import org.quartz.impl.StdSchedulerFactory; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - import org.apache.zeppelin.conf.ZeppelinConfiguration; import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars; import org.apache.zeppelin.display.AngularObject; import org.apache.zeppelin.display.AngularObjectRegistry; +import org.apache.zeppelin.interpreter.*; +import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry; import org.apache.zeppelin.notebook.repo.NotebookRepo; import org.apache.zeppelin.notebook.repo.NotebookRepo.Revision; import org.apache.zeppelin.notebook.repo.NotebookRepoSync; @@ -61,6 +35,14 @@ import org.apache.zeppelin.scheduler.SchedulerFactory; import org.apache.zeppelin.search.SearchService; import org.apache.zeppelin.user.AuthenticationInfo; import org.apache.zeppelin.user.Credentials; +import org.quartz.*; +import org.quartz.impl.StdSchedulerFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.*; +import java.util.concurrent.TimeUnit; /** * Collection of Notes. http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepo.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepo.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepo.java new file mode 100644 index 0000000..6052e5f --- /dev/null +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepo.java @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zeppelin.notebook.repo; + +import org.apache.zeppelin.conf.ZeppelinConfiguration; +import org.apache.zeppelin.user.AuthenticationInfo; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.PullCommand; +import org.eclipse.jgit.api.PushCommand; +import org.eclipse.jgit.api.RemoteAddCommand; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.transport.URIish; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.net.URISyntaxException; + +/** + * GitHub integration to store notebooks in a GitHub repository. + * It uses the same simple logic implemented in @see + * {@link org.apache.zeppelin.notebook.repo.GitNotebookRepo} + * + * The logic for updating the local repository from the remote repository is the following: + * - When the <code>GitHubNotebookRepo</code> is initialized + * - When pushing the changes to the remote repository + * + * The logic for updating the remote repository on GitHub from local repository is the following: + * - When commit the changes (saving the notebook) + */ +public class GitHubNotebookRepo extends GitNotebookRepo { + private static final Logger LOG = LoggerFactory.getLogger(GitNotebookRepo.class); + private ZeppelinConfiguration zeppelinConfiguration; + private Git git; + + public GitHubNotebookRepo(ZeppelinConfiguration conf) throws IOException { + super(conf); + + this.git = super.getGit(); + this.zeppelinConfiguration = conf; + + configureRemoteStream(); + pullFromRemoteStream(); + } + + @Override + public Revision checkpoint(String pattern, String commitMessage, AuthenticationInfo subject) { + Revision revision = super.checkpoint(pattern, commitMessage, subject); + + updateRemoteStream(); + + return revision; + } + + private void configureRemoteStream() { + try { + LOG.debug("Setting up remote stream"); + RemoteAddCommand remoteAddCommand = git.remoteAdd(); + remoteAddCommand.setName(zeppelinConfiguration.getZeppelinNotebookGitRemoteOrigin()); + remoteAddCommand.setUri(new URIish(zeppelinConfiguration.getZeppelinNotebookGitURL())); + remoteAddCommand.call(); + } catch (GitAPIException e) { + LOG.error("Error configuring GitHub", e); + } catch (URISyntaxException e) { + LOG.error("Error in GitHub URL provided", e); + } + } + + private void updateRemoteStream() { + LOG.debug("Updating remote stream"); + + pullFromRemoteStream(); + pushToRemoteSteam(); + } + + private void pullFromRemoteStream() { + try { + LOG.debug("Pull latest changed from remote stream"); + PullCommand pullCommand = git.pull(); + pullCommand.setCredentialsProvider( + new UsernamePasswordCredentialsProvider( + zeppelinConfiguration.getZeppelinNotebookGitUsername(), + zeppelinConfiguration.getZeppelinNotebookGitAccessToken() + ) + ); + + pullCommand.call(); + + } catch (GitAPIException e) { + LOG.error("Error when pulling latest changes from remote repository", e); + } + } + + private void pushToRemoteSteam() { + try { + LOG.debug("Push latest changed from remote stream"); + PushCommand pushCommand = git.push(); + pushCommand.setCredentialsProvider( + new UsernamePasswordCredentialsProvider( + zeppelinConfiguration.getZeppelinNotebookGitUsername(), + zeppelinConfiguration.getZeppelinNotebookGitAccessToken() + ) + ); + + pushCommand.call(); + } catch (GitAPIException e) { + LOG.error("Error when pushing latest changes from remote repository", e); + } + } +} http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java index 21183da..2ac4c72 100644 --- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java +++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java @@ -47,7 +47,8 @@ import com.google.common.collect.Lists; * * This impl intended to be simple and straightforward: * - does not handle branches - * - only basic local git file repo, no remote Github push\pull yet + * - only basic local git file repo, no remote Github push\pull. GitHub integration is + * implemented in @see {@link org.apache.zeppelin.notebook.repo.GitHubNotebookRepo} * * TODO(bzz): add default .gitignore */ @@ -177,7 +178,7 @@ public class GitNotebookRepo extends VFSNotebookRepo { } //DI replacements for Tests - Git getGit() { + protected Git getGit() { return git; } http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java new file mode 100644 index 0000000..49a5cbd --- /dev/null +++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java @@ -0,0 +1,207 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zeppelin.notebook.repo; + + +import com.google.common.base.Joiner; +import org.apache.commons.io.FileUtils; +import org.apache.zeppelin.conf.ZeppelinConfiguration; +import org.apache.zeppelin.interpreter.InterpreterFactory; +import org.apache.zeppelin.notebook.Note; +import org.apache.zeppelin.notebook.Paragraph; +import org.apache.zeppelin.user.AuthenticationInfo; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.internal.storage.file.FileRepository; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.util.Iterator; + +import static org.mockito.Mockito.mock; + +/** + * This tests the remote Git tracking for notebooks. The tests uses two local Git repositories created locally + * to handle the tracking of Git actions (pushes and pulls). The repositories are: + * 1. The first repository is considered as a remote that mimics a remote GitHub directory + * 2. The second repository is considered as the local notebook repository + */ +public class GitHubNotebookRepoTest { + private static final Logger LOG = LoggerFactory.getLogger(GitNotebookRepoTest.class); + + private static final String TEST_NOTE_ID = "2A94M5J1Z"; + + private File remoteZeppelinDir; + private File localZeppelinDir; + private String localNotebooksDir; + private String remoteNotebooksDir; + private ZeppelinConfiguration conf; + private GitHubNotebookRepo gitHubNotebookRepo; + private RevCommit firstCommitRevision; + private Git remoteGit; + + @Before + public void setUp() throws Exception { + conf = ZeppelinConfiguration.create(); + + String remoteRepositoryPath = System.getProperty("java.io.tmpdir") + "/ZeppelinTestRemote_" + + System.currentTimeMillis(); + String localRepositoryPath = System.getProperty("java.io.tmpdir") + "/ZeppelinTest_" + + System.currentTimeMillis(); + + // Create a fake remote notebook Git repository locally in another directory + remoteZeppelinDir = new File(remoteRepositoryPath); + remoteZeppelinDir.mkdirs(); + + // Create a local repository for notebooks + localZeppelinDir = new File(localRepositoryPath); + localZeppelinDir.mkdirs(); + + // Notebooks directory (for both the remote and local directories) + localNotebooksDir = Joiner.on(File.separator).join(localRepositoryPath, "notebook"); + remoteNotebooksDir = Joiner.on(File.separator).join(remoteRepositoryPath, "notebook"); + + File notebookDir = new File(localNotebooksDir); + notebookDir.mkdirs(); + + // Copy the test notebook directory from the test/resources/2A94M5J1Z folder to the fake remote Git directory + String remoteTestNoteDir = Joiner.on(File.separator).join(remoteNotebooksDir, TEST_NOTE_ID); + FileUtils.copyDirectory( + new File( + GitHubNotebookRepoTest.class.getResource( + Joiner.on(File.separator).join("", TEST_NOTE_ID) + ).getFile() + ), new File(remoteTestNoteDir) + ); + + // Create the fake remote Git repository + Repository remoteRepository = new FileRepository(Joiner.on(File.separator).join(remoteNotebooksDir, ".git")); + remoteRepository.create(); + + remoteGit = new Git(remoteRepository); + remoteGit.add().addFilepattern(".").call(); + firstCommitRevision = remoteGit.commit().setMessage("First commit from remote repository").call(); + + // Set the Git and Git configurations + System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME.getVarName(), remoteZeppelinDir.getAbsolutePath()); + System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_DIR.getVarName(), notebookDir.getAbsolutePath()); + + // Set the GitHub configurations + System.setProperty( + ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_STORAGE.getVarName(), + "org.apache.zeppelin.notebook.repo.GitHubNotebookRepo"); + System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_URL.getVarName(), + remoteNotebooksDir + File.separator + ".git"); + System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_USERNAME.getVarName(), "token"); + System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_GIT_REMOTE_ACCESS_TOKEN.getVarName(), + "access-token"); + + // Create the Notebook repository (configured for the local repository) + gitHubNotebookRepo = new GitHubNotebookRepo(conf); + } + + @After + public void tearDown() throws Exception { + // Cleanup the temporary folders uses as Git repositories + File[] temporaryFolders = { remoteZeppelinDir, localZeppelinDir }; + + for(File temporaryFolder : temporaryFolders) { + if (!FileUtils.deleteQuietly(temporaryFolder)) + LOG.error("Failed to delete {} ", temporaryFolder.getName()); + } + } + + @Test + /** + * Test the case when the Notebook repository is created, it pulls the latest changes from the remote repository + */ + public void pullChangesFromRemoteRepositoryOnLoadingNotebook() throws IOException, GitAPIException { + NotebookRepo.Revision firstHistoryRevision = gitHubNotebookRepo.revisionHistory(TEST_NOTE_ID, null).get(0); + + assert(this.firstCommitRevision.getName().equals(firstHistoryRevision.id)); + } + + @Test + /** + * Test the case when the check-pointing (add new files and commit) it also pulls the latest changes from the + * remote repository + */ + public void pullChangesFromRemoteRepositoryOnCheckpointing() throws GitAPIException, IOException { + // Create a new commit in the remote repository + RevCommit secondCommitRevision = remoteGit.commit().setMessage("Second commit from remote repository").call(); + + // Add a new paragraph to the local repository + addParagraphToNotebook(TEST_NOTE_ID); + + // Commit and push the changes to remote repository + NotebookRepo.Revision thirdCommitRevision = gitHubNotebookRepo.checkpoint( + TEST_NOTE_ID, "Third commit from local repository", null); + + // Check all the commits as seen from the local repository. The commits are ordered chronologically. The last + // commit is the first in the commit logs. + Iterator<RevCommit> revisions = gitHubNotebookRepo.getGit().log().all().call().iterator(); + + revisions.next(); // The Merge `master` commit after pushing to the remote repository + + assert(thirdCommitRevision.id.equals(revisions.next().getName())); // The local commit after adding the paragraph + + // The second commit done on the remote repository + assert(secondCommitRevision.getName().equals(revisions.next().getName())); + + // The first commit done on the remote repository + assert(firstCommitRevision.getName().equals(revisions.next().getName())); + } + + @Test + /** + * Test the case when the check-pointing (add new files and commit) it pushes the local commits to the remote + * repository + */ + public void pushLocalChangesToRemoteRepositoryOnCheckpointing() throws IOException, GitAPIException { + // Add a new paragraph to the local repository + addParagraphToNotebook(TEST_NOTE_ID); + + // Commit and push the changes to remote repository + NotebookRepo.Revision secondCommitRevision = gitHubNotebookRepo.checkpoint( + TEST_NOTE_ID, "Second commit from local repository", null); + + // Check all the commits as seen from the remote repository. The commits are ordered chronologically. The last + // commit is the first in the commit logs. + Iterator<RevCommit> revisions = remoteGit.log().all().call().iterator(); + + assert(secondCommitRevision.id.equals(revisions.next().getName())); // The local commit after adding the paragraph + + // The first commit done on the remote repository + assert(firstCommitRevision.getName().equals(revisions.next().getName())); + } + + private void addParagraphToNotebook(String noteId) throws IOException { + Note note = gitHubNotebookRepo.get(TEST_NOTE_ID, null); + note.setInterpreterFactory(mock(InterpreterFactory.class)); + Paragraph paragraph = note.addNewParagraph(AuthenticationInfo.ANONYMOUS); + paragraph.setText("%md text"); + gitHubNotebookRepo.save(note, null); + } +} http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2be8f350/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java ---------------------------------------------------------------------- diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java index 2276c25..72ea439 100644 --- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java +++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/GitNotebookRepoTest.java @@ -70,9 +70,19 @@ public class GitNotebookRepoTest { String testNoteDir = Joiner.on(File.separator).join(notebooksDir, TEST_NOTE_ID); String testNoteDir2 = Joiner.on(File.separator).join(notebooksDir, TEST_NOTE_ID2); - FileUtils.copyDirectory(new File(Joiner.on(File.separator).join("src", "test", "resources", TEST_NOTE_ID)), - new File(testNoteDir)); - FileUtils.copyDirectory(new File(Joiner.on(File.separator).join("src", "test", "resources", TEST_NOTE_ID2)), + FileUtils.copyDirectory( + new File( + GitHubNotebookRepoTest.class.getResource( + Joiner.on(File.separator).join("", TEST_NOTE_ID) + ).getFile() + ), + new File(testNoteDir)); + FileUtils.copyDirectory( + new File( + GitHubNotebookRepoTest.class.getResource( + Joiner.on(File.separator).join("", TEST_NOTE_ID2) + ).getFile() + ), new File(testNoteDir2) );