Repository: zeppelin Updated Branches: refs/heads/branch-0.8 29cf9186c -> 9cf254218
[Zeppelin 3388] Refactor documentation for ZeppelinContext ### What is this PR for? This PR is a refactoring of the `ZepplinContext` documentation. The description of `ZepplinContext` is now almost entirely contained within the _Spark_ interpreter's documentation ([spark.md](https://github.com/apache/zeppelin/blob/master/docs/interpreter/spark.md)). But `ZepplinContext` has many generic features that are available to all interpreters, and it is important for `ZepplinContext` to have a more visible and independent presence in the Zeppelin documentation. ### What type of PR is it? [Documentation] ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3388 ### How should this be tested? Only documentation. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No (This _is_ documentation) Author: Sanjay Dasgupta <sanjay.dasgu...@gmail.com> Closes #2953 from sanjaydasgupta/zeppelin-3388 and squashes the following commits: bb8aa0c [Sanjay Dasgupta] Changes to address Felix Cheung's review https://github.com/apache/zeppelin/pull/2953#pullrequestreview-116196552 7360b50 [Sanjay Dasgupta] Correcting deleted hyperlink urls 64e4d7e [Sanjay Dasgupta] Initial version of files for 3388 (cherry picked from commit 0c3260e91f4c1ec58e67856d466ea9cba89d0085) Signed-off-by: Jeff Zhang <zjf...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/9cf25421 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/9cf25421 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/9cf25421 Branch: refs/heads/branch-0.8 Commit: 9cf25421812e89c4ba7ecc9e6e53ff65c95dbe12 Parents: 29cf918 Author: Sanjay Dasgupta <sanjay.dasgu...@gmail.com> Authored: Mon Apr 30 16:11:26 2018 +0530 Committer: Jeff Zhang <zjf...@apache.org> Committed: Wed May 2 15:03:28 2018 +0800 ---------------------------------------------------------------------- docs/_includes/themes/zeppelin/_navigation.html | 2 + docs/index.md | 2 + docs/interpreter/shell.md | 4 +- docs/interpreter/spark.md | 154 +----------- docs/usage/other_features/zeppelin_context.md | 233 +++++++++++++++++++ 5 files changed, 241 insertions(+), 154 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9cf25421/docs/_includes/themes/zeppelin/_navigation.html ---------------------------------------------------------------------- diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html index 796b8bc..24b3cde 100644 --- a/docs/_includes/themes/zeppelin/_navigation.html +++ b/docs/_includes/themes/zeppelin/_navigation.html @@ -63,6 +63,7 @@ <li><a href="{{BASE_PATH}}/usage/other_features/customizing_homepage.html">Customizing Zeppelin Homepage</a></li> <li><a href="{{BASE_PATH}}/usage/other_features/notebook_actions.html">Notebook Actions</a></li> <li><a href="{{BASE_PATH}}/usage/other_features/cron_scheduler.html">Cron Scheduler</a></li> + <li><a href="{{BASE_PATH}}/usage/other_features/zeppelin_context.html">Zeppelin Context</a></li> <li role="separator" class="divider"></li> <li class="title"><span>REST API</span></li> <li><a href="{{BASE_PATH}}/usage/rest_api/interpreter.html">Interpreter API</a></li> @@ -182,3 +183,4 @@ </nav><!--/.navbar-collapse --> </div> </div> + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9cf25421/docs/index.md ---------------------------------------------------------------------- diff --git a/docs/index.md b/docs/index.md index f00571e..788a979 100644 --- a/docs/index.md +++ b/docs/index.md @@ -74,6 +74,7 @@ limitations under the License. * [Personalized Mode](./usage/other_features/personalized_mode.html) * [Customizing Zeppelin Homepage](./usage/other_features/customizing_homepage.html) with one of your notebooks * [Notebook actions](./usage/other_features/notebook_actions.html) + * [Zeppelin-Context](./usage/other_features/zeppelin_context.html) * REST API: available REST API list in Apache Zeppelin * [Interpreter API](./usage/rest_api/interpreter.html) * [Zeppelin Server API](./usage/rest_api/zeppelin_server.html) @@ -160,3 +161,4 @@ limitations under the License. * [Mailing List](https://zeppelin.apache.org/community.html) * [Apache Zeppelin Wiki](https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home) * [Stackoverflow Questions about Zeppelin (tag: `apache-zeppelin`)](http://stackoverflow.com/questions/tagged/apache-zeppelin) + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9cf25421/docs/interpreter/shell.md ---------------------------------------------------------------------- diff --git a/docs/interpreter/shell.md b/docs/interpreter/shell.md index 25349aa..9ab4036 100644 --- a/docs/interpreter/shell.md +++ b/docs/interpreter/shell.md @@ -107,5 +107,5 @@ val members = spark.read.parquet(z.get("dataFileName")) Object interpolation is disabled by default, and can be enabled (for the Shell interpreter) by setting the value of the property `zeppelin.shell.interpolation` to `true` (see _Configuration_ above). -More details of this feature can be found in the Spark interpreter documentation under -[Object Interpolation](spark.html#object-interpolation) +More details of this feature can be found in [Zeppelin-Context](../usage/other_features/zeppelin_context.html) + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9cf25421/docs/interpreter/spark.md ---------------------------------------------------------------------- diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md index 3999e3a..85b2981 100644 --- a/docs/interpreter/spark.md +++ b/docs/interpreter/spark.md @@ -323,158 +323,7 @@ z.load("groupId:artifactId:version").local() ## ZeppelinContext Zeppelin automatically injects `ZeppelinContext` as variable `z` in your Scala/Python environment. `ZeppelinContext` provides some additional functions and utilities. - -### Exploring Spark DataFrames -`ZeppelinContext` provides a `show` method, which, using Zeppelin's `table` feature, can be used to nicely display a Spark DataFrame: - -``` -df = spark.read.csv('/path/to/csv') -z.show(df) -``` - -### Object Exchange -`ZeppelinContext` extends map and it's shared between Scala and Python environment. -So you can put some objects from Scala and read it from Python, vice versa. - -<div class="codetabs"> - <div data-lang="scala" markdown="1"> - -{% highlight scala %} -// Put object from scala -%spark -val myObject = ... -z.put("objName", myObject) - -// Exchanging data frames -myScalaDataFrame = ... -z.put("myScalaDataFrame", myScalaDataFrame) - -val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame] -{% endhighlight %} - - </div> - <div data-lang="python" markdown="1"> - -{% highlight python %} -# Get object from python -%spark.pyspark -myObject = z.get("objName") - -# Exchanging data frames -myPythonDataFrame = ... -z.put("myPythonDataFrame", postsDf._jdf) - -myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext) -{% endhighlight %} - - </div> -</div> - -### Object Interpolation -Some interpreters can interpolate object values from `z` into the paragraph text by using the -`{variable-name}` syntax. The value of any object previously `put` into `z` can be -interpolated into a paragraph text by using such a pattern containing the object's name. -The following example shows one use of this facility: - -####In Scala cell: -``` -z.put("minAge", 35) -``` - -####In later SQL cell: -``` -%sql select * from members where age >= {minAge} -``` - -The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name. -But the pattern is left unchanged if the named object does not exist in `z`. -Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur -- -translation of only some of the patterns in a paragraph text is never done. - -In some situations, it is necessary to use { and } characters in a paragraph text without invoking the -object interpolation mechanism. For these cases an escaping mechanism is available -- -doubled braces {{ and }} should be used. The following example shows the use of {{ and }} for passing a -regular expression containing just { and } into the paragraph text. - -``` -%sql select * from members where name rlike '[aeiou]{{3}}' -``` - -To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefined -object of the specified name exists. Additionally, all such patterns within the paragraph text should also -be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`. -These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conform -to one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way -(than `{var-name}` and `{{any-text}}`) is used as-is without any changes. -No error is flagged in any case. This behavior is identical to the implementation of a similar feature in -Jupyter's shell invocation using the `!` magic command. - -This feature is disabled by default, and must be explicitly turned on for each interpreter independently -by setting the value of an interpreter-specific property to `true`. -Consult the _Configuration_ section of each interpreter's documentation -to find out if object interpolation is implemented, and the name of the parameter that must be set to `true` to -enable the feature. The name of the parameter used to enable this feature it is different for each interpreter. -For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and -`zeppelin.shell.interpolation` respectively. - -At present only the SparkSQL and Shell interpreters support object interpolation. - -### Form Creation - -`ZeppelinContext` provides functions for creating forms. -In Scala and Python environments, you can create forms programmatically. -<div class="codetabs"> - <div data-lang="scala" markdown="1"> - -{% highlight scala %} -%spark -/* Create text input form */ -z.input("formName") - -/* Create text input form with default value */ -z.input("formName", "defaultValue") - -/* Create select form */ -z.select("formName", Seq(("option1", "option1DisplayName"), - ("option2", "option2DisplayName"))) - -/* Create select form with default value*/ -z.select("formName", "option1", Seq(("option1", "option1DisplayName"), - ("option2", "option2DisplayName"))) -{% endhighlight %} - - </div> - <div data-lang="python" markdown="1"> - -{% highlight python %} -%spark.pyspark -# Create text input form -z.input("formName") - -# Create text input form with default value -z.input("formName", "defaultValue") - -# Create select form -z.select("formName", [("option1", "option1DisplayName"), - ("option2", "option2DisplayName")]) - -# Create select form with default value -z.select("formName", [("option1", "option1DisplayName"), - ("option2", "option2DisplayName")], "option1") -{% endhighlight %} - - </div> -</div> - -In sql environment, you can create form in simple template. - -```sql -%spark.sql -select * from ${table=defaultTableName} where text like '%${search}%' -``` - -To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html). - +See [Zeppelin-Context](../usage/other_features/zeppelin_context.html) for more details. ## Matplotlib Integration (pyspark) Both the `python` and `pyspark` interpreters have built-in support for inline visualization using `matplotlib`, @@ -518,3 +367,4 @@ This is to make the server communicate with KDC. > **NOTE:** If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the Spark Interpreter setting through the Interpreter tab in the Zeppelin UI. 4. That's it. Play with Zeppelin! + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9cf25421/docs/usage/other_features/zeppelin_context.md ---------------------------------------------------------------------- diff --git a/docs/usage/other_features/zeppelin_context.md b/docs/usage/other_features/zeppelin_context.md new file mode 100644 index 0000000..5a77727 --- /dev/null +++ b/docs/usage/other_features/zeppelin_context.md @@ -0,0 +1,233 @@ +--- +layout: page +title: "Zeppelin-Context" +description: "The Zeppelin-Context is a system-wide container for a variety of user-specific settings and parameters that are accessible across notebooks, cells, and interpreters." +group: usage/other_features +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +{% include JB/setup %} + +# Zeppelin-Context + +<div id="toc"></div> + +The zeppelin-context is a system-wide container for common utility functions and +user-specific data. It implements functions for data input, data display, etc. that are +often needed but are not uniformly available in all interpreters. +Its single per-user instance is accessible across all of the user's notebooks and cells, +enabling data exchange between cells - even in different notebooks. +But the way in which the zeppelin-context is used, and the functionality available differs +depending on whether or not the associated interpreter is based on a programming language. +Details of how the zeppelin-context is used for different purposes and in different +environments is described below. + +## Usage in Programming Language Cells + +In many programming-language interpreters (e.g. Apache Spark, Python, R) the zeppelin-context is available +as a predefined variable `z` that can be used by directly invoking its methods. +The methods available on the `z` object are described below. +Other interpreters based on programming languages like spark.dep, Apache Beam, etc. also provide the +predefined variable `z`. + +### Exploring Spark DataFrames +In the Apache Spark interpreter, the zeppelin-context provides a `show` method, which, +using Zeppelin's `table` feature, can be used to nicely display a Spark DataFrame: + +``` +df = spark.read.csv('/path/to/csv') +z.show(df) +``` + +This display functionality using the `show` method is planned to be extended uniformly to +other interpreters that can access the `z` object. + +### Object Exchange +`ZeppelinContext` extends map and it's shared between the Apache Spark and Python environments. +So you can put some objects using Scala (in an Apache Spark cell) and read it from Python, and vice versa. + +<div class="codetabs"> + <div data-lang="scala" markdown="1"> + +{% highlight scala %} +// Put object from scala +%spark +val myObject = ... +z.put("objName", myObject) + +// Exchanging data frames +myScalaDataFrame = ... +z.put("myScalaDataFrame", myScalaDataFrame) + +val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame] +{% endhighlight %} + + </div> + <div data-lang="python" markdown="1"> + +{% highlight python %} +# Get object from python +%spark.pyspark +myObject = z.get("objName") + +# Exchanging data frames +myPythonDataFrame = ... +z.put("myPythonDataFrame", postsDf._jdf) + +myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext) +{% endhighlight %} + + </div> +</div> + +### Form Creation + +`ZeppelinContext` provides functions for creating forms. +In Scala and Python environments, you can create forms programmatically. +<div class="codetabs"> + <div data-lang="scala" markdown="1"> + +{% highlight scala %} +%spark +/* Create text input form */ +z.input("formName") + +/* Create text input form with default value */ +z.input("formName", "defaultValue") + +/* Create select form */ +z.select("formName", Seq(("option1", "option1DisplayName"), + ("option2", "option2DisplayName"))) + +/* Create select form with default value*/ +z.select("formName", "option1", Seq(("option1", "option1DisplayName"), + ("option2", "option2DisplayName"))) +{% endhighlight %} + + </div> + <div data-lang="python" markdown="1"> + +{% highlight python %} +%spark.pyspark +# Create text input form +z.input("formName") + +# Create text input form with default value +z.input("formName", "defaultValue") + +# Create select form +z.select("formName", [("option1", "option1DisplayName"), + ("option2", "option2DisplayName")]) + +# Create select form with default value +z.select("formName", [("option1", "option1DisplayName"), + ("option2", "option2DisplayName")], "option1") +{% endhighlight %} + + </div> +</div> + +In sql environment, you can create form in simple template. + +```sql +%spark.sql +select * from ${table=defaultTableName} where text like '%${search}%' +``` + +To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html). + +### Interpreter-Specific Functions + +Some interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific functions. +For example functions of the dependency loader (%spark.dep) can be invoked as `z.addRepo()`, `z.load()`, etc. +Such interpreter-specific functions are described within each interpreter's documentation. + +## Usage with Embedded Commands + +In certain interpreters (see table below) zeppelin-context features may be invoked by embedding +command strings into the paragraph text. Such embedded command strings are used to invoke +dynamic-forms and object-interpolation as described below. + +| Interpreters that use Embedded Commands | +|-------------------------------------------------------------------| +|spark.sql (\*), bigquery, cassandra, elasticsearch, file, hbase, ignite, jdbc (\*), kylin, livy, markdown, neo4j, pig, python, shell (\*), zengine | + +Dynamic forms are available in all of the interpreters in the table above, +but object interpolation is only available in a small, but growing, list of interpreters +(marked with an asterisk in the table above). +Both these zeppelin-context features are described below. + +### Dynamic Forms + +Patterns of the form ${ ... } are used to dynamically create additional HTML elements +for requesting user input (that replaces the corresponding pattern in the paragraph text). +Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text), +[select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with +[options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and +[checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are supported. + +Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html). + +### Object Interpolation +Some interpreters can interpolate object values from `z` into the paragraph text by using the +`{variable-name}` syntax. The value of any object previously `put` into `z` can be +interpolated into a paragraph text by using such a pattern containing the object's name. +The following example shows one use of this facility: + +####In Scala cell: +``` +z.put("minAge", 35) +``` + +####In later SQL cell: +``` +%sql select * from members where age >= {minAge} +``` + +The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name. +But the pattern is left unchanged if the named object does not exist in `z`. +Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur -- +translation of only some of the patterns in a paragraph text is never done. + +In some situations, it is necessary to use { and } characters in a paragraph text without invoking the +object interpolation mechanism. For these cases an escaping mechanism is available -- +doubled braces {{ and }} should be used. The following example shows the use of {{ and }} for passing a +regular expression containing just { and } into the paragraph text. + +``` +%sql select * from members where name rlike '[aeiou]{{3}}' +``` + +To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefined +object of the specified name exists. Additionally, all such patterns within the paragraph text should also +be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`. +These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conform +to one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way +(than `{var-name}` and `{{any-text}}`) is used as-is without any changes. +No error is flagged in any case. This behavior is identical to the implementation of a similar feature in +Jupyter's shell invocation using the `!` magic command. + +This feature is disabled by default, and must be explicitly turned on for each interpreter independently +by setting the value of an interpreter-specific property to `true`. +Consult the _Configuration_ section of each interpreter's documentation +to find out if object interpolation is implemented, and the name of the parameter that must be set to `true` to +enable the feature. The name of the parameter used to enable this feature it is different for each interpreter. +For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and +`zeppelin.shell.interpolation` respectively. + +At present only the SparkSQL, JDBC, and Shell interpreters support object interpolation. + + + +
