This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c1c6485 [SPARK-28793][DOC][SQL] Document CREATE FUNCTION in SQL
Reference
c1c6485 is described below
commit c1c64851ed4b8dac3ca4becaea9e6721eb25c589
Author: Dilip Biswal <[email protected]>
AuthorDate: Tue Oct 22 08:56:44 2019 -0500
[SPARK-28793][DOC][SQL] Document CREATE FUNCTION in SQL Reference
### What changes were proposed in this pull request?
Document CREATE FUNCTION statement in SQL Reference Guide.
### Why are the changes needed?
Currently Spark lacks documentation on the supported SQL constructs causing
confusion among users who sometimes have to look at the code to understand
the
usage. This is aimed at addressing this issue.
### Does this PR introduce any user-facing change?
Yes.
**Before:**
There was no documentation for this.
**After.**
<img width="1260" alt="Screen Shot 2019-09-22 at 3 01 52 PM"
src="https://user-images.githubusercontent.com/14225158/65395036-5bdc6680-dd4a-11e9-9873-0a1da88706a8.png";>
<img width="1260" alt="Screen Shot 2019-09-22 at 3 02 11 PM"
src="https://user-images.githubusercontent.com/14225158/65395037-5bdc6680-dd4a-11e9-964f-c02d23803b68.png";>
<img width="1260" alt="Screen Shot 2019-09-22 at 3 02 39 PM"
src="https://user-images.githubusercontent.com/14225158/65395038-5bdc6680-dd4a-11e9-831b-6ba1d041893d.png";>
<img width="1260" alt="Screen Shot 2019-09-22 at 3 04 04 PM"
src="https://user-images.githubusercontent.com/14225158/65395040-5bdc6680-dd4a-11e9-8226-250f77dfeaf3.png";>
### How was this patch tested?
Tested using jykyll build --serve
Closes #25894 from dilipbiswal/sql-ref-create-function.
Authored-by: Dilip Biswal <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
docs/sql-getting-started.md | 3 +
docs/sql-ref-syntax-ddl-create-function.md | 151 ++++++++++++++++++++++++++++-
2 files changed, 153 insertions(+), 1 deletion(-)
diff --git a/docs/sql-getting-started.md b/docs/sql-getting-started.md
index 5d18c48..0ded265 100644
--- a/docs/sql-getting-started.md
+++ b/docs/sql-getting-started.md
@@ -346,6 +346,9 @@ For example:
</div>
+## Scalar Functions
+(to be filled soon)
+
## Aggregations
The [built-in DataFrames
functions](api/scala/index.html#org.apache.spark.sql.functions$) provide common
diff --git a/docs/sql-ref-syntax-ddl-create-function.md
b/docs/sql-ref-syntax-ddl-create-function.md
index f95a9eb..4c09eba 100644
--- a/docs/sql-ref-syntax-ddl-create-function.md
+++ b/docs/sql-ref-syntax-ddl-create-function.md
@@ -19,4 +19,153 @@ license: |
limitations under the License.
---
-**This page is under construction**
+### Description
+The `CREATE FUNCTION` statement is used to create a temporary or permanent
function
+in Spark. Temporary functions are scoped at a session level where as permanent
+functions are created in the persistent catalog and are made available to
+all sessions. The resources specified in the `USING` clause are made available
+to all executors when they are executed for the first time. In addition to the
+SQL interface, spark allows users to create custom user defined scalar and
+aggregate functions using Scala, Python and Java APIs. Please refer to
+[scalar_functions](sql-getting-started.html#scalar-functions) and
+[aggregate functions](sql-getting-started#aggregations) for more information.
+
+### Syntax
+{% highlight sql %}
+CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [ IF NOT EXISTS ]
+ function_name AS class_name [ resource_locations ]
+{% endhighlight %}
+
+### Parameters
+<dl>
+ <dt><code><em>OR REPLACE</em></code></dt>
+ <dd>
+ If specified, the resources for the function are reloaded. This is mainly
useful
+ to pick up any changes made to the implementation of the function. This
+ parameter is mutually exclusive to <code>IF NOT EXISTS</code> and can not
+ be specified together.
+ </dd>
+ <dt><code><em>TEMPORARY</em></code></dt>
+ <dd>
+ Indicates the scope of function being created. When <code>TEMPORARY</code>
is specified, the
+ created function is valid and visible in the current session. No persistent
+ entry is made in the catalog for these kind of functions.
+ </dd>
+ <dt><code><em>IF NOT EXISTS</em></code></dt>
+ <dd>
+ If specified, creates the function only when it does not exist. The
creation
+ of function succeeds (no error is thrown) if the specified function already
+ exists in the system. This parameter is mutually exclusive to <code> OR
REPLACE</code>
+ and can not be specified together.
+ </dd>
+ <dt><code><em>function_name</em></code></dt>
+ <dd>
+ Specifies a name of funnction to be created. The function name may be
+ optionally qualified with a database name. <br><br>
+ <b>Syntax:</b>
+ <code>
+ [database_name.]function_name
+ </code>
+ </dd>
+ <dt><code><em>class_name</em></code></dt>
+ <dd>
+ Specifies the name of the class that provides the implementation for
function to be created.
+ The implementing class should extend one of the base classes as follows:
+ <ul>
+ <li>Should extend <code>UDF</code> or <code>UDAF</code> in
<code>org.apache.hadoop.hive.ql.exec</code> package.</li>
+ <li>Should extend <code>AbstractGenericUDAFResolver</code>,
<code>GenericUDF</code>, or
+ <code>GenericUDTF</code> in
<code>org.apache.hadoop.hive.ql.udf.generic</code> package.</li>
+ <li>Should extend <code>UserDefinedAggregateFunction</code> in
<code>org.apache.spark.sql.expressions</code> package.</li>
+ </ul>
+ </dd>
+ <dt><code><em>resource_locations</em></code></dt>
+ <dd>
+ Specifies the list of resources that contain the implementation of the
function
+ along with its dependencies. <br><br>
+ <b>Syntax:</b>
+ <code>
+ USING { { (JAR | FILE ) resource_uri} , ...}
+ </code>
+ </dd>
+</dl>
+
+### Examples
+{% highlight sql %}
+-- 1. Create a simple UDF `SimpleUdf` that increments the supplied integral
value by 10.
+-- import org.apache.hadoop.hive.ql.exec.UDF;
+-- public class SimpleUdf extends UDF {
+-- public int evaluate(int value) {
+-- return value + 10;
+-- }
+-- }
+-- 2. Compile and place it in a JAR file called `SimpleUdf.jar` in /tmp.
+
+-- Create a table called `test` and insert two rows.
+CREATE TABLE test(c1 INT);
+INSERT INTO test VALUES (1), (2);
+
+-- Create a permanent function called `simple_udf`.
+CREATE FUNCTION simple_udf AS 'SimpleUdf'
+ USING JAR '/tmp/SimpleUdf.jar';
+
+-- Verify that the function is in the registry.
+SHOW USER FUNCTIONS;
+ +------------------+
+ | function|
+ +------------------+
+ |default.simple_udf|
+ +------------------+
+
+-- Invoke the function. Every selected value should be incremented by 10.
+SELECT simple_udf(c1) AS function_return_value FROM t1;
+ +---------------------+
+ |function_return_value|
+ +---------------------+
+ | 11|
+ | 12|
+ +---------------------+
+
+-- Created a temporary function.
+CREATE TEMPORARY FUNCTION simple_temp_udf AS 'SimpleUdf'
+ USING JAR '/tmp/SimpleUdf.jar';
+
+-- Verify that the newly created temporary function is in the registry.
+-- Please note that the temporary function does not have a qualified
+-- database associated with it.
+SHOW USER FUNCTIONS;
+ +------------------+
+ | function|
+ +------------------+
+ |default.simple_udf|
+ | simple_temp_udf|
+ +------------------+
+
+-- 1. Modify `SimpleUdf`'s implementation to add supplied integral value by 20.
+-- import org.apache.hadoop.hive.ql.exec.UDF;
+
+-- public class SimpleUdfR extends UDF {
+-- public int evaluate(int value) {
+-- return value + 20;
+-- }
+-- }
+-- 2. Compile and place it in a jar file called `SimpleUdfR.jar` in /tmp.
+
+-- Replace the implementation of `simple_udf`
+CREATE OR REPLACE FUNCTION simple_udf AS 'SimpleUdfR'
+ USING JAR '/tmp/SimpleUdfR.jar';
+
+-- Invoke the function. Every selected value should be incremented by 20.
+SELECT simple_udf(c1) AS function_return_value FROM t1;
++---------------------+
+|function_return_value|
++---------------------+
+| 21|
+| 22|
++---------------------+
+
+{% endhighlight %}
+
+### Related statements
+- [SHOW FUNCTIONS](sql-ref-syntax-aux-show-functions.html)
+- [DESCRIBE FUNCTION](sql-ref-syntax-aux-describe-function.html)
+- [DROP FUNCTION](sql-ref-syntax-ddl-drop-function.html)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/14225158/65395036-5bdc6680-dd4a-11e9-9873-0a1da88706a8.png"](https://user-images.githubusercontent.com/14225158/65395036-5bdc6680-dd4a-11e9-9873-0a1da88706a8.png"){kind=link}
![https://user-images.githubusercontent.com/14225158/65395037-5bdc6680-dd4a-11e9-964f-c02d23803b68.png"](https://user-images.githubusercontent.com/14225158/65395037-5bdc6680-dd4a-11e9-964f-c02d23803b68.png"){kind=link}
![https://user-images.githubusercontent.com/14225158/65395038-5bdc6680-dd4a-11e9-831b-6ba1d041893d.png"](https://user-images.githubusercontent.com/14225158/65395038-5bdc6680-dd4a-11e9-831b-6ba1d041893d.png"){kind=link}
![https://user-images.githubusercontent.com/14225158/65395040-5bdc6680-dd4a-11e9-8226-250f77dfeaf3.png"](https://user-images.githubusercontent.com/14225158/65395040-5bdc6680-dd4a-11e9-8226-250f77dfeaf3.png"){kind=link}