This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-statefun.git
The following commit(s) were added to refs/heads/master by this push:
new e1bb151 [FLINK-23600] [docs] Restructure module configuration docs
e1bb151 is described below
commit e1bb1518d309be62260f758b34c808eb90cb81e1
Author: Tzu-Li (Gordon) Tai <tzuli...@apache.org>
AuthorDate: Fri Aug 27 14:11:11 2021 +0800
[FLINK-23600] [docs] Restructure module configuration docs
This closes #263
---
docs/content/docs/deployment/module.md | 301 ---------------------
docs/content/docs/deployment/overview.md | 21 +-
docs/content/docs/io-module/overview.md | 96 -------
docs/content/docs/{io-module => modules}/_index.md | 2 +-
.../docs/{deployment => modules}/embedded.md | 6 +-
docs/content/docs/modules/http-endpoint.md | 207 ++++++++++++++
.../docs/{io-module => modules/io}/_index.md | 8 +-
.../docs/{io-module => modules/io}/apache-kafka.md | 68 ++---
.../docs/{io-module => modules/io}/aws-kinesis.md | 88 +++---
.../{io-module => modules/io}/flink-connectors.md | 4 +-
docs/content/docs/modules/io/overview.md | 36 +++
docs/content/docs/modules/overview.md | 64 +++++
docs/content/docs/sdk/flink-datastream.md | 2 +-
docs/content/docs/sdk/golang.md | 2 +-
docs/content/docs/sdk/java.md | 2 +-
docs/content/docs/sdk/python.md | 2 +-
16 files changed, 399 insertions(+), 510 deletions(-)
diff --git a/docs/content/docs/deployment/module.md
b/docs/content/docs/deployment/module.md
deleted file mode 100644
index 0c46956..0000000
--- a/docs/content/docs/deployment/module.md
+++ /dev/null
@@ -1,301 +0,0 @@
----
-title: Module Configuration
-weight: 2
-type: docs
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Module Configuration
-
-An application's module configuration contains all necessary runtime
information to configure the Stateful Functions runtime for a specific
application. It includes the endpoints where functions can be reached along
with ingress and egress definitions.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- endpoints:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTempalte: https://bar.foo.com/{function.name}
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kafka/ingress
- id: com.example/my-ingress
- spec:
- address: kafka-broker:9092
- consumerGroupId: my-consumer-group
- startupPosition:
- type: earliest
- topics:
- - topic: message-topic
- valueType: io.statefun.types/string
- targets:
- - com.example/greeter
- egresses:
- - egress:
- meta:
- type: io.statefun.kafka/egress
- id: example/output-messages
- spec:
- address: kafka-broker:9092
- deliverySemantic:
- type: exactly-once
- transactionTimeout: 15min
-```
-
-## Endpoint Definition
-
-`module.spec.endpoints` declares a list of `endpoint` objects containing all
metadata the Stateful Function runtime needs to invoke a function. The
following is an example of an endpoint definition for a single function.
-
-```yaml
-endpoints:
- - endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTemplate: https://bar.foo.com/{function.name}
-```
-
-In this example, an endpoint for a function within the logical namespace
`com.example` is declared.
-The runtime will invoke all functions under this namespace with the endpoint
URL template.
-
-## URL Template
-
-The URL template name may contain template parameters that are filled in based
on the function's specific type.
-For example, a message sent to message type `com.example/greeter` will be sent
to `http://bar.foo.com/greeter`.
-
-```yaml
-endpoints:
- - endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
- urlPathTemplate: https://bar.foo.com/{function.name}
-```
-
-Templating parameterization works well with load balancers and service
gateways.
-Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server,
you can use the different paths to physical systems. Users may now deploy some
functions on Kubernetes, others AWS Lambda, while others still on physical
servers.
-
-{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
-
-#### Example Using NGINX
-
-```
-http {
- server {
- listen 80;
- server_name proxy;
-
- location /greeter {
- proxy_pass http://greeter:8000/service;
- }
-
- location /emailsender {
- proxy_pass http://emailsender:8000/service;
- }
- }
-}
-
-events {}
-```
-
-This pattern makes the function's physical deployment transparent to the
Stateful Functions runtime.
-They can be deployed, upgraded, rolled back and scaled transparently from the
cluster.
-
-### Full Options
-
-#### Protocol
-
-The RPC protocol used to communicate by the runtime to communicate with the
function.
-``http`` is currently the only supported value.
-
-```yaml
-endpoint:
- meta:
- kind: http
- spec:
-```
-
-#### Typename
-
-The meta `typename` is the logical type name used to match a function
invocation with a physical endpoint.
-Typenames are composed of a namespace (required) and name (optional).
-Endpoints containing a namespace but no name will match all function types in
that namespace.
-
-{{< hint info >}}
-It is recommended to have endpoints only specified against a namespace to
enable dynamic function registration.
-{{< /hint >}}
-
-```yaml
-endpoint:
- meta:
- kind: http
- spec:
- functions: com.example/*
-```
-
-#### Url Path Template
-
-The `urlPathTemplate` is the physical path to be resolved when an endpoint is
matched.
-It may contain templated parameters for dynamic routing.
-
-```yaml
-endpoint:
- meta:
- spec:
- urlPathTemplate: http://bar.foo.com/{function.name}
-```
-
-Supported schemes:
-* ``http``
-* ``https``
-
-Transport via UNIX domain sockets is supported by using the schemes
``http+unix`` or ``https+unix``.
-When using UNIX domain sockets, the endpoint format is:
``http+unix://<socket-file-path>/<serve-url-path>``. For example,
``http+unix:///uds.sock/path/of/url``.
-For example, ``http+unix:///uds.sock/path/of/url``.
-
-#### Max Batch Requests
-
-The maximum number of records that can be processed by a function for a
particular address (typename + id) before invoking backpressure on the system.
The default value is `1000`.
-
-```yaml
-endpoint:
- meta:
- spec:
- maxNumBatchRequests: 1000
-```
-
-#### Timeouts
-
-Timeouts is an optional object containing various timeouts for a request
before a function is considered unreachable.
-
-**Call**
-
-The timeout for a complete function call.
-This configuration spans the entire call: resolving DNS, connecting, writing
the request body, server processing, and reading the response body.
-If the call requires redirects or retries all must complete within one timeout
period.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- call: 1 min
-```
-
-**Connect**
-
-The default connect timeout for new connections.
-The connect timeout is applied when connecting a TCP socket to the target host.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- connect: 10 sec
-```
-
-**Read**
-
-The default read timeout for new connections.
-The read timeout is applied to both the TCP socket and for individual read IO
operations.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- read: 10 sec
-```
-
-**Write**
-
-The default write timeout for new connections.
-The write timeout is applied for individual write IO operations.
-
-```yaml
-endpoint:
- meta:
- spec:
- timeout:
- write: 10 sec
-```
-
-## Ingress
-
-An ingress is an input point where data is consumed from an external system
and forwarded to zero or more functions.
-It is defined via an identifier and specification.
-
-An ingress identifier, similar to a function type, uniquely identifies an
ingress by specifying its [input type]({{< ref "docs/sdk/appendix#types" >}}),
a namespace, and a name.
-
-The spec defines the details of how to connect to the external system, which
is specific to each individual I/O module. Each identifier-spec pair is bound
to the system inside an stateful function module.
-
-See [IO Modules]{{< ref "docs/io-module/overview" >}} for more information on
configuring an ingress.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- id: example/my-ingress
- type: # ingress type
- spec: # ingress specific configurations
-```
-
-## Egress
-
-An egress is an output point where data is written to an external system and
forwarded to zero or more functions.
-It is defined via an identifier and specification.
-
-An egress identifier, similar to a function type, uniquely identifies an
egress.
-
-The spec defines the details of how to connect to the external system, which
is specific to each individual I/O module.
-Each identifier-spec pair is bound to the system inside a stateful function
module.
-
-See [IO Modules]{{< ref "docs/io-module/overview" >}} for more information on
configuring an egress.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- id: example/my-egress
- type: # egress type
- spec: # egress specific configurations
-```
diff --git a/docs/content/docs/deployment/overview.md
b/docs/content/docs/deployment/overview.md
index e6bc3b1..e5e1dd4 100644
--- a/docs/content/docs/deployment/overview.md
+++ b/docs/content/docs/deployment/overview.md
@@ -39,7 +39,7 @@ This way, user code does not need to package any Apache Flink
components.
The provided image allows teams to package their applications with all the
necessary runtime dependencies quickly.
The community provides images containing the entire Stateful Functions
runtime: `apache/flink-statefun:{{< version >}}`.
-All you need to do is attach your [module configurations]({{< ref
"docs/deployment/module" >}}) to the container.
+All you need to do is attach your [module configurations]({{< ref
"docs/modules/overview" >}}) to the container.
{{< unstable >}}
{{< hint info >}}
@@ -79,7 +79,7 @@ $ kubectl delete -f application-module.yaml
{{< details "application-module.yaml" >}}
The application function and io module configuration.
-See the full [documentation]({{< ref "docs/deployment/module" >}}) for more
details.
+See the full [documentation]({{< ref "docs/modules/overview" >}}) for more
details.
```yaml
apiVersion: v1
@@ -91,14 +91,15 @@ metadata:
app: statefun
data:
module.yaml: |+
- version: "3.0"
- module:
- meta:
- type: remote
- spec:
- endpoints:
- ingresses:
- egresses:
+ kind: ...
+ spec:
+ ...
+ ---
+ kind: ...
+ spec:
+ ...
+ ---
+ ...
```
{{< /details >}}
diff --git a/docs/content/docs/io-module/overview.md
b/docs/content/docs/io-module/overview.md
deleted file mode 100644
index b21e826..0000000
--- a/docs/content/docs/io-module/overview.md
+++ /dev/null
@@ -1,96 +0,0 @@
----
-title: 'Overview'
-weight: 1
-type: docs
-aliases:
- - /io-module/
-permalink: /io-module/index.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# I/O Module
-
-Stateful Functions' I/O modules allow functions to receive and send messages
to external systems.
-Based on the concept of Ingress (input) and Egress (output) points, and built
on top of the Apache Flink® connector ecosystem, I/O modules enable functions
to interact with the outside world through the style of message passing.
-
-Commonly used I/O modules are bundled into the runtime by default and can be
configured direclty via the applications [module configuration]({{< ref
"docs/deployment/module" >}}).
-Additionally, custom connectors for other systems can be [plugged in]({{< ref
"docs/io-module/flink-connectors" >}}) to the runtime.
-
-Keep in mind that to use one of these connectors in an application, additional
third party components are usually required, e.g. servers for the data stores
or message queues.
-
-## Ingress
-
-An Ingress is an input point where data is consumed from an external system
and forwarded to zero or more functions.
-It is defined via an ``IngressIdentifier`` and an ``IngressSpec``.
-
-An ingress identifier, similar to a function type, uniquely identifies an
ingress by specifying its input type, a namespace, and a name.
-
-The spec defines the details of how to connect to the external system, which
is specific to each individual I/O module. Each identifier-spec pair is bound
to the system inside an stateful function module.
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- id: example/user-ingress
- type: # ingress type
- spec: # ingress specific configurations
-```
-
-## Router
-
-A router is a stateless operator that takes each record from an ingress and
routes it to zero or more functions.
-Routers are bound to the system via a stateful function module, and unlike
other components, an ingress may have any number of routers.
-
-Routers are defined by a list of function types.
-The ``id`` component of the address is pulled from the key associated with
each record in its underlying source implementation.
-```yaml
-targets:
- - example-namespace/my-function-1
- - example-namespace/my-function-2
-```
-
-## Egress
-
-Egress is the opposite of ingress; it is a point that takes messages and
writes them to external systems.
-Each egress is defined using two components, an ``EgressIdentifier`` and an
``EgressSpec``.
-
-An egress identifier uniquely identifies an egress based on a namespace, name,
and producing type.
-An egress spec defines the details of how to connect to the external system,
the details are specific to each individual I/O module.
-Each identifier-spec pair is bound to the system inside a Stateful Functions
module.
-
-```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- id: example/user-egress
- type: # egress type
- spec: # egress specific configurations
-```
diff --git a/docs/content/docs/io-module/_index.md
b/docs/content/docs/modules/_index.md
similarity index 97%
copy from docs/content/docs/io-module/_index.md
copy to docs/content/docs/modules/_index.md
index aacee54..5b98067 100644
--- a/docs/content/docs/io-module/_index.md
+++ b/docs/content/docs/modules/_index.md
@@ -1,5 +1,5 @@
---
-title: I/O Modules
+title: Modules
icon: <i class="fa fa-random title maindish" aria-hidden="true"></i>
bold: true
bookCollapseSection: true
diff --git a/docs/content/docs/deployment/embedded.md
b/docs/content/docs/modules/embedded.md
similarity index 96%
rename from docs/content/docs/deployment/embedded.md
rename to docs/content/docs/modules/embedded.md
index de988dd..3ac92e6 100644
--- a/docs/content/docs/deployment/embedded.md
+++ b/docs/content/docs/modules/embedded.md
@@ -1,6 +1,6 @@
---
-title: Embedded Module
-weight: 3
+title: 'Embedded Modules'
+weight: 5
type: docs
---
<!--
@@ -25,7 +25,7 @@ under the License.
# Embedded Module Configuration
Embedded modules allow users to load code into the Stateful Functions runtime
that is executed directly within the cluster.
-This is usually to allow plugging in [custom ingress and egress
implementations]({{< ref "docs/io-module/flink-connectors">}}).
+This is usually to allow plugging in [custom ingress and egress
implementations]({{< ref "docs/modules/io/flink-connectors">}}).
Additionally, and embedded module may include [embedded functions]({{< ref
"docs/sdk/flink-datastream#embedded-functions" >}}) that run within the
cluster.
Embedded modules should be used with care, they cannot be deployed or scaled
without downtime and can effect the performance and stability of the entire
cluster.
diff --git a/docs/content/docs/modules/http-endpoint.md
b/docs/content/docs/modules/http-endpoint.md
new file mode 100644
index 0000000..5e7e915
--- /dev/null
+++ b/docs/content/docs/modules/http-endpoint.md
@@ -0,0 +1,207 @@
+---
+title: 'HTTP Function Endpoint'
+weight: 2
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HTTP Function Endpoint
+
+An HTTP Function Endpoint component defines the endpoint URL that the Stateful
Functions runtime should connect to for
+invoking a given function, or for a more dynamic deployment, functions within
a specified namespace.
+
+Below is an example of an HTTP endpoint definition in an application's module
configuration:
+
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v2 (latest)" >}}
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com:8080/{function.name}
+ transport:
+ timeouts:
+ call: 1 min
+ read: 10 sec
+ write: 10 sec
+```
+{{< /tab >}}
+{{< tab "v1" >}}
+```yaml
+kind: io.statefun.endpoints.v1/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com:8080/{function.name}
+ timeouts:
+ call: 1 min
+ read: 10 sec
+ write: 10 sec
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example, an endpoint for a function within the logical namespace
`com.example` is declared.
+The runtime will invoke all functions under this namespace with the endpoint
URL template.
+
+### URL Template
+
+The URL template name may contain template parameters filled in dynamically
based on the function's specific type.
+In the example below, a message sent to message type `com.example/greeter`
will be sent to `http://bar.foo.com/greeter`.
+
+```yaml
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com/{function.name}
+```
+
+Templating parameterization works well with load balancers and service
gateways.
+Suppose `http://bar.foo.com` was an [NGINX](https://www.nginx.com/) server;
you can use the different paths to physical systems. Users may now deploy some
functions on Kubernetes, others AWS Lambda, while others are still on physical
servers.
+
+{{< img src="/fig/dispatch.png" alt="function dispatch" width="75%" >}}
+
+### Transport
+
+Switching between different transport clients is supported since `v2` of HTTP
function endpoint definitions.
+
+The transport client to use is specified using `spec.transport.type`. If not
specified, by default, Stateful Functions uses
[OkHttp](https://square.github.io/okhttp/).
+
+All fields under `spec.transport` is used as the properties to configure the
transport client. For example, the example below configures various timeout
settings for the default `OkHttp` transport:
+
+```yaml
+spec:
+ transport:
+ timeouts:
+ call: 1 min
+ read: 30 sec
+ write: 20 sec
+```
+
+#### Asynchronous HTTP transport (Beta)
+
+Alternatively, Stateful Functions also ships a transport option based on
asynchronous non-blocking IO, implemented with [Netty](https://netty.io/).
+This transport enables much higher resource utilization, higher throughput,
and lower remote function invocation latency.
+
+Below is a complete example of the `transport` section of an HTTP function
endpoint definition, if you want to use this transport type:
+
+```yaml
+spec:
+ transport:
+ type: io.statefun.transports.v1/async
+ call: 2m
+ connect: 20s
+ pool_ttl: 15s
+ pool_size: 1024
+ payload_max_bytes: 33554432
+```
+
+Please see the full spec options below for a description of each property.
+
+## Full Spec Options
+
+#### Target functions (or function)
+
+The meta `typename` is the logical type name used to match a function
invocation with a physical endpoint.
+Typenames are composed of a namespace (required) and name (optional).
+Endpoints containing a namespace but no name will match all function types in
that namespace.
+
+{{< hint info >}}
+It is recommended to have endpoints only specified against a namespace to
enable dynamic function registration.
+{{< /hint >}}
+
+```yaml
+spec:
+ functions: com.example/*
+```
+
+#### Url Path Template
+
+The `urlPathTemplate` is the physical path to be resolved when an endpoint is
matched.
+It may contain templated parameters for dynamic routing.
+
+```yaml
+spec:
+ urlPathTemplate: http://bar.foo.com/{function.name}
+```
+
+Supported schemes:
+* ``http``
+* ``https``
+
+Transport via UNIX domain sockets is supported by using the schemes
``http+unix`` or ``https+unix``.
+When using UNIX domain sockets, the endpoint format is:
``http+unix://<socket-file-path>/<serve-url-path>``. For example,
``http+unix:///uds.sock/path/of/url``.
+For example, ``http+unix:///uds.sock/path/of/url``.
+
+#### Max Batch Requests
+
+The maximum number of records that can be processed by a function for a
particular address (typename + id) before invoking backpressure on the system.
The default value is `1000`.
+
+```yaml
+spec:
+ maxNumBatchRequests: 1000
+```
+
+#### Transport
+
+The transport client type to use for sending requests to the function. If not
specified, by default, Stateful Functions uses
[OkHttp](https://square.github.io/okhttp/).
+
+Below is a full example for configuring the default transport:
+
+```yaml
+spec:
+ transport:
+ timeouts:
+ call: 2m
+ connect: 20s
+ read: 10s
+ write: 10s
+```
+
+* `call`: The timeout for a complete function call. This configuration spans
the entire call: resolving DNS, connecting,
+ writing the request body, server processing, and reading the response body.
If the call requires redirects or retries
+ all must complete within one timeout period. Default value is 1 minute.
+* `connect`: The default connect timeout for new connections. The connect
timeout is applied when connecting a TCP socket to the target host. Default
value is 10 seconds.
+* `read`: The default read timeout for new connections. The read timeout is
applied to both the TCP socket and for individual read IO operations. Default
value is 10 seconds.
+* `write`: The default write timeout for new connections. The write timeout is
applied for individual write IO operations. Default value is 10 seconds.
+
+Alternatively, a transport client based on asynchronous non-blocking IO is
supported:
+
+```yaml
+spec:
+ transport:
+ type: io.statefun.transports.v1/async
+ call: 2m
+ connect: 20s
+ pool_ttl: 15s
+ pool_size: 1024
+ payload_max_bytes: 33554432
+```
+
+* `call`: total duration of a single request (including retries, and
backoffs). After this duration, the call is considered failed.
+* `connect`: the total amount of time to wait for a successful TCP connection.
After that amount of time, an attempt is considered failed, and if the total
call time has not elapsed, an additional attempt will be scheduled (after a
backoff).
+* `pool_ttl`: the amount of time a connection will live in the connection
pool. Set to 0 to disable, otherwise the connection will be evicted from the
pool after (approximately) that time. If a connection is evicted while it is
serving a request, that connection will be only marked for eviction and will be
dropped from the pool once the request returns.
+* `pool_size`: the maximum pool size.
+* `payload_max_bytes`: the maximum size for a request or response payload
size. The default is set to 32MB.
+
+{{< hint info >}}
+We highly recommend setting `statefun.async.max-per-task` to a much higher
value (see [Configurations]({{< ref "docs/deployment/configurations">}}))
+when using the asynchronous transport type. This configuration dictates the
maximum amount of in-flight requests sent to
+the function before applying back pressure. The default value of this
configuration is `1024`, which makes sense only for
+the default threaded blocking `OkHttp` transport. For the asynchronous
transport, a much higher value is recommended.
+{{< /hint >}}
\ No newline at end of file
diff --git a/docs/content/docs/io-module/_index.md
b/docs/content/docs/modules/io/_index.md
similarity index 87%
rename from docs/content/docs/io-module/_index.md
rename to docs/content/docs/modules/io/_index.md
index aacee54..16a9e62 100644
--- a/docs/content/docs/io-module/_index.md
+++ b/docs/content/docs/modules/io/_index.md
@@ -1,9 +1,9 @@
---
-title: I/O Modules
-icon: <i class="fa fa-random title maindish" aria-hidden="true"></i>
-bold: true
+title: IO Components
+weight: 3
bookCollapseSection: true
-weight: 4
+aliases:
+ - /modules/io/
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
diff --git a/docs/content/docs/io-module/apache-kafka.md
b/docs/content/docs/modules/io/apache-kafka.md
similarity index 82%
rename from docs/content/docs/io-module/apache-kafka.md
rename to docs/content/docs/modules/io/apache-kafka.md
index f331dc2..8a2c643 100644
--- a/docs/content/docs/io-module/apache-kafka.md
+++ b/docs/content/docs/modules/io/apache-kafka.md
@@ -3,7 +3,7 @@ title: Apache Kafka
weight: 2
type: docs
aliases:
-- /io-module/apache-kafka.html
+- /modules/io/apache-kafka.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -28,35 +28,30 @@ under the License.
Stateful Functions offers an Apache Kafka I/O Module for reading from and
writing to Kafka topics.
It is based on Apache Flink's universal [Kafka
connector](https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html)
and provides exactly-once processing semantics.
-Kafka is configured in the [module specification]({{< ref
"docs/deployment/module" >}}) of your application.
+Kafka is configured in the [module specification]({{< ref
"docs/modules/overview" >}}) of your application.
## Kafka Ingress Spec
A Kafka ingress defines an input point that reads records from one or more
topics.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf42" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
+kind: io.statefun.kafka.v1/ingress
spec:
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kafka/ingress
- id: com.example/users
- spec:
- address: kafka-broker:9092
- consumerGroupId: my-consumer-group
- startupPosition:
- type: earliest
- topics:
- - topic: messages-1
- valueType: com.example/User
- targets:
- - com.example.fns/greeter
+ id: com.example/users
+ address: kafka-broker:9092
+ consumerGroupId: my-consumer-group
+ startupPosition:
+ type: earliest
+ topics:
+ - topic: messages-1
+ valueType: com.example/User
+ targets:
+ - com.example.fns/greeter
```
+{{< /tab >}}
+{{< /tabs >}}
The ingress also accepts properties to directly configure the Kafka client,
using ``ingress.spec.properties``.
Please refer to the Kafka [consumer
configuration](https://docs.confluent.io/current/installation/configuration/consumer-configs.html)
documentation for the full list of available properties.
@@ -124,26 +119,21 @@ By default, this is set to be the `latest` position.
A Kafka egress defines an input point where functions can write out records to
one or more topics.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
+kind: io.statefun.kafka.v1/egress
spec:
- egresses:
- - egress:
- meta:
- type: io.statefun.kafka/egress
- id: example/output-messages
- spec:
- address: kafka-broker:9092
- deliverySemantic:
- type: exactly-once
- transactionTimeout: 15min
- properties:
- - foo.config: bar
+ id: com.example/users
+ address: kafka-broker:9092
+ deliverySemantic:
+ type: exactly-once
+ transactionTimeout: 15min
+ properties:
+ - foo.config: bar
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kafka [producer
configuration](https://docs.confluent.io/current/installation/configuration/producer-configs.html)
documentation for the full list of available properties.
diff --git a/docs/content/docs/io-module/aws-kinesis.md
b/docs/content/docs/modules/io/aws-kinesis.md
similarity index 75%
rename from docs/content/docs/io-module/aws-kinesis.md
rename to docs/content/docs/modules/io/aws-kinesis.md
index c8947c1..ebc44aa 100644
--- a/docs/content/docs/io-module/aws-kinesis.md
+++ b/docs/content/docs/modules/io/aws-kinesis.md
@@ -3,7 +3,7 @@ title: AWS Kinesis
weight: 3
type: docs
aliases:
-- /io-module/aws-kinesis.html
+- /modules/io/aws-kinesis.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,42 +29,35 @@ under the License.
Stateful Functions offers an AWS Kinesis I/O Module for reading from and
writing to Kinesis streams.
It is based on Apache Flink's [Kinesis
connector](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/kinesis.html).
-Kinesis is configured in the [module specification]({{< ref
"docs/deployment/module" >}}) of your application.
+Kinesis is configured in the [module specification]({{< ref
"docs/modules/overview" >}}) of your application.
## Kinesis Ingress Spec
A Kinesis ingress defines an input point that reads records from one or more
streams.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf42" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- ingresses:
- - ingress:
- meta:
- type: io.statefun.kinesis/ingress
- id: com.example/users
- spec:
- awsRegion:
- type: specific
- id: eu-west-1
- startupPosition:
- type: latest
- streams:
- - stream: user-stream
- valueType: com.example/User
- targets:
- - com.example.fn/greeter
- clientConfigProperties:
- - SocketTimeout: 9999
- - MaxConnections: 15
- type: statefun.kinesis.io/routable-protobuf-ingress
- id: example-namespace/messages
+kind: io.statefun.kinesis.v1/ingress
+spec:
+ id: com.example/users
+ awsRegion:
+ type: specific
+ id: eu-west-1
+ startupPosition:
+ type: latest
+ streams:
+ - stream: user-stream
+ valueType: com.example/User
+ targets:
+ - com.example.fn/greeter
+ clientConfigProperties:
+ - SocketTimeout: 9999
+ - MaxConnections: 15
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kinesis [client
configuration](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html)
documentation for the full list of available properties.
Note that configuration passed using named methods will have higher precedence
and overwrite their respective settings in the provided properties.
@@ -105,29 +98,24 @@ startupPosition:
A Kinesis egress defines an input point where functions can write out records
to one or more streams.
+{{< tabs "8951ef0a-cdd4-40d1-bda8-dec1299aaf41" >}}
+{{< tab "v1 (latest)" >}}
```yaml
-version: "3.0"
-
-module:
- meta:
- type: remote
- spec:
- egresses:
- - egress:
- meta:
- type: io.statefun.kinesis/egress
- id: com.example/out
- spec:
- awsRegion:
- type: specific
- id: eu-west-1
- awsCredentials:
- type: default
- maxOutstandingRecords: 9999
- clientConfigProperties:
- - ThreadingModel: POOLED
- - ThreadPoolSize: 10
+kind: io.statefun.kinesis.v1/egress
+spec:
+ id: com.example/out
+ awsRegion:
+ type: specific
+ id: eu-west-1
+ awsCredentials:
+ type: default
+ maxOutstandingRecords: 9999
+ clientConfigProperties:
+ - ThreadingModel: POOLED
+ - ThreadPoolSize: 10
```
+{{< /tab >}}
+{{< /tabs >}}
Please refer to the Kinesis [producer default configuration
properties](https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties)
documentation for the full list of available properties.
diff --git a/docs/content/docs/io-module/flink-connectors.md
b/docs/content/docs/modules/io/flink-connectors.md
similarity index 98%
rename from docs/content/docs/io-module/flink-connectors.md
rename to docs/content/docs/modules/io/flink-connectors.md
index 330cc4a..5f2699a 100644
--- a/docs/content/docs/io-module/flink-connectors.md
+++ b/docs/content/docs/modules/io/flink-connectors.md
@@ -3,7 +3,7 @@ title: Flink Connectors
weight: 4
type: docs
aliases:
- - /io-module/flink-connectors.html
+ - /modules/io/flink-connectors.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,7 +29,7 @@ under the License.
The source-sink I/O module allows you to plug in existing, or custom, Flink
connectors that are not already integrated into a dedicated I/O module.
Please see the official Apache Flink documentation for a full list of
[available
connectors](https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/)
as well as details on how to build your own.
-Connectors can be plugged into the runtime via an [embedded module]({{< ref
"docs/deployment/embedded" >}})
+Connectors can be plugged into the runtime via an [embedded module]({{< ref
"docs/modules/embedded" >}})
## Dependency
diff --git a/docs/content/docs/modules/io/overview.md
b/docs/content/docs/modules/io/overview.md
new file mode 100644
index 0000000..a49462e
--- /dev/null
+++ b/docs/content/docs/modules/io/overview.md
@@ -0,0 +1,36 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+ - /modules/io/
+permalink: /modules/io/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# I/O Module
+
+Stateful Functions' I/O modules allow functions to receive and send messages
to external systems.
+Based on the concept of Ingress (input) and Egress (output) points, and built
on top of the Apache Flink® connector ecosystem, I/O modules enable functions
to interact with the outside world through the style of message passing.
+
+Commonly used I/O modules are bundled into the runtime by default and can be
configured directly via the applications [module configuration]({{< ref
"docs/modules/overview" >}}).
+Additionally, custom connectors for other systems can be [plugged in]({{< ref
"docs/modules/io/flink-connectors" >}}) to the runtime.
+
+Remember, to use one of these connectors in an application, third-party
components are usually required, e.g., servers for the data stores or message
queues.
diff --git a/docs/content/docs/modules/overview.md
b/docs/content/docs/modules/overview.md
new file mode 100644
index 0000000..1cc3492
--- /dev/null
+++ b/docs/content/docs/modules/overview.md
@@ -0,0 +1,64 @@
+---
+title: 'Overview'
+weight: 1
+type: docs
+aliases:
+ - /modules/
+permalink: /modules/index.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Application Modules
+
+An application module consists of multiple [components]({{< ref
"docs/concepts/application-building-blocks" >}})
+that take part in a StateFun application. It includes the endpoints where the
runtime can reach functions, along with ingress and egress definitions.
+
+Modules are defined using a YAML file. For example, below is a module that
defines an HTTP function endpoint as well as
+a Kafka ingress and egress:
+
+```yaml
+kind: io.statefun.endpoints.v2/http
+spec:
+ functions: com.example/*
+ urlPathTemplate: https://bar.foo.com/{function.name}
+---
+kind: io.statefun.kafka.v1/ingress
+spec:
+ id: com.example/my-ingress
+ address: kafka-broker:9092
+ consumerGroupId: my-consumer-group
+ topics:
+ - topic: message-topic
+ valueType: io.statefun.types/string
+ targets:
+ - com.example/greeter
+---
+kind: io.statefun.kafka.v1/egress
+spec:
+ id: com.example/my-egress
+ address: kafka-broker:9092
+ deliverySemantic:
+ type: exactly-once
+ transactionTimeout: 15min
+---
+```
+
+A module YAML file can contain multiple YAML documents, separated by `---`,
each representing a component to be included in the application.
+Each component is defined by a kind typename string and a spec object
containing the component's properties.
diff --git a/docs/content/docs/sdk/flink-datastream.md
b/docs/content/docs/sdk/flink-datastream.md
index f222487..ba5d217 100644
--- a/docs/content/docs/sdk/flink-datastream.md
+++ b/docs/content/docs/sdk/flink-datastream.md
@@ -104,7 +104,7 @@ StatefulFunctionEgressStreams egresses =
As you can see, instead of binding functions, ingresses, and egresses through
modules as you would with a typical Stateful
Functions application, you bind them directly to the ``DataStream`` job using
a ``StatefulFunctionDataStreamBuilder``:
-* Remote functions are bound using the `withRequestReplyRemoteFunction`
method. [Specification of the remote function]({{< ref "docs/deployment/module"
>}})
+* Remote functions are bound using the `withRequestReplyRemoteFunction`
method. [Specification of the remote function]({{< ref
"docs/modules/http-endpoint" >}})
such as service endpoint and various connection configurations can be set
using the provided ``RequestReplyFunctionBuilder``.
* [Embedded functions](#embedded-functions) are bound using
``withFunctionProvider``.
* Egress identifiers used by functions need to be bound with the
`withEgressId` method.
diff --git a/docs/content/docs/sdk/golang.md b/docs/content/docs/sdk/golang.md
index e88a3c2..f2ef316 100644
--- a/docs/content/docs/sdk/golang.md
+++ b/docs/content/docs/sdk/golang.md
@@ -359,4 +359,4 @@ func main() {
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref
"docs/io-module/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref
"docs/modules/io/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
diff --git a/docs/content/docs/sdk/java.md b/docs/content/docs/sdk/java.md
index bfa1ec7..ccb0ed0 100644
--- a/docs/content/docs/sdk/java.md
+++ b/docs/content/docs/sdk/java.md
@@ -504,4 +504,4 @@ public class FunctionUnderTestTest {
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref
"docs/io-module/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref
"docs/modules/io/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
diff --git a/docs/content/docs/sdk/python.md b/docs/content/docs/sdk/python.md
index c9cde7d..e67ad96 100644
--- a/docs/content/docs/sdk/python.md
+++ b/docs/content/docs/sdk/python.md
@@ -306,4 +306,4 @@ if __name__ == '__main__':
## Next Steps
-Keep learning with information on setting up [I/O modules]({{< ref
"docs/io-module/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
+Keep learning with information on setting up [I/O components]({{< ref
"docs/modules/io/overview" >}}) and configuring the [Stateful Functions
runtime]({{< ref "docs/deployment/overview" >}}).
