[jira] [Created] (FLINK-32455) Breaking change in TypeSerializerUpgradeTestBase prevents flink-connector-kafka from building against 1.18-SNAPSHOT
Tzu-Li (Gordon) Tai created FLINK-32455:
---
Summary: Breaking change in TypeSerializerUpgradeTestBase prevents
flink-connector-kafka from building against 1.18-SNAPSHOT
Key: FLINK-32455
URL: https://issues.apache.org/jira/browse/FLINK-32455
Project: Flink
Issue Type: Sub-task
Components: Connectors / Kafka, Test Infrastructure
Affects Versions: 1.18.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: 1.18.0
FLINK-27518 introduced a breaking signature change to the abstract class
{{TypeSerializerUpgradeTestBase}}, specifically the abstract
{{createTestSpecifications}} method signature was changed. This breaks
downstream test code in externalized connector repos, e.g.
flink-connector-kafka's {{KafkaSerializerUpgradeTest}}
Moreover, {{fink-migration-test-utils}} needs to be transitively pulled in by
downstream test code that depends on flink-core test-jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32453) flink-connector-kafka does not build against Flink 1.18-SNAPSHOT
Tzu-Li (Gordon) Tai created FLINK-32453:
---
Summary: flink-connector-kafka does not build against Flink
1.18-SNAPSHOT
Key: FLINK-32453
URL: https://issues.apache.org/jira/browse/FLINK-32453
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.18.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: 1.18.0
There are a few breaking changes in test utility code that prevents
{{apache/flink-connector-kafka}} from building against Flink 1.18-SNAPSHOT.
This umbrella ticket captures all breaking changes, and should only be closed
once we make things build again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31740) Allow setting boundedness for upsert-kafka SQL connector
Tzu-Li (Gordon) Tai created FLINK-31740: --- Summary: Allow setting boundedness for upsert-kafka SQL connector Key: FLINK-31740 URL: https://issues.apache.org/jira/browse/FLINK-31740 Project: Flink Issue Type: New Feature Components: Connectors / Kafka Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai With FLINK-24456, we added boundedness options for streaming mode to the SQL Kafka Connector. This was mostly just an exposure of existing functionality that was already available at the DataStream API level. We should do the same for the SQL Upsert Kafka Connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-23853) Update StateFun's Flink dependency to 1.13.2
Tzu-Li (Gordon) Tai created FLINK-23853: --- Summary: Update StateFun's Flink dependency to 1.13.2 Key: FLINK-23853 URL: https://issues.apache.org/jira/browse/FLINK-23853 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23762) Revisit RequestReplyFunctionTest unit tests
Tzu-Li (Gordon) Tai created FLINK-23762:
---
Summary: Revisit RequestReplyFunctionTest unit tests
Key: FLINK-23762
URL: https://issues.apache.org/jira/browse/FLINK-23762
Project: Flink
Issue Type: Technical Debt
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
There's some tech debt piling up in the {{RequestReplyFunctionTest}}. We need
to revisit how we're unit testing the {{RequestReplyFunction}}.
Some outstanding issues:
* We're explicitly calling `invoke` with async results to simulate function
responses. With changes in FLINK-20574, that's no longer always the case (the
first request is a blocking call). This hints the fact that those unit tests
are leaking implementation detail and thus making them hard to extend.
* State restore is not properly mocked, which becomes apparent in
{{retryBatchOnUnkownAsyncResponseAfterRestore}}. The states "batch" and
"requestState" starts again from fresh in that test. To properly simulate a
snapshot and restore, we probably want to move all persisted state of
{{RequestReplyFunction}}, including "batch", "requestState", and the remote
function values, all into a separate wrapper class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23718) Manage StateFun Python SDK as a Maven module
Tzu-Li (Gordon) Tai created FLINK-23718: --- Summary: Manage StateFun Python SDK as a Maven module Key: FLINK-23718 URL: https://issues.apache.org/jira/browse/FLINK-23718 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai As of now, the StateFun Python SDK lives as a "dangling" directory in the repository that is not managed by Maven. We'd like include the directory as a Maven module. To start things simple, we can have the Maven POM do nothing - the purpose is just so that the directory is included in the build process (e.g. for ASF license checks on the source files). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23717) Allow setting configs using plain strings in StatefulFunctionsAppContainers
Tzu-Li (Gordon) Tai created FLINK-23717:
---
Summary: Allow setting configs using plain strings in
StatefulFunctionsAppContainers
Key: FLINK-23717
URL: https://issues.apache.org/jira/browse/FLINK-23717
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
The {{StatefulFunctionsAppContainers}} has a {{withConfiguration(ConfigOption,
value)}} method that allows setting Flink configurations.
While this is useful, having to include external dependencies (most of the
time, core) just to get access to the ConfigOption is often too much. It would
be nice if the utility supports setting configs using plain strings as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23714) Expose both master and worker logs when using StatefulFunctionsAppContainers
Tzu-Li (Gordon) Tai created FLINK-23714:
---
Summary: Expose both master and worker logs when using
StatefulFunctionsAppContainers
Key: FLINK-23714
URL: https://issues.apache.org/jira/browse/FLINK-23714
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
We currently only support exposing master logs when using the
{{StatefulFunctionsAppContainers}} E2E test utility. A lot of the times, the
worker logs are also insightful, e.g. checking logs of state restore operations.
Lets extend the {{exposeMasterLogs(Logger)}} method to simply be
{{exposeLogs(Logger)}} which exposes both the master and worker logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23709) Remove SanityVerificationE2E and ExactlyOnceRemoteE2E
Tzu-Li (Gordon) Tai created FLINK-23709:
---
Summary: Remove SanityVerificationE2E and ExactlyOnceRemoteE2E
Key: FLINK-23709
URL: https://issues.apache.org/jira/browse/FLINK-23709
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Over time, the smoke E2E tests have proven to be extensive enough to subsume
the {{SanityVerificationE2E}} and {{ExactlyOnceRemoteE2E}}, which are far less
covering in terms of test scope. As a matter of fact, a large majority (if not
all) if the more important bugs we have discovered over the last few releases
were surfaced by the smoke E2Es.
As the build times are growing larger and larger in StateFun, we suggest to
remove {{SanityVerificationE2E}} and {{ExactlyOnceRemoteE2E}} to be
conservative on the build times.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23600) Rework StateFun's remote module parsing and binding
Tzu-Li (Gordon) Tai created FLINK-23600:
---
Summary: Rework StateFun's remote module parsing and binding
Key: FLINK-23600
URL: https://issues.apache.org/jira/browse/FLINK-23600
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, we have a {{JsonModule}} class that is responsible for parsing
user's module YAML specifications, resolving the specification into application
components (i.e. function providers, ingresses, routers, and egresses) that is
then bound to the application universe.
Over time, the {{JsonModule}} class has overgrown with several changes as we
progressively adapted the YAML format.
* The class handles ALL kinds of components, including ingresses / functions /
egresses etc. The code is extremely fragile and becoming hard to extend.
* Users have no access to extend this class, if they somehow need to plugin
custom components (e.g. adding an unsupported ingress / egress, custom protocol
implementations etc).
We aim to rework this with the following goals in mind:
# The system should only handle {{module.yaml}} parsing up to the point where
it extracts a list of JSON objects that individually represent an application
component.
# The system has no knowledge of what each JSON objects contains, other than
its {{TypeName}} which would map to a corresponding {{ComponentBinder}}.
# A {{ComponentBinder}} is essentially an extension bound to the system that
knows how to parse a specific JSON object, and bind components to the
application universe.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23296) Add RequestReplyClientFactory as a pluggable extension
Tzu-Li (Gordon) Tai created FLINK-23296:
---
Summary: Add RequestReplyClientFactory as a pluggable extension
Key: FLINK-23296
URL: https://issues.apache.org/jira/browse/FLINK-23296
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.1.0
Currently, we ship and always use by default a {{RequestReplyClient}}
implementation based on OkHttp. We'd like to allow users to use their own
implementations of the {{RequestReplyClient}} for remote invocations. This is
strictly for extending the means of transport, and should not leave room to
touch the invocation protocol.
The way this would look like in the module YAML files for remote modules would
be:
{code}
module:
spec:
endpoints:
- endpoint:
meta:
kind: http
spec:
functions: com.foo.bar/*
urlPathTemplate:
http://bar.foo.com:8080/functions/{function.name}
transport:
extension: com.foo.bar/some.custom.client
prop1: foobar
prop2:
- k: v
- k2: v2
{code}
The important part is the transport section. If not specified, then the default
OkHttp implementation will be used. Otherwise, if specified, an extension with
the specified typename must be bound and exist in the application, and that
extension needs to be a {{RequestReplyClientFactory}}:
{code}
interface RequestReplyClientFactory {
RequestReplyClient create(JSONNode properties, URI endpointUrl);
}
{code}
The provided JSON node properties will be as is provided in the {{transport}}
block of the module YAML endpoint specification.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23295) Introduce extension module SPI to StateFun
Tzu-Li (Gordon) Tai created FLINK-23295:
---
Summary: Introduce extension module SPI to StateFun
Key: FLINK-23295
URL: https://issues.apache.org/jira/browse/FLINK-23295
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.1.0
To support pluggable extensions, e.g. custom HTTP clients for remote
invocation, we'd like to add a generic way for users to provide their own
implementations of various pluggable components.
This will take the form of {{ExtensionModule}}s, where a user that is providing
an extension implements such a module to be included into the StateFun app.
Each {{ExtensionModule}} may bind multiple extensions identified by unique
typenames. Other components of the application, such as functions and IO
modules, may access these extensions through their typenames.
The SPI would likely look like this:
{code}
public interface ExtensionModule {
void configure(Map globalConfiguration, Binder binder);
interface Binder {
void bindExtension(TypeName typeName, T extension);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23293) Support pluggable / extendable HTTP transport for StateFun remote invocations
Tzu-Li (Gordon) Tai created FLINK-23293: --- Summary: Support pluggable / extendable HTTP transport for StateFun remote invocations Key: FLINK-23293 URL: https://issues.apache.org/jira/browse/FLINK-23293 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.1.0 For some of our advanced users, it is required for them to use their own HTTP client implementations for remote function invocations. Towards that end, we'd like to support a generic way to plugin custom implementations, with the HTTP client being one of the initially supported extensions. This includes a few separate sub-tasks that will be added under this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22529) StateFun Kinesis ingresses should support configs that are available via FlinkKinesisConsumer's ConsumerConfigConstants
Tzu-Li (Gordon) Tai created FLINK-22529:
---
Summary: StateFun Kinesis ingresses should support configs that
are available via FlinkKinesisConsumer's ConsumerConfigConstants
Key: FLINK-22529
URL: https://issues.apache.org/jira/browse/FLINK-22529
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
The Kinesis ingress should support the configs that are available in
{{FlinkKinesisConsumer}}'s {{ConsumerConfigConstants}}. Instead, currently, all
property keys provided to the Kinesis ingress are assumed to be AWS-client
related keys, and therefore have all been appended with the `aws.clientconfigs`
string.
I'd suggest to avoid mixing the {{ConsumerConfigConstants}} configs within the
properties as well. Having named methods on the {{KinesisIngressBuilder}} for
those configuration would provide a cleaner solution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22023) Remove outdated StateFun quickstart archetype
Tzu-Li (Gordon) Tai created FLINK-22023: --- Summary: Remove outdated StateFun quickstart archetype Key: FLINK-22023 URL: https://issues.apache.org/jira/browse/FLINK-22023 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 The StateFun Maven quickstart archetype should be removed, because it is outdated (only works for embedded functions). We can add a quickstart archetype for Java remote SDKs in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21904) parseJmJvmArgsAndExportLogs: command not found warning when starting StateFun
Tzu-Li (Gordon) Tai created FLINK-21904:
---
Summary: parseJmJvmArgsAndExportLogs: command not found warning
when starting StateFun
Key: FLINK-21904
URL: https://issues.apache.org/jira/browse/FLINK-21904
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-2.2.2
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
I'm seeing this warning in E2E logs:
{code}
11:37:12,572 ERROR
org.apache.flink.statefun.e2e.remote.ExactlyOnceWithRemoteFnE2E -
/opt/flink/bin/standalone-job.sh: line 43: parseJmJvmArgsAndExportLogs: command
not found
{code}
This was caused by FLINK-19662, which renamed {{parseJmJvmArgsAndExportLogs}}
to {{parseJmArgsAndExportLogs}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21901) Update StateFun version to 3.0-SNAPSHOT
Tzu-Li (Gordon) Tai created FLINK-21901: --- Summary: Update StateFun version to 3.0-SNAPSHOT Key: FLINK-21901 URL: https://issues.apache.org/jira/browse/FLINK-21901 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 Our version is still on 2.3-SNAPSHOT in the main repository, since directly jumping to 3.0-SNAPSHOT was something that was decided during the development cycle. To prepare for the upcoming release, we should update the main branch to 3.0-SNAPSHOT already. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21865) Add a Docker Compose greeter example to StateFun playgrounds
Tzu-Li (Gordon) Tai created FLINK-21865:
---
Summary: Add a Docker Compose greeter example to StateFun
playgrounds
Key: FLINK-21865
URL: https://issues.apache.org/jira/browse/FLINK-21865
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
This example is intended as a follow-up after completion of the Java SDK
Showcase Tutorial (FLINK-21862).
If users are already familiar with the Java SDK fundamentals and would like to
get a better understanding of how a realistic StateFun application looks like,
then this would be the example they start with. Otherwise, we would recommend
users to take a look at the Showcase tutorial first.
This example works with Docker Compose, and runs a few services that build up
an end-to-end StateFun application:
- Functions service that runs functions and expose them through an HTTP
endpoint.
- StateFun runtime processes (a manager plus workers) that will handle ingress,
egress, and inter-function messages as well as function state storage in a
consistent and fault-tolerant manner.
- Apache Kafka broker for the application ingress and egress.
To motivate this example, we'll implement a simple user greeter application,
which has two functions - a {{UserFn}} that expects {{UserLogin}} JSON events
from an ingress and keeps in state storage information about users, and a
{{GreetingsFn}} that accepts user information to generate personalized greeting
messages that are sent to users via an egress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21862) Add a Java SDK showcase tutorial to StateFun playground
Tzu-Li (Gordon) Tai created FLINK-21862: --- Summary: Add a Java SDK showcase tutorial to StateFun playground Key: FLINK-21862 URL: https://issues.apache.org/jira/browse/FLINK-21862 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 This goal of the showcase project is intended for new StateFun users that would like to start implementing their StateFun application functions using Java (or any other JVM language). The tutorial should be streamlined and split into a few parts which we recommend to go through a specific order. Each part can demonstrate with some code snippets plus Javadocs and comments to guide new users through the SDK fundamentals. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21705) Add a SliceType to the StateFun Java SDK
Tzu-Li (Gordon) Tai created FLINK-21705:
---
Summary: Add a SliceType to the StateFun Java SDK
Key: FLINK-21705
URL: https://issues.apache.org/jira/browse/FLINK-21705
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
A {{SliceType}} is a type that simply treats the value as raw bytes. The
serializer implementation simply forwards the bytes as is, without any copying.
Since slices are immutable, the type is considered immutable.
This type can come in handy if users would like to avoid serialization /
deserialization in certain scenarios, e.g. sending state values as a message to
another function.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21678) Rename Expiration.expiresAfterReadOrWrite to Expiration.expiresAfterCall in StateFun
Tzu-Li (Gordon) Tai created FLINK-21678:
---
Summary: Rename Expiration.expiresAfterReadOrWrite to
Expiration.expiresAfterCall in StateFun
Key: FLINK-21678
URL: https://issues.apache.org/jira/browse/FLINK-21678
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
For remote functions in StateFun, a managed remote state with Flink expiration
settings {{expireAfterReadAndWrite}} actually means that a value's TTL timer
will always be refreshed when attempting to invoke the function. This is
because invoking a remote function always entails reading the state from the
state backends (as defined by the request reply protocol).
To avoid any confusion for users, we've decided to rename this expiration
configuration as {{expireAfterCall}} for remote functions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21676) Ensure that state names are valid identifiers in the StateFun Java SDK
Tzu-Li (Gordon) Tai created FLINK-21676: --- Summary: Ensure that state names are valid identifiers in the StateFun Java SDK Key: FLINK-21676 URL: https://issues.apache.org/jira/browse/FLINK-21676 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 While the Java SDK only access state via state name strings, other SDKs, like the Python SDK, may access state as properties on the storage instance. To ensure that users can properly access previous state after swapping languages for their functions, we should enforce that specified state names are valid identifiers, following the rules below: * Must start with either an alphabet letter [a-z] / [A-Z], or an underscore "_". * Characters other than the first, can be either an alphabet letter [a-z] / [A-Z], a digit [0-9], or an underscore "_". * Must not contain spaces. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21663) Reject duplicate StatefulFunction registrations under the same typename
Tzu-Li (Gordon) Tai created FLINK-21663:
---
Summary: Reject duplicate StatefulFunction registrations under the
same typename
Key: FLINK-21663
URL: https://issues.apache.org/jira/browse/FLINK-21663
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
Likewise to FLINK-21662, in StateFun, a single function typename should be
associated with exactly one {{StatefulFunction}}. Duplicate registrations
should be caught eagerly when users create a {{StatefulFunctions}} object.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21662) Reject duplicate ValueSpec registrations under the same state name
Tzu-Li (Gordon) Tai created FLINK-21662:
---
Summary: Reject duplicate ValueSpec registrations under the same
state name
Key: FLINK-21662
URL: https://issues.apache.org/jira/browse/FLINK-21662
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
StateFun's remote Java SDK currently does not throw if a user builds a
{{StatefulFunctionSpec}} with multiple {{ValueSpec}}s that are registered under
the same name.
A single state name is associated with a single {{ValueSpec}}, so we should
restrict and catch this early from the API level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21577) SimpleType.simpleTypeFrom(...) complains with "Collection is empty"
Tzu-Li (Gordon) Tai created FLINK-21577:
---
Summary: SimpleType.simpleTypeFrom(...) complains with "Collection
is empty"
Key: FLINK-21577
URL: https://issues.apache.org/jira/browse/FLINK-21577
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
This is caused by the {{EnumSet.copyOf}} method call at:
https://github.com/apache/flink-statefun/blob/master/statefun-sdk-java/src/main/java/org/apache/flink/statefun/sdk/java/types/SimpleType.java#L57
That expects the collection to be non-empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21530) Precompute TypeName's canonical string representation
Tzu-Li (Gordon) Tai created FLINK-21530:
---
Summary: Precompute TypeName's canonical string representation
Key: FLINK-21530
URL: https://issues.apache.org/jira/browse/FLINK-21530
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
There's room for improvement in StateFun's {{PersistedRemoteFunctionValues}},
where we currently concatenate strings to build the typename string for each
state value we attach to a {{ToFunction}} message.
This extra work can be easily avoided by precomputing the canonical typename
string, since {{TypeName}}'s are immutable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21498) Avoid copying when converting byte[] to ByteString in StateFun
Tzu-Li (Gordon) Tai created FLINK-21498: --- Summary: Avoid copying when converting byte[] to ByteString in StateFun Key: FLINK-21498 URL: https://issues.apache.org/jira/browse/FLINK-21498 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai There's a few places in StateFun where we can be more efficient with byte[] to Protobuf ByteString conversions, by just wrapping the byte[] instead of copying, since we know that the byte array can no longer be mutated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21496) Upgrade Testcontainers to 1.15.1 in Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-21496: --- Summary: Upgrade Testcontainers to 1.15.1 in Stateful Functions Key: FLINK-21496 URL: https://issues.apache.org/jira/browse/FLINK-21496 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 The E2E tests in CI is currently failing for StateFun, started failing recently due to Github Actions upgrading their Docker version to 20.10.2+. Due to this upgrade, our current Testcontainers version 1.12.x is no longer compatible since that version relies on a deprecated Docker API that no longer exists in Docker version 10.10.2 (API version 1.41). Full description of the issue: https://github.com/testcontainers/testcontainers-java/issues/3574 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21491) Properly shade Protobuf in statefun-sdk-java
Tzu-Li (Gordon) Tai created FLINK-21491:
---
Summary: Properly shade Protobuf in statefun-sdk-java
Key: FLINK-21491
URL: https://issues.apache.org/jira/browse/FLINK-21491
Project: Flink
Issue Type: Task
Components: Build System / Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
Currently, although we shade Protobuf in {{statefun-sdk-java}}, Protobuf is
still treated as a transitive compile dependency. This can be checked by
executing {{mvn dependency:tree}} on the {{statefun-sdk-java}} module.
In our case, in order to properly shade this and remove the transitive
dependency, we need to introduce a separate module specifically containing only
the relocated protobuf classes, and let {{statefun-sdk-java}} depend on that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21459) Implement remote Java SDK for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-21459: --- Summary: Implement remote Java SDK for Stateful Functions Key: FLINK-21459 URL: https://issues.apache.org/jira/browse/FLINK-21459 Project: Flink Issue Type: Task Components: Build System / Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Igal Shilman Fix For: statefun-3.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21276) flink-statefun/statefun-sdk-java should mention Protobuf in NOTICE and bundle its license
Tzu-Li (Gordon) Tai created FLINK-21276: --- Summary: flink-statefun/statefun-sdk-java should mention Protobuf in NOTICE and bundle its license Key: FLINK-21276 URL: https://issues.apache.org/jira/browse/FLINK-21276 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21171) Introduce TypedValue to the StateFun request-reply protocol
Tzu-Li (Gordon) Tai created FLINK-21171:
---
Summary: Introduce TypedValue to the StateFun request-reply
protocol
Key: FLINK-21171
URL: https://issues.apache.org/jira/browse/FLINK-21171
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-3.0.0
Currently, all values being passed around via the request-reply protocol, are
of the Protobuf {{Any}} type. This includes payloads of outgoing messages to
other functions, and also state values.
This has a few shortcomings:
* All user records are strictly required to be modeled and wrapped as a
Protobuf message - even for simple primitive type. This makes it awkward to
work with for many common types of messages, for example JSON.
* For data persisted as state, with each state value being a Protobuf {{Any}},
each value would also redundantly store the type urls associated with each
Protobuf message.
Instead, we'd like to introduced a {{TypedValue}} construct that replaces
{{Any}} everywhere in the protocol, for both messages and state values:
{code}
message TypedValue {
string typename = 1;
bytes value = 2;
}
{code}
The {{typename}} here directly maps to the type concept introduced in
FLINK-21061.
For state, we directly write the value bytes of a {{TypedValue}} into state,
and the {{typename}} is the meta information snapshotted by the state
serializer (see FLINK-21061).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21061) Introduce RemotePersistedValue construct in Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-21061:
---
Summary: Introduce RemotePersistedValue construct in Stateful
Functions
Key: FLINK-21061
URL: https://issues.apache.org/jira/browse/FLINK-21061
Project: Flink
Issue Type: Task
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
To prepare for the new cross-language type system upcoming in Stateful
Functions, we need a new {{RemotePersistedValue}} state construct to support
this in the runtime.
A {{RemotePersistedValue}} should translate to a Flink {{ValueState}} that is
essentially a byte array value, and persists in state serializer snapshots a
{{typeUrl}} metainfo to represent the type of the remote value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20987) Upgrade Stateful Functions to use Flink 1.12
Tzu-Li (Gordon) Tai created FLINK-20987: --- Summary: Upgrade Stateful Functions to use Flink 1.12 Key: FLINK-20987 URL: https://issues.apache.org/jira/browse/FLINK-20987 Project: Flink Issue Type: New Feature Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20742) Skip deployment of StateFun example artifacts
Tzu-Li (Gordon) Tai created FLINK-20742: --- Summary: Skip deployment of StateFun example artifacts Key: FLINK-20742 URL: https://issues.apache.org/jira/browse/FLINK-20742 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-3.0.0 Starting from the next Stateful Functions release, we'd like to stop publishing Maven artifacts for the examples. We never expect users to be trying out examples through this artifacts, and therefore releasing them is not required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20741) StateFun's DataStream interop SDK module is missing a valid NOTICE file
Tzu-Li (Gordon) Tai created FLINK-20741:
---
Summary: StateFun's DataStream interop SDK module is missing a
valid NOTICE file
Key: FLINK-20741
URL: https://issues.apache.org/jira/browse/FLINK-20741
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.2
The {{statefun-flink-datastream}} module bundles a fat jar, just like the
{{statefun-flink-distribution}} module. Likewise, for legal purposes, all
bundled dependencies need to be acknowledged in the NOTICE file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20689) Upgrade StateFun to Flink 1.11.3
Tzu-Li (Gordon) Tai created FLINK-20689: --- Summary: Upgrade StateFun to Flink 1.11.3 Key: FLINK-20689 URL: https://issues.apache.org/jira/browse/FLINK-20689 Project: Flink Issue Type: Task Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.2.2 Now that Flink 1.11.3 is out, we should upgrade StateFun to use 1.11.3. This is important for FLINK-19740. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20636) Require unaligned checkpoints to be disabled in StateFun applications
Tzu-Li (Gordon) Tai created FLINK-20636: --- Summary: Require unaligned checkpoints to be disabled in StateFun applications Key: FLINK-20636 URL: https://issues.apache.org/jira/browse/FLINK-20636 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.2.2 Due to how StateFun has feedback loops, with unaligned checkpoints a function dispatcher operator may receive a feedback checkpoint barrier (from other parallel subtasks that process the checkpoint barrier first) before it receives its own checkpoint barrier. We need to further investigate how to properly support unaligned checkpointing in StateFun. For the time being, we should strictly require aligned checkpointing for StateFun apps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20574) Throttle number of remote invocation requests on startup or restores with large backlogs
Tzu-Li (Gordon) Tai created FLINK-20574:
---
Summary: Throttle number of remote invocation requests on startup
or restores with large backlogs
Key: FLINK-20574
URL: https://issues.apache.org/jira/browse/FLINK-20574
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
On startup or restores, the {{RequestReplyFunction}} may heavily load the
remote functions with multiple concurrent invocation requests if there is a
large backlog of restored or historical events to process through.
The new protocol introduced by FLINK-20265 emphasizes this much more due to the
nature of extra invocation roundtrips if the function has state declarations
(i.e., the first hoard of concurrent invocations would all fail with an
{{IncompleteInvocationContext}} and requires invocation patching + state
registrations).
We should think about how to apply throttling to mitigate these scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20336) RequestReplyFunction should not silently ignore UNRECOGNIZED state value mutations types
Tzu-Li (Gordon) Tai created FLINK-20336:
---
Summary: RequestReplyFunction should not silently ignore
UNRECOGNIZED state value mutations types
Key: FLINK-20336
URL: https://issues.apache.org/jira/browse/FLINK-20336
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-2.2.1, statefun-2.1.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0, statefun-2.2.2
If a function's response has a {{PersistedValueMutation}} type that is
{{UNRECOGNIZED}}, we currently just silently ignore that mutation:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/reqreply/PersistedRemoteFunctionValues.java#L84
This is incorrect. The {{UNRECOGNIZED}} enum constant is a pre-defined constant
used by the Protobuf Java SDK, to represent a constant that was unable to be
deserialized (because the the serialized constant does not match any enums
defined in the protobuf message).
Therefore, it should be handled by throwing an exception, preferably indicating
that there is some sort of version mismatch between the function's Protobuf
message definitions, and StateFun's Protobuf message definitions (i.e. most
likely a mismatch in the invocation protocol versions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20335) Remove support for eager state specifications in module YAML definitions
Tzu-Li (Gordon) Tai created FLINK-20335:
---
Summary: Remove support for eager state specifications in module
YAML definitions
Key: FLINK-20335
URL: https://issues.apache.org/jira/browse/FLINK-20335
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
With FLINK-20265, we now support declaring state in StateFun functions, and
that can change dynamically without any system downtime.
It can be confusing for users if we continued to support the legacy way of
statically declaring state specifications in the module YAML definitions.
Therefore, we propose to completely remove that by:
* No longer support module YAML format versions <= 2.0.
* Remove the {{PersistedRemoteFunctionValues}} constructor that accepts a list
of eager state specifications
This would be a breaking change:
* Users upgrading to version 2.3.0 have to rewrite their module YAMLs to
conform to format version 3.0
* They also have to correspondingly update their functions to use SDKs of
version 2.3.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20334) Introduce function endpoint path templating in module YAML specifications
Tzu-Li (Gordon) Tai created FLINK-20334:
---
Summary: Introduce function endpoint path templating in module
YAML specifications
Key: FLINK-20334
URL: https://issues.apache.org/jira/browse/FLINK-20334
Project: Flink
Issue Type: Sub-task
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
In the current module specifications, function endpoints are defined like so:
{code}
functions:
- function:
meta:
kind: http
type: com.foo/world
spec:
endpoint: http://localhost:5959/statefun
{code}
A list of functions and their corresponding service endpoints are listed
statically in the module specification file, which is loaded once on system
startup. The system may only route messages to functions that have been
defined. This prevents users from adding new functions to the application,
without having to restart the system and reload new module specifications.
We propose that instead of specifying functions, users should specify a
"family" of function endpoints, like so:
{code}
functionEndpoints:
- functionEndpoint:
meta:
kind: http
spec:
target:
typename:
namespace: com.foo.bar
function: * # (can be wildcard * or a specific name)
urlPathTemplate: "https://bar.foo.com:8000/{typename.function};
connectTimeout: 1min
# ... (other connection related configs that is shared for this
endpoint family)
{code}
Note how users no longer define eager state per individual function. This is
made possible by FLINK-20265, where state is now defined in the functions
instead of in the module specifications.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20268) Update Python SDK to implement new invocation protocol
Tzu-Li (Gordon) Tai created FLINK-20268:
---
Summary: Update Python SDK to implement new invocation protocol
Key: FLINK-20268
URL: https://issues.apache.org/jira/browse/FLINK-20268
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
The Python SDK should be updated to implement the new protocol introduced by
FLINK-20265.
Users should be able to declare states the Python function uses, and with
information, the Python SDK {{RequestReplyHandler}} can then match on the
provided states values in invocation requests against the declared states of a
target function:
Proposed API:
{code}
@functions.bind(
typename="example/greeter",
states=[StateSpec("seen_count")])
def greet(context, greet_request: GreetRequest):
state = context.state("seen_count").unpack(SeenCount)
...
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20265) Extend invocation protocol to allow functions to indicate incomplete state context
Tzu-Li (Gordon) Tai created FLINK-20265:
---
Summary: Extend invocation protocol to allow functions to indicate
incomplete state context
Key: FLINK-20265
URL: https://issues.apache.org/jira/browse/FLINK-20265
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
Currently, users declare the states a function will access with a module YAML
definition file. The modules are loaded once when starting a StateFun cluster,
meaning that the state specifications remain static throughout the cluster's
execution lifetime.
We propose that state specifications should be declared by the function
themselves via the language SDKs, instead of being declared in the module YAMLs.
The state specifications, now living in the functions, can be made discoverable
by the StateFun runtime through the invocation request-reply protocol.
Brief simplified sketch of the extended protocol:
- StateFun dispatches an invocation request, with states {A, B}.
- Function receives request, but since it requires {A, B, C, D}, it responds
with a IncompleteInvocationContext response indicating that state values for C,
D is missing
- StateFun receives this response, and registers new Flink state handles for
{C, D}.
- Then, a new invocation request with the same input messages, but "patched"
with new states to contain all values for {A, B, C, D} is resent to the
function.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20264) Zero-downtime / dynamic function upgrades in Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-20264: --- Summary: Zero-downtime / dynamic function upgrades in Stateful Functions Key: FLINK-20264 URL: https://issues.apache.org/jira/browse/FLINK-20264 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Fix For: statefun-2.3.0 Currently, due to how functions can be executed as stateless deployments separate to the StateFun runtime, they can be easily upgraded with zero-downtime. However, up to now there are still some restrictions to what can be done without restarting StateFun processes: * Can't upgrade existing functions to declare new persisted state * Can't add new functions to an existing StateFun application, and have messages routed to it The end goal of this epic is to enable the above operations for function deployments, without the need to restart the StateFun runtime. Further details can be found in subtasks of this JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20189) Restored feedback events may be silently dropped if per key-group header bytes were not fully read
Tzu-Li (Gordon) Tai created FLINK-20189:
---
Summary: Restored feedback events may be silently dropped if per
key-group header bytes were not fully read
Key: FLINK-20189
URL: https://issues.apache.org/jira/browse/FLINK-20189
Project: Flink
Issue Type: Task
Components: Stateful Functions
Affects Versions: statefun-2.2.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0, statefun-2.2.2
The attempt to read the per key-group header bytes here does not guarantee the
header bytes are fully-read:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/logger/UnboundedFeedbackLogger.java#L163
What could happen is the following:
* Say the input stream actually has the header bytes written in there
* Less then {{HEADER_BYTES.length}} number of bytes was read into the read
buffer, in the above reference code line.
* The {{if (bytesRead > 0 && !Arrays.equals(header, HEADER_BYTES))}} check
would be true, because the read byte array != the expected header bytes.
* We would mistakenly think that the header bytes are not in the input stream,
and pushback. i.e. the header bytes were not being skipped, and the following
reads would see the header bytes first.
* Most importantly, since the header bytes are not being skipped in this case,
the {{STATEFUN_VERSION}} (which is {{0}}) is being incorrectly read by
{{KeyGroupStream.readFrom(...)}} as the number of feedback elements to read:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/logger/KeyGroupStream.java#L57
* The end result of all of this is in this scenario: some checkpointed feedback
events would be silently dropped.
Although it is hard to say how possible this would happen in reality, and would
also depend on the actual implementation of the {{InputStream}}, from the
general contracts of {{InputStream#read(byte[])}} this is definitely something
that should be addressed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20182) Add JDK 11 build to StateFun's CI
Tzu-Li (Gordon) Tai created FLINK-20182: --- Summary: Add JDK 11 build to StateFun's CI Key: FLINK-20182 URL: https://issues.apache.org/jira/browse/FLINK-20182 Project: Flink Issue Type: Task Components: Build System / CI, Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai We'd like to officially support building Stateful Function's with Java 11 (note, we still release Java 8 artifacts, which works with Java 11 as well). This should be covered in the per-push / per-PR CI builds in StateFun. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20161) Consider switching from Travis CI to Github Actions for flink-statefun's CI workflows
Tzu-Li (Gordon) Tai created FLINK-20161:
---
Summary: Consider switching from Travis CI to Github Actions for
flink-statefun's CI workflows
Key: FLINK-20161
URL: https://issues.apache.org/jira/browse/FLINK-20161
Project: Flink
Issue Type: Task
Components: Build System / CI, Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Travis-CI.com recently announced a new pricing model on Nov. 2, which affects
public open source projects:
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
While its a bit unclear if Travis CI repos under
{{travis-ci.com/github/apache}} is affected by this, this will definitely
affect contributors who fork our repositories and enable Travis CI on their
fork for development purposes.
Github Actions seems to be a popular alternative nowadays:
* No limited test time with its hosted builders, if repo is public
* Activation is automatic - one step / click less for contributors to get CI
running for their forks
Given that the CI workflows in {{flink-statefun}} is very minimal right now, we
propose to make the switch to Github Actions as the efforts to do that should
be relatively trivial.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20085) Remove RemoteFunctionStateMigrator code paths from StateFun
Tzu-Li (Gordon) Tai created FLINK-20085:
---
Summary: Remove RemoteFunctionStateMigrator code paths from
StateFun
Key: FLINK-20085
URL: https://issues.apache.org/jira/browse/FLINK-20085
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.3.0
The {{RemoteFunctionStateMigrator}} was added to allow savepoints with versions
<= 2.1.0 to have a migration path for upgrading to versions >= 2.2.0. The
binary format of remote function state was changed due to demultiplexed remote
state, introduced in 2.2.0.
With 2.2.0 already released with the new formats, it is now safe to fully
remove this migration path.
For users, what this means that it would not be possible to directly upgrade
from 2.0.x / 2.1.x to 2.3.x+. They'd have to perform incremental upgrades via
2.2.x, by restoring first with 2.2.x and then taking another savepoint, before
upgrading to 2.3.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19748) StateFun's UnboundedFeedbackLogger should call startNewKeyGroup for all assigned key groups
Tzu-Li (Gordon) Tai created FLINK-19748:
---
Summary: StateFun's UnboundedFeedbackLogger should call
startNewKeyGroup for all assigned key groups
Key: FLINK-19748
URL: https://issues.apache.org/jira/browse/FLINK-19748
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-2.2.0, statefun-2.1.0, statefun-2.0.0
Reporter: Tzu-Li (Gordon) Tai
Currently, on commit the {{UnboundedFeedbackLogger}} only calls
{{startNewKeyGroup}} on the raw keyed stream for key groups that actually have
logged messages:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/logger/UnboundedFeedbackLogger.java#L102
This means that it might skip some key groups, if a key group doesn't have any
logged messages.
This doesn't conform with the expected usage of Flink's
{{KeyedStateCheckpointOutputStream}}, where it expects that for ALL key groups
within the range, {{startNewKeyGroup}} needs to be invoked.
The reason for this is that underneath, calling {{startNewKeyGroup}} would also
record the starting stream offset position for the key group.
However, when iterating through a raw keyed stream, the key group offsets
iterator {{KeyGroupRangeOffsets#KeyGroupOffsetsIterator}} doesn't take into
account that some key groups weren't written and therefore do not have offsets
defined, and the streams will be seeked to incorrect positions.
Ultimately, if some key groups were skipped while writing to the raw keyed
stream, the following error will be thrown on restore:
{code}
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:204)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.IOException: position out of bounds
at
org.apache.flink.runtime.state.StatePartitionStreamProvider.getStream(StatePartitionStreamProvider.java:58)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.internalTimeServiceManager(StreamTaskStateInitializerImpl.java:235)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:167)
... 9 more
Caused by: java.io.IOException: position out of bounds
at
org.apache.flink.runtime.state.memory.ByteStreamStateHandle$ByteStateHandleInputStream.seek(ByteStreamStateHandle.java:124)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl$KeyGroupStreamIterator.next(StreamTaskStateInitializerImpl.java:442)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl$KeyGroupStreamIterator.next(StreamTaskStateInitializerImpl.java:395)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.internalTimeServiceManager(StreamTaskStateInitializerImpl.java:228)
... 10 more
{code}
**Possible solutions**
There are 2 possible solutions, either by fixing in StateFun or in Flink:
- This can be fixed in StateFun by ensuring that the feedback logger starts a
new key group for all key groups in range, by doing:
{code}
for (int keyGroupId : rawKeyedStream.getKeyGroupList()) {
rawKeyedStream.startNewKeyGroup(keyGroupId);
// write to stream if there are logged messages for this key group
}
{code}
- Or, alternatively, we change the
{{KeyGroupRangeOffsets#KeyGroupOffsetsIterator}} in Flink to skip key groups
that don't have a defined offset (i.e. {{startNewKeyGroup}} wasn't called for
these key groups).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19741) InternalTimeServiceManager fails to restore if there are other users of raw keyed state streams
Tzu-Li (Gordon) Tai created FLINK-19741:
---
Summary: InternalTimeServiceManager fails to restore if there are
other users of raw keyed state streams
Key: FLINK-19741
URL: https://issues.apache.org/jira/browse/FLINK-19741
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.11.2, 1.10.2, 1.9.3
Reporter: Tzu-Li (Gordon) Tai
Currently, when restoring a {{InternalTimeServiceManager}}, we always attempt
to read from the provided raw keyed state streams (using
{{InternalTimerServiceSerializationProxy}}):
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManagerImpl.java#L117
This is incorrect, since we don't write with the
{{InternalTimerServiceSerializationProxy}} if the timers do not require legacy
synchronous snapshots:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManagerImpl.java#L192
(we currently only require that when users use RocksDB backend + heap timers).
Therefore, the {{InternalTimeServiceManager}} can fail to be created on restore
due to corrupt reads in the case where:
* a checkpoint was taken where {{useLegacySynchronousSnapshots}} is false
(hence nothing was written, and the time service manager does not use the raw
keyed stream)
* the raw keyed stream is used elsewhere (e.g. in the Flink application's user
code)
* on restore from the checkpoint, {{InternalTimeServiceManagerImpl.create()}}
attempts to read from the raw keyed stream with the
{{InternalTimerServiceSerializationProxy}}.
The fix would be to also respect the {{useLegacySynchronousSnapshots}} flag in:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManagerImpl.java#L231
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19620) Merge StateFun's ExactlyOnceE2E and RemoteModuleE2E
Tzu-Li (Gordon) Tai created FLINK-19620:
---
Summary: Merge StateFun's ExactlyOnceE2E and RemoteModuleE2E
Key: FLINK-19620
URL: https://issues.apache.org/jira/browse/FLINK-19620
Project: Flink
Issue Type: Test
Components: Stateful Functions, Tests
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, we have:
- {{ExactlyOnceE2E}}, which verifies end-to-end exactly-once in the presence of
TM failures, but uses embedded functions
- {{RemoteModuleE2E}} which runs functions remotely and verifies that the
communication between StateFun + the functions is correct, messages are routed
correctly.
A recent issue (https://github.com/apache/flink-statefun/pull/159) suggested
that we should add a E2E with remote functions + TM failures.
With this in mind, it is worth considering merging these 2 E2Es into one, as
together they should cover equal functionality (remote functions are built on
top of embedded functions), and having them merged also saves test time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19408) Update flink-statefun-docker release scripts for cross release Java 8 and 11
Tzu-Li (Gordon) Tai created FLINK-19408:
---
Summary: Update flink-statefun-docker release scripts for cross
release Java 8 and 11
Key: FLINK-19408
URL: https://issues.apache.org/jira/browse/FLINK-19408
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.0
Currently, the {{add-version.sh}} script in the {{flink-statefun-docker}} repo
does not generate Dockerfiles for different Java versions.
Since we have decided to cross-release images for Java 8 and 11, that script
needs to be updated as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19290) Add documentation for Stateful Function's Flink DataStream SDK
Tzu-Li (Gordon) Tai created FLINK-19290: --- Summary: Add documentation for Stateful Function's Flink DataStream SDK Key: FLINK-19290 URL: https://issues.apache.org/jira/browse/FLINK-19290 Project: Flink Issue Type: Task Components: Documentation, Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai The new Flink DataStream integration SDK for Stateful Functions is still lacking documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19256) Validate Stateful Functions configuration only where necessary
Tzu-Li (Gordon) Tai created FLINK-19256:
---
Summary: Validate Stateful Functions configuration only where
necessary
Key: FLINK-19256
URL: https://issues.apache.org/jira/browse/FLINK-19256
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, the Flink configuration is validated to contain necessary settings,
such as parent-first classloading patterns whereever
{{StatefulFunctionsConfig}} is instantiated.
This validation should not be part of creating a {{StatefulFunctionsConfig}},
and should be refactored out as a separate utility method that is called only
on necessary execution paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19199) Add execution attempt ID to feedback channel keys
Tzu-Li (Gordon) Tai created FLINK-19199:
---
Summary: Add execution attempt ID to feedback channel keys
Key: FLINK-19199
URL: https://issues.apache.org/jira/browse/FLINK-19199
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
The feedback brokers identify a single pair of producer / consumer with a
{{SubtaskFeedbackKey}}, which is uniquely identified within a job by only the
subtask index.
This can potentially become an issue in job restart scenarios, since the
feedback brokers are static fields, and are durable across non-TM failure job
restarts.
Currently it is not a problem because we do clear broker channels on close, but
adding an extra execution attempt ID into the {{SubtaskFeedbackKey}} can make
things safer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19197) Add documentation for PersistedStateRegistry
Tzu-Li (Gordon) Tai created FLINK-19197:
---
Summary: Add documentation for PersistedStateRegistry
Key: FLINK-19197
URL: https://issues.apache.org/jira/browse/FLINK-19197
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Documentation for the new {{PersistedStateRegistry}} state construct is still
missing. This should be added as a new sub-section beneath
https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/java.html#persistence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19130) Expose backpressure metrics / logs for function dispatcher operator
Tzu-Li (Gordon) Tai created FLINK-19130: --- Summary: Expose backpressure metrics / logs for function dispatcher operator Key: FLINK-19130 URL: https://issues.apache.org/jira/browse/FLINK-19130 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai As of now, there is no visibility on why or how backpressure is applied in Stateful Functions. This JIRA attemps to add two metrics as an initial effort of providing more visibility: - Total number of addresses that have asked to be blocked - Total number of inflight pending async operations -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19107) Add basic checkpoint and recovery config keys to template flink-conf.yaml
Tzu-Li (Gordon) Tai created FLINK-19107: --- Summary: Add basic checkpoint and recovery config keys to template flink-conf.yaml Key: FLINK-19107 URL: https://issues.apache.org/jira/browse/FLINK-19107 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.2.0 How to enable checkpointing in Stateful Functions seems to be a recurring question. Adding the relevant configuration keys to the template flink-conf.yaml in StateFun's Docker images could help with this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19106) Add more timeout options for remote function specs
Tzu-Li (Gordon) Tai created FLINK-19106:
---
Summary: Add more timeout options for remote function specs
Key: FLINK-19106
URL: https://issues.apache.org/jira/browse/FLINK-19106
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.0
As of now, we only support setting the call timeout for remote functions, which
spans a complete call including connection, writing request, server-side
processing, and reading response times.
To allow more fine-grained control of this, we propose to introduce
configuration keys for {{connectTimeout}} / {{readTimeout}} / {{writeTimeout}}
to remote function specs.
By default, these values should be 10 to be coherent with the current behaviour.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19102) Make StateBinder a per-FunctionType entity
Tzu-Li (Gordon) Tai created FLINK-19102:
---
Summary: Make StateBinder a per-FunctionType entity
Key: FLINK-19102
URL: https://issues.apache.org/jira/browse/FLINK-19102
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.0
Currently, a single {{StateBinder}} instance is used across multiple
{{FunctionType}} s for binding state objects to Flink state.
While this is fine, it creates a leak of internal runtime information to the
{{PersistedStateRegistry}} class. To fix this, we need to make {{FunctionType}}
a property of each individual state binder, etc. each state binder binds state
for a single {{FunctionType}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19096) Simplify PersistedStateRegistry state registration methods
Tzu-Li (Gordon) Tai created FLINK-19096:
---
Summary: Simplify PersistedStateRegistry state registration methods
Key: FLINK-19096
URL: https://issues.apache.org/jira/browse/FLINK-19096
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.0
Currently, to register for example a {{PersistedValue}} with a
{{PersistedStateRegistry}}, you create it by doing
{{registry.registerValue(name, type, expiration)}}.
This leads to duplicate signatures with the state class constructors, and
therefore requires separate synchronization across the methods.
We should change the syntax to accept a {{PersistedValue}} object directly, to
simplify things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19095) Add expire mode for remote function state TTL
Tzu-Li (Gordon) Tai created FLINK-19095: --- Summary: Add expire mode for remote function state TTL Key: FLINK-19095 URL: https://issues.apache.org/jira/browse/FLINK-19095 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.2.0 We did not allow setting expire mode for each remote function state before due to FLINK-18316. Now that remote function state is de-multiplexed, we can now easily support this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18340) Support directly providing bytes instead of a Protobuf messages when writing to Kafka / Kinesis with the Python SDK
Tzu-Li (Gordon) Tai created FLINK-18340:
---
Summary: Support directly providing bytes instead of a Protobuf
messages when writing to Kafka / Kinesis with the Python SDK
Key: FLINK-18340
URL: https://issues.apache.org/jira/browse/FLINK-18340
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Environment: State
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.2.0
This was insight from this ML thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Any-python-example-with-json-data-from-Kafka-using-flink-statefun-td42520.html
Currently, the {{kafka_egress_record}} and {{kinesis_egress_record}} methods in
the Python SDK only support providing a Protobuf message to be written to Kafka
/ Kinesis.
We should make this more flexible so that a user can directly supply bytes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18316) Add a dynamic state registration primitive for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-18316:
---
Summary: Add a dynamic state registration primitive for Stateful
Functions
Key: FLINK-18316
URL: https://issues.apache.org/jira/browse/FLINK-18316
Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, using the {{PersistedValue}} / {{PersistedTable}} /
{{PersistedAppendingBuffer}} primitives, the user can only eagerly define
states prior to function instance activation using the {{Persisted}} field
annotation.
We propose to add a primitive that allows them to register states dynamically
after activation (i.e. during runtime), along the lines of:
{code}
public MyStateFn implements StatefulFunction {
@Persisted
private final PersistedStateProvider provider = new
PersistedStateProvider();
public MyStateFn() {
PersistedValue valueState = provider.getValue(...);
}
void invoke(Object input) {
PersistedValue anotherValueState = provider.getValue(...);
}
}
{code}
Note how you can register state during instantiation (in the constructor) and
in the invoke method. Both registrations should be picked up by the runtime and
bound to Flink state.
This will be useful for a few scenarios:
- Could enable us to get rid of eager state spec definitions in the YAML
modules for remote functions in the future.
- Will allow new state to be registered in remote functions, without shutting
down the StateFun cluster.
- Moreover, this approach allows us to differentiate which functions have
dynamic state and which ones have only eager state, which might be handy in the
future in case there is a need to differentiate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18177) Document all YAML format versions for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-18177: --- Summary: Document all YAML format versions for Stateful Functions Key: FLINK-18177 URL: https://issues.apache.org/jira/browse/FLINK-18177 Project: Flink Issue Type: Task Reporter: Tzu-Li (Gordon) Tai In https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/modules.html#specification, we currently only document the latest format version. It could make sense to document multiple versions here, especially as we add more versions. Perhaps a starter example for referencing how we might structure the pages would be the Docker Compose version docs: https://docs.docker.com/compose/compose-file/compose-versioning/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18069) Scaladocs not building since inner Java interfaces cannot be recognized
Tzu-Li (Gordon) Tai created FLINK-18069:
---
Summary: Scaladocs not building since inner Java interfaces cannot
be recognized
Key: FLINK-18069
URL: https://issues.apache.org/jira/browse/FLINK-18069
Project: Flink
Issue Type: Bug
Components: API / Scala, Documentation
Affects Versions: 1.11.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Error:
{code}
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/Tuple2CaseClassSerializerSnapshot.java:98:
error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility
resolveOuterSchemaCompatibility(ScalaCaseClassSerializer>
newSerializer) {
^
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/TraversableSerializerSnapshot.java:101:
error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility
resolveOuterSchemaCompatibility(TraversableSerializer newSerializer) {
^
/home/buildslave/slave/flink-docs-master/build/flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/ScalaCaseClassSerializerSnapshot.java:106:
error: not found: type OuterSchemaCompatibility
protected OuterSchemaCompatibility
resolveOuterSchemaCompatibility(ScalaCaseClassSerializer newSerializer) {
^
{code}
This is a similar issue as reported here:
https://github.com/scala/bug/issues/10509.
This seems to be a problem with Scala 2.12.x. The only workaround is to
redundantly add the full-length qualifiers for such interfaces.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17997) Revert manual merging of AWS KPL's THIRD_PARTY_NOTICE files content in Stateful Functions distribution artifact
Tzu-Li (Gordon) Tai created FLINK-17997: --- Summary: Revert manual merging of AWS KPL's THIRD_PARTY_NOTICE files content in Stateful Functions distribution artifact Key: FLINK-17997 URL: https://issues.apache.org/jira/browse/FLINK-17997 Project: Flink Issue Type: Task Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.0.1, statefun-2.1.0 We manually merged the contents in FLINK-16901, because at the time the upstream Flink Kinesis connector wasn't yet properly handling the content. Now that this is fixed upstream, we can revert the fix in StateFun. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17963) Revert execution environment patching in StatefulFunctionsJob (FLINK-16926)
Tzu-Li (Gordon) Tai created FLINK-17963:
---
Summary: Revert execution environment patching in
StatefulFunctionsJob (FLINK-16926)
Key: FLINK-17963
URL: https://issues.apache.org/jira/browse/FLINK-17963
Project: Flink
Issue Type: Task
Components: Stateful Functions
Affects Versions: statefun-2.0.1, statefun-2.1.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
In FLINK-16926, we explicitly "patched" the {{StreamExecutionEnvironment}} due
to FLINK-16560.
Now that we have upgraded the Flink version in StateFun to 1.10.1 which
includes a fix for FLINK-16560, we can now revert the patching of
{{StreamExecutionEnvironment}} in the {{StatefulFunctionsJob}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17954) Do not multiplex remote function state into single PersistedTable
Tzu-Li (Gordon) Tai created FLINK-17954:
---
Summary: Do not multiplex remote function state into single
PersistedTable
Key: FLINK-17954
URL: https://issues.apache.org/jira/browse/FLINK-17954
Project: Flink
Issue Type: Task
Components: Stateful Functions
Affects Versions: statefun-2.0.1, statefun-2.1.0
Reporter: Tzu-Li (Gordon) Tai
We are currently multiplexing multiple remote function's user value states into
a single {{PersistedTable}}, using the state name as the table key.
This is not nice since:
- It does not allow individual states to have different properties, such as TTL
expiration.
- We are restricted to only value states for remote functions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17875) Support state TTL for remote functions
Tzu-Li (Gordon) Tai created FLINK-17875: --- Summary: Support state TTL for remote functions Key: FLINK-17875 URL: https://issues.apache.org/jira/browse/FLINK-17875 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.1.0 With FLINK-17644, we now have support for embedded functions. This should be extended to remote functions, by allowing the module specs to define the TTL for declared remote function state. With this, it is also likely that we need to uptick the version for the YAML module spec. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17712) Upgrade Flink version to 1.10.1 in Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-17712: --- Summary: Upgrade Flink version to 1.10.1 in Stateful Functions Key: FLINK-17712 URL: https://issues.apache.org/jira/browse/FLINK-17712 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.0.1, statefun-2.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17684) Run containerized end-to-end tests for all Travis builds
Tzu-Li (Gordon) Tai created FLINK-17684:
---
Summary: Run containerized end-to-end tests for all Travis builds
Key: FLINK-17684
URL: https://issues.apache.org/jira/browse/FLINK-17684
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
The end-to-end tests in Stateful Functions, so far, seems to be quite stable,
and do not take up much time.
We want to enable them (using the {{-Prun-e2e-tests}} profile) for all builds
per commit / PR for the time being.
If build time for Stateful Functions does get longer in the future, we may
still consider other approaches e.g. only running e2e tests with cron jobs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17605) Consolidate flink.version in Stateful Functions POM files
Tzu-Li (Gordon) Tai created FLINK-17605:
---
Summary: Consolidate flink.version in Stateful Functions POM files
Key: FLINK-17605
URL: https://issues.apache.org/jira/browse/FLINK-17605
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Right now, we have definitions of {{flink.version}} property scattered in
multiple POM files in Stateful Functions. They should be consolidated in the
root parent POM, as that should not likely change across modules.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17520) Extend CompositeTypeSerializerSnapshot to allow composite serializers to signal migration based on outer configuration
Tzu-Li (Gordon) Tai created FLINK-17520:
---
Summary: Extend CompositeTypeSerializerSnapshot to allow composite
serializers to signal migration based on outer configuration
Key: FLINK-17520
URL: https://issues.apache.org/jira/browse/FLINK-17520
Project: Flink
Issue Type: Improvement
Components: API / Type Serialization System
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Compatibility of composite serializers is governed by the overall resolved
compatibility of all its nested serializers, as well as any additional
configuration (or what we call the "outer configuration" or "outer snapshot").
The compatibility resolution logic for these composite serializers is
implemented in the {{CompositeTypeSerializerSnapshot}} abstract class.
One current limitation of this base class is that the implementation assumes
that the outer configuration is always either compatible, or incompatible.
We should relax this to also allow signaling migration, purely based on the
outer configuration. This is already a requirement by FLINK-16998.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17519) Add Java State Bootstrapping E2E test for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-17519:
---
Summary: Add Java State Bootstrapping E2E test for Stateful
Functions
Key: FLINK-17519
URL: https://issues.apache.org/jira/browse/FLINK-17519
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
Add an Stateful Functions E2E test that writes a savepoint using the state
bootstrapping API, which is compatible to be restored by the greeter example.
Then, deploy a Stateful Functions app using the
{{StatefulFunctionsAppContainers}} and restoring from the written savepoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17518) Add HTTP-based request reply protocol E2E test for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-17518:
---
Summary: Add HTTP-based request reply protocol E2E test for
Stateful Functions
Key: FLINK-17518
URL: https://issues.apache.org/jira/browse/FLINK-17518
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
The E2E test should contain of a standalone deployed containerized remote
function, e.g. using the Python SDK + Flask, as well as a Flink Stateful
Functions cluster deployed using the {{StatefulFunctionsAppsContainers}}
utility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17517) Add failure recovery / exactly-once E2E for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-17517:
---
Summary: Add failure recovery / exactly-once E2E for Stateful
Functions
Key: FLINK-17517
URL: https://issues.apache.org/jira/browse/FLINK-17517
Project: Flink
Issue Type: Sub-task
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
Implementation should be based on top of the {{StatefulFunctionsAppContainers}}
test containers utility.
The verification should write some records to a Stateful Functions app (via a
Kafka ingress), simulate failure by restarting containers, and assert that the
result is not corrupt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17516) Add more end-to-end tests for Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-17516: --- Summary: Add more end-to-end tests for Stateful Functions Key: FLINK-17516 URL: https://issues.apache.org/jira/browse/FLINK-17516 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Fix For: statefun-2.1.0 Umbrella ticket to track adding a few more end-to-end tests for Stateful Functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17060) "{flink}" isn't being replaced by the Apache Flink trademark in Stateful Functions docs
Tzu-Li (Gordon) Tai created FLINK-17060:
---
Summary: "{flink}" isn't being replaced by the Apache Flink
trademark in Stateful Functions docs
Key: FLINK-17060
URL: https://issues.apache.org/jira/browse/FLINK-17060
Project: Flink
Issue Type: Bug
Components: Documentation, Stateful Functions
Affects Versions: statefun-2.0.0
Reporter: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
See
https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/modules.html#embedded-module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16985) Specify job names for Stateful Functions examples
Tzu-Li (Gordon) Tai created FLINK-16985: --- Summary: Specify job names for Stateful Functions examples Key: FLINK-16985 URL: https://issues.apache.org/jira/browse/FLINK-16985 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Tzu-Li (Gordon) Tai Fix For: statefun-2.0.1 The StateFun examples all use the default job name "StatefulFunctions". It would be nice if they had specific job names, like "Greeter Example" or "Shopping Cart Example", etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16928) Switch back to new scheduler in Stateful Functions
Tzu-Li (Gordon) Tai created FLINK-16928: --- Summary: Switch back to new scheduler in Stateful Functions Key: FLINK-16928 URL: https://issues.apache.org/jira/browse/FLINK-16928 Project: Flink Issue Type: Task Components: Stateful Functions Affects Versions: statefun-2.0.0 Reporter: Tzu-Li (Gordon) Tai Fix For: statefun-2.0.1 This JIRA tracks switching back to using the new scheduler in Stateful Functions. We will be temporarily using the legacy scheduler in StateFun 2.0.0 due to FLINK-16927. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16927) Stateful Function jobs fail to restore after losing task slots using the new Flink scheduler
Tzu-Li (Gordon) Tai created FLINK-16927:
---
Summary: Stateful Function jobs fail to restore after losing task
slots using the new Flink scheduler
Key: FLINK-16927
URL: https://issues.apache.org/jira/browse/FLINK-16927
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-2.0.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
The following exception is being thrown indefinitely once a worker is lost:
{code}
org.apache.flink.util.FlinkException: Multi task slot is not local and, thus,
does not fulfill the co-location constraint.
{code}
This is caused by FLINK-16139, which is fixed in Flink 1.10.1.
To unblock the release for Stateful Functions 2.0.0, which depends on Flink
1.10.0,
for StateFun 2.0.0 we can use the old legacy scheduler for the time being.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16902) commons-codec dependency pulls in LICENSE.txt that should be excluded
Tzu-Li (Gordon) Tai created FLINK-16902:
---
Summary: commons-codec dependency pulls in LICENSE.txt that should
be excluded
Key: FLINK-16902
URL: https://issues.apache.org/jira/browse/FLINK-16902
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Affects Versions: statefun-2.0.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
This should be excluded from the {{statefun-flink-distribution}} packaging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16901) Flink Kinesis connector NOTICE should have contents of AWS KPL's THIRD_PARTY_NOTICES file manually merged in
Tzu-Li (Gordon) Tai created FLINK-16901:
---
Summary: Flink Kinesis connector NOTICE should have contents of
AWS KPL's THIRD_PARTY_NOTICES file manually merged in
Key: FLINK-16901
URL: https://issues.apache.org/jira/browse/FLINK-16901
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Affects Versions: 1.10.0
Reporter: Tzu-Li (Gordon) Tai
The Flink Kinesis connector artifact bundles AWS KPL's
[THIRD_PARTY_NOTICES|https://github.com/awslabs/amazon-kinesis-producer/blob/master/THIRD_PARTY_NOTICES]
under the {{META-INF}} folder.
The contents of this should be manually merged into the artifact's own NOTICE
file, and the {{THIRD_PARTY_NOTICES}} file itself excluded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16900) Exclude NOTCE and LICENSE files of Kafka dependencies for statefun-flink-distribution
Tzu-Li (Gordon) Tai created FLINK-16900:
---
Summary: Exclude NOTCE and LICENSE files of Kafka dependencies for
statefun-flink-distribution
Key: FLINK-16900
URL: https://issues.apache.org/jira/browse/FLINK-16900
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.0
As how we do it in the {{flink-sql-connector-kafka}} modules, the NOTICE and
LICENSE files pulled in from the Kafka dependencies of
{{statefun-flink-distribution}} and {{statefun-ridesharing-example-simulator}}
should be excluded from packaging.
They do not contain any relevant, as it cites a binary dependency on jersey,
but this is neither reflected in the dependency graph, nor are any jersey files
bundled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16880) Adapt Dockerfiles from flink-statefun to be added to flink-statefun-docker
Tzu-Li (Gordon) Tai created FLINK-16880:
---
Summary: Adapt Dockerfiles from flink-statefun to be added to
flink-statefun-docker
Key: FLINK-16880
URL: https://issues.apache.org/jira/browse/FLINK-16880
Project: Flink
Issue Type: Task
Components: Release System / Docker, Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.0
The scripts under {{tools/docker/}} in the flink-statefun repo contains scripts
that prepare the build context for building snapshot StateFun images that we
use for development purposes.
Those can be adapted so that they may be used for submission to the Docker Hub
official images repo:
- The build context should be readily in shape as is in the Dockerfile repo,
instead of relying on a build script
- {{statefun-flink-distribution}} and {{statefun-flink-core}} jars should be
downloaded from Maven + signature verification needs to be performed
- We need some tooling in place that allows us to automate the release workflow
of adding a new version for the images. We should be able to adapt those from
the existing Flink Dockerfiles repo.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16879) Disable the source-release-assembly execution goal when using the apache-release build profile
Tzu-Li (Gordon) Tai created FLINK-16879:
---
Summary: Disable the source-release-assembly execution goal when
using the apache-release build profile
Key: FLINK-16879
URL: https://issues.apache.org/jira/browse/FLINK-16879
Project: Flink
Issue Type: Task
Components: Release System, Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0.1, statefun-2.1.0
The {{apache-release}} profile defined in the Apache Parent POM defines a
{{source-release-assembly}} execution that packages a source release
distribution to be published to Maven.
We should disable this, because we use our own tools to package source release
distributions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16843) Python SDK distribution is missing LICENSE and NOTICE files
Tzu-Li (Gordon) Tai created FLINK-16843: --- Summary: Python SDK distribution is missing LICENSE and NOTICE files Key: FLINK-16843 URL: https://issues.apache.org/jira/browse/FLINK-16843 Project: Flink Issue Type: Bug Components: Stateful Functions Affects Versions: statefun-2.0 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Fix For: statefun-2.0 The Python SDK distributions for Stateful Functions do not bundle any LICENSE or NOTICE files. This should be fixed, as these are required to be included in all ASF-released distributions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16842) Ridesharing example simulator built artifact is missing NOTICE / LICENSE for bundled dependencies
Tzu-Li (Gordon) Tai created FLINK-16842:
---
Summary: Ridesharing example simulator built artifact is missing
NOTICE / LICENSE for bundled dependencies
Key: FLINK-16842
URL: https://issues.apache.org/jira/browse/FLINK-16842
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-2.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0
The {{statefun-ridesharing-example-simulator}} artifact bundles {{spring-boot}}
as a dependency, which in turn pulls in some other dependencies that are
non-ASLv2.
We should add NOTICE / LICENSE files to the built artifact for those.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16841) Stateful Function artifacts jars should not bundle proto sources
Tzu-Li (Gordon) Tai created FLINK-16841:
---
Summary: Stateful Function artifacts jars should not bundle proto
sources
Key: FLINK-16841
URL: https://issues.apache.org/jira/browse/FLINK-16841
Project: Flink
Issue Type: Bug
Components: Build System, Stateful Functions
Affects Versions: statefun-2.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Fix For: statefun-2.0
These protobuf definition files are being bundled in built artifacts:
{code}
google/protobuf/any.proto
google/protobuf/api.proto
google/protobuf/descriptor.proto
google/protobuf/duration.proto
google/protobuf/empty.proto
google/protobuf/field_mask.proto
google/protobuf/source_context.proto
google/protobuf/struct.proto
google/protobuf/timestamp.proto
google/protobuf/type.proto
google/protobuf/wrappers.proto
{code}
This is caused by the {{addProtoSources}} configuration of the
{{protoc-jar-maven-plugin}}.
We should remove those, because:
- Bundling those will require licensing acknowledgement to Protobuf in our
artifacts.
- Those definition files are not used directly by Stateful Functions at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16838) Stateful Functions Quickstart archetype Dockerfile should reference a specific version tag
Tzu-Li (Gordon) Tai created FLINK-16838:
---
Summary: Stateful Functions Quickstart archetype Dockerfile should
reference a specific version tag
Key: FLINK-16838
URL: https://issues.apache.org/jira/browse/FLINK-16838
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Currently, the quickstart archetype provides a skeleton Dockerfile that always
builds on top of the latest image:
{code}
FROM statefun
{code}
While it happens to work for the first release since the `latest` tag will
(coincidentally) point to the correct version,
once we have multiple releases this will no longer be correct.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16738) Add convenience Kinesis generic egress record builder to the Python SDK
Tzu-Li (Gordon) Tai created FLINK-16738:
---
Summary: Add convenience Kinesis generic egress record builder to
the Python SDK
Key: FLINK-16738
URL: https://issues.apache.org/jira/browse/FLINK-16738
Project: Flink
Issue Type: Task
Components: Stateful Functions
Affects Versions: statefun-2.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
Similar to the {{kafka_egress_record}} builder method, we're missing a
counterpart for the Kinesis generic egress in the StateFun Python SDK.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16706) Update Stateful Functions master branch version to 2.0-SNAPSHOT
Tzu-Li (Gordon) Tai created FLINK-16706:
---
Summary: Update Stateful Functions master branch version to
2.0-SNAPSHOT
Key: FLINK-16706
URL: https://issues.apache.org/jira/browse/FLINK-16706
Project: Flink
Issue Type: Task
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
As discussed in
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Update-on-Flink-Stateful-Functions-amp-what-are-the-next-steps-td38646.html,
the community wants the next (and first Apache-endorsed) Stateful Functions
release number to be 2.0.
We should update the master branch snapshot version, as well as the {{Affects
Version}} and {{Fixed Version}} fields of all Stateful Functions JIRAs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16680) Add documentation for Stateful Function's State Bootstrapping API
Tzu-Li (Gordon) Tai created FLINK-16680: --- Summary: Add documentation for Stateful Function's State Bootstrapping API Key: FLINK-16680 URL: https://issues.apache.org/jira/browse/FLINK-16680 Project: Flink Issue Type: Task Components: Documentation, Stateful Functions Affects Versions: statefun-1.1 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai The documentation of StateFun is missing documentation for its State Bootstrapping API. I propose to initially add it as a sub-page under the "Deployment and Operations" category: https://ci.apache.org/projects/flink/flink-statefun-docs-master/deployment_operations/index.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16569) Allow empty keys in Kafka egress messages for Statefun Python SDK
Tzu-Li (Gordon) Tai created FLINK-16569: --- Summary: Allow empty keys in Kafka egress messages for Statefun Python SDK Key: FLINK-16569 URL: https://issues.apache.org/jira/browse/FLINK-16569 Project: Flink Issue Type: Task Components: Stateful Functions Affects Versions: statefun-1.1 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai The generic YAML-ized Kafka egress does allow messages to not have keys defined. However, the convenience Kafka egress messages builder in the Python SDK requires keys to always be set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16558) Reword Stateful Functions doc's tagline
Tzu-Li (Gordon) Tai created FLINK-16558: --- Summary: Reword Stateful Functions doc's tagline Key: FLINK-16558 URL: https://issues.apache.org/jira/browse/FLINK-16558 Project: Flink Issue Type: Task Components: Documentation, Stateful Functions Affects Versions: statefun-1.1 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai The current tagline is "A framework for stateful distributed applications by the original creators of Apache Flink®." The part about "by the original creators of Apache Flink" reads a bit out-of-place now, since the project is now maintained by the Apache Flink community. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16557) Document YAML-ized Kafka egresses / ingresses in Stateful Functions documentation
Tzu-Li (Gordon) Tai created FLINK-16557: --- Summary: Document YAML-ized Kafka egresses / ingresses in Stateful Functions documentation Key: FLINK-16557 URL: https://issues.apache.org/jira/browse/FLINK-16557 Project: Flink Issue Type: Task Components: Stateful Functions Affects Versions: statefun-1.1 Reporter: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai The Stateful Functions documentation is still missing information about YAML-ized egresses / ingresses. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16520) Move Kafka client properties resolution in KafkaSinkProvider to KafkaEgressBuilder
Tzu-Li (Gordon) Tai created FLINK-16520:
---
Summary: Move Kafka client properties resolution in
KafkaSinkProvider to KafkaEgressBuilder
Key: FLINK-16520
URL: https://issues.apache.org/jira/browse/FLINK-16520
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
We've moved Kafka client properties resolution from the {{KafkaSourceProvider}}
to be consolidated in the {{KafkaIngressBuilder}} in FLINK-15769.
The same should also be done for the producer side, i.e. for
{{KafkaSinkProvider}}, by moving the properties resolution logic to
{{KafkaEgressBuilder}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16518) Stateful Function's KafkaSinkProvider should use `setProperty` instead of `put` for resolving client properties
Tzu-Li (Gordon) Tai created FLINK-16518:
---
Summary: Stateful Function's KafkaSinkProvider should use
`setProperty` instead of `put` for resolving client properties
Key: FLINK-16518
URL: https://issues.apache.org/jira/browse/FLINK-16518
Project: Flink
Issue Type: Bug
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
The {{put}} method is strongly discourage to be used on {{Properties}} as a bad
practice, since it allows putting non-string values.
This has already caused a bug, where a long was put into the properties, while
Kafka was expecting an integer:
{code}
org.apache.kafka.common.config.ConfigException: Invalid value 10 for
configuration transaction.timeout.ms: Expected value to be a 32-bit integer,
but it was a java.lang.Long
at
org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:669)
at
org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:471)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:464)
at
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
at
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75)
at
org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:396)
at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:326)
at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
at
org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.(FlinkKafkaInternalProducer.java:76)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
