[jira] [Created] (FLINK-26378) caller's address isn't set to None properly when a message is coming from an ingress
Igal Shilman created FLINK-26378: Summary: caller's address isn't set to None properly when a message is coming from an ingress Key: FLINK-26378 URL: https://issues.apache.org/jira/browse/FLINK-26378 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25936) MutableTypeCell is not setting correctly the value's type_name
Igal Shilman created FLINK-25936: Summary: MutableTypeCell is not setting correctly the value's type_name Key: FLINK-25936 URL: https://issues.apache.org/jira/browse/FLINK-25936 Project: Flink Issue Type: Improvement Reporter: Igal Shilman In the remote Java SDK, when a type doesn't specify an IMMUTABLE_VALUE type characteristics, (which is the case for custom type implementations by default) the type_name field is not set correctly on the resulting TypedValue. A MutableTypeCell was assuming incorrectly that the backend will send a TypedValue with a type_name field set even if the value is missing (TypedValue.has_value = false). This is not the case, and hence the type_name needs to be set explicitly. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25935) Add a harness based entry point to simply getting started.
Igal Shilman created FLINK-25935: Summary: Add a harness based entry point to simply getting started. Key: FLINK-25935 URL: https://issues.apache.org/jira/browse/FLINK-25935 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman It would be nicer to improve the getting started experience by providing an additional entry point in the StateFun distribution that is built on the Harness. This can be as simple as providing a Main function that configures RocksDB and starts the StateFun Flink job. The rest of the configurations needs to come from the module.yaml Having something like that will allow us to simplfy the playground even further by reducing the start time and the memory requirements for a docker-compose based example. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25934) Modernize statefun playground examples
Igal Shilman created FLINK-25934: Summary: Modernize statefun playground examples Key: FLINK-25934 URL: https://issues.apache.org/jira/browse/FLINK-25934 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman It is about time to touch up abit the examples in playground. Most of the docker-compose/docker files are pretty old and there are a lot of room for improvement. # use redpanda instead of kafka+zk - from local experiments it seems to cut the start time and the memory requirements significantly. In addition it also comes with a REST proxy, which can improve the interactivity with the examples quite a lot. # For the Java examples, there is no reason to use java8 for the remote functions. We can use at least 11, if not higher. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25933) Allow configuring different transports in RequestReplyFunctionBuilder
Igal Shilman created FLINK-25933: Summary: Allow configuring different transports in RequestReplyFunctionBuilder Key: FLINK-25933 URL: https://issues.apache.org/jira/browse/FLINK-25933 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently it is not possible to configure the type of the transport used while using the data stream integration. It would be useful to do so. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25866) Support additional TLS configuration.
Igal Shilman created FLINK-25866: Summary: Support additional TLS configuration. Key: FLINK-25866 URL: https://issues.apache.org/jira/browse/FLINK-25866 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently the default HTTP client used to invoke remote functions does not support customising the TLS settings as part of the endpoint spec definition. This includes using self-signed certificates, and providing client side certificates for authentication (which is a slightly different requirement). This issue is about including additional TLS settings to the default endpoint resource definition, and supporting them in statefun-core. User mailing list threads: * [client cert auth in remote function|https://lists.apache.org/thread/97nw245kxqp32qglwfynhhgyhgp2pxvg] * [endpoint self-signed certificate problem|https://lists.apache.org/thread/y2m2bpwg4n71rxfont6pgky2t8m19n7w] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25752) Create a migration e2e test
Igal Shilman created FLINK-25752:
Summary: Create a migration e2e test
Key: FLINK-25752
URL: https://issues.apache.org/jira/browse/FLINK-25752
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Igal Shilman
A separate e2e migration test is needed that verifies that a save point taken
with prior statefun versions is successfully opened in the current version.
We need this test to run within its own migration profile, that includes at
least a Kafka ingress/egress pair.
It can be completely based remote functions, without any verification/failure
injection aspects to it, and this probably should be based on
{{testcontainers}} instead of the harness.
having test containers will allows the test to create a save point in an
arbitrary previous version (say 3.1.1)
based on the previously released official docker images, and load it with the
current version.
The test would be bit more time consuming, and that's why I would recommend
that to run under a different profile.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25367) Upgrade Flink dependency to 1.13.5
Igal Shilman created FLINK-25367: Summary: Upgrade Flink dependency to 1.13.5 Key: FLINK-25367 URL: https://issues.apache.org/jira/browse/FLINK-25367 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman This applies to the 3.1 release series. Flink 1.13.5 addresses log4j CVEs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25355) Upgrade Flink dependency to 1.13.3
Igal Shilman created FLINK-25355: Summary: Upgrade Flink dependency to 1.13.3 Key: FLINK-25355 URL: https://issues.apache.org/jira/browse/FLINK-25355 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-3.1.0 Reporter: Igal Shilman Fix For: statefun-3.1.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25309) Upgrade to the latest Flink version
Igal Shilman created FLINK-25309: Summary: Upgrade to the latest Flink version Key: FLINK-25309 URL: https://issues.apache.org/jira/browse/FLINK-25309 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25308) Support different remote functions module names
Igal Shilman created FLINK-25308: Summary: Support different remote functions module names Key: FLINK-25308 URL: https://issues.apache.org/jira/browse/FLINK-25308 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently StateFun will search in the class path, for a file named module.yaml. But it seems it might be useful to specifiy a different name (for example based on the environment: module-prod.yaml vs module-dev.yaml) See the following thread: https://lists.apache.org/thread/dpp45wr81gsmnjcoqtotprnt6gx9comq -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24534) Set async based transport for remote functions default
Igal Shilman created FLINK-24534: Summary: Set async based transport for remote functions default Key: FLINK-24534 URL: https://issues.apache.org/jira/browse/FLINK-24534 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Currently, in an endpoint definition, if a transport field is not specified we assume that the transport is the synchronous one (io.statefun.transports.v1/okhttp), we should make the asynchronous transport (io.statefun.transports.v1/async) to be the default one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24364) Add a delayed message cancellation to the embedded smoke test
Igal Shilman created FLINK-24364: Summary: Add a delayed message cancellation to the embedded smoke test Key: FLINK-24364 URL: https://issues.apache.org/jira/browse/FLINK-24364 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman StateFun supports sending delayed messages with a user defined cancellation token, and in addition canceling delayed messaging by providing this token. This feature is based on best-effort and it is non-deterministic, therefore it is difficult to craft an e2e test to capture it (it is covered with a unit test, where the exact conditions can be fixed) I propose non the less, to add it to the smoke as a leaf command, meaning, a command that doesn't result in state increment either). But nevertheless it basically simulates a user that is "clicking around". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24322) Improve TypeScript support for the JS SDK
Igal Shilman created FLINK-24322: Summary: Improve TypeScript support for the JS SDK Key: FLINK-24322 URL: https://issues.apache.org/jira/browse/FLINK-24322 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Ingo Bürk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24284) Add a greeter and a showcase for the JavaScript SDK
Igal Shilman created FLINK-24284: Summary: Add a greeter and a showcase for the JavaScript SDK Key: FLINK-24284 URL: https://issues.apache.org/jira/browse/FLINK-24284 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman We need to add a greeter and a showcase for the Javascript SDK to the playground. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24256) Add JavaScript SDK for Stateful Functions
Igal Shilman created FLINK-24256: Summary: Add JavaScript SDK for Stateful Functions Key: FLINK-24256 URL: https://issues.apache.org/jira/browse/FLINK-24256 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24132) directory-maven-plugin fails on Github action
Igal Shilman created FLINK-24132:
Summary: directory-maven-plugin fails on Github action
Key: FLINK-24132
URL: https://issues.apache.org/jira/browse/FLINK-24132
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Igal Shilman
Starting at around Thu Sep 2 17:10:23 CEST 2021,
StateFun master and previously green branches started to fail with
{code:java}
Error: Failed to execute goal
org.commonjava.maven.plugins:directory-maven-plugin:0.1:highest-basedir
(directories) on project statefun-parent: Cannot find a single highest
directory for this project set. First two candidates directories don't share a
common root. -> [Help 1]
3273Error:
3274Error: To see the full stack trace of the errors, re-run Maven with the -e
switch.
3275Error: Re-run Maven using the -X switch to enable full debug logging.
3276Error:
3277Error: For more information about the errors and possible solutions,
please read the following articles:
3278Error: [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
3279Error: Process completed with exit code 1.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24107) StateFun doesn't open correctly in intellij when started with JDK11
Igal Shilman created FLINK-24107: Summary: StateFun doesn't open correctly in intellij when started with JDK11 Key: FLINK-24107 URL: https://issues.apache.org/jira/browse/FLINK-24107 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman After merging [https://github.com/apache/flink-statefun/pull/262] , the IntelliJ IDE (that is started with java11) mistakenly classifies modules as having a source version 11. This happens because the profile specified here: [https://github.com/apache/flink-statefun/blob/21ac5f6c8a49fab4598832372636b345cc365921/pom.xml#L137,L151] Is activated (it picks up InteliJ's JDK version) and it sets automatically the version to be 11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23966) Don't try to add swifts filesystem plugins in the Dockerfile
Igal Shilman created FLINK-23966: Summary: Don't try to add swifts filesystem plugins in the Dockerfile Key: FLINK-23966 URL: https://issues.apache.org/jira/browse/FLINK-23966 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman It seems like the swiftfs filesystem plugin is not part of Flink's base docker image anymore, therefore we need to remove it from the derived statefun image as-well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23964) Add Shaded Netty TCnative to lib/ in the docker image
Igal Shilman created FLINK-23964: Summary: Add Shaded Netty TCnative to lib/ in the docker image Key: FLINK-23964 URL: https://issues.apache.org/jira/browse/FLINK-23964 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Flink base image contains flink-shaded-netty-tcnative-dynamic-*.jar under opt/ Since we are now using netty as a transport for remote functions, we better move that artifact to lib/ so that it will be present at runtime, and hence providing access to openssl. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23712) Improve Protobuf serialization performance
Igal Shilman created FLINK-23712: Summary: Improve Protobuf serialization performance Key: FLINK-23712 URL: https://issues.apache.org/jira/browse/FLINK-23712 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Profiling allocations shows that a quarter of the allocations happens while parsing Protobuf, this happens because the CodedInputStream/CodedOutputStream allocate a 4kb buffer on construction. This can be easily avoided by caching them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23711) Add an asynchronous HTTP transport for the RequestReply function
Igal Shilman created FLINK-23711: Summary: Add an asynchronous HTTP transport for the RequestReply function Key: FLINK-23711 URL: https://issues.apache.org/jira/browse/FLINK-23711 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Currently StateFun uses a synchronous threaded HTTP client to dispatch invocations to the remote functions. This is highly undesirable, and can be easily changed. We already use and shade Netty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23039) Support pluggable transports for HTTP endpoints
Igal Shilman created FLINK-23039:
Summary: Support pluggable transports for HTTP endpoints
Key: FLINK-23039
URL: https://issues.apache.org/jira/browse/FLINK-23039
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Igal Shilman
Fix For: statefun-3.1.0
We've recently learned about a use case that requires using a custom client
that dispatches the HTTP requests (due to some internal reasons).
This can be a useful addition to further customizing the exact client code,
that suites the user's need. (for example adding company specific tracing
information)
This is technically feasible as well, as all it takes is:
1) provide an implementation of this
[interface|https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/reqreply/RequestReplyClient.java]
2) extend the ability to configure it from information present at the
module.yaml.
The current proposal is to add an optional "transport" section to endpoint
definition:
{code:java}
- endpoint:
meta:
kind: http
spec:
functions: com.foo.bar/*
transport:
provider_class: com.foo.bar.ClientProvider
some:
internal:
property: 123
urlPathTemplate: http://bar.foo.com:8080/functions/{function.name}
maxNumBatchRequests: 1 {code}
If the transport is not present we assume that the StateFun's pre-bundled
transport is present.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22809) Improve the exception message when a KafkaIngress receives a NULL key.
Igal Shilman created FLINK-22809: Summary: Improve the exception message when a KafkaIngress receives a NULL key. Key: FLINK-22809 URL: https://issues.apache.org/jira/browse/FLINK-22809 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Fix For: statefun-3.1.0 Mailing list discussion: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Statefun-2-2-2-Checkpoint-restore-NPE-td44022.html The following line throws a generic NPE, instead it should a bit more descriptive. https://github.com/apache/flink-statefun/blob/release-2.2/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/RoutableProtobufKafkaIngressDeserializer.java#L48 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22633) Add a Harness example to flink-statefun-playground
Igal Shilman created FLINK-22633: Summary: Add a Harness example to flink-statefun-playground Key: FLINK-22633 URL: https://issues.apache.org/jira/browse/FLINK-22633 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Fix For: statefun-3.1.0 We need to add an easy, in the IDE, getting started example for Java. This example can be based by the Harness and the remote Java SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22584) Use protobuf-shaded in StateFun core.
Igal Shilman created FLINK-22584:
Summary: Use protobuf-shaded in StateFun core.
Key: FLINK-22584
URL: https://issues.apache.org/jira/browse/FLINK-22584
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Igal Shilman
We have *statefun-protobuf-shaded* module, that is used by the remote java sdk,
we can use it to shade protobuf internally.
The major hurdle we need to overcome is that, in embedded functions, we have to
be able to accept instances of protobuf generated messages by the user.
For example:
{code:java}
UserProfile userProfile = UserProfile.newBilder().build();
context.send(..., userProfile) {code}
If we will simply use the shaded Protobuf version, we will get immediately a
class cast exception.
One way to overcome this is to use reflection and find the well known methods
on the generated classes and call toBytes() / parseFrom() reflectively.
This however will cause a significant slow down, even by using MethodHandles.
A small experiment that I've previously done with ByteBuddy mitigates this, by
generating
accessors, in pre-flight:
{code:java}
package org.apache.flink.statefun.flink.common.protobuf.serde;
import static net.bytebuddy.matcher.ElementMatchers.named;import
java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import net.bytebuddy.ByteBuddy;
import net.bytebuddy.dynamic.DynamicType;
import net.bytebuddy.implementation.FixedValue;
import net.bytebuddy.implementation.MethodCall;
import net.bytebuddy.implementation.bytecode.assign.Assigner;final class
ReflectiveProtobufSerde { @SuppressWarnings({"unchecked", "rawtypes"})
static ProtobufSerde ofProtobufGeneratedType(Class type) {
try {
DynamicType.Unloaded unloaded = configureByteBuddy(type);
Class writer =
unloaded.load(type.getClassLoader()).getLoaded(); return
(ProtobufSerde) writer.getDeclaredConstructor().newInstance();
} catch (Throwable e) {
throw new IllegalArgumentException();
}
} @SuppressWarnings("rawtypes")
private static DynamicType.Unloaded
configureByteBuddy(Class type)
throws NoSuchMethodException, InvocationTargetException,
IllegalAccessException {
Method writeToMethod = type.getMethod("writeTo", OutputStream.class);
Method parseFromMethod = type.getMethod("parseFrom", InputStream.class);
Method getSerializedSizeMethod = type.getMethod("getSerializedSize");//
get the message full name
Method getDescriptorMethod = type.getMethod("getDescriptor");
Object descriptor = getDescriptorMethod.invoke(null);
Method getFullNameMethod = descriptor.getClass().getMethod("getFullName");
String messageFullName = (String) getFullNameMethod.invoke(descriptor);
return new ByteBuddy()
.subclass(ProtobufSerde.class)
.typeVariable("M", type)
.method(named("writeTo"))
.intercept(
MethodCall.invoke(writeToMethod)
.onArgument(0)
.withArgument(1)
.withAssigner(Assigner.DEFAULT, Assigner.Typing.DYNAMIC))
.method(named("parseFrom"))
.intercept(MethodCall.invoke(parseFromMethod).withArgument(0))
.method(named("getSerializedSize"))
.intercept(
MethodCall.invoke(getSerializedSizeMethod)
.onArgument(0)
.withAssigner(Assigner.DEFAULT, Assigner.Typing.DYNAMIC))
.method(named("getMessageFullName"))
.intercept(FixedValue.value(messageFullName))
.make();
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22552) Rebase StateFun on Flink 1.13
Igal Shilman created FLINK-22552: Summary: Rebase StateFun on Flink 1.13 Key: FLINK-22552 URL: https://issues.apache.org/jira/browse/FLINK-22552 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Following the recent release of Flink 1.13, StateFun master needs to be rebased on that version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22533) Allow creating custom metrics
Igal Shilman created FLINK-22533: Summary: Allow creating custom metrics Key: FLINK-22533 URL: https://issues.apache.org/jira/browse/FLINK-22533 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently it is not possible to create custom metrics in StateFun. Let us consider supporting these. Mailing list thread: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-metrics-in-Stateful-Functions-td43282.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22532) Improve the support for remote functions in the DataStream integration
Igal Shilman created FLINK-22532: Summary: Improve the support for remote functions in the DataStream integration Key: FLINK-22532 URL: https://issues.apache.org/jira/browse/FLINK-22532 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-3.0.0 Reporter: Igal Shilman While looking at [RoutableMessabeBuilder.java|https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/RoutableMessageBuilder.java#L57] it is not that clear that the argument for a remote function needs to be of type TypedValue. We need to think of how to improve the end experience for this use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22531) Improve the support for finite streaming jobs with async operations.
Igal Shilman created FLINK-22531: Summary: Improve the support for finite streaming jobs with async operations. Key: FLINK-22531 URL: https://issues.apache.org/jira/browse/FLINK-22531 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-3.0.0 Reporter: Igal Shilman Finite streaming jobs will terminate even in the presence of asynchronous in flight operations. Looking at the AsyncWait operator, it seems that it can be mitigated by using the following interface: [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/BoundedOneInput.java#L27] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22185) Add k8s example to flink-statefun-playground
Igal Shilman created FLINK-22185: Summary: Add k8s example to flink-statefun-playground Key: FLINK-22185 URL: https://issues.apache.org/jira/browse/FLINK-22185 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21961) Add JSON StateFun type to the Python SDK
Igal Shilman created FLINK-21961: Summary: Add JSON StateFun type to the Python SDK Key: FLINK-21961 URL: https://issues.apache.org/jira/browse/FLINK-21961 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Since the json module is builtin in Python, we can support a builtin type constructor that is backed by JSON. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21960) Simplify module.yaml spec
Igal Shilman created FLINK-21960: Summary: Simplify module.yaml spec Key: FLINK-21960 URL: https://issues.apache.org/jira/browse/FLINK-21960 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman There are some fields that are inconsistent/confusing with the new SDKs. Let's simplify what we can. For example: spec.endpoints.endpoint.typename could be changed to spec.endpoints.endpoint.targetFunction: /* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21959) Rename Kafka/Kinesis ingress/egress types.
Igal Shilman created FLINK-21959: Summary: Rename Kafka/Kinesis ingress/egress types. Key: FLINK-21959 URL: https://issues.apache.org/jira/browse/FLINK-21959 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman The ingresses for Kinesis and Kafka are a bit strange at the moment, lets simplify the names to: io.statefun.kafka/ingress io.statefun.kafka/egress io.statefun.kinesis/ingress > io.statefun.kafka/egress -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21958) Rename statefun-sdk to statefun-sdk-embedded
Igal Shilman created FLINK-21958: Summary: Rename statefun-sdk to statefun-sdk-embedded Key: FLINK-21958 URL: https://issues.apache.org/jira/browse/FLINK-21958 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman having statefun-sdk and statefun-sdk-java is a bit confusing, lets statefun-sdk to reflect that this is the SDK for the embedded functions only. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21957) Rename statefun-python-sdk => statefun-sdk-python
Igal Shilman created FLINK-21957: Summary: Rename statefun-python-sdk => statefun-sdk-python Key: FLINK-21957 URL: https://issues.apache.org/jira/browse/FLINK-21957 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman This is not consistent with the other SDKs, let us rename that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21956) NPE in Kafka ingress when the key is missing
Igal Shilman created FLINK-21956: Summary: NPE in Kafka ingress when the key is missing Key: FLINK-21956 URL: https://issues.apache.org/jira/browse/FLINK-21956 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Our Kafka ingress requires a utf8 key, but if this key is missing, we need to display a meaningful message. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21955) Remove the old examples
Igal Shilman created FLINK-21955: Summary: Remove the old examples Key: FLINK-21955 URL: https://issues.apache.org/jira/browse/FLINK-21955 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman We currently have few examples under statefun-examples/ which are outdated. Some of them are ported to the new repository: apache/flink-statefun-playground We need to delete them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21932) Add a Python SDK showcase to the playground
Igal Shilman created FLINK-21932: Summary: Add a Python SDK showcase to the playground Key: FLINK-21932 URL: https://issues.apache.org/jira/browse/FLINK-21932 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman We need a Python showcase SDK that demonstrates various SDK concepts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21882) Create a Python based greeter example
Igal Shilman created FLINK-21882: Summary: Create a Python based greeter example Key: FLINK-21882 URL: https://issues.apache.org/jira/browse/FLINK-21882 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21828) Add an AWS-λ example to stateful-playground
Igal Shilman created FLINK-21828: Summary: Add an AWS-λ example to stateful-playground Key: FLINK-21828 URL: https://issues.apache.org/jira/browse/FLINK-21828 Project: Flink Issue Type: Task Reporter: Igal Shilman Assignee: Konstantin Knauf It could be nice to add an example to: [https://github.com/apache/flink-statefun-playground] under deployments/aws -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21827) Migrate the Python examples to the new SDK
Igal Shilman created FLINK-21827: Summary: Migrate the Python examples to the new SDK Key: FLINK-21827 URL: https://issues.apache.org/jira/browse/FLINK-21827 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Our current examples for the Python SDK are using the previous SDK, we need to migrate them to the new SDK, and move them to the new repo: flink-statefun-playground -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21826) Migrate the shopping-cart-example to the new SDK.
Igal Shilman created FLINK-21826: Summary: Migrate the shopping-cart-example to the new SDK. Key: FLINK-21826 URL: https://issues.apache.org/jira/browse/FLINK-21826 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Alexander Fedulov We have a shopping cart example: [https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-shopping-cart-example] Witten against the old embedded SDK, it would be nice if we can migrate it to the new remote SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21824) Add a simple Json Kafka producer script
Igal Shilman created FLINK-21824: Summary: Add a simple Json Kafka producer script Key: FLINK-21824 URL: https://issues.apache.org/jira/browse/FLINK-21824 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Add a simple script to be used from various flink-statefun-playground examples. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21721) Add a generic egress message builder to the SDK
Igal Shilman created FLINK-21721: Summary: Add a generic egress message builder to the SDK Key: FLINK-21721 URL: https://issues.apache.org/jira/browse/FLINK-21721 Project: Flink Issue Type: Task Reporter: Igal Shilman The Java SDK is missing a generic egress message builder (an egress that is not Kinesis or Kafka). This exists in the Python SDK: [https://github.com/apache/flink-statefun/blob/master/statefun-python-sdk/statefun/messages.py#L178] And it is used to send messages to custom egresses. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21692) Update the Python SDK to reflect the latest changes in the request-reply protocol
Igal Shilman created FLINK-21692: Summary: Update the Python SDK to reflect the latest changes in the request-reply protocol Key: FLINK-21692 URL: https://issues.apache.org/jira/browse/FLINK-21692 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Following the recent effort of standardizing the SDK surface for all the languages, we need to update the Python SDK to reflect these changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21642) RequestReplyFunction recovery fails with a remote SDK
Igal Shilman created FLINK-21642: Summary: RequestReplyFunction recovery fails with a remote SDK Key: FLINK-21642 URL: https://issues.apache.org/jira/browse/FLINK-21642 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman While extending our smoke e2e test to use the remote SDKS I've stumbled upon a bug in the RequestReplyFunction. We get a unknown state exception after recovery. The exact scenario that trigger that bug is: # There was request in flight. # A failure occurs that causes the job to restart. # On restore, we start with no managed state # But we try to re-send to the SDK exactly the same ToFunction message. # That ToFunction contains state definitions from the previous attempt. (before the failure) # The SDK processes this message normally (it has all the state definitions that it knows) # The SDK responds with a state mutation. # The PersistedRemoteFunctionValues fails with unknown state. We need to treat the ToFunction messages as a retryBatch, instead of sending it as-is. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21483) Disable the spot bugs plugin.
Igal Shilman created FLINK-21483: Summary: Disable the spot bugs plugin. Key: FLINK-21483 URL: https://issues.apache.org/jira/browse/FLINK-21483 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman The spotbugs plugin is enabled by default, and adds a significant increases at build time with a little value. Most of the issues found by the plugin we eventually have to suppress because it is unaware of the wider execution context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21457) Add support to differentiate a zero length value bytes and non existing value.
Igal Shilman created FLINK-21457: Summary: Add support to differentiate a zero length value bytes and non existing value. Key: FLINK-21457 URL: https://issues.apache.org/jira/browse/FLINK-21457 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21280) Investigate a way to disable object reuse in StateFun
Igal Shilman created FLINK-21280: Summary: Investigate a way to disable object reuse in StateFun Key: FLINK-21280 URL: https://issues.apache.org/jira/browse/FLINK-21280 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Currently in StateFun, we require object reuse to be enabled. We are doing it because Flink sources can emit arbitrary objects, but the first occurrence of the user defined payload serializer happens in a IngressRouterOperator. This works due to the fact that this operator is chained to the user, and with object reuse, the source output is passed as reference (without any Flink serialization) to the IngressRouterOperator. Unfortunately JdbcSink requires object reuse to be disabled, and therefore can't be used directly with StateFun. This issue was first reported on the mailing list: [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Stateful-Functions-JDBC-Sink-Problems-td41265.html] This issue is about exploring a way to remove the object reuse requirement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21243) Add Java SDK maven module
Igal Shilman created FLINK-21243: Summary: Add Java SDK maven module Key: FLINK-21243 URL: https://issues.apache.org/jira/browse/FLINK-21243 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Add the Skelton maven module that will contain the Java SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21230) Add protobuf wrapper types for the StateFun SDK types.
Igal Shilman created FLINK-21230: Summary: Add protobuf wrapper types for the StateFun SDK types. Key: FLINK-21230 URL: https://issues.apache.org/jira/browse/FLINK-21230 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Add primitive wrapper types to be used for messaging and state as part of the new type system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21154) Move the shared request-reply protobuf files to a separate artifact.
Igal Shilman created FLINK-21154: Summary: Move the shared request-reply protobuf files to a separate artifact. Key: FLINK-21154 URL: https://issues.apache.org/jira/browse/FLINK-21154 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Currently the request-reply.proto file is located in the statefun-flink-core artifact. This file is also used to manually generate the Python SDK request_reply_pb2.py file. Since we are about to add additional SDKS and expand the number of proto files, generating these by hand can get quickly out of hand. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20699) Feedback invocation_id must not be constant.
Igal Shilman created FLINK-20699: Summary: Feedback invocation_id must not be constant. Key: FLINK-20699 URL: https://issues.apache.org/jira/browse/FLINK-20699 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-2.2.1, statefun-2.3.0 Reporter: Igal Shilman Assignee: Igal Shilman Fix For: statefun-2.2.2 To support iteration in StateFun (we call them feedback) we use a key (FeedbackKey) that connects a feedback producer with a feedback consumer. The feedback broker will connect a consumer with a producer given a supplied key. A key has two complements: pipeline name and an invocation id, To guard against accidental reuse, the invocation id part should be unique. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20303) Add a SmokeE2E test
Igal Shilman created FLINK-20303: Summary: Add a SmokeE2E test Key: FLINK-20303 URL: https://issues.apache.org/jira/browse/FLINK-20303 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman We need an E2E test that mimics random stateful function applications, that creates random failures. This test should also verify that messages and state are consistent. This test should be run: # in a dockerized environment (for example via test containers) # via the IDE (in a mini cluster) for debuggability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19975) Bump testconatiners version
Igal Shilman created FLINK-19975: Summary: Bump testconatiners version Key: FLINK-19975 URL: https://issues.apache.org/jira/browse/FLINK-19975 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The latest Docker version causes this: [https://github.com/testcontainers/testcontainers-java/issues/700] when trying to run end to end tests. Upgrading to the latest version solves the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19840) Add a rocksdb and heap timers configuration validation
Igal Shilman created FLINK-19840: Summary: Add a rocksdb and heap timers configuration validation Key: FLINK-19840 URL: https://issues.apache.org/jira/browse/FLINK-19840 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Both stateful function applications and heap based timers are using the raw keyed stream for fault tolerance. We need to exclude configurations that set heap based timers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19827) Allow the harness to start a MiniCluster with a user provided Flink configuration
Igal Shilman created FLINK-19827: Summary: Allow the harness to start a MiniCluster with a user provided Flink configuration Key: FLINK-19827 URL: https://issues.apache.org/jira/browse/FLINK-19827 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Currently the Harness does not use the user supplied flink-configuration fully. We only use it to configure some aspects of the execution environments, but other aspects as save point location etc, are effectively ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19826) StateFun Dockerfile copies plugins with a specific version instead of a wildcard
Igal Shilman created FLINK-19826: Summary: StateFun Dockerfile copies plugins with a specific version instead of a wildcard Key: FLINK-19826 URL: https://issues.apache.org/jira/browse/FLINK-19826 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman The Dockerfile under tools/docker currently requires changing the versions, of the individual plugins while copying from opt -> plugins. The rest of the Dockerfile is using wildcards to do it, and doing so allows switching between Flink version by only changing the FROM statement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19451) Add HELM chart distribution to StateFun release process
Igal Shilman created FLINK-19451: Summary: Add HELM chart distribution to StateFun release process Key: FLINK-19451 URL: https://issues.apache.org/jira/browse/FLINK-19451 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Helm charts can be packaged and hosted publicly, see here: [https://helm.sh/docs/topics/chart_repository/#create-a-chart-repository] As part of our release process we release a source distribution that includes the Helm charts, but we can also add a step that hosts them in an Apache compatible way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19399) Add Python AsyncRequestReplyHandler docs
Igal Shilman created FLINK-19399: Summary: Add Python AsyncRequestReplyHandler docs Key: FLINK-19399 URL: https://issues.apache.org/jira/browse/FLINK-19399 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Document the new handler. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19380) Add support for a gRPC transport for the RequestReply protocol.
Igal Shilman created FLINK-19380: Summary: Add support for a gRPC transport for the RequestReply protocol. Key: FLINK-19380 URL: https://issues.apache.org/jira/browse/FLINK-19380 Project: Flink Issue Type: Task Components: Stateful Functions Reporter: Igal Shilman Fix For: statefun-2.3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19330) Recovery with async operations fails due to unitialized runtimeContext
Igal Shilman created FLINK-19330:
Summary: Recovery with async operations fails due to unitialized
runtimeContext
Key: FLINK-19330
URL: https://issues.apache.org/jira/browse/FLINK-19330
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Igal Shilman
In Flink 1.11, the AbstractStreamOperator's runtimeContext is not fully
initialized when executing `AbstractStreamOperator#intializeState()` in
particular KeyedStateStore is set after intializeState was finished.
See:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L258,L259
This behaviour was changed from Flink 1.10->Flink 1.11.
StateFun's FunctionGroupOperator performs its initialization logic at
initalizeState, and it requires an already initialized runtimeContext.
This situation causes the following failure after recovery:
{code}
java.lang.RuntimeException: java.lang.NullPointerException: Keyed state can
only be used on a 'keyed stream', i.e., after a 'keyBy()' operation.
at
org.apache.flink.runtime.state.AbstractKeyedStateBackend.lambda$applyToAllKeys$0(AbstractKeyedStateBackend.java:256)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_265]
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
~[?:1.8.0_265]
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
~[?:1.8.0_265]
at
org.apache.flink.runtime.state.AbstractKeyedStateBackend.applyToAllKeys(AbstractKeyedStateBackend.java:249)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.statefun.flink.core.functions.AsyncOperationFailureNotifier.fireExpiredAsyncOperations(AsyncOperationFailureNotifier.java:42)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.flink.core.functions.FunctionGroupOperator.initializeState(FunctionGroupOperator.java:160)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_265]
Caused by: java.lang.NullPointerException: Keyed state can only be used on a
'keyed stream', i.e., after a 'keyBy()' operation.
at
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:75)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.StreamingRuntimeContext.checkPreconditionsAndGetKeyedStateStore(StreamingRuntimeContext.java:223)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getState(StreamingRuntimeContext.java:188)
~[flink-dist_2.12-1.11.1.jar:1.11.1]
at
org.apache.flink.statefun.flink.core.state.FlinkState.createFlinkStateAccessor(FlinkState.java:69)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.flink.core.state.FlinkStateBinder.bindValue(FlinkStateBinder.java:48)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.sdk.state.StateBinder.bind(StateBinder.java:30)
~[statefun-flink-distribution.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.flink.core.state.PersistedStates.findReflectivelyAndBind(PersistedStates.java:46)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.flink.core.functions.StatefulFunctionRepository.load(StatefulFunctionRepository.java:74)
~[statefun-flink-core.jar:2.3-SNAPSHOT]
at
org.apache.flink.statefun.flink.core.functions.StatefulFunctionRepository.g
[jira] [Created] (FLINK-19329) FunctionGroupOperator#dispose might throw an NPE
Igal Shilman created FLINK-19329: Summary: FunctionGroupOperator#dispose might throw an NPE Key: FLINK-19329 URL: https://issues.apache.org/jira/browse/FLINK-19329 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman The dispose method of an operator can be called without a successful call to initalizeState. (for example a failure to load the checkpoint data, or any user exception during initializeState) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19327) Helm charts specify a too low value for the job manager's heap size
Igal Shilman created FLINK-19327:
Summary: Helm charts specify a too low value for the job manager's
heap size
Key: FLINK-19327
URL: https://issues.apache.org/jira/browse/FLINK-19327
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Reporter: Igal Shilman
Currently our Helm chart specfiy the value of
jobmanager.memory.process.size to be 500mb which causes the JobManager to crush
on start with:
{code:java}
INFO [] - Loading configuration property: jobmanager.memory.process.size, 500m
INFO [] - Loading configuration property: taskmanager.memory.process.size, 4g
INFO [] - Loading configuration property: parallelism.default, 3
INFO [] - The derived from fraction jvm overhead memory (50.000mb (52428800
bytes)) is less than its min value 192.000mb (201326592 bytes), min value will
be used instead
Exception in thread "main"
org.apache.flink.configuration.IllegalConfigurationException: The configured
Total Flink Memory (52.000mb (54525952 bytes)) is less than the configured
Off-heap Memory (128.000mb (134217728 bytes)).
at
org.apache.flink.runtime.util.config.memory.jobmanager.JobManagerFlinkMemoryUtils.deriveFromTotalFlinkMemory(JobManagerFlinkMemoryUtils.java:107)
at
org.apache.flink.runtime.util.config.memory.jobmanager.JobManagerFlinkMemoryUtils.deriveFromTotalFlinkMemory(JobManagerFlinkMemoryUtils.java:36)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveProcessSpecWithTotalProcessMemory(ProcessMemoryUtils.java:105)
at
org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.memoryProcessSpecFromConfig(ProcessMemoryUtils.java:79)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfig(JobManagerProcessUtils.java:76)
at
org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:71)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.getJmResourceParams(BashJavaUtils.java:102)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.runCommand(BashJavaUtils.java:73)
at
org.apache.flink.runtime.util.bash.BashJavaUtils.main(BashJavaUtils.java:61)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19296) RetryingCallback is not aware of task cancaltion
Igal Shilman created FLINK-19296: Summary: RetryingCallback is not aware of task cancaltion Key: FLINK-19296 URL: https://issues.apache.org/jira/browse/FLINK-19296 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman Currently RetryingCallback would be retrying until the maximum timeout elapses, unaware of rather or not the task was canceled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19279) Remove StatefulFunctionUnvierse cache
Igal Shilman created FLINK-19279: Summary: Remove StatefulFunctionUnvierse cache Key: FLINK-19279 URL: https://issues.apache.org/jira/browse/FLINK-19279 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Currently we cache the provided universe, per JVM. The cache exists to avoid scanning the class path, whenever accessing the universe across different operators executing at the same JVM, however: # Each operator access the universe during its open phase. # Having such a cache result in a shared universe across attempts, and this is undesirable. For example accessing previously released resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19220) Add a way to close internal resources
Igal Shilman created FLINK-19220: Summary: Add a way to close internal resources Key: FLINK-19220 URL: https://issues.apache.org/jira/browse/FLINK-19220 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently the internal Http stateful function, obtains transitively few resources like a thread pool and a connection pool via an http client. But it is unaware of a task cancellation and redeploying creating the potential for these resources to leak. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19203) Use Flink-*-scala2.12 variants for StateFun
Igal Shilman created FLINK-19203: Summary: Use Flink-*-scala2.12 variants for StateFun Key: FLINK-19203 URL: https://issues.apache.org/jira/browse/FLINK-19203 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman StateFun compiles and runs successfully with Scala2.12, we should use that instead of Scala2.11 as it is too old, and forces StateFun users to stick with an older version of Scala without any good reason. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19192) Set higher limit on the HTTP connection pool
Igal Shilman created FLINK-19192: Summary: Set higher limit on the HTTP connection pool Key: FLINK-19192 URL: https://issues.apache.org/jira/browse/FLINK-19192 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The default size of the connection pool is too low, we should set it to a higher value and let servers to decide if they will keep the connection alive or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19191) Reduce the default number for async operations
Igal Shilman created FLINK-19191: Summary: Reduce the default number for async operations Key: FLINK-19191 URL: https://issues.apache.org/jira/browse/FLINK-19191 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman The default upper limit for async operations per task slot is currently set to 10 million, and it is unrealistically high, we should set it to a more realistic value. A closer example would be the recommend value in Flink's AsyncWait operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19190) Use camelCase for metric names
Igal Shilman created FLINK-19190: Summary: Use camelCase for metric names Key: FLINK-19190 URL: https://issues.apache.org/jira/browse/FLINK-19190 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Rename StateFun metrics to use camelCase instead of a snake-case, to conform with Flink metrics convention. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19129) Helm charts are missing the latest log4j-console file
Igal Shilman created FLINK-19129: Summary: Helm charts are missing the latest log4j-console file Key: FLINK-19129 URL: https://issues.apache.org/jira/browse/FLINK-19129 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Fix For: statefun-2.2.0 Our Helm charts still contain the old log4j file (from flink-1.10.x) but logging was changed in Flink 1.11.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19020) Add more metrics around async operations and backpressure
Igal Shilman created FLINK-19020: Summary: Add more metrics around async operations and backpressure Key: FLINK-19020 URL: https://issues.apache.org/jira/browse/FLINK-19020 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19019) Add HDFS / S3 / GCS support to Flink-StateFun Docker image.
Igal Shilman created FLINK-19019: Summary: Add HDFS / S3 / GCS support to Flink-StateFun Docker image. Key: FLINK-19019 URL: https://issues.apache.org/jira/browse/FLINK-19019 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The 3rd party filesystem support is not enabled by default. To make checkpointing work in s3 / gcs etc' we need to add the relevant plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19018) Add connection timeout to remote functions.
Igal Shilman created FLINK-19018: Summary: Add connection timeout to remote functions. Key: FLINK-19018 URL: https://issues.apache.org/jira/browse/FLINK-19018 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Currently we only have total request timeout which also incorporates connection timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19017) Increases the visibility of a remote function call retry
Igal Shilman created FLINK-19017: Summary: Increases the visibility of a remote function call retry Key: FLINK-19017 URL: https://issues.apache.org/jira/browse/FLINK-19017 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Right now it is difficult to see if a remote function call is being retired, this issue would want to improve that situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19001) Add data-stream integration for stateful functions
Igal Shilman created FLINK-19001: Summary: Add data-stream integration for stateful functions Key: FLINK-19001 URL: https://issues.apache.org/jira/browse/FLINK-19001 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18790) Set a connection timeout that is lower than the request timeout for remote functions
Igal Shilman created FLINK-18790: Summary: Set a connection timeout that is lower than the request timeout for remote functions Key: FLINK-18790 URL: https://issues.apache.org/jira/browse/FLINK-18790 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Fix For: statefun-2.2.0 Currently for remote functions, the connection timeout is identical to the whole request timeout. A problem with this happens when a remote function is behind a NAT/load balancer/or in general behind anything that holds the port open, even tho the remote function is not present or was relocated. In that case the entire request budget would be spent on waiting for a connection. This in particularly the case in Kubernetes where pods behind a service, were ungracefully killed at once. To fix that issue, I propose: 1) by default use 10% of the total request timeout for the connection timeout. 2) expose a configuration parameter explicitly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18645) Add community driven SDKs to StateFun documentations.
Igal Shilman created FLINK-18645: Summary: Add community driven SDKs to StateFun documentations. Key: FLINK-18645 URL: https://issues.apache.org/jira/browse/FLINK-18645 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Now that we have an external [Rust SDK|https://github.com/aljoscha/statefun-rust], (by [~aljoscha]), We should link it from [https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/] Under "community" / "3rd party". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18518) Add Async RequestReply handler for the Python SDK
Igal Shilman created FLINK-18518:
Summary: Add Async RequestReply handler for the Python SDK
Key: FLINK-18518
URL: https://issues.apache.org/jira/browse/FLINK-18518
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Affects Versions: statefun-2.1.0
Reporter: Igal Shilman
I/O bound stateful functions can benefit from the built-in async/io support in
Python, but the
RequestReply handler is not an async-io compatible. See
[this|https://stackoverflow.com/questions/62640283/flink-stateful-functions-async-calls-with-the-python-sdk]
question on stackoverflow.
Having an asyncio compatible handler will open the door to the usage of aiohttp
for example:
{code:java}
import aiohttp
import asyncio
...
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
@function.bind("example/hello")
async def hello(context, message):
async with aiohttp.ClientSession() as session:
html = await fetch(session, 'http://python.org')
context.pack_and_reply(SomeProtobufMessage(html))
from aiohttp import webhandler
handler = AsyncRequestReplyHandler(functions)
async def handle(request):
req = await request.read()
res = await handler(req)
return web.Response(body=res, content_type="application/octet-stream'")
app = web.Application()
app.add_routes([web.post('/statefun', handle)])
if __name__ == '__main__':
web.run_app(app, port=5000)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17690) Python function wrapper omits docstr
Igal Shilman created FLINK-17690: Summary: Python function wrapper omits docstr Key: FLINK-17690 URL: https://issues.apache.org/jira/browse/FLINK-17690 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Statefun Python SDK has a connivance bind method, that wraps a functions. The [wrapper|https://github.com/apache/flink-statefun/blob/master/statefun-python-sdk/statefun/core.py#L182] would omit the docstr of the wrapper function. A common practice would be to use [https://docs.python.org/3/library/functools.html#functools.wraps] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17644) Add support for state TTL.
Igal Shilman created FLINK-17644: Summary: Add support for state TTL. Key: FLINK-17644 URL: https://issues.apache.org/jira/browse/FLINK-17644 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The DataStream API supports[state TTL|[https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl]], and it can be made accessible to stateful functions users. To facilitate uses cases as described in [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Window-processing-in-Stateful-Functions-td34966.html] The proposed approach would extend the PersistedValue, PersistedTable and PersistedBuffer with another constructor that accepts ExpireAfter object which has: * java time duration * refresh on read (boolean) (we should never return an expired entry) In addition, we need to extend the remote function state to support state expiration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17533) Add support for concurrent checkpoints in SateFun
Igal Shilman created FLINK-17533: Summary: Add support for concurrent checkpoints in SateFun Key: FLINK-17533 URL: https://issues.apache.org/jira/browse/FLINK-17533 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-2.0.0 Reporter: Igal Shilman Assignee: Igal Shilman Fix For: statefun-2.1.0 This issue is about adding support for concurrent checkpoints to stateful functions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16950) Kafka console consumer displays uncommitted records.
Igal Shilman created FLINK-16950: Summary: Kafka console consumer displays uncommitted records. Key: FLINK-16950 URL: https://issues.apache.org/jira/browse/FLINK-16950 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Kafka console consumer by default displays uncommitted records. This might confuse users that try out our examples, in respect with exactly once. we need to add --isolation-level read_committed to our docs and readmes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16930) Drop StatefulFunctionsClusterEntryPoint
Igal Shilman created FLINK-16930: Summary: Drop StatefulFunctionsClusterEntryPoint Key: FLINK-16930 URL: https://issues.apache.org/jira/browse/FLINK-16930 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Stateful functions is using a custom ClusterEntryPoint that is based on org.apache.flink.runtime.entrypoint.JobClusterEntrypoint. The reason we have our own entry point is that we need to setup the userClassPath (all the artifacts found under /opt/statefun/modules/*) But having our own entry point might lead to a code rot and miss important improvements (for example [1]) or bugfixes. This issue is to track how can we remove our custom entry point and reuse the existing ones (perhaps with enhancement) [1]https://issues.apache.org/jira/browse/FLINK-16658 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16926) Values present at flink-conf.yaml are not respected via the StreamPlanEnvironment
Igal Shilman created FLINK-16926: Summary: Values present at flink-conf.yaml are not respected via the StreamPlanEnvironment Key: FLINK-16926 URL: https://issues.apache.org/jira/browse/FLINK-16926 Project: Flink Issue Type: Bug Components: Stateful Functions Affects Versions: statefun-2.0.0 Reporter: Igal Shilman Fix For: statefun-2.0.0 Values present at flink-conf.yaml are not respected via the StreamPlanEnvironment, this seems to be caused by https://issues.apache.org/jira/browse/FLINK-16560 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16899) Adjust module.yaml of python-k8s example to match python-worker-service
Igal Shilman created FLINK-16899: Summary: Adjust module.yaml of python-k8s example to match python-worker-service Key: FLINK-16899 URL: https://issues.apache.org/jira/browse/FLINK-16899 Project: Flink Issue Type: Improvement Reporter: Igal Shilman Assignee: Igal Shilman Fix For: statefun-2.0.1 The Python k8s example's module.yaml, defines an endpoint that is different than what is specified in python-worker-service.yaml -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16898) Helm chart worker deployment is using the wrong port for lives checks
Igal Shilman created FLINK-16898: Summary: Helm chart worker deployment is using the wrong port for lives checks Key: FLINK-16898 URL: https://issues.apache.org/jira/browse/FLINK-16898 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Fix For: statefun-2.0.1 The deployment chart of a statefun worker is using the port 6123 for liveness checks, instead of 6122, causing the container to exit with SIGTERM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16886) Helm chart worker/master image name is using the wrong template value
Igal Shilman created FLINK-16886: Summary: Helm chart worker/master image name is using the wrong template value Key: FLINK-16886 URL: https://issues.apache.org/jira/browse/FLINK-16886 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Fix For: statefun-2.0.1 Helm chart worker/master image name is using the wrong template value. The templated value should be .Values.master.image / .Values.worker.image. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16785) Add a README to the Python SDK
Igal Shilman created FLINK-16785: Summary: Add a README to the Python SDK Key: FLINK-16785 URL: https://issues.apache.org/jira/browse/FLINK-16785 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16783) Add Polyglot Docker-compose example
Igal Shilman created FLINK-16783: Summary: Add Polyglot Docker-compose example Key: FLINK-16783 URL: https://issues.apache.org/jira/browse/FLINK-16783 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16757) Refactor Helm charts
Igal Shilman created FLINK-16757: Summary: Refactor Helm charts Key: FLINK-16757 URL: https://issues.apache.org/jira/browse/FLINK-16757 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman The Helm charts provided as part of the python-k8s example are useful as a general resource and can be reused both in the greeter example and in the python-k8s example. Moreover they provide a template for k8s resource generation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16756) Move Bootstrap API example to statefun-examples/
Igal Shilman created FLINK-16756: Summary: Move Bootstrap API example to statefun-examples/ Key: FLINK-16756 URL: https://issues.apache.org/jira/browse/FLINK-16756 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman GreetStatefulFunctionBootstrapExample should be moved to statefun-examples/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16752) Ridesharing example doesn't start
Igal Shilman created FLINK-16752: Summary: Ridesharing example doesn't start Key: FLINK-16752 URL: https://issues.apache.org/jira/browse/FLINK-16752 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman The ridesharing simulator doesn't start, after changing the parent from spring-boot to statefun. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16723) Move Python SDK example out of statefun-python-sdk
Igal Shilman created FLINK-16723: Summary: Move Python SDK example out of statefun-python-sdk Key: FLINK-16723 URL: https://issues.apache.org/jira/browse/FLINK-16723 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16722) Add Python SDK walkthrough
Igal Shilman created FLINK-16722: Summary: Add Python SDK walkthrough Key: FLINK-16722 URL: https://issues.apache.org/jira/browse/FLINK-16722 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16685) Add an example of how to deploy a multi-language function to Kubernetes.
Igal Shilman created FLINK-16685: Summary: Add an example of how to deploy a multi-language function to Kubernetes. Key: FLINK-16685 URL: https://issues.apache.org/jira/browse/FLINK-16685 Project: Flink Issue Type: Improvement Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Add an example of how to deploy a multi-language function to Kubernetes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16617) Automatic Any unpacking is broken with type deduction.
Igal Shilman created FLINK-16617: Summary: Automatic Any unpacking is broken with type deduction. Key: FLINK-16617 URL: https://issues.apache.org/jira/browse/FLINK-16617 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman Assignee: Igal Shilman Automatic Any unpacking is broken when a function is annotated with type hints. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16603) statefun-quickstart generates invalid pom.xml
Igal Shilman created FLINK-16603:
Summary: statefun-quickstart generates invalid pom.xml
Key: FLINK-16603
URL: https://issues.apache.org/jira/browse/FLINK-16603
Project: Flink
Issue Type: Bug
Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Igal Shilman
Generating a stateful functions via the mvn archetype statefun-quickstart
doesn't generate a valid pom.xml,
steps to reproduce:
1)
{code:java}
mvn archetype:generate\
-DarchetypeGroupId=org.apache.flink \
-DarchetypeArtifactId=statefun-quickstart \
-DarchetypeVersion=1.1-SNAPSHOT
{code}
2)
{code:java}
cd
mvn clean install {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
