Hello,
I am using the Maven Avro plugin to create SpecificRecord classes for a bunch
of entities specified in .avsc files.
I use the generated Java classes directly in a few MapStates.
My state is now growing very large. I wanted to know whether there is any
config or code to specify Snappy
>
> I'm afraid that it's because the configuration of your maven plugins like
> maven-assembly or maven-shaded to include the classes.
>
> Best
> Yun Tang
> From: Sandeep khanzode
> Sent: Saturday, August 7, 2021 10:20
> To: Yun Tang
> Cc: Stephan Ewen ; user
ses within the application
> jar package, and it might cause problems if the client has different flink
> version with servers.
>
>
> Best,
> Yun Tang
> From: Stephan Ewen mailto:se...@apache.org>>
> Sent: Wednesday, August 4, 2021 19:10
> To: Yun Tang mailto:myas...@live
Hello,
I tried to add the bloom filter functionality as mentioned here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html
Hello,
With the default memory settings, after about 5000 records in my
KafkaFlinkConsumer, and some operators in my pipeline, I get the below error:
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
at
orted-data-types-for-schema-evolution>
>
> Best,
> Guowei
>
>
> On Fri, Jun 25, 2021 at 3:04 AM Sandeep khanzode <mailto:sand...@shiftright.ai>> wrote:
> Hello,
>
> 1.] Can someone please share a working example of how to read
> ValueState and MapSt
Hello,
1.] Can someone please share a working example of how to read ValueState
and MapState from a checkpoint and update it? I tried to assemble a
working Java example but there are bit and pieces of info around.
2.] I am using Avro 1.7.7 with Flink for state entities since versions belong
Hello,
Can someone please assist for this query? Thanks!
Thanks,
Sandeep
> On 06-May-2021, at 10:30 AM, Sandeep khanzode wrote:
>
> Hi,
>
> Is there a working example somewhere that I can refer for writing Avro
> entities in Flink state as well as Avro serializaition i
html
>
> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#avro
>
> I hope this helps.
>
> Regards,
> Timo
>
>
> On 30.04.21 13:20, Sandeep khanzode wrote:
>> Hi Timo,
>> Thanks! I will take a look at the links.
>> Can you p
/runtime/PojoSerializer.java#L599
>
> For the POJO logic.
>
> By the way, usually we recommend Avro for state data structures if schema
> evolution is a topic.
>
> Regards,
> Timo
>
>
>
> On 29.04.21 18:10, Sandeep khanzode wrote:
>> Hello,
>
Hello,
Is there a working example of a TypeSerializer for a Java type stored in the
State?
My requirement is that I should be able to store the Java POJO entity in the
MapState. The state is backed by RocksDBBackend.
If I update the entity with a new member variable, I am unable to
Hello,
Stuck at this time. Any help will be appreciated.
I am able to create a queryable state and also query the state. Everything
works correctly.
KeyedStream, Key> stream = sourceStream.keyBy(t2 -> t2.f0);
stream.asQueryableState("queryableVO");
I deploy this on a Kubernetes cluster with
Hello
I was reading this:
https://stackoverflow.com/questions/61010970/flink-resume-from-externalised-checkpoint-question
I am trying to run a standalone job on my local with a single job manager and
task manager.
I have enabled checkpointing as below:
env.setStateBackend(new
Hi,
I have a master/reference data that needs to come in through a
FlinkKafkaConsumer to be broadcast to all nodes and subsequently joined with
the actual stream for enriching content.
The Kafka consumer gets CDC-type records from database changes. All this works
well.
My question is how do
ci.apache.org/projects/flink/flink-docs-stable/dev/project-configuration.html#setting-up-a-project-basic-dependencies>
> On Tue, Mar 23, 2021 at 5:28 AM Sandeep khanzode <mailto:sand...@shiftright.ai>> wrote:
> Hi Arvid,
>
> I copy the JAR to the usrlib folder. This works in t
problem?
>
> Best,
> Matthias
>
> On Tue, Mar 23, 2021 at 5:28 AM Sandeep khanzode <mailto:sand...@shiftright.ai>> wrote:
> Hi,
>
> I have a stream that exposes the state for Queryable State.
>
> I am using the key as follows:
>
> public class
jar should be ~1MB.
>
> On Fri, Mar 19, 2021 at 8:58 PM Sandeep khanzode <mailto:sand...@shiftright.ai>> wrote:
> Hello,
>
> I have a fat JAR compiled using the Man Shade plugin and everything works
> correctly when I deploy it on a standalone local cluster i.e. one
Hi,
I have a stream that exposes the state for Queryable State.
I am using the key as follows:
public class MyKey {
private Long first;
private EnumType myType;
private Long second;
private TreeMap map;
@Override
public boolean equals(Object o) {
if (this == o)
Hello,
I have a fat JAR compiled using the Man Shade plugin and everything works
correctly when I deploy it on a standalone local cluster i.e. one job and one
task manager node.
But I installed Minikube and the same JAR file packaged into a docker image
fails with weird serialization
Hello,
I am deploying a standalone-job cluster (cluster with a single Job and Task
Manager instance instantiated with a —job-classname and —job-id).
I have map/flatmap/process functions being executed in the various stream
functions in the Taskmanager for which I need access to the Job Id and
k/flink-docs-release-1.12/dev/stream/state/queryable_state.html
>
> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/queryable_state.html>
>
> On February 9, 2021 at 22:09:29, Sandeep khanzode (sand...@shiftright.ai
> <mailto:sand...@shif
gether and keyBy them. This way you can have all three
> streams’ states in downstream without duplication.
>
> Best,
> Kezhu Wang
> On February 11, 2021 at 20:49:20, Sandeep khanzode (sand...@shiftright.ai
> <mailto:sand...@shiftright.ai>) wrote:
>
>> Hello
g/apache/flink/test/streaming/runtime/MultipleInputITCase.java#L113
>
> <https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/MultipleInputITCase.java#L113>
>
> On February 10, 2021 at 17:19:15, Sandeep khanzode (sand...@s
able/dev/stream/state/broadcast_state.html
>
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html>
>
> Best,
> Kezhu Wang
>
> On February 10, 2021 at 13:03:36, Sandeep khanzode (sand...@shiftright.ai
> <mailto:sand...@s
tate/queryable_state.html>
>
> On February 9, 2021 at 22:09:29, Sandeep khanzode (sand...@shiftright.ai
> <mailto:sand...@shiftright.ai>) wrote:
>
>> Hello,
>>
>> I am creating a class that extends RichCoFlatMapFunction. I need to
>> connect() tw
Hello,
I am creating a class that extends RichCoFlatMapFunction. I need to connect()
two streams to basically share the state of one stream in another.
This is what I do:
private transient MapState state;
@Override
public void open(Configuration parameters) throws Exception {
26 matches
Mail list logo
