Hi Dan,
Usually broadcast state needs more network buffers, the network buffer used
to exchange data records between tasks would request a portion of direct
memory[1], I think it is possible to get the “Direct buffer memory” OOM
errors in this scenarios. Maybe you can try to increase
大家好！
最近在开发一个项目时，在使用CDC表和维表表做Temporal Table
JOIN时，发现2个表Join时join字段的类型必须一致，否则提交时提示如下的错误
The main method caused an error: Temporal table join requires an equality
condition on fields of table.
为了解决上述问题，我们做了如下尝试：
1：在join时，对维表要关联的字段使用cast转换，如： JOIN ON CAST(tableA.filedA AS
Hi all, I am trying to figure out how Datastream.broadcast() and
DataStream.broadcast(MapStateDescriptor) differ.
My use case:
I have 2 streams:
Stream 1 contains updates, which collectively build up a state
Stream 2 is keyed and every parallel instance need to connect with EVERY update
from
Thanks, I'll check it out.
On Fri, Oct 21, 2022, 18:20 Piotr Nowojski wrote:
> Hi,
>
> Yes and no. StateProcessor API can read any Flink state, but you have to
> describe the state you want it to access. Take a look at the example in the
> docs [1].
>
> First you have an example of a
Great, thanks!
Kind regards,
Levan Huyen
On Fri, 21 Oct 2022 at 00:53, Biao Geng wrote:
> You are right.
> It contains the python package `pyflink` and some dependencies like py4j
> and cloudpickle but does not contain all relevant dependencies(e.g.
> `google.protobuf` as the error log shows,
Hi,
Yes and no. StateProcessor API can read any Flink state, but you have to
describe the state you want it to access. Take a look at the example in the
docs [1].
First you have an example of a theoretical production function
`StatefulFunctionWithTime`, which state you want to modify. Note the
I have question on this. Different connector can have different
serialisation and de-serlisation technique right?. Wouldn't that impact?.
If I use StateProcessor API, would that be agnostic to all the sources and
sinks?.
On Fri, Oct 21, 2022, 18:00 Piotr Nowojski wrote:
> ops
>
> >
ops
> Alternatively, you can modify a code of your function/operator for which
you want to modify the state. For example in the
`org.apache.flink.streaming.api.checkpoint.CheckpointedFunction#initializeState`
method you could add some code that would do a migration of your old state
to a new one.
Hi Nick,
I would say open a PR so one of the maintainers can have a look :)
Thanks,
Martijn
On Sun, Oct 9, 2022 at 3:16 AM Nick Levandoski
wrote:
> Hello,
>
> I've had an issue similar to the one reported in ticket FLINK-27054. I
> would like to pass the SSLContext to the ElasticSearchSink
Thanks !. Will try this.
On Fri, Oct 21, 2022 at 2:22 PM Piotr Nowojski wrote:
> Hi Sriram,
>
> You can read and modify savepoints using StateProcessor API [1].
>
> Alternatively, you can modify a code of your function/operator for which
> you want to modify the state. For example in the
>
Hi Sriram,
You can read and modify savepoints using StateProcessor API [1].
Alternatively, you can modify a code of your function/operator for which
you want to modify the state. For example in the
`org.apache.flink.streaming.api.checkpoint.CheckpointedFunction#initializeState`
method you could
Hi All,
I am working on a scenario where I need to modify the existing savepoint
operator state. Ex: Wanted to remove some offset of the savepoint.
What is the better practice for these scenarios?. Could you please help me
with any example as such?
Thanks in advance.
--
*Sriram G*
*Tech*
Hi. My team recently added broadcast state to our Flink jobs. We've
started hitting this OOM "Direct buffer memory" error. Is this a common
problem with broadcast state? Or is it likely a different problem?
Thanks! - Dan
13 matches
Mail list logo
