Re: [Flink K8s operator] HA metadata not available to restore from last state

[Dongwon Kim]

Hi Gyula :-)

Okay, we're gonna upgrade to 1.15 and see what happens.

Thanks a lot for the quick feedback and the detailed explanation!

Best,

Dongwon


On Tue, Nov 22, 2022 at 5:57 PM Gyula Fóra  wrote:

> Hi Dongwon!
>
> This error mostly occurs when using Flink 1.14 and the Flink cluster goes
> into a terminal state. If a Flink job is FAILED/FINISHED (such as it
> exhausted the retry strategy), in Flink 1.14 the cluster shuts itself down
> and removes the HA metadata.
>
> In these cases the operator will only see that the cluster completely
> disappeared and there is no HA metadata and it will throw the error you
> mentioned. It does not know what happened and doesn't have any way to
> recover checkpoint information.
>
> This is fixed in Flink 1.15 where even after terminal FAILED/FINISHED
> states, the jobmanager would not shut down. This allows the operator to
> observe this terminal state and actually recover the job even if the HA
> metadata was removed.
>
> To summarize, this is mostly caused by Flink 1.14 behaviour that the
> operator cannot control. Upgrading to 1.15 allows much more robustness and
> should eliminate most of these cases.
>
> Cheers,
> Gyula
>
> On Tue, Nov 22, 2022 at 9:43 AM Dongwon Kim  wrote:
>
>> Hi,
>>
>> While using a last-state upgrade mode on flink-k8s-operator-1.2.0 and
>> flink-1.14.3, we're occasionally facing the following error:
>>
>> Status:
>>>   Cluster Info:
>>> Flink - Revision: 98997ea @ 2022-01-08T23:23:54+01:00
>>> Flink - Version:  1.14.3
>>>   Error:  HA metadata not available to restore
>>> from last state. It is possible that the job has finished or terminally
>>> failed, or the configmaps have been deleted. Manual restore required.
>>>   Job Manager Deployment Status:  ERROR
>>>   Job Status:
>>> Job Id:e8dd04ea4b03f1817a4a4b9e5282f433
>>> Job Name:  flinktest
>>> Savepoint Info:
>>>   Last Periodic Savepoint Timestamp:  0
>>>   Savepoint History:
>>>   Trigger Id:
>>>   Trigger Timestamp:  0
>>>   Trigger Type:   UNKNOWN
>>> Start Time:   1668660381400
>>> State:RECONCILING
>>> Update Time:  1668994910151
>>>   Reconciliation Status:
>>> Last Reconciled Spec:  ...
>>> Reconciliation Timestamp:  1668660371853
>>> State: DEPLOYED
>>>   Task Manager:
>>> Label Selector:  component=taskmanager,app=flinktest
>>> Replicas:1
>>> Events:
>>>   Type ReasonAge From
>>> Message
>>>    --
>>> ---
>>>   Normal   JobStatusChanged  30m Job
>>> Job status changed from RUNNING to RESTARTING
>>>   Normal   JobStatusChanged  29m Job
>>> Job status changed from RESTARTING to CREATED
>>>   Normal   JobStatusChanged  28m Job
>>> Job status changed from CREATED to RESTARTING
>>>   Warning  Missing   26m JobManagerDeployment
>>> Missing JobManager deployment
>>>   Warning  RestoreFailed 9s (x106 over 26m)  JobManagerDeployment
>>> HA metadata not available to restore from last state. It is possible that
>>> the job has finished or terminally failed, or the configmaps have been
>>> deleted. Manual restore required.
>>>   Normal   Submit9s (x106 over 26m)  JobManagerDeployment
>>> Starting deployment
>>
>>
>> We're happy with the last state mode most of the time, but we face it
>> occasionally.
>>
>> We found that it's not easy to reproduce the problem; we tried to kill
>> JMs and TMs and even shutdown the nodes on which JMs and TMs are running.
>>
>> We also checked that the file size is not zero.
>>
>> Thanks,
>>
>> Dongwon
>>
>>
>>

[Flink K8s operator] HA metadata not available to restore from last state

[Dongwon Kim]

Hi,

While using a last-state upgrade mode on flink-k8s-operator-1.2.0 and
flink-1.14.3, we're occasionally facing the following error:

Status:
>   Cluster Info:
> Flink - Revision: 98997ea @ 2022-01-08T23:23:54+01:00
> Flink - Version:  1.14.3
>   Error:  HA metadata not available to restore
> from last state. It is possible that the job has finished or terminally
> failed, or the configmaps have been deleted. Manual restore required.
>   Job Manager Deployment Status:  ERROR
>   Job Status:
> Job Id:e8dd04ea4b03f1817a4a4b9e5282f433
> Job Name:  flinktest
> Savepoint Info:
>   Last Periodic Savepoint Timestamp:  0
>   Savepoint History:
>   Trigger Id:
>   Trigger Timestamp:  0
>   Trigger Type:   UNKNOWN
> Start Time:   1668660381400
> State:RECONCILING
> Update Time:  1668994910151
>   Reconciliation Status:
> Last Reconciled Spec:  ...
> Reconciliation Timestamp:  1668660371853
> State: DEPLOYED
>   Task Manager:
> Label Selector:  component=taskmanager,app=flinktest
> Replicas:1
> Events:
>   Type ReasonAge From
> Message
>    --
> ---
>   Normal   JobStatusChanged  30m Job
> Job status changed from RUNNING to RESTARTING
>   Normal   JobStatusChanged  29m Job
> Job status changed from RESTARTING to CREATED
>   Normal   JobStatusChanged  28m Job
> Job status changed from CREATED to RESTARTING
>   Warning  Missing   26m JobManagerDeployment
> Missing JobManager deployment
>   Warning  RestoreFailed 9s (x106 over 26m)  JobManagerDeployment  HA
> metadata not available to restore from last state. It is possible that the
> job has finished or terminally failed, or the configmaps have been
> deleted. Manual restore required.
>   Normal   Submit9s (x106 over 26m)  JobManagerDeployment
> Starting deployment


We're happy with the last state mode most of the time, but we face it
occasionally.

We found that it's not easy to reproduce the problem; we tried to kill JMs
and TMs and even shutdown the nodes on which JMs and TMs are running.

We also checked that the file size is not zero.

Thanks,

Dongwon

Re: [State Processor API] unable to load the state of a trigger attached to a session window

[Dongwon Kim]

Hi Juntao,

Thanks a lot for taking a look at this.

After a little inspection, I found that elements (window state) are stored
> in namespace TimeWindow{start=1,end=11}, in your case, and trigger count
> (trigger state) is stored in namespace TimeWindow{start=1,end=15}, but
> WindowReaderOperator only tries to find keys and namespaces related to
> window state.


So is there any chance for me to get the trigger state using the correct
name space? Or should it be considered as a bug to be fixed?

Best,

Dongwon

On Thu, May 12, 2022 at 5:51 PM Juntao Hu  wrote:

> Sorry to make the previous mail private.
> My response reposted here:
> "
> After a little inspection, I found that elements (window state) are stored
> in namespace TimeWindow{start=1,end=11}, in your case, and trigger count
> (trigger state) is stored in namespace TimeWindow{start=1,end=15}, but
> WindowReaderOperator only tries to find keys and namespaces related to
> window state.
> "
>
> Juntao Hu  于2022年5月12日周四 11:47写道：
>
>> After a little inspection, I found that elements (window state) are
>> stored in namespace TimeWindow{start=1,end=11}, in your case, and trigger
>> count (trigger state) is stored in namespace TimeWindow{start=1,end=15},
>> but WindowReaderOperator only tries to find keys and namespaces related to
>> window state.
>>
>> Dongwon Kim  于2022年4月19日周二 15:29写道：
>>
>>> Hi,
>>>
>>> I'm using Flink-1.14.4 and failed to load in WindowReaderFunction the
>>> state of a stateful trigger attached to a session window.
>>> I found that the following data become available in WindowReaderFunction:
>>> - the state defined in the ProcessWindowFunction
>>> - the registered timers of the stateful trigger attached to the session
>>> window
>>> - all the elements of the window
>>> , but the state of the stateful trigger attached to the session window
>>> is not available when using State Processor API.
>>> In other words, the following code always returns null when used with
>>> session windows:
>>>
>>>> ReducingState state =
>>>> context.triggerState(triggerCountDesc);
>>>> Long val = state.get();
>>>>
>>> On the other hand, the above code snippet returns expected data when
>>> used with sliding and tumbling windows.
>>>
>>> To explain the problem, I made up an example in a similar spirit to
>>> o.a.f.state.api.SavepointWindowReaderITCase.
>>> Here you can find three test cases each with different types of
>>> event-time windows: Session, Sliding, and Tumbling.
>>> With sliding and tumbling windows, I can read the state of the trigger
>>> attached to the windows in WindowReaderFunction.
>>> However, with a session window, I cannot read the state of the trigger
>>> in WindowReaderFunction.
>>>
>>> Is it a bug, or did I miss something?
>>>
>>> Best,
>>>
>>> Dongwon
>>>
>>>

Re: [State Processor API] unable to load the state of a trigger attached to a session window

[Dongwon Kim]

Is the library, State Processor API, not widely used and no longer
maintained?

On Mon, Apr 25, 2022 at 3:16 PM Dongwon Kim  wrote:

> Can anyone help me with this?
>
> Thanks in advance,
>
> On Tue, Apr 19, 2022 at 4:28 PM Dongwon Kim  wrote:
>
>> Hi,
>>
>> I'm using Flink-1.14.4 and failed to load in WindowReaderFunction the
>> state of a stateful trigger attached to a session window.
>> I found that the following data become available in WindowReaderFunction:
>> - the state defined in the ProcessWindowFunction
>> - the registered timers of the stateful trigger attached to the session
>> window
>> - all the elements of the window
>> , but the state of the stateful trigger attached to the session window is
>> not available when using State Processor API.
>> In other words, the following code always returns null when used with
>> session windows:
>>
>>> ReducingState state =
>>> context.triggerState(triggerCountDesc);
>>> Long val = state.get();
>>>
>> On the other hand, the above code snippet returns expected data when used
>> with sliding and tumbling windows.
>>
>> To explain the problem, I made up an example in a similar spirit to
>> o.a.f.state.api.SavepointWindowReaderITCase.
>> Here you can find three test cases each with different types of
>> event-time windows: Session, Sliding, and Tumbling.
>> With sliding and tumbling windows, I can read the state of the trigger
>> attached to the windows in WindowReaderFunction.
>> However, with a session window, I cannot read the state of the trigger in
>> WindowReaderFunction.
>>
>> Is it a bug, or did I miss something?
>>
>> Best,
>>
>> Dongwon
>>
>>

Re: [State Processor API] unable to load the state of a trigger attached to a session window

[Dongwon Kim]

Can anyone help me with this?

Thanks in advance,

On Tue, Apr 19, 2022 at 4:28 PM Dongwon Kim  wrote:

> Hi,
>
> I'm using Flink-1.14.4 and failed to load in WindowReaderFunction the
> state of a stateful trigger attached to a session window.
> I found that the following data become available in WindowReaderFunction:
> - the state defined in the ProcessWindowFunction
> - the registered timers of the stateful trigger attached to the session
> window
> - all the elements of the window
> , but the state of the stateful trigger attached to the session window is
> not available when using State Processor API.
> In other words, the following code always returns null when used with
> session windows:
>
>> ReducingState state =
>> context.triggerState(triggerCountDesc);
>> Long val = state.get();
>>
> On the other hand, the above code snippet returns expected data when used
> with sliding and tumbling windows.
>
> To explain the problem, I made up an example in a similar spirit to
> o.a.f.state.api.SavepointWindowReaderITCase.
> Here you can find three test cases each with different types of event-time
> windows: Session, Sliding, and Tumbling.
> With sliding and tumbling windows, I can read the state of the trigger
> attached to the windows in WindowReaderFunction.
> However, with a session window, I cannot read the state of the trigger in
> WindowReaderFunction.
>
> Is it a bug, or did I miss something?
>
> Best,
>
> Dongwon
>
>

[State Processor API] unable to load the state of a trigger attached to a session window

[Dongwon Kim]

Hi,

I'm using Flink-1.14.4 and failed to load in WindowReaderFunction the state
of a stateful trigger attached to a session window.
I found that the following data become available in WindowReaderFunction:
- the state defined in the ProcessWindowFunction
- the registered timers of the stateful trigger attached to the session
window
- all the elements of the window
, but the state of the stateful trigger attached to the session window is
not available when using State Processor API.
In other words, the following code always returns null when used with
session windows:

> ReducingState state =
> context.triggerState(triggerCountDesc);
> Long val = state.get();
>
On the other hand, the above code snippet returns expected data when used
with sliding and tumbling windows.

To explain the problem, I made up an example in a similar spirit to
o.a.f.state.api.SavepointWindowReaderITCase.
Here you can find three test cases each with different types of event-time
windows: Session, Sliding, and Tumbling.
With sliding and tumbling windows, I can read the state of the trigger
attached to the windows in WindowReaderFunction.
However, with a session window, I cannot read the state of the trigger in
WindowReaderFunction.

Is it a bug, or did I miss something?

Best,

Dongwon


SavepointWindowReaderTestForDifferentWindows.java
Description: Binary data

Re: Weird Flink Kafka source watermark behavior

[Dongwon Kim]

I totally agree with Schwalbe that per-partition watermarking allows #
source tasks < # kafka partitions.

Otherwise, Dan, you should suspect other possibilities like what Schwalbe
said.

Best,

Dongwon

On Fri, Mar 18, 2022 at 5:01 PM Schwalbe Matthias <
matthias.schwa...@viseca.ch> wrote:

> Hi San, Dongwon,
>
>
>
> I share the opinion that when per-partition watermarking is enabled, you
> should observe correct behavior … would be interesting to see why it does
> not work for you.
>
>
>
> I’d like to clear one tiny misconception here when you write:
>
>
>
> >> - The same issue happens even if I use an idle watermark.
>
>
>
> You would expect to see glitches with watermarking when you enable
> idleness.
>
> Idleness sort of trades watermark correctness for reduces latency when
> processing timers (much simplified).
>
> With idleness enabled you have no guaranties whatsoever as to the quality
> of watermarks (which might be ok in some cases).
>
> BTW we dominantly use a mix of fast and slow sources (that only update
> once a day) which hand-pimped watermarking and late event processing, and
> enabling idleness would break everything.
>
>
>
> Oversight put aside things should work the way you implemented it.
>
>
>
> One thing I could imagine to be a cause is
>
>- that over time the kafka partitions get reassigned  to different
>consumer subtasks which would probably stress correct recalculation of
>watermarks. Hence #partition == number subtask might reduce the problem
>- can you enable logging of partition-consumer assignment, to see if
>that is the cause of the problem
>- also involuntary restarts of the job can cause havoc as this resets
>watermarking
>
>
>
> I’ll be off next week, unable to take part in the active discussion …
>
>
>
> Sincere greetings
>
>
>
> Thias
>
>
>
>
>
>
>
>
>
> *From:* Dan Hill 
> *Sent:* Freitag, 18. März 2022 08:23
> *To:* Dongwon Kim 
> *Cc:* user 
> *Subject:* Re: Weird Flink Kafka source watermark behavior
>
>
>
> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>
>
>
> I'll try forcing # source tasks = # partitions tomorrow.
>
>
>
> Thank you, Dongwon, for all of your help!
>
>
>
> On Fri, Mar 18, 2022 at 12:20 AM Dongwon Kim 
> wrote:
>
> I believe your job with per-partition watermarking should be working okay
> even in a backfill scenario.
>
>
>
> BTW, is the problem still observed even with # sour tasks = # partitions?
>
>
>
> For committers:
>
> Is there a way to confirm that per-partition watermarking is used in TM
> log?
>
>
>
> On Fri, Mar 18, 2022 at 4:14 PM Dan Hill  wrote:
>
> I hit this using event processing and no idleness detection.  The same
> issue happens if I enable idleness.
>
>
>
> My code matches the code example for per-partition watermarking
> <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector>
> .
>
>
>
> On Fri, Mar 18, 2022 at 12:07 AM Dongwon Kim 
> wrote:
>
> Hi Dan,
>
>
>
> I'm quite confused as you already use per-partition watermarking.
>
>
>
> What I meant in the reply is
>
> - If you don't use per-partition watermarking, # tasks < # partitions can
> cause the problem for backfill jobs.
>
> - If you don't use per-partition watermarking, # tasks = # partitions is
> going to be okay even for backfill jobs.
>
> - If you use per-partition watermarking, # tasks < # partitions shouldn't
> cause any problems unless you turn on the idleness detection.
>
>
>
> Regarding the idleness detection which is based on processing time, what
> is your setting? If you set the value to 10 seconds for example, you'll
> face the same problem unless the watermark of your backfill job catches
> up real-time within 10 seconds. If you increase the value to 1 minute, your
> backfill job should catch up real-time within 1 minute.
>
>
>
> Best,
>
>
>
> Dongwon
>
>
>
>
>
> On Fri, Mar 18, 2022 at 3:51 PM Dan Hill  wrote:
>
> Thanks Dongwon!
>
>
>
> Wow.  Yes, I'm using per-partition watermarking [1].  Yes, my # source
> tasks < # kafka partitions.  This should be called out in the docs or the
> bug should be fixed.
>
>
>
> On Thu, Mar 17, 2022 at 10:54 PM Dongwon Kim 
> wrote:
>
> Hi Dan,
>
>
>
> Do you use the per-partition watermarking explained in [1]?
>
> I've also experienced a similar problem when running backfill jobs
> specifically when # source tasks

Re: Weird Flink Kafka source watermark behavior

[Dongwon Kim]

I believe your job with per-partition watermarking should be working okay
even in a backfill scenario.

BTW, is the problem still observed even with # sour tasks = # partitions?

For committers:
Is there a way to confirm that per-partition watermarking is used in TM log?

On Fri, Mar 18, 2022 at 4:14 PM Dan Hill  wrote:

> I hit this using event processing and no idleness detection.  The same
> issue happens if I enable idleness.
>
> My code matches the code example for per-partition watermarking
> <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector>
> .
>
>
> On Fri, Mar 18, 2022 at 12:07 AM Dongwon Kim 
> wrote:
>
>> Hi Dan,
>>
>> I'm quite confused as you already use per-partition watermarking.
>>
>> What I meant in the reply is
>> - If you don't use per-partition watermarking, # tasks < # partitions can
>> cause the problem for backfill jobs.
>> - If you don't use per-partition watermarking, # tasks = # partitions is
>> going to be okay even for backfill jobs.
>> - If you use per-partition watermarking, # tasks < # partitions shouldn't
>> cause any problems unless you turn on the idleness detection.
>>
>> Regarding the idleness detection which is based on processing time, what
>> is your setting? If you set the value to 10 seconds for example, you'll
>> face the same problem unless the watermark of your backfill job catches
>> up real-time within 10 seconds. If you increase the value to 1 minute, your
>> backfill job should catch up real-time within 1 minute.
>>
>> Best,
>>
>> Dongwon
>>
>>
>> On Fri, Mar 18, 2022 at 3:51 PM Dan Hill  wrote:
>>
>>> Thanks Dongwon!
>>>
>>> Wow.  Yes, I'm using per-partition watermarking [1].  Yes, my # source
>>> tasks < # kafka partitions.  This should be called out in the docs or the
>>> bug should be fixed.
>>>
>>> On Thu, Mar 17, 2022 at 10:54 PM Dongwon Kim 
>>> wrote:
>>>
>>>> Hi Dan,
>>>>
>>>> Do you use the per-partition watermarking explained in [1]?
>>>> I've also experienced a similar problem when running backfill jobs
>>>> specifically when # source tasks < # kafka partitions.
>>>> - When # source tasks = # kafka partitions, the backfill job works as
>>>> expected.
>>>> - When # source tasks < # kafka partitions, a Kafka consumer consumes
>>>> multiple partitions. This case can destroying the per-partition patterns as
>>>> explained in [2].
>>>>
>>>> Hope this helps.
>>>>
>>>> p.s. If you plan to use the per-partition watermarking, be aware that
>>>> idleness detection [3] can cause another problem when you run a backfill
>>>> job. Kafka source tasks in a backfill job seem to read a batch of records
>>>> from Kafka and then wait for downstream tasks to catch up the progress,
>>>> which can be counted as idleness.
>>>>
>>>> [1]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#using-watermark-strategie
>>>> [2]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
>>>> [3]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources
>>>>
>>>> Best,
>>>>
>>>> Dongwon
>>>>
>>>> On Fri, Mar 18, 2022 at 2:35 PM Dan Hill  wrote:
>>>>
>>>>> I'm following the example from this section:
>>>>>
>>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
>>>>>
>>>>> On Thu, Mar 17, 2022 at 10:26 PM Dan Hill 
>>>>> wrote:
>>>>>
>>>>>> Other points
>>>>>> - I'm using the kafka timestamp as event time.
>>>>>> - The same issue happens even if I use an idle watermark.
>>>>>>
>>>>>> On Thu, Mar 17, 2022 at 10:17 PM Dan Hill 
>>>>>> wrote:
>>>>>>
>>>>>>> There are 12 Kafka partitions (to keep the structure similar to
>>>>>>> other low traffic environments).
>>>>>>>
>>>>>&g

Re: Weird Flink Kafka source watermark behavior

[Dongwon Kim]

Hi Dan,

I'm quite confused as you already use per-partition watermarking.

What I meant in the reply is
- If you don't use per-partition watermarking, # tasks < # partitions can
cause the problem for backfill jobs.
- If you don't use per-partition watermarking, # tasks = # partitions is
going to be okay even for backfill jobs.
- If you use per-partition watermarking, # tasks < # partitions shouldn't
cause any problems unless you turn on the idleness detection.

Regarding the idleness detection which is based on processing time, what is
your setting? If you set the value to 10 seconds for example, you'll face
the same problem unless the watermark of your backfill job catches
up real-time within 10 seconds. If you increase the value to 1 minute, your
backfill job should catch up real-time within 1 minute.

Best,

Dongwon


On Fri, Mar 18, 2022 at 3:51 PM Dan Hill  wrote:

> Thanks Dongwon!
>
> Wow.  Yes, I'm using per-partition watermarking [1].  Yes, my # source
> tasks < # kafka partitions.  This should be called out in the docs or the
> bug should be fixed.
>
> On Thu, Mar 17, 2022 at 10:54 PM Dongwon Kim 
> wrote:
>
>> Hi Dan,
>>
>> Do you use the per-partition watermarking explained in [1]?
>> I've also experienced a similar problem when running backfill jobs
>> specifically when # source tasks < # kafka partitions.
>> - When # source tasks = # kafka partitions, the backfill job works as
>> expected.
>> - When # source tasks < # kafka partitions, a Kafka consumer consumes
>> multiple partitions. This case can destroying the per-partition patterns as
>> explained in [2].
>>
>> Hope this helps.
>>
>> p.s. If you plan to use the per-partition watermarking, be aware that
>> idleness detection [3] can cause another problem when you run a backfill
>> job. Kafka source tasks in a backfill job seem to read a batch of records
>> from Kafka and then wait for downstream tasks to catch up the progress,
>> which can be counted as idleness.
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#using-watermark-strategie
>> [2]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
>> [3]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources
>>
>> Best,
>>
>> Dongwon
>>
>> On Fri, Mar 18, 2022 at 2:35 PM Dan Hill  wrote:
>>
>>> I'm following the example from this section:
>>>
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
>>>
>>> On Thu, Mar 17, 2022 at 10:26 PM Dan Hill  wrote:
>>>
>>>> Other points
>>>> - I'm using the kafka timestamp as event time.
>>>> - The same issue happens even if I use an idle watermark.
>>>>
>>>> On Thu, Mar 17, 2022 at 10:17 PM Dan Hill 
>>>> wrote:
>>>>
>>>>> There are 12 Kafka partitions (to keep the structure similar to other
>>>>> low traffic environments).
>>>>>
>>>>> On Thu, Mar 17, 2022 at 10:13 PM Dan Hill 
>>>>> wrote:
>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> I'm running a backfill from a kafka topic with very few records
>>>>>> spread across a few days.  I'm seeing a case where the records coming 
>>>>>> from
>>>>>> a kafka source have a watermark that's more recent (by hours) than the
>>>>>> event time.  I haven't seen this before when running this.  This violates
>>>>>> what I'd assume the kafka source would do.
>>>>>>
>>>>>> Example problem:
>>>>>> 1. I have kafka records at ts=1000, 2000, ... 50.  The actual
>>>>>> times are separated by a longer time period.
>>>>>> 2.  My first operator after the FlinkKafkaConsumer sees:
>>>>>>context.timestamp() = 1000
>>>>>>context.timerService().currentWatermark() = 50
>>>>>>
>>>>>> Details about how I'm running this:
>>>>>> - I'm on Flink 1.12.3 that's running on EKS and using MSK as the
>>>>>> source.
>>>>>> - I'm using FlinkKafkaConsumer
>>>>>> - I'm using WatermarkStrategy.forBoundedOutOfOrderness(5s).  No
>>>>>> idleness 

Re: Weird Flink Kafka source watermark behavior

[Dongwon Kim]

Hi Dan,

Do you use the per-partition watermarking explained in [1]?
I've also experienced a similar problem when running backfill jobs
specifically when # source tasks < # kafka partitions.
- When # source tasks = # kafka partitions, the backfill job works as
expected.
- When # source tasks < # kafka partitions, a Kafka consumer consumes
multiple partitions. This case can destroying the per-partition patterns as
explained in [2].

Hope this helps.

p.s. If you plan to use the per-partition watermarking, be aware that
idleness detection [3] can cause another problem when you run a backfill
job. Kafka source tasks in a backfill job seem to read a batch of records
from Kafka and then wait for downstream tasks to catch up the progress,
which can be counted as idleness.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#using-watermark-strategie
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources

Best,

Dongwon

On Fri, Mar 18, 2022 at 2:35 PM Dan Hill  wrote:

> I'm following the example from this section:
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector
>
> On Thu, Mar 17, 2022 at 10:26 PM Dan Hill  wrote:
>
>> Other points
>> - I'm using the kafka timestamp as event time.
>> - The same issue happens even if I use an idle watermark.
>>
>> On Thu, Mar 17, 2022 at 10:17 PM Dan Hill  wrote:
>>
>>> There are 12 Kafka partitions (to keep the structure similar to other
>>> low traffic environments).
>>>
>>> On Thu, Mar 17, 2022 at 10:13 PM Dan Hill  wrote:
>>>
 Hi.

 I'm running a backfill from a kafka topic with very few records spread
 across a few days.  I'm seeing a case where the records coming from a kafka
 source have a watermark that's more recent (by hours) than the event time.
 I haven't seen this before when running this.  This violates what I'd
 assume the kafka source would do.

 Example problem:
 1. I have kafka records at ts=1000, 2000, ... 50.  The actual times
 are separated by a longer time period.
 2.  My first operator after the FlinkKafkaConsumer sees:
context.timestamp() = 1000
context.timerService().currentWatermark() = 50

 Details about how I'm running this:
 - I'm on Flink 1.12.3 that's running on EKS and using MSK as the source.
 - I'm using FlinkKafkaConsumer
 - I'm using WatermarkStrategy.forBoundedOutOfOrderness(5s).  No
 idleness settings.
 - I'm running similar code in all the environments.  The main
 difference is low traffic.  I have not been able to reproduce this out of
 the environment.


 I put the following process function right after my kafka source.

 

 AfterSource
 ts=1647274892728
 watermark=1647575140007
 record=...


 public static class TextLog extends ProcessFunction {
 private final String label;
 public TextLogDeliveryLog(String label) {
 this.label = label;
 }
 @Override
 public void processElement(Record record, Context context,
 Collector collector) throws Exception {
 LOGGER.info("{}\nts={}\nwatermark={}\nrecord={}",
 label, context.timestamp(),
 context.timerService().currentWatermark(), record);
 collector.collect(deliveryLog);
 }
 }

>>>

Re: Anyone trying to adopt Scotty on the recent Flink versions?

[Dongwon Kim]

Hi David,

Scotty's approach to sliding windows seems really interesting ;)

I also agree that the idea is very interesting. It will be very effective
for our use case as we create a sliding window of size 10 minutes that
slides by 1 minutes for every road of the country.

 Looking at the code [1], it seems to be no longer maintained. It's both
> compiled and tested against Flink 1.8 so I wouldn't really expect it to be
> compatible with 1.14.x :(
>
Yeah, we tried to compile it with Flink 1.14.x but it failed. Scotty
compiled with Flink 1.8 nonetheless seems to work well for about 10~20
minutes when we executed the job on a cluster, so we thought that Flink's
runtime, especially windows, hasn't changed significantly since 1.8.

I found that Flink committers previously discussed implanting the core idea
to Flink [1] but it seems not discussed anymore, unfortunately.

Thanks for the reply :-)

Best,

Dongwon

On Wed, Dec 29, 2021 at 5:53 PM David Morávek  wrote:

> Hi Dongwon,
>
> Scotty's approach to sliding windows seems really interesting ;) Looking
> at the code [1], it seems to be no longer maintained. It's both compiled
> and tested against Flink 1.8 so I wouldn't really expect it to be
> compatible with 1.14.x :(
>
> [1] https://github.com/TU-Berlin-DIMA/scotty-window-processor
>
> Best,
> D.
>
> On Tue, Dec 28, 2021 at 4:15 AM Dongwon Kim  wrote:
>
>> Hi community,
>>
>> We're recently trying to adopt Scotty to overcome the poor performance
>> caused by too many sliding windows.
>>
>> We're facing the following exception on the latest Flink-1.14.2:
>>
>> switched from RUNNING to FAILED with failure cause: 
>> java.lang.ArrayIndexOutOfBoundsException: -1
>>  at java.util.ArrayList.elementData(ArrayList.java:424)
>>  at java.util.ArrayList.get(ArrayList.java:437)
>>  at 
>> de.tub.dima.scotty.slicing.aggregationstore.LazyAggregateStore.getSlice(LazyAggregateStore.java:53)
>>  at 
>> de.tub.dima.scotty.slicing.aggregationstore.LazyAggregateStore.insertValueToSlice(LazyAggregateStore.java:64)
>>  at 
>> de.tub.dima.scotty.slicing.SliceManager.processElement(SliceManager.java:76)
>>  at 
>> de.tub.dima.scotty.slicing.SlicingWindowOperator.processElement(SlicingWindowOperator.java:43)
>>  at 
>> de.tub.dima.scotty.flinkconnector.KeyedScottyWindowOperator.processElement(KeyedScottyWindowOperator.java:62)
>>  at 
>> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
>>  at 
>> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
>>  at 
>> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
>>  at 
>> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
>>  at 
>> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
>>  at 
>> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
>>  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
>>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
>>  at java.lang.Thread.run(Thread.java:748)
>>
>> We create a window operator as follows:
>>
>> import de.tub.dima.scotty.core.windowType.{SlidingWindow, WindowMeasure}
>> import de.tub.dima.scotty.flinkconnector.KeyedScottyWindowOperator
>> import de.tub.dima.scotty.flinkconnector.{_}
>>
>> val windowOp = new KeyedScottyWindowOperator[(java.lang.String, Long), 
>> NaviGpsProcessable, NaviTrafficUserResult](new 
>> NaviTrafficUserAggregationScotty())
>> windowOp.addWindow(new SlidingWindow(WindowMeasure.Time, 600_000, 
>> 60_000))
>>
>> val userAggStream = stream
>>   .keyBy(el => (el.id, el.trafficId))
>>   .process(windowOp)
>>   .map(_.getAggValues.get(0));
>>
>> Can I get any advice on this?
>>
>> Best,
>>
>> Dongwon
>>
>

Anyone trying to adopt Scotty on the recent Flink versions?

[Dongwon Kim]

Hi community,

We're recently trying to adopt Scotty to overcome the poor performance
caused by too many sliding windows.

We're facing the following exception on the latest Flink-1.14.2:

switched from RUNNING to FAILED with failure cause:
java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:424)
at java.util.ArrayList.get(ArrayList.java:437)
at 
de.tub.dima.scotty.slicing.aggregationstore.LazyAggregateStore.getSlice(LazyAggregateStore.java:53)
at 
de.tub.dima.scotty.slicing.aggregationstore.LazyAggregateStore.insertValueToSlice(LazyAggregateStore.java:64)
at 
de.tub.dima.scotty.slicing.SliceManager.processElement(SliceManager.java:76)
at 
de.tub.dima.scotty.slicing.SlicingWindowOperator.processElement(SlicingWindowOperator.java:43)
at 
de.tub.dima.scotty.flinkconnector.KeyedScottyWindowOperator.processElement(KeyedScottyWindowOperator.java:62)
at 
org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
at 
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at 
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)

We create a window operator as follows:

import de.tub.dima.scotty.core.windowType.{SlidingWindow, WindowMeasure}
import de.tub.dima.scotty.flinkconnector.KeyedScottyWindowOperator
import de.tub.dima.scotty.flinkconnector.{_}

val windowOp = new KeyedScottyWindowOperator[(java.lang.String,
Long), NaviGpsProcessable, NaviTrafficUserResult](new
NaviTrafficUserAggregationScotty())
windowOp.addWindow(new SlidingWindow(WindowMeasure.Time, 600_000, 60_000))

val userAggStream = stream
  .keyBy(el => (el.id, el.trafficId))
  .process(windowOp)
  .map(_.getAggValues.get(0));

Can I get any advice on this?

Best,

Dongwon

Re: [DISCUSS] Drop Zookeeper 3.4

[Dongwon Kim]

When should I prepare for upgrading ZK to 3.5 or newer?
We're operating a Hadoop cluster w/ ZK 3.4.6 for running only Flink jobs.
Just hope that the rolling update is not that painful - any advice on this?

Best,

Dongwon

On Tue, Dec 7, 2021 at 3:22 AM Chesnay Schepler  wrote:

> Current users of ZK 3.4 and below would need to upgrade their Zookeeper
> installation that is used by Flink to 3.5+.
>
> Whether K8s users are affected depends on whether they use ZK or not. If
> they do, see above, otherwise they are not affected at all.
>
> On 06/12/2021 18:49, Arvid Heise wrote:
>
> Could someone please help me understand the implications of the upgrade?
>
> As far as I understood this upgrade would only affect users that have a
> zookeeper shared across multiple services, some of which require ZK 3.4-? A
> workaround for those users would be to run two ZKs with different versions,
> eventually deprecating old ZK, correct?
>
> If that is the only limitation, I'm +1 for the proposal since ZK 3.4 is
> already EOL.
>
> How are K8s users affected?
>
> Best,
>
> Arvid
>
> On Mon, Dec 6, 2021 at 2:00 PM Chesnay Schepler 
> wrote:
>
>> ping @users; any input on how this would affect you is highly appreciated.
>>
>> On 25/11/2021 22:39, Chesnay Schepler wrote:
>> > I included the user ML in the thread.
>> >
>> > @users Are you still using Zookeeper 3.4? If so, were you planning to
>> > upgrade Zookeeper in the near future?
>> >
>> > I'm not sure about ZK compatibility, but we'd also upgrade Curator to
>> > 5.x, which doesn't support ookeeperK 3.4 anymore.
>> >
>> > On 25/11/2021 21:56, Till Rohrmann wrote:
>> >> Should we ask on the user mailing list whether anybody is still using
>> >> ZooKeeper 3.4 and thus needs support for this version or can a
>> ZooKeeper
>> >> 3.5/3.6 client talk to a ZooKeeper 3.4 cluster? I would expect that
>> >> not a
>> >> lot of users depend on it but just to make sure that we aren't
>> >> annoying a
>> >> lot of our users with this change. Apart from that +1 for removing it
>> if
>> >> not a lot of user depend on it.
>> >>
>> >> Cheers,
>> >> Till
>> >>
>> >> On Wed, Nov 24, 2021 at 11:03 AM Matthias Pohl > >
>> >> wrote:
>> >>
>> >>> Thanks for starting this discussion, Chesnay. +1 from my side. It's
>> >>> time to
>> >>> move forward with the ZK support considering the EOL of 3.4 you
>> already
>> >>> mentioned. The benefits we gain from upgrading Curator to 5.x as a
>> >>> consequence is another plus point. Just for reference on the
>> >>> inconsistent
>> >>> state issue you mentioned: FLINK-24543 [1].
>> >>>
>> >>> Matthias
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/FLINK-24543
>> >>>
>> >>> On Wed, Nov 24, 2021 at 10:19 AM Chesnay Schepler > >
>> >>> wrote:
>> >>>
>>  Hello,
>> 
>>  I'd like to drop support for Zookeeper 3.4 in 1.15, upgrading the
>>  default to 3.5 with an opt-in for 3.6.
>> 
>>  Supporting Zookeeper 3.4 (which is already EOL) prevents us from
>>  upgrading Curator to 5.x, which would allow us to properly fix an
>>  issue
>>  with inconsistent state. It is also required to eventually support ZK
>> >>> 3.6.
>> >
>> >
>>
>>
>

Converting DataStream of Avro SpecificRecord to Table

[Dongwon Kim]

Hi community,

I'm currently converting a DataStream of Avro SpecificRecord type into
Table using the following method:

public static  Table
toTable(StreamTableEnvironment tEnv,
   DataStream dataStream,
   Class cls) {
  RichMapFunction avroSpecific2RowConverter = new RichMapFunction<>() {
private transient AvroSerializationSchema avro2bin = null;
private transient AvroRowDeserializationSchema bin2row = null;

@Override
public void open(Configuration parameters) throws Exception {
  avro2bin = AvroSerializationSchema.forSpecific(cls);
  bin2row = new AvroRowDeserializationSchema(cls);
}

@Override
public Row map(T value) throws Exception {
  byte[] bytes = avro2bin.serialize(value);
  Row row = bin2row.deserialize(bytes);
  return row;
}
  };

  SingleOutputStreamOperator rows =
dataStream.map(avroSpecific2RowConverter)
// https://issues.apache.org/jira/browse/FLINK-23885
.returns(AvroSchemaConverter.convertToTypeInfo(cls));

  return tEnv.fromDataStream(rows);
}

I'm wondering whether there's a pre-defined utility for that or a better
way to do so in Flink-1.14.

Best,

Dongwon

Re: how to expose the current in-flight async i/o requests as metrics?

[Dongwon Kim]

Hi Fabian,

Can you maybe share more about the setup and how you use the AsyncFunction
> with
> the Kafka client?

Oops, I didn't mention in the reply to David that the kafka producer has
nothing to do with the AsyncFunction!
I interact with Redis and a Spring boot app. in the AsyncFunction, not
Kafka.
My pipeline looks like "KafkaSource -> AsyncFunction -> Window ->
(KafkaSink1, KafkaSink2)" and the kafka producer is the last one.

As David already pointed out it could be indeed a Kafka bug but it could
> also
> mean that your defined async function leaks direct memory by not freeing
> some
> resources.

The problem seems to be the KafkaSink2 which I recently added to the
pipeline in order to write large records (~10MB) for a debugging purpose.
I just launched the pipeline without KafkaSink2 to see whether or not my
conjecture is right or wrong.
If so, I'd rather give more direct memory to each task manager to avoid
this problem.

We can definitely improve the metrics for the AsyncFunction and expose the
> current queue size as a followup.

Really looking forward to it. I spent many hours debugging AsyncFunction
without metrics. It would be great to have how many records are timed out
as a metric as well.

Thanks,

Dongwon


On Tue, Nov 9, 2021 at 7:55 PM Fabian Paul  wrote:

> Hi Dongwan,
>
> Can you maybe share more about the setup and how you use the AsyncFunction
> with
> the Kafka client?
>
> As David already pointed out it could be indeed a Kafka bug but it could
> also
> mean that your defined async function leaks direct memory by not freeing
> some
> resources.
>
> We can definitely improve the metrics for the AsyncFunction and expose the
> current queue size as a followup.
>
> Best,
> Fabian

Re: how to expose the current in-flight async i/o requests as metrics?

[Dongwon Kim]

Hi Jun,

Did you override AsyncFunction#timeout()?  If so, did you call
> resultFuture.complete()/completeExceptionally() in your override?  Not
> calling them can result in checkpoint timeout.

No, I've only called resultFuture.complete() in AsyncFunction.asyncInvoke()
and didn't know much about  AsyncFunction.timeout(). However, looking at
the default implementation of AsyncFunction.timeout(), I'd rather get the
timeout exception as my streaming pipeline is going to be fail-fast which
is what I prefer the most.

I think I understand what you are concerned about but, as I wrote in the
reply to David, the problem seems to be the kafka sink which I recently
added to the pipeline in order to write large records (~10MB) for the
debugging purpose.

Anyway thanks a lot for letting me know the possibility of overriding
AsyncFunction.timeout().

Best,

Dongwon

On Tue, Nov 9, 2021 at 5:53 PM Jun Qin  wrote:

> Hi Dongwon
>
> Did you override AsyncFunction#timeout()?  If so, did you call
> resultFuture.complete()/completeExceptionally() in your override?  Not
> calling them can result in checkpoint timeout.
>
> Thanks
> Jun
>
>
> On Nov 9, 2021, at 7:37 AM, Dongwon Kim  wrote:
>
> Hi David,
>
> There are currently no metrics for the async work-queue size (you should
>> be able to see the queue stats with debug logs enabled though [1]).
>
> Thanks for the input but scraping DEBUG messages into, for example,
> ElasticSearch for monitoring on Grafana is not possible in my current
> environment.
> I just defined two counters in RichAsyncFunction for tracking # sent
> requests and # finished/failed requests, respectively, and used the two
> counters to calculate the inflight requests from Prometheus.
>
> As far as I can tell from looking at the code, the async operator is able
>> to checkpoint even if the work-queue is exhausted.
>
> Oh, I didn't know that! As you pointed out and I'm going to explain below,
> the async operator might not be the source of the problem.
>
> I just hit the same situation and found that
> - # of inflight records are zero when the backpressure is getting high
> - A taskmanager complains the following error message around the time when
> the backpressure is getting high (all the others don't do):
>
>> 2021-11-09 13:20:40,601 ERROR org.apache.kafka.common.utils.KafkaThread
>>   [] - Uncaught exception in thread
>> 'kafka-producer-network-thread | producer-8':
>> java.lang.OutOfMemoryError: Direct buffer memory
>> at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
>> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:118)
>> ~[?:?]
>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
>> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[?:?]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[?:?]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[?:?]
>> at
>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493) ~[?:?]
>> at java.nio.channels.SocketChannel.write(SocketChannel.java:507)
>> ~[?:?]
>> at
>> org.apache.kafka.common.network.PlaintextTransportLayer.write(PlaintextTransportLayer.java:152)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.common.network.ByteBufferSend.writeTo(ByteBufferSend.java:60)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:429)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:399)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:589)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.common.network.Selector.poll(Selector.java:483)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at
>> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
>> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>> at java.lang.Thread.run(Thread.java:829) [?:?]
>>
>
> Can it be the reason why my pipeline is stalled and ends up with the
> checkout timeout? I guess all the upstream tasks m

Re: how to expose the current in-flight async i/o requests as metrics?

[Dongwon Kim]

Hi David,

There are currently no metrics for the async work-queue size (you should be
> able to see the queue stats with debug logs enabled though [1]).

Thanks for the input but scraping DEBUG messages into, for example,
ElasticSearch for monitoring on Grafana is not possible in my current
environment.
I just defined two counters in RichAsyncFunction for tracking # sent
requests and # finished/failed requests, respectively, and used the two
counters to calculate the inflight requests from Prometheus.

As far as I can tell from looking at the code, the async operator is able
> to checkpoint even if the work-queue is exhausted.

Oh, I didn't know that! As you pointed out and I'm going to explain below,
the async operator might not be the source of the problem.

I just hit the same situation and found that
- # of inflight records are zero when the backpressure is getting high
- A taskmanager complains the following error message around the time when
the backpressure is getting high (all the others don't do):

> 2021-11-09 13:20:40,601 ERROR org.apache.kafka.common.utils.KafkaThread
>   [] - Uncaught exception in thread
> 'kafka-producer-network-thread | producer-8':
>
> java.lang.OutOfMemoryError: Direct buffer memory
>
> at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
>
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:118)
> ~[?:?]
>
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
>
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[?:?]
>
> at sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[?:?]
>
> at sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[?:?]
>
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
> ~[?:?]
>
> at java.nio.channels.SocketChannel.write(SocketChannel.java:507)
> ~[?:?]
>
> at
> org.apache.kafka.common.network.PlaintextTransportLayer.write(PlaintextTransportLayer.java:152)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.common.network.ByteBufferSend.writeTo(ByteBufferSend.java:60)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.common.network.KafkaChannel.send(KafkaChannel.java:429)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:399)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:589)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.common.network.Selector.poll(Selector.java:483)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
> ~[drivinghabit-stream-calculator-2.0-SNAPSHOT.jar:?]
>
> at java.lang.Thread.run(Thread.java:829) [?:?]
>

Can it be the reason why my pipeline is stalled and ends up with the
checkout timeout? I guess all the upstream tasks might fail to send data to
the failed kafka producer and records are stacking up in buffers, which
could result in the back-pressure. If so, is there no mechanism in Flink to
detect such an error and send it to the job manager for debugging purposes?

Best,

Dongwon


On Mon, Nov 8, 2021 at 9:21 PM David Morávek  wrote:

> Hi Dongwon,
>
> There are currently no metrics for the async work-queue size (you should
> be able to see the queue stats with debug logs enabled though [1]). As far
> as I can tell from looking at the code, the async operator is able to
> checkpoint even if the work-queue is exhausted.
>
> Arvid can you please validate the above? (the checkpoints not being
> blocked by the work queue part)
>
> [1]
> https://github.com/apache/flink/blob/release-1.14.0/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/queue/OrderedStreamElementQueue.java#L109
>
> Best,
> D.
>
> On Sun, Nov 7, 2021 at 10:41 AM Dongwon Kim  wrote:
>
>> Hi community,
>>
>> While using Flink's async i/o for interacting with an external system, I
>> got the following exception:
>>
>> 2021-11-06 10:38:35,270 INFO  
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Triggering 
>> checkpoint 54 (type=CHECKPOINT) @ 1636162715262 for job 
>> f168a44ea331

how to expose the current in-flight async i/o requests as metrics?

[Dongwon Kim]

Hi community,

While using Flink's async i/o for interacting with an external system, I
got the following exception:

2021-11-06 10:38:35,270 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] -
Triggering checkpoint 54 (type=CHECKPOINT) @ 1636162715262 for job
f168a44ea33198cd71783824d49f9554.
2021-11-06 10:38:47,031 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] -
Completed checkpoint 54 for job f168a44ea33198cd71783824d49f9554
(11930992707 bytes, checkpointDuration=11722 ms, finalizationTime=47
ms).
2021-11-06 10:58:35,270 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] -
Triggering checkpoint 55 (type=CHECKPOINT) @ 1636163915262 for job
f168a44ea33198cd71783824d49f9554.
2021-11-06 11:08:35,271 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] -
Checkpoint 55 of job f168a44ea33198cd71783824d49f9554 expired before
completing.
2021-11-06 11:08:35,287 INFO
org.apache.flink.runtime.jobmaster.JobMaster [] -
Trying to recover from a global failure.


- FYI, I'm using 1.14.0 and enabled unaligned checkpointing and buffer
debloating
- the 55th ckpt failed to complete within 10 mins (which is the value of
execution.checkpointing.timeout)
- the below graph shows that backpressure skyrocketed around the time the
55th ckpt began
[image: image.png]

What I suspect is the capacity of the asynchronous operation because
limiting the value can cause back-pressure once the capacity is exhausted
[1].

Although I could increase the value, I want to monitor the current
in-flight async i/o requests like the above back-pressure graph on Grafana.
[2] does not introduce any system metric specific to async i/o.

Best,

Dongwon

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#async-io-api
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/metrics/#system-metrics

Re: How to deserialize Avro enum type in Flink SQL?

[Dongwon Kim]

>
> Can you provide a minimal reproducer (without confluent schema registry)
> with a valid input?
>

Please download and unzip the attached file.

   - src/main/avro/MyProtocol.avdl
  - MyRecord, MyEntry, and the MyEnumType is defined
  - "mvn generate-sources" will auto-generate Java classes under
  "target/generated-sources"
   - "org.example.fs" contains
  - "org.example.fs.Writer" which writes a single record of MyRecord
  type to "output.avro"
  - "org.example.fs.Reader" which reads the record from "output.avro"
  - "org.example.fs.ExampleFromFileSystem" executes CREATE TABLE
  defined in "my_table.ddl" and shows that it successfully deserialize
  MyRecord from a Avro record written in a file as you mentioned.
   - "org.example.kafka.ExampleFromKafkaAndSR" does almost the same as
   "org.example.fs.ExampleFromFileSystem" except that it reads from Kafka and
   looks up the schema from Schema Registry
  - However, it produces the same exception unlike
  ExampleFromFileSystem
  - What I produced to a Kafka topic is {"entries": [{"type":
  "TypeVal1"}, {"type": "TypeVal2"}, {"type": "TypeVal2"}]} which is a Avro
  record saved on output.avro.
  - The size of "output.avro" is 321 bytes on the disk while the size
  of the value of a Kafka record is 10 bytes.

Hope this provides enough information.

Best,

Dongwon

On Wed, Oct 13, 2021 at 4:50 PM Francesco Guardiani 
wrote:

> First of all, are you sure the input data is correct? From the stacktrace
> it seems to me the issue might be that the input data is invalid.
>
> Looking at the code of AvroToRowDataConverters, It sounds like STRING
> should work with avro enums. Can you provide a minimal reproducer (without
> confluent schema registry) with a valid input?
>
> On Tue, Oct 12, 2021 at 6:19 PM Dongwon Kim  wrote:
>
>> Hi community,
>>
>> Can I get advice on this question?
>>
>> Another user just sent me an email asking whether I found a solution or a
>> workaround for this question, but I'm still stuck there.
>>
>> Any suggestions?
>>
>> Thanks in advance,
>>
>> Dongwon
>>
>> -- Forwarded message -
>> From: Dongwon Kim 
>> Date: Mon, Aug 9, 2021 at 7:26 PM
>> Subject: How to deserialize Avro enum type in Flink SQL?
>> To: user 
>>
>>
>> Hi community,
>>
>> I have a Kafka topic where the schema of its values is defined by the
>> "MyRecord" record in the following Avro IDL and registered to the Confluent
>> Schema Registry.
>>
>>> @namespace("my.type.avro")
>>> protocol MyProtocol {
>>>   enum MyEnumType {
>>> TypeVal1, TypeVal2
>>>   }
>>>   record MyEntry {
>>> MyEnumType type;
>>>   }
>>>   record MyRecord {
>>> array entries;
>>>   }
>>> }
>>
>>
>> To read from the topic, I've defined the following DDL:
>>
>>> CREATE TABLE my_table
>>
>> (
>>> `entries` ARRAY>> *`type` ??? (This is the main question)*
>>> >>
>>> ) WITH (
>>> 'connector' = 'kafka',
>>> 'topic' = 'my-topic',
>>> 'properties.bootstrap.servers' = '...:9092',
>>> 'scan.startup.mode' = 'latest-offset',
>>> 'value.format' = 'avro-confluent',
>>> 'value.avro-confluent.schema-registry.url' = 'http://...:8081'
>>>
>> )
>>
>>
>> And I run the following query :
>>
>>> SELECT * FROM my_table
>>
>>
>> Now I got the following messages in Flink-1.13.1 when I use *STRING* for
>> the type:
>>
>>> *Caused by: java.io.IOException: Failed to deserialize Avro record.*
>>>   at
>>> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
>>>   at
>>> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
>>>   at
>>> org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
>>>   at
>>> org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:113)
>>>   at
>>> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
>>>   at
>>> org

Fwd: How to deserialize Avro enum type in Flink SQL?

[Dongwon Kim]

Hi community,

Can I get advice on this question?

Another user just sent me an email asking whether I found a solution or a
workaround for this question, but I'm still stuck there.

Any suggestions?

Thanks in advance,

Dongwon

-- Forwarded message -
From: Dongwon Kim 
Date: Mon, Aug 9, 2021 at 7:26 PM
Subject: How to deserialize Avro enum type in Flink SQL?
To: user 


Hi community,

I have a Kafka topic where the schema of its values is defined by the
"MyRecord" record in the following Avro IDL and registered to the Confluent
Schema Registry.

> @namespace("my.type.avro")
> protocol MyProtocol {
>   enum MyEnumType {
> TypeVal1, TypeVal2
>   }
>   record MyEntry {
> MyEnumType type;
>   }
>   record MyRecord {
> array entries;
>   }
> }


To read from the topic, I've defined the following DDL:

> CREATE TABLE my_table

(
> `entries` ARRAY *`type` ??? (This is the main question)*
> >>
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'my-topic',
> 'properties.bootstrap.servers' = '...:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'value.format' = 'avro-confluent',
> 'value.avro-confluent.schema-registry.url' = 'http://...:8081'
>
)


And I run the following query :

> SELECT * FROM my_table


Now I got the following messages in Flink-1.13.1 when I use *STRING* for
the type:

> *Caused by: java.io.IOException: Failed to deserialize Avro record.*
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
>   at
> org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
>   at
> org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:113)
>   at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
>   at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
>   at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
>   at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
>   at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
>   at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
> *Caused by: org.apache.avro.AvroTypeException: Found
> my.type.avro.MyEnumType, expecting union*
>   at
> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:308)
>   at org.apache.avro.io.parsing.Parser.advance(Parser.java:86)
>   at
> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:275)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
>   at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:298)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:183)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
>   at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
>   at
> org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:81)
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
>   ... 9 more

The reason I use the STRING type is just for fast-prototyping.

While reading through [1], I've been thinking about using *RAW('class',
'snapshot')* where 'class' is my.type.avro.MyEnumType, but I'm not sure
whether it is a good idea and if so, what can be a value for the snapshot.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/types/#raw

Thanks in advance,

Dongwon

How to deserialize Avro enum type in Flink SQL?

[Dongwon Kim]

Hi community,

I have a Kafka topic where the schema of its values is defined by the
"MyRecord" record in the following Avro IDL and registered to the Confluent
Schema Registry.

> @namespace("my.type.avro")
> protocol MyProtocol {
>   enum MyEnumType {
> TypeVal1, TypeVal2
>   }
>   record MyEntry {
> MyEnumType type;
>   }
>   record MyRecord {
> array entries;
>   }
> }


To read from the topic, I've defined the following DDL:

> CREATE TABLE my_table

(
> `entries` ARRAY *`type` ??? (This is the main question)*
> >>
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'my-topic',
> 'properties.bootstrap.servers' = '...:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'value.format' = 'avro-confluent',
> 'value.avro-confluent.schema-registry.url' = 'http://...:8081'
>
)


And I run the following query :

> SELECT * FROM my_table


Now I got the following messages in Flink-1.13.1 when I use *STRING* for
the type:

> *Caused by: java.io.IOException: Failed to deserialize Avro record.*
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
>   at
> org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
>   at
> org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:113)
>   at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
>   at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
>   at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
>   at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
>   at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
>   at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
> *Caused by: org.apache.avro.AvroTypeException: Found
> my.type.avro.MyEnumType, expecting union*
>   at
> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:308)
>   at org.apache.avro.io.parsing.Parser.advance(Parser.java:86)
>   at
> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:275)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
>   at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:298)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:183)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
>   at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
>   at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
>   at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
>   at
> org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:81)
>   at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
>   ... 9 more

The reason I use the STRING type is just for fast-prototyping.

While reading through [1], I've been thinking about using *RAW('class',
'snapshot')* where 'class' is my.type.avro.MyEnumType, but I'm not sure
whether it is a good idea and if so, what can be a value for the snapshot.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/types/#raw

Thanks in advance,

Dongwon

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

[Dongwon Kim]

Hi Khachatryan,

Thanks for the explanation and the input!

1. Use the State Processor API to create a new snapshot [1]

I haven't used it. but does the API prevent the class of a specific
serializer from being loaded?

2. If the operator has only this state then changing uid (together with
> allowNonRestoredState) should help

Very unfortunately, I have another per-key state defined on the operator
which is very important and cannot be abandoned T.T

3. Probably just changing POJO to an empty class will suffice in your case?

Yeah, I might be bringing the class definition for a while.

Best,

Dongwon


On Tue, Feb 9, 2021 at 2:35 AM Khachatryan Roman <
khachatryan.ro...@gmail.com> wrote:

> Hi,
>
> I'm pulling Yun Tang who is familiar with StateBackends and RocksDB in
> particular.
>
> From what I see, the 2nd snapshot (sp2) is built using the same set of
> states obtained from the starting savepoint/checkpoint (sp1) to write its
> metadata. This metadata includes serializers snapshots, including
> PojoSerializer for your custom type. On restore, this metadata is read, and
> POJO class itself is loaded.
>
> I see the following ways to overcome this issue:
> 1. Use the State Processor API to create a new snapshot [1]
> 2. If the operator has only this state then changing uid (together with
> allowNonRestoredState) should help
> 3. Probably just changing POJO to an empty class will suffice in your case?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> Regards,
> Roman
>
>
> On Mon, Feb 8, 2021 at 4:31 PM Dongwon Kim  wrote:
>
>> Hi 张静,
>>
>> Q1: By default, a savepoint restore will try to match all state
>>> back to the restored job. `AllowNonRestoredState` cannot avoid
>>> recovery all state from savepoint, but only skip match all of the
>>> restore state back to the restored job. So `ClassNotFoundException `
>>> could not be avoid.
>>
>> okay
>>
>>Q2: Not really. After you recover new job from the savepoint
>>> (savepoint1)based on (1), you could do a new savepoint (savepoint2),
>>> then remove the definition of the POJO type. then you can restore from
>>> savepoint2.
>>>
>> I did it but it ends up with the same ClassNotFoundException :-(
>>
>> What I did exactly are
>> (1) Trigger sp1 from v1
>> (2) Start v2-1 (w/ the definition of the POJO but do not use it at all)
>> from sp1
>> (3) Trigger sp2 from v2-1
>> (4) Start v2-2 (w/o the definition of the POJO)  from sp2
>> (5) v2-2 failed with the same ClassNotFoundException regarding the POJO
>> type
>>
>> Should v2-2 successfully start from sp2?
>>
>> Best,
>>
>> Dongwon
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 8, 2021 at 11:48 PM 张静  wrote:
>>
>>> Hi, Dongwon,
>>>  Q1: By default, a savepoint restore will try to match all state
>>> back to the restored job. `AllowNonRestoredState` cannot avoid
>>> recovery all state from savepoint, but only skip match all of the
>>> restore state back to the restored job. So `ClassNotFoundException `
>>> could not be avoid.
>>>  Q2: Not really. After you recover new job from the savepoint
>>> (savepoint1)based on (1), you could do a new savepoint (savepoint2),
>>> then remove the definition of the POJO type. then you can restore from
>>> savepoint2.
>>> Correct me please if I'm wrong. Thanks.
>>>
>>> Best,
>>> Beyond1920
>>>
>>> Dongwon Kim  于2021年2月8日周一 下午9:43写道：
>>> >
>>> > Hi,
>>> >
>>> > I have an original job (say v1) and I want to start a new job (say v2)
>>> from a savepoint of v1.
>>> >
>>> > An operator of v1 used to have per-key states of a POJO type, but I
>>> want to remove the states together with the definition of the POJO type.
>>> >
>>> > When I start v2 from a savepoint of v1, I specified
>>> "--allowNonRestoredState" but  I got the following exception:
>>> >
>>> > 2021-02-08 22:07:28,324 INFO
>>> org.apache.flink.runtime.executiongraph.ExecutionGraph   [] -
>>> input-to-idata (5/8) (168520e96e73dbf6c2e3c5342b324764) switched from
>>> RUNNING to FAILED on container_e02_1607261469522_0242_01_08 @
>>> mobdata-flink-dn29.dakao.io (dataPort=45505).
>>> > java.lang.Exception: Exception while creating
>>> StreamOperatorStateContext.
>>> > at
>>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperator

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

[Dongwon Kim]

Hi 张静,

Q1: By default, a savepoint restore will try to match all state
> back to the restored job. `AllowNonRestoredState` cannot avoid
> recovery all state from savepoint, but only skip match all of the
> restore state back to the restored job. So `ClassNotFoundException `
> could not be avoid.

okay

   Q2: Not really. After you recover new job from the savepoint
> (savepoint1)based on (1), you could do a new savepoint (savepoint2),
> then remove the definition of the POJO type. then you can restore from
> savepoint2.
>
I did it but it ends up with the same ClassNotFoundException :-(

What I did exactly are
(1) Trigger sp1 from v1
(2) Start v2-1 (w/ the definition of the POJO but do not use it at all)
from sp1
(3) Trigger sp2 from v2-1
(4) Start v2-2 (w/o the definition of the POJO)  from sp2
(5) v2-2 failed with the same ClassNotFoundException regarding the POJO type

Should v2-2 successfully start from sp2?

Best,

Dongwon






On Mon, Feb 8, 2021 at 11:48 PM 张静  wrote:

> Hi, Dongwon,
>  Q1: By default, a savepoint restore will try to match all state
> back to the restored job. `AllowNonRestoredState` cannot avoid
> recovery all state from savepoint, but only skip match all of the
> restore state back to the restored job. So `ClassNotFoundException `
> could not be avoid.
>  Q2: Not really. After you recover new job from the savepoint
> (savepoint1)based on (1), you could do a new savepoint (savepoint2),
> then remove the definition of the POJO type. then you can restore from
> savepoint2.
> Correct me please if I'm wrong. Thanks.
>
> Best,
> Beyond1920
>
> Dongwon Kim  于2021年2月8日周一 下午9:43写道：
> >
> > Hi,
> >
> > I have an original job (say v1) and I want to start a new job (say v2)
> from a savepoint of v1.
> >
> > An operator of v1 used to have per-key states of a POJO type, but I want
> to remove the states together with the definition of the POJO type.
> >
> > When I start v2 from a savepoint of v1, I specified
> "--allowNonRestoredState" but  I got the following exception:
> >
> > 2021-02-08 22:07:28,324 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph   [] -
> input-to-idata (5/8) (168520e96e73dbf6c2e3c5342b324764) switched from
> RUNNING to FAILED on container_e02_1607261469522_0242_01_08 @
> mobdata-flink-dn29.dakao.io (dataPort=45505).
> > java.lang.Exception: Exception while creating StreamOperatorStateContext.
> > at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
> > Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for
> CoBroadcastWithKeyedOperator_692eb9021f6319ecb59ea7c8901de92a_(5/8) from
> any of the 1 provided restore options.
> > at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
> ~[flink-dist_2.11-1.12.1.jar:1.12.1]
> > ... 9 more
> > Caused by: org.apache.flink.runtime.state.BackendBuildingException:
> Caught unexpecte

Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

[Dongwon Kim]

Hi,

I have an original job (say v1) and I want to start a new job (say v2) from
a savepoint of v1.

An operator of v1 used to have per-key states of a POJO type, but I want to
remove the states together with the definition of the POJO type.

When I start v2 from a savepoint of v1, I specified
"--allowNonRestoredState" but  I got the following exception:

2021-02-08 22:07:28,324 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph   [] -
input-to-idata (5/8) (168520e96e73dbf6c2e3c5342b324764) switched from
RUNNING to FAILED on container_e02_1607261469522_0242_01_08 @
mobdata-flink-dn29.dakao.io (dataPort=45505).
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
Caused by: org.apache.flink.util.FlinkException: Could not restore
keyed state backend for
CoBroadcastWithKeyedOperator_692eb9021f6319ecb59ea7c8901de92a_(5/8)
from any of the 1 provided restore options.
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
... 9 more
Caused by: org.apache.flink.runtime.state.BackendBuildingException:
Caught unexpected exception.
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:361)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
... 9 more
Caused by: java.io.IOException: Could not find class
'com.kakaomobility.drivinghabit.stream.IDataConverter$SpeedAndLoc' in
classpath.
at 
org.apache.flink.util.InstantiationUtil.resolveClassByName(InstantiationUtil.java:756)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.util.InstantiationUtil.resolveClassByName(InstantiationUtil.java:731)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 
org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshotData.readSnapshotData(PojoSerializerSnapshotData.java:214)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at 

Re: How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

[Dongwon Kim]

Hi Arvid,

Okay, I'd better get back to Avro as you suggested!
Thanks for the tips regarding Avro.

Best,

Dongwon

On Wed, Jan 13, 2021 at 3:28 AM Arvid Heise  wrote:

> Do you think Scala is a better option in that regard?
>
> I'm not sure if scala is better in this regard. Sure you could use sealed
> classes but I don't know if the schema inference is using it. Maybe @Timo
> Walther  knows more?
>
> I used to define Avro records in .avro files and generate Java classes to
>> use in my Flink application, but recently decided to move to POJOs for some
>> reasons (e.g. custom Java annotations, etc).
>>
> If it's just about annotations: You can use custom Java annotations with
> Avro generated classes by setting "javaAnnotations" property. [1] You can
> use it on records and fields.
> You can even provide your own velocity template [2] to add more features
> to the generation.
>
> [1]
> https://www.javacodegeeks.com/2020/07/it-is-never-enough-of-them-enriching-apache-avro-generated-classes-with-custom-java-annotations.html
> [2]
> https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/record.vm
>
> On Tue, Jan 12, 2021 at 11:16 AM Dongwon Kim 
> wrote:
>
>> Hi Arvid,
>>
>> Thanks for the very detailed explanation and tips!
>>
>> inferring the type information of Java classes is quite messy. At first,
>>> it seems like that should work out the box as you are only using  as the
>>> type of the list, right? However, there is no way of knowing if you didn't
>>> use a subclass of A. Of course, if A was final, it might be possible to
>>> exclude this case but you quickly go down a rabbit hole.
>>>
>> Do you think Scala is a better option in that regard?
>>
>> To avoid such limitations, I always recommend schema-first approaches
>>> when moving a PoC to production code. First figure out what kind of data
>>> you actually want to transfer. Then, settle for a serializer [2]. Then,
>>> create the schema and let the classes be generated (on-the-fly).
>>> I usually do it in two ways: if I write a rather generic program, I try
>>> to use Table API, which optimizes everything in a Row has one of the most
>>> memory efficient representations.
>>
>> I have to work with DataStream API as I need a custom trigger which is
>> not supported in Table API AFAIK.
>>
>> If Table API is not sufficient, I fall back to Avro and use Avro specific
>>> records [3].
>>
>> I used to define Avro records in .avro files and generate Java classes to
>> use in my Flink application, but recently decided to move to POJOs for some
>> reasons (e.g. custom Java annotations, etc).
>>
>> So it seems like I'd better satisfy with BTypeFactory in my original
>> question unless I'm willing to move to Scala or Avro, huh?
>>
>> Best,
>>
>> Dongwon
>>
>> On Fri, Jan 8, 2021 at 6:22 PM Arvid Heise  wrote:
>>
>>> Hi Dongwon,
>>>
>>> inferring the type information of Java classes is quite messy. At first,
>>> it seems like that should work out the box as you are only using  as the
>>> type of the list, right? However, there is no way of knowing if you didn't
>>> use a subclass of A. Of course, if A was final, it might be possible to
>>> exclude this case but you quickly go down a rabbit hole.
>>> It gets especially bad if you consider that your classes evolve over
>>> time. What happens if A is first final, but you later decide to subclass
>>> it? How should Flink map old data to the new hierarchy? Flink falls back to
>>> Kryo for most cases, which is why you need generic types. However, that is
>>> rather inefficient unless you register all possible classes beforehand [1].
>>>
>>> To avoid such limitations, I always recommend schema-first approaches
>>> when moving a PoC to production code. First figure out what kind of data
>>> you actually want to transfer. Then, settle for a serializer [2]. Then,
>>> create the schema and let the classes be generated (on-the-fly).
>>>
>>> I usually do it in two ways: if I write a rather generic program, I try
>>> to use Table API, which optimizes everything in a Row has one of the most
>>> memory efficient representations. If Table API is not sufficient, I fall
>>> back to Avro and use Avro specific records [3].
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/execution_configuration.html
>>> [2]
>>> https://flink.apa

Re: How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

[Dongwon Kim]

Hi Arvid,

Thanks for the very detailed explanation and tips!

inferring the type information of Java classes is quite messy. At first, it
> seems like that should work out the box as you are only using  as the
> type of the list, right? However, there is no way of knowing if you didn't
> use a subclass of A. Of course, if A was final, it might be possible to
> exclude this case but you quickly go down a rabbit hole.
>
Do you think Scala is a better option in that regard?

To avoid such limitations, I always recommend schema-first approaches when
> moving a PoC to production code. First figure out what kind of data you
> actually want to transfer. Then, settle for a serializer [2]. Then, create
> the schema and let the classes be generated (on-the-fly).
> I usually do it in two ways: if I write a rather generic program, I try to
> use Table API, which optimizes everything in a Row has one of the most
> memory efficient representations.

I have to work with DataStream API as I need a custom trigger which is not
supported in Table API AFAIK.

If Table API is not sufficient, I fall back to Avro and use Avro specific
> records [3].

I used to define Avro records in .avro files and generate Java classes to
use in my Flink application, but recently decided to move to POJOs for some
reasons (e.g. custom Java annotations, etc).

So it seems like I'd better satisfy with BTypeFactory in my original
question unless I'm willing to move to Scala or Avro, huh?

Best,

Dongwon

On Fri, Jan 8, 2021 at 6:22 PM Arvid Heise  wrote:

> Hi Dongwon,
>
> inferring the type information of Java classes is quite messy. At first,
> it seems like that should work out the box as you are only using  as the
> type of the list, right? However, there is no way of knowing if you didn't
> use a subclass of A. Of course, if A was final, it might be possible to
> exclude this case but you quickly go down a rabbit hole.
> It gets especially bad if you consider that your classes evolve over time.
> What happens if A is first final, but you later decide to subclass it? How
> should Flink map old data to the new hierarchy? Flink falls back to Kryo
> for most cases, which is why you need generic types. However, that is
> rather inefficient unless you register all possible classes beforehand [1].
>
> To avoid such limitations, I always recommend schema-first approaches when
> moving a PoC to production code. First figure out what kind of data you
> actually want to transfer. Then, settle for a serializer [2]. Then, create
> the schema and let the classes be generated (on-the-fly).
>
> I usually do it in two ways: if I write a rather generic program, I try to
> use Table API, which optimizes everything in a Row has one of the most
> memory efficient representations. If Table API is not sufficient, I fall
> back to Avro and use Avro specific records [3].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/execution_configuration.html
> [2]
> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
> [3] https://avro.apache.org/docs/current/gettingstartedjava.html
>
>
> On Mon, Jan 4, 2021 at 6:49 PM Dongwon Kim  wrote:
>
>> Any advice would be appreciated :)
>>
>> Thanks,
>>
>> Dongwon
>>
>> -- Forwarded message -
>> From: Dongwon Kim 
>> Date: Mon, Dec 14, 2020 at 11:27 PM
>> Subject: How to gracefully avoid "Generic types have been disabled in the
>> ExecutionConfig and type java.util.List is treated as a generic type"?
>> To: user 
>>
>>
>> Hi,
>>
>> The following program compiles and runs w/o exceptions:
>>
>>> public class Test {
>>>
>>>   public static class A {
>>> private int n;
>>>
>>> public A() { }
>>> public int getN() {  return n;  }
>>> public void setN(int n) {  this.n = n;  }
>>>   }
>>>
>>>   public static class B {
>>> private List lst;
>>>
>>> public B() { }
>>> public List getLst() {  return lst;  }
>>> public void setLst(List lst) {  this.lst = lst;  }
>>>   }
>>>
>>>   public static void main(String[] args) throws Exception {
>>> StreamExecutionEnvironment env =
>>> StreamExecutionEnvironment.createLocalEnvironment();
>>>
>>> env.fromElements(new B())
>>>   .print();
>>>
>>> env.execute();
>>>   }
>>> }
>>>
>>
>> When I add the following line,
>>
>>> env.getConfig().disableGenericTypes();
>>
>> then the program shows me an exception:
>>
>>> Exception in thread "

Re: "flink list" fails when zk-based HA is enabled in flink-conf.yaml

[Dongwon Kim]

Thanks Yang for the very detailed explanation!

Wow, I really appreciate it.

Best,

Dongwon

On Wed, Jan 6, 2021 at 10:17 PM Yang Wang  wrote:

> Hi Dongwon,
>
> Please find the answer inline.
>
> > why CLI accessed ZK?
> This is a good question. Currently, when the HA is enabled, even though we
> could get the JobManager rest endpoint from Yarn application report, we
> still have to retrieve the leader information from ZooKeeper. Please find
> more information in the class *RestClusterClient*. I am not aware of any
> potential issues if we directly retrieve rest endpoint from Yarn
> application report. And I think this could be a minor improvement.
>
> > Specify the "high-availability.cluster-id" to list jobs
> I have created a ticket for updating the documentation[1].
>
> > About the "-m yarn-cluster"
> You are right. "--target yarn-per-job" is the recommended way to start a
> perjob cluster. The backend cli option parser is *GenericCLI*. It is also
> used for application mode and K8s deployment. "-m yarn-cluster" is the old
> way. All the cli options are parsed by FlinkYarnSessionCli. Since it is
> widely used, it could not be deprecated or removed very soon. "--executor"
> has the exactly same effect with "--target". The only different is the
> naming.
>
> [1]. https://issues.apache.org/jira/browse/FLINK-20866
>
> Best,
> Yang
>
>
> Dongwon Kim  于2021年1月6日周三 下午6:49写道：
>
>> Hi Yang,
>>
>> I was wondering why CLI accessed ZK because, as shown in the following
>> lines, CLI seemed to know the address of JM by contacting AHS before
>> connecting to ZK.
>>
>> 2021-01-06 18:35:32,351 INFO  org.apache.flink.client.cli.CliFrontend
>>   [] - Running 'list' command.
>>
>> 2021-01-06 18:35:32,682 INFO  org.apache.hadoop.yarn.client.AHSProxy
>>   [] - Connecting to Application History server at
>> mobdata-devflink-nm02.dakao.io/10.93.0.91:10200
>>
>> 2021-01-06 18:35:32,763 INFO  org.apache.flink.yarn.YarnClusterDescriptor
>> [] - Found Web Interface
>> mobdata-devflink-dn03.dakao.io:37098 of application
>> 'application_1600163418174_0127'.
>>
>> 2021-01-06 18:35:32,773 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
>> [] - Enforcing default ACL for ZK connections
>>
>> 2021-01-06 18:35:32,774 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
>> [] - Using '/driving-habits/default' as Zookeeper
>> namespace.
>> Anyway CLI needs to know where the leader (=active) JM is located via a
>> ZK node and GenericCLI has to be informed of high-availability.cluster-id.
>> Thanks for the heads up!
>>
>>
>> You could also specify the "high-availability.cluster-id" so that leader
>>> retrieval could get the correct JobManager address.
>>> *flink list --target yarn-per-job -Dyarn.application.id
>>> <http://dyarn.application.id/>=$application_id
>>> -Dhigh-availability.cluster-id=$application_id*
>>
>> Okay, it checked that it works. Thank you very much :-) It will be nice
>> for other users if your answer is also explained on [1].
>>
>>
>> And the following
>>> command should work with/without ZooKeeper HA configured.
>>> *./bin/flink list -m yarn-cluster -yid $applicationId*
>>
>> I'm very confused as there's different ways to specify YARN per-job
>> clusters:
>> - "--target yarn-per-job" is explained in the current documentation [1]
>> and it looks like the most recent one, so I'd rather use this one with
>> "-Dhigh-availability.cluster-id=$application_id"
>> - Is "--jobmanater yarn-per-job" a preferred way of specifying per-job
>> clusters and is it going to co-exist with "--target yarn-per-job" for the
>> future releases? but It looks old-school to me.
>> - There's also "--executor yarn-per-job" which seems to be deprecated
>> soon (explained in "flink help")
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/yarn.html#per-job-cluster-mode
>>
>> Best,
>>
>> Dongwon
>>
>> On Wed, Jan 6, 2021 at 12:33 PM Yang Wang  wrote:
>>
>>> Hi Dongwon,
>>>
>>> I think the root cause is that GenericCLI do not override the
>>> "high-availability.cluster-id" with specified application id.
>>> The GenericCLI is activated by "--target yarn-per-job". In
>>> the FlinkYarnSessionCli, we have done this. And the follow

Re: "flink list" fails when zk-based HA is enabled in flink-conf.yaml

[Dongwon Kim]

Hi Yang,

I was wondering why CLI accessed ZK because, as shown in the following
lines, CLI seemed to know the address of JM by contacting AHS before
connecting to ZK.

2021-01-06 18:35:32,351 INFO  org.apache.flink.client.cli.CliFrontend
[] - Running 'list' command.

2021-01-06 18:35:32,682 INFO  org.apache.hadoop.yarn.client.AHSProxy
[] - Connecting to Application History server at
mobdata-devflink-nm02.dakao.io/10.93.0.91:10200

2021-01-06 18:35:32,763 INFO  org.apache.flink.yarn.YarnClusterDescriptor
[] - Found Web Interface
mobdata-devflink-dn03.dakao.io:37098 of application
'application_1600163418174_0127'.

2021-01-06 18:35:32,773 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
  [] - Enforcing default ACL for ZK connections

2021-01-06 18:35:32,774 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
  [] - Using '/driving-habits/default' as Zookeeper namespace.
Anyway CLI needs to know where the leader (=active) JM is located via a ZK
node and GenericCLI has to be informed of high-availability.cluster-id.
Thanks for the heads up!


You could also specify the "high-availability.cluster-id" so that leader
> retrieval could get the correct JobManager address.
> *flink list --target yarn-per-job -Dyarn.application.id
> <http://dyarn.application.id/>=$application_id
> -Dhigh-availability.cluster-id=$application_id*

Okay, it checked that it works. Thank you very much :-) It will be nice for
other users if your answer is also explained on [1].


And the following
> command should work with/without ZooKeeper HA configured.
> *./bin/flink list -m yarn-cluster -yid $applicationId*

I'm very confused as there's different ways to specify YARN per-job
clusters:
- "--target yarn-per-job" is explained in the current documentation [1] and
it looks like the most recent one, so I'd rather use this one with
"-Dhigh-availability.cluster-id=$application_id"
- Is "--jobmanater yarn-per-job" a preferred way of specifying per-job
clusters and is it going to co-exist with "--target yarn-per-job" for the
future releases? but It looks old-school to me.
- There's also "--executor yarn-per-job" which seems to be deprecated soon
(explained in "flink help")

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/yarn.html#per-job-cluster-mode

Best,

Dongwon

On Wed, Jan 6, 2021 at 12:33 PM Yang Wang  wrote:

> Hi Dongwon,
>
> I think the root cause is that GenericCLI do not override the
> "high-availability.cluster-id" with specified application id.
> The GenericCLI is activated by "--target yarn-per-job". In
> the FlinkYarnSessionCli, we have done this. And the following
> command should work with/without ZooKeeper HA configured.
>
>
> *./bin/flink list -m yarn-cluster -yid $applicationId*
>
> You could also specify the "high-availability.cluster-id" so that leader
> retrieval could get the correct JobManager address.
>
>
> *flink list --target yarn-per-job -Dyarn.application.id
> <http://Dyarn.application.id>=$application_id
> -Dhigh-availability.cluster-id=$application_id*
>
> BTW, this is not a new introduced behavior change in Flink 1.12. I believe
> it also could not work in 1.11 and 1.10.
>
>
> Best,
> Yang
>
>
> Dongwon Kim  于2021年1月5日周二 下午11:22写道：
>
>> Hi,
>>
>> I'm using Flink-1.12.0 and running on Hadoop YARN.
>>
>> After setting HA-related properties in flink-conf.yaml,
>>
>> high-availability: zookeeper
>>
>> high-availability.zookeeper.path.root: /recovery
>>
>> high-availability.zookeeper.quorum: nm01:2181,nm02:2181,nm03:2181
>>
>> high-availability.storageDir: hdfs:///flink/recovery
>>
>> the following command hangs and fails:
>>
>> $ flink list --target yarn-per-job -Dyarn.application.id=$application_id
>>
>> Before setting the properties, I can see the following lines after
>> executing the above command:
>>
>> 2021-01-06 00:11:48,961 INFO  
>> org.apache.flink.runtime.security.modules.HadoopModule
>>   [] - Hadoop user set to deploy (auth:SIMPLE)
>>
>> 2021-01-06 00:11:48,968 INFO  
>> org.apache.flink.runtime.security.modules.JaasModule
>> [] - Jaas file will be created as
>> /tmp/jaas-8522045433029410483.conf.
>>
>> 2021-01-06 00:11:48,976 INFO  org.apache.flink.client.cli.CliFrontend
>>   [] - Running 'list' command.
>>
>> 2021-01-06 00:11:49,316 INFO  org.apache.hadoop.yarn.client.AHSProxy
>>   [] - Connecting to Application History server at nm02/
>> 10.93.0.91:10200
>>
>> 2021-01-06 00:11:49,324 INFO  org.apac

"flink list" fails when zk-based HA is enabled in flink-conf.yaml

[Dongwon Kim]

Hi,

I'm using Flink-1.12.0 and running on Hadoop YARN.

After setting HA-related properties in flink-conf.yaml,

high-availability: zookeeper

high-availability.zookeeper.path.root: /recovery

high-availability.zookeeper.quorum: nm01:2181,nm02:2181,nm03:2181

high-availability.storageDir: hdfs:///flink/recovery

the following command hangs and fails:

$ flink list --target yarn-per-job -Dyarn.application.id=$application_id

Before setting the properties, I can see the following lines after
executing the above command:

2021-01-06 00:11:48,961 INFO
org.apache.flink.runtime.security.modules.HadoopModule
  [] - Hadoop user set to deploy (auth:SIMPLE)

2021-01-06 00:11:48,968 INFO
org.apache.flink.runtime.security.modules.JaasModule
[] - Jaas file will be created as
/tmp/jaas-8522045433029410483.conf.

2021-01-06 00:11:48,976 INFO  org.apache.flink.client.cli.CliFrontend
[] - Running 'list' command.

2021-01-06 00:11:49,316 INFO  org.apache.hadoop.yarn.client.AHSProxy
[] - Connecting to Application History server at nm02/
10.93.0.91:10200

2021-01-06 00:11:49,324 INFO  org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location
of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

2021-01-06 00:11:49,333 WARN  org.apache.flink.yarn.YarnClusterDescriptor
[] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR
environment variable is set.The Flink YARN Client needs one of these to be
set to properly load the Hadoop configuration for accessing YARN.

2021-01-06 00:11:49,404 INFO  org.apache.flink.yarn.YarnClusterDescriptor
[] - Found Web Interface dn03:37098 of application
'application_1600163418174_0127'.

2021-01-06 00:11:49,758 INFO  org.apache.flink.client.cli.CliFrontend
[] - Waiting for response...

Waiting for response...

2021-01-06 00:11:49,863 INFO  org.apache.flink.client.cli.CliFrontend
[] - Successfully retrieved list of jobs

-- Running/Restarting Jobs ---

31.12.2020 01:22:34 : 76fc265c44ef44ae343ab15868155de6 : stream calculator
(RUNNING)

--

No scheduled jobs.

After:

2021-01-06 00:06:38,971 INFO
org.apache.flink.runtime.security.modules.HadoopModule
  [] - Hadoop user set to deploy (auth:SIMPLE)

2021-01-06 00:06:38,976 INFO
org.apache.flink.runtime.security.modules.JaasModule
[] - Jaas file will be created as
/tmp/jaas-3613274701724362777.conf.

2021-01-06 00:06:38,982 INFO  org.apache.flink.client.cli.CliFrontend
[] - Running 'list' command.

2021-01-06 00:06:39,304 INFO  org.apache.hadoop.yarn.client.AHSProxy
[] - Connecting to Application History server at nm02/
10.93.0.91:10200

2021-01-06 00:06:39,312 INFO  org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location
of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

2021-01-06 00:06:39,320 WARN  org.apache.flink.yarn.YarnClusterDescriptor
[] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR
environment variable is set.The Flink YARN Client needs one of these to be
set to properly load the Hadoop configuration for accessing YARN.

2021-01-06 00:06:39,388 INFO  org.apache.flink.yarn.YarnClusterDescriptor
[] - Found Web Interface dn03:37098 of application
'application_1600163418174_0127'.

2021-01-06 00:06:39,399 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
  [] - Enforcing default ACL for ZK connections

2021-01-06 00:06:39,399 INFO  org.apache.flink.runtime.util.ZooKeeperUtils
  [] - Using '/recovery/default' as Zookeeper namespace.

2021-01-06 00:06:39,425 INFO
org.apache.flink.shaded.curator4.org.apache.curator.utils.Compatibility
[] - Running in ZooKeeper 3.4.x compatibility mode

2021-01-06 00:06:39,425 INFO
org.apache.flink.shaded.curator4.org.apache.curator.utils.Compatibility
[] - Using emulated InjectSessionExpiration

2021-01-06 00:06:39,447 INFO
org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl
[] - Starting

2021-01-06 00:06:39,455 INFO
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper
[] - Initiating client connection, connectString=nm01:2181,

nm02:2181,nm03:2181 sessionTimeout=6
watcher=org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState@7668d560

2021-01-06 00:06:39,466 INFO
org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl
[] - Default schema

2021-01-06 00:06:39,466 WARN
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn
[] - SASL configuration failed: javax.security.auth.login.LoginException:
No JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-3613274701724362777.conf'. Will continue

Fwd: How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

[Dongwon Kim]

Any advice would be appreciated :)

Thanks,

Dongwon

-- Forwarded message -
From: Dongwon Kim 
Date: Mon, Dec 14, 2020 at 11:27 PM
Subject: How to gracefully avoid "Generic types have been disabled in the
ExecutionConfig and type java.util.List is treated as a generic type"?
To: user 


Hi,

The following program compiles and runs w/o exceptions:

> public class Test {
>
>   public static class A {
> private int n;
>
> public A() { }
> public int getN() {  return n;  }
> public void setN(int n) {  this.n = n;  }
>   }
>
>   public static class B {
> private List lst;
>
> public B() { }
> public List getLst() {  return lst;  }
> public void setLst(List lst) {  this.lst = lst;  }
>   }
>
>   public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment();
>
> env.fromElements(new B())
>   .print();
>
> env.execute();
>   }
> }
>

When I add the following line,

> env.getConfig().disableGenericTypes();

then the program shows me an exception:

> Exception in thread "main" java.lang.UnsupportedOperationException:
> Generic types have been disabled in the ExecutionConfig and type
> java.util.List is treated as a generic type.
> at
> org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:86)
> at
> org.apache.flink.api.java.typeutils.PojoTypeInfo.createPojoSerializer(PojoTypeInfo.java:319)
> at
> org.apache.flink.api.java.typeutils.PojoTypeInfo.createSerializer(PojoTypeInfo.java:311)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:970)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromElements(StreamExecutionEnvironment.java:871)
> at Test.main(Test.java:29)


To avoid this exception, I found that I have to declare a type factory like:

>   public static class BTypeFactory extends TypeInfoFactory {
> @Override
> public TypeInformation createTypeInfo(Type t, Map TypeInformation> genericParameters) {
>   return Types.POJO(
> B.class,
> ImmutableMap.>builder()
>   .put("lst", Types.LIST(Types.POJO(A.class)))
> .build()
>   );
> }
>   }

and give it to class B as follows:

>   @TypeInfo(BTypeFactory.class)
>   public static class B {


Is there no other way but to declare BTypeFactory in such cases?
I don't like the way I have to type a field name twice, one for a member
variable and the other for an Map entry in TypeInfoFactory.

Thanks in advance,

Dongwon

Re: failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

[Dongwon Kim]

core.AbstractRedisClient.closeClientResources(AbstractRedisClient.java:569)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at
> io.lettuce.core.AbstractRedisClient.lambda$shutdownAsync$5(AbstractRedisClient.java:521)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at
> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
> ~[?:1.8.0_222]
> at
> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
> ~[?:1.8.0_222]
> at
> io.lettuce.core.AbstractRedisClient.shutdownAsync(AbstractRedisClient.java:521)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at
> io.lettuce.core.AbstractRedisClient.shutdown(AbstractRedisClient.java:485)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at
> io.lettuce.core.AbstractRedisClient.shutdown(AbstractRedisClient.java:453)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at com.kakaomobility.drivinghabit.stream.Enricher.close(Enricher.java:60)
> ~[blob_p-ad05abf47895f79601474d976f6b3a3d57c22c87-cb151990c181788b52fc860c990e0c96:?]
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:740)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:720)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:643)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:552)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-dist_2.11-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
>

The exception seems to come from another operator, not Kafka, and this
operator performs async io using Lettuce, an async Redis client API.

Best,

Dongwon

On Wed, Dec 16, 2020 at 6:07 PM Yang Wang  wrote:

> Hi Dongwon,
>
> For application mode, the job submission happens in the JobManager side.
> We are using an embedded client
> to submit the job. So the user jar will be added to distributed cache.
> When deploying a task to TaskManager,
> it will be downloaded again and run in user classloader even though we
> already have it in the system classpath.
>
> I think it might be the reason why these classes are loaded by different
> classloaders.
>
> For per-job mode, we are recovering the job and the user jars will not be
> added to distributed cache.
>
> Could you add "-Dyarn.per-job-cluster.include-user-jar=DISABLED" to your
> command and have a try? After that, we
> will disable the user jars including in the system classpath.
>
>
> Best,
> Yang
>
>
>
> Dongwon Kim  于2020年12月16日周三 下午4:20写道：
>
>> Robert,
>>
>> But if Kafka is really only available in the user jar, then this error
>>> still should not occur.
>>
>> I think so too; it should not occur.
>> I scan through all the jar files in the classpath using `jar tf` but no
>> jar contains org.apache.kafka.common.serialization.Deserializer with a
>> different version.
>>
>> In your case it seems that the classes are loaded from different
>>> classloaders.
>>
>> Hmm, why did the artifact work fine with per-job cluster mode?
>>
>> p.s. Another user seems to face the same problem:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplication-error-on-Kafka-Connector-Libraries-td39805.html#a39812
>>
>> Thanks,
>>
>> Dongwon
>>
>>
>>
>> On Wed, Dec 16, 2020 at 4:45 PM Robert Metzger 
>> wrote:
>>
>>> Hey Dongwon,
>>>
>>> I don't think this is the intended behavior.
>>> I believe in application mode, we are adding the user jar into the
>>> system classloader as well. In your case it seems that the classes are
>>> loaded from different classloaders.
>>> But if Kafka is really only available in the user jar, then this error
>>> still should not occur.
>>>
>>>
>&

Re: failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

[Dongwon Kim]

Robert,

But if Kafka is really only available in the user jar, then this error
> still should not occur.

I think so too; it should not occur.
I scan through all the jar files in the classpath using `jar tf` but no jar
contains org.apache.kafka.common.serialization.Deserializer with a
different version.

In your case it seems that the classes are loaded from different
> classloaders.

Hmm, why did the artifact work fine with per-job cluster mode?

p.s. Another user seems to face the same problem:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplication-error-on-Kafka-Connector-Libraries-td39805.html#a39812

Thanks,

Dongwon



On Wed, Dec 16, 2020 at 4:45 PM Robert Metzger  wrote:

> Hey Dongwon,
>
> I don't think this is the intended behavior.
> I believe in application mode, we are adding the user jar into the system
> classloader as well. In your case it seems that the classes are loaded from
> different classloaders.
> But if Kafka is really only available in the user jar, then this error
> still should not occur.
>
>
> On Wed, Dec 16, 2020 at 8:22 AM Dongwon Kim  wrote:
>
>> I just added the following option to the script:
>>
>>
>> -Dclassloader.parent-first-patterns.additional=org.apache.kafka.common.serialization
>>
>> Now it seems to work.
>>
>> Why do the application mode and the per-job cluster mode behave
>> differently when it comes to the classloading?
>>
>> Is it a bug? or intended?
>>
>> Best,
>>
>> Dongwon
>>
>> On Wed, Dec 16, 2020 at 3:59 PM Dongwon Kim 
>> wrote:
>>
>>> Hi,
>>>
>>> I have an artifact which works perfectly fine with Per-Job Cluster Mode
>>> with the following bash script:
>>>
>>> #!/bin/env bash
>>>
>>> export FLINK_CONF_DIR=./conf
>>>
>>> export HADOOP_CLASSPATH=`hadoop classpath`
>>>
>>>
>>> $FLINK_HOME/bin/flink run -t yarn-per-job myjar.jar myconf.conf
>>>
>>> I tried Application Mode [1] using the exact same artifact with the
>>> following script:
>>>
>>> #!/bin/env bash
>>>
>>>
>>> export FLINK_CONF_DIR=./conf
>>>
>>> export HADOOP_CLASSPATH=`hadoop classpath`
>>>
>>>
>>> $FLINK_HOME/bin/flink run-application -t yarn-application \
>>>
>>> 
>>> -Dyarn.provided.lib.dirs='hdfs:///flink-dists/flink-1.12.0/lib;hdfs:///flink-dists/flink-1.12.0/plugins'
>>> \
>>>
>>> -Dyarn.ship-files=myconf.conf \
>>>
>>> hdfs:///jars/myjar.jar myconf.conf
>>>
>>> but the job fails with the following exception
>>>
>>> 2020-12-16 15:52:25,364 WARN  org.apache.flink.runtime.taskmanager.Task
>>>   [] - session-window -> (Sink: kafka-sink, Sink:
>>> session-window-late-data) (1/1)#0 (ee9fc1aa21833c749e3c271fd52cbfd4)
>>> switched from RUNNING to FAILED.
>>>
>>> org.apache.kafka.common.KafkaException: Failed to construct kafka
>>> producer
>>>
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:432)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.internals.FlinkKafkaInternalProducer.(FlinkKafkaInternalProducer.java:78)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.createProducer(FlinkKafkaProducer.java:1158)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initProducer(FlinkKafkaProducer.java:1259)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initNonTransactionalProducer(FlinkKafkaProducer.java:1255)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:950)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:100)
>>> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>>>
>>> at
>&g

Re: failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

[Dongwon Kim]

I just added the following option to the script:

-Dclassloader.parent-first-patterns.additional=org.apache.kafka.common.serialization

Now it seems to work.

Why do the application mode and the per-job cluster mode behave differently
when it comes to the classloading?

Is it a bug? or intended?

Best,

Dongwon

On Wed, Dec 16, 2020 at 3:59 PM Dongwon Kim  wrote:

> Hi,
>
> I have an artifact which works perfectly fine with Per-Job Cluster Mode
> with the following bash script:
>
> #!/bin/env bash
>
> export FLINK_CONF_DIR=./conf
>
> export HADOOP_CLASSPATH=`hadoop classpath`
>
>
> $FLINK_HOME/bin/flink run -t yarn-per-job myjar.jar myconf.conf
>
> I tried Application Mode [1] using the exact same artifact with the
> following script:
>
> #!/bin/env bash
>
>
> export FLINK_CONF_DIR=./conf
>
> export HADOOP_CLASSPATH=`hadoop classpath`
>
>
> $FLINK_HOME/bin/flink run-application -t yarn-application \
>
> 
> -Dyarn.provided.lib.dirs='hdfs:///flink-dists/flink-1.12.0/lib;hdfs:///flink-dists/flink-1.12.0/plugins'
> \
>
> -Dyarn.ship-files=myconf.conf \
>
> hdfs:///jars/myjar.jar myconf.conf
>
> but the job fails with the following exception
>
> 2020-12-16 15:52:25,364 WARN  org.apache.flink.runtime.taskmanager.Task
>   [] - session-window -> (Sink: kafka-sink, Sink:
> session-window-late-data) (1/1)#0 (ee9fc1aa21833c749e3c271fd52cbfd4)
> switched from RUNNING to FAILED.
>
> org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:432)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.internals.FlinkKafkaInternalProducer.(FlinkKafkaInternalProducer.java:78)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.createProducer(FlinkKafkaProducer.java:1158)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initProducer(FlinkKafkaProducer.java:1259)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initNonTransactionalProducer(FlinkKafkaProducer.java:1255)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:950)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:100)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:398)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:389)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1128)
> ~[stream-calculator-0.1-SNAPSHOT.jar:?]
>
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:107)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:264)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:400)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:507)
> ~[flink-dist_2.11-1.12.0.jar:1.12.0]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runT

failed w/ Application Mode but succeeded w/ Per-Job Cluster Mode

[Dongwon Kim]

Hi,

I have an artifact which works perfectly fine with Per-Job Cluster Mode
with the following bash script:

#!/bin/env bash

export FLINK_CONF_DIR=./conf

export HADOOP_CLASSPATH=`hadoop classpath`


$FLINK_HOME/bin/flink run -t yarn-per-job myjar.jar myconf.conf

I tried Application Mode [1] using the exact same artifact with the
following script:

#!/bin/env bash


export FLINK_CONF_DIR=./conf

export HADOOP_CLASSPATH=`hadoop classpath`


$FLINK_HOME/bin/flink run-application -t yarn-application \


-Dyarn.provided.lib.dirs='hdfs:///flink-dists/flink-1.12.0/lib;hdfs:///flink-dists/flink-1.12.0/plugins'
\

-Dyarn.ship-files=myconf.conf \

hdfs:///jars/myjar.jar myconf.conf

but the job fails with the following exception

2020-12-16 15:52:25,364 WARN  org.apache.flink.runtime.taskmanager.Task
[] - session-window -> (Sink: kafka-sink, Sink:
session-window-late-data) (1/1)#0 (ee9fc1aa21833c749e3c271fd52cbfd4)
switched from RUNNING to FAILED.

org.apache.kafka.common.KafkaException: Failed to construct kafka producer

at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:432)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.internals.FlinkKafkaInternalProducer.(FlinkKafkaInternalProducer.java:78)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.createProducer(FlinkKafkaProducer.java:1158)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initProducer(FlinkKafkaProducer.java:1259)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initNonTransactionalProducer(FlinkKafkaProducer.java:1255)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:950)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:100)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:398)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:389)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1128)
~[stream-calculator-0.1-SNAPSHOT.jar:?]

at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:107)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:264)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:400)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:507)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
~[flink-dist_2.11-1.12.0.jar:1.12.0]

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
[flink-dist_2.11-1.12.0.jar:1.12.0]

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
[flink-dist_2.11-1.12.0.jar:1.12.0]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

Caused by: org.apache.kafka.common.KafkaException: class
org.apache.kafka.common.serialization.ByteArraySerializer is not an
instance of org.apache.kafka.common.serialization.Serializer

at
org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:374)

How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

[Dongwon Kim]

Hi,

The following program compiles and runs w/o exceptions:

> public class Test {
>
>   public static class A {
> private int n;
>
> public A() { }
> public int getN() {  return n;  }
> public void setN(int n) {  this.n = n;  }
>   }
>
>   public static class B {
> private List lst;
>
> public B() { }
> public List getLst() {  return lst;  }
> public void setLst(List lst) {  this.lst = lst;  }
>   }
>
>   public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment();
>
> env.fromElements(new B())
>   .print();
>
> env.execute();
>   }
> }
>

When I add the following line,

> env.getConfig().disableGenericTypes();

then the program shows me an exception:

> Exception in thread "main" java.lang.UnsupportedOperationException:
> Generic types have been disabled in the ExecutionConfig and type
> java.util.List is treated as a generic type.
> at
> org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:86)
> at
> org.apache.flink.api.java.typeutils.PojoTypeInfo.createPojoSerializer(PojoTypeInfo.java:319)
> at
> org.apache.flink.api.java.typeutils.PojoTypeInfo.createSerializer(PojoTypeInfo.java:311)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:970)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromElements(StreamExecutionEnvironment.java:871)
> at Test.main(Test.java:29)


To avoid this exception, I found that I have to declare a type factory like:

>   public static class BTypeFactory extends TypeInfoFactory {
> @Override
> public TypeInformation createTypeInfo(Type t, Map TypeInformation> genericParameters) {
>   return Types.POJO(
> B.class,
> ImmutableMap.>builder()
>   .put("lst", Types.LIST(Types.POJO(A.class)))
> .build()
>   );
> }
>   }

and give it to class B as follows:

>   @TypeInfo(BTypeFactory.class)
>   public static class B {


Is there no other way but to declare BTypeFactory in such cases?
I don't like the way I have to type a field name twice, one for a member
variable and the other for an Map entry in TypeInfoFactory.

Thanks in advance,

Dongwon

Re: What happens when all input partitions become idle

[Dongwon Kim]

Hi Benchao,

Thanks for the input.

The code is self-explanatory.

Best,

Dongwon


On Thu, Dec 10, 2020 at 12:20 PM Benchao Li  wrote:

> Hi Dongwon,
>
> I think you understand it correctly.
> You can find this logic here[1]
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/streamstatus/StatusWatermarkValve.java#L108
>
> Dongwon Kim  于2020年12月10日周四 上午12:21写道：
>
>> Hi,
>>
>> Let's consider two operators: A (parallelism=2) and B (parallelism=1).
>> B has two input partitions, B_A1 and B_A2, which are connected to A1 and
>> A2 respectively.
>>
>> At some point,
>> - B_A1's watermark : 12
>> - B_A2's watermark : 10
>> - B's event-time clock : 10 = min(12, 10)
>> - B has registered a timer at 12
>> - No data will be fed into the pipeline for the next few hours, but I
>> want the timer to be fired after a few seconds if no data is coming.
>>
>> After adopting a watermark strategy explained in [1], I found that the
>> timer is fired as wished! That's awesome!
>>
>> But I want to know what happens inside in detail.
>> Based on my current understanding of how watermark is calculated [2], I
>> cannot come up with what happens inside when idleness is considered.
>> If B_A2 is marked idle earlier than B_A1, is B's event-time clock
>> calculated as min(12, MAX_WATERMARK)?
>>
>> Thanks,
>>
>> Dongwon
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html#dealing-with-idle-sources
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html#how-operators-process-watermarks
>> [3]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners
>>
>
>
> --
>
> Best,
> Benchao Li
>

What happens when all input partitions become idle

[Dongwon Kim]

Hi,

Let's consider two operators: A (parallelism=2) and B (parallelism=1).
B has two input partitions, B_A1 and B_A2, which are connected to A1 and A2
respectively.

At some point,
- B_A1's watermark : 12
- B_A2's watermark : 10
- B's event-time clock : 10 = min(12, 10)
- B has registered a timer at 12
- No data will be fed into the pipeline for the next few hours, but I want
the timer to be fired after a few seconds if no data is coming.

After adopting a watermark strategy explained in [1], I found that the
timer is fired as wished! That's awesome!

But I want to know what happens inside in detail.
Based on my current understanding of how watermark is calculated [2], I
cannot come up with what happens inside when idleness is considered.
If B_A2 is marked idle earlier than B_A1, is B's event-time clock
calculated as min(12, MAX_WATERMARK)?

Thanks,

Dongwon

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html#dealing-with-idle-sources
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html#how-operators-process-watermarks
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners

Re: queryLogicalType != sinkLogicalType when UDAF returns List

[Dongwon Kim]

Hi Timo,

Okay, then the aggregate function should look like this:

>  public static class Agg extends AggregateFunction ArrayList> {
> @Override
> public ArrayList createAccumulator() {
>   return new ArrayList<>();
> }
> @Override
> public Integer[] getValue(ArrayList acc) {
>   return acc.toArray(new Integer[0]);
> }
> public void accumulate(ArrayList acc, int i) {
>   acc.add(i);
> }
> @Override
> public TypeInformation getResultType() {
>   return OBJECT_ARRAY(Types.INT);
> }
>   }


Now the program outputs:

> 2> +I([1, 2])


Thanks,

Dongwon

On Fri, Nov 27, 2020 at 5:38 PM Timo Walther  wrote:

> Hi,
>
> first of all we don't support ListTypeInfo in Table API. Therefore, it
> is treated as a RAW type. The exception during exception creation is a
> bug that should be fixed in future version. But the mismatch is valid:
>
> ARRAY is not a list type info but `Types.OBJECT_ARRAY(Types.INT)`.
> Can you try this as the result type of your aggregate function.
>
> Reagrds,
> Timo
>
>
> On 26.11.20 18:13, Dongwon Kim wrote:
> > Hello,
> >
> > I'm using Flink-1.11.2.
> >
> > Let's assume that I want to store on a table the result of the following
> > UDAF:
> >
> >public class Agg extends AggregateFunction,
> > List> {
> >  @Override
> >  public List createAccumulator() {
> >return new LinkedList<>();
> >  }
> >  @Override
> >  public List getValue(List acc) {
> >return acc;
> >  }
> >  public void accumulate(List acc, int i) {
> >acc.add(i);
> >  }
> >  @Override
> >  public TypeInformation> getResultType() {
> >return new ListTypeInfo<>(Integer.class);
> >  }
> >}
> >
> >
> > The main program looks as follow:
> >
> > public class TestMain {
> >public static void main(String[] args) {
> >  EnvironmentSettings settings = EnvironmentSettings.newInstance()
> >.inBatchMode()
> >.build();
> >  TableEnvironment tEnv = TableEnvironment.create(settings);
> >  tEnv.executeSql(
> >"CREATE TEMPORARY FUNCTION agg AS '" + Agg.class.getName() +
> "'"
> >  );
> >  Table t = tEnv.sqlQuery(
> >"SELECT agg(c2)\n" +
> >  "FROM (VALUES (ROW('a',1)), (ROW('a',2))) AS T(c1,c2)\n" +
> >  "GROUP BY c1"
> >  );
> >  tEnv.executeSql(
> >"CREATE TABLE output (a ARRAY) WITH ('connector' =
> 'print')"
> >  );
> >  /**
> >   * root
> >   *  |-- EXPR$0: RAW('java.util.List', ?)
> >   */
> >  t.printSchema();
> >  t.executeInsert("output" );
> >}
> > }
> >
> >
> > This program fails with the following exception:
> >
> > Exception in thread "main"
> > org.apache.flink.table.api.TableException: A raw type backed by type
> > information has no serializable string representation. It needs to
> > be resolved into a proper raw type.
> > at
> >
>  
> org.apache.flink.table.types.logical.TypeInformationRawType.asSerializableString(TypeInformationRawType.java:101)
> > at
> >
>  
> org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$2.apply(TableSinkUtils.scala:92)
> > at
> >
>  
> org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$2.apply(TableSinkUtils.scala:92)
> > at
> >
>  
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> > at
> >
>  
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> > at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> > at
> >
>  
> org.apache.flink.table.planner.sinks.TableSinkUtils$.validateSchemaAndApplyImplicitCast(TableSinkUtil

Re: Caching

[Dongwon Kim]

Hi Navneeth,

I didn't quite understand how async io can be used here. It would be great
> if you can share some info on it.

You need to add an async operator in the middle of your pipeline in order
to enrich your input data. [1] and [2] will help you.

Also how are you propagating any changes to values?

I need to maintain the mapping of road ID to various attributes of each
road, and the mapping is updated every week.
I use keys for versioning and I use Hash [3] for value in order to store a
mapping.
When a new mapping is prepared I'm uploading it using a fresh key while the
previous version is being served to Flink (via async io).
Such concurrent read/write is possible in Redis when you turn off
transaction when creating Redis client's pipeline.
When the new mapping is completely uploaded, I inform my Flink pipeline of
the new mapping via Kafka.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
[2] https://www.youtube.com/watch?v=UParyxe-2Wc
[3] https://redis.io/topics/data-types#hashes
[4] https://github.com/andymccurdy/redis-py#pipelines

Best,

Dongwon

On Fri, Nov 27, 2020 at 4:31 PM Navneeth Krishnan 
wrote:

> Thanks Dongwon. It was extremely helpful. I didn't quite understand how
> async io can be used here. It would be great if you can share some info on
> it.
>
> Also how are you propagating any changes to values?
>
> Regards,
> Navneeth
>
> On Thu, Nov 26, 2020 at 6:26 AM Dongwon Kim  wrote:
>
>> Oops, I forgot to mention that when doing bulk insert into Redis, you'd
>> better open a pipeline with a 'transaction' property set to False [1].
>>
>> Otherwise, API calls from your Flink job will be timeout.
>>
>> [1] https://github.com/andymccurdy/redis-py#pipelines
>>
>> On Thu, Nov 26, 2020 at 11:09 PM Dongwon Kim 
>> wrote:
>>
>>> Hi Navneeth,
>>>
>>> I reported a similar issue to yours before [1] but I took the
>>> broadcasting approach at first.
>>>
>>> As you already anticipated, broadcasting is going to use more memory
>>> than your current approach based on a static object on each TM .
>>>
>>> And the broadcasted data will be treated as operator state and will be
>>> periodically checkpointed with serialization overhead & garbage collections.
>>> These are not negligible at all if you're not carefully choosing
>>> serialization strategy as explained in [2].
>>> Even with the proper one, I've experienced mild back pressure whenever
>>> - checkpoint is in progress (AFAIK, incremental checkpoint has nothing
>>> to do with operator states)
>>> - cache is being broadcasted
>>>
>>> For that reason, I decided to populate data on Redis but it also calls
>>> for design decisions:
>>> - which Java client to use? Jedis [3]? Lettuce [4]?
>>> - how to invoke APIs calls inside Flink? synchronously or asynchronously?
>>>
>>> Currently I'm very satisfied with Lettuce with Flink's async io [5] with
>>> very small memory footprint and without worrying about serialization
>>> overhead and garbage collections.
>>> Lettuce supports asynchronous communication so it works perfectly with
>>> Flink's async io.
>>> I bet you'll be very disappointed with invoking Jedis synchronously
>>> inside ProcessFunction.
>>>
>>> Best,
>>>
>>> Dongwon
>>>
>>> [1]
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Better-way-to-share-large-data-across-task-managers-td38231.html
>>> [2]
>>> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
>>> [3] https://github.com/redis/jedis
>>> [4] https://lettuce.io/
>>> [5]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>>>
>>> On Thu, Nov 26, 2020 at 5:31 PM Navneeth Krishnan <
>>> reachnavnee...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> We have a flink streaming job processing around 200k events per second.
>>>> The job requires a lot of less frequently changing data (sort of static but
>>>> there will be some changes over time, say 5% change once per day or so).
>>>> There are about 12 caches with some containing approximately 20k
>>>> entries whereas a few with about 2 million entries.
>>>>
>>>> In the current implementation we are using in-memory lazy loading
>>>> static cache to populate the data and the initialization happens in open
>>>> function. The reason to choose this approach is because we

queryLogicalType != sinkLogicalType when UDAF returns List

[Dongwon Kim]

Hello,

I'm using Flink-1.11.2.

Let's assume that I want to store on a table the result of the following
UDAF:

>   public class Agg extends AggregateFunction, List>
> {
> @Override
> public List createAccumulator() {
>   return new LinkedList<>();
> }
> @Override
> public List getValue(List acc) {
>   return acc;
> }
> public void accumulate(List acc, int i) {
>   acc.add(i);
> }
> @Override
> public TypeInformation> getResultType() {
>   return new ListTypeInfo<>(Integer.class);
> }
>   }


The main program looks as follow:

> public class TestMain {
>   public static void main(String[] args) {
> EnvironmentSettings settings = EnvironmentSettings.newInstance()
>   .inBatchMode()
>   .build();
> TableEnvironment tEnv = TableEnvironment.create(settings);
> tEnv.executeSql(
>   "CREATE TEMPORARY FUNCTION agg AS '" + Agg.class.getName() + "'"
> );
> Table t = tEnv.sqlQuery(
>   "SELECT agg(c2)\n" +
> "FROM (VALUES (ROW('a',1)), (ROW('a',2))) AS T(c1,c2)\n" +
> "GROUP BY c1"
> );
> tEnv.executeSql(
>   "CREATE TABLE output (a ARRAY) WITH ('connector' = 'print')"
> );
> /**
>  * root
>  *  |-- EXPR$0: RAW('java.util.List', ?)
>  */
> t.printSchema();
> t.executeInsert("output" );
>   }
> }


This program fails with the following exception:

> Exception in thread "main" org.apache.flink.table.api.TableException: A
> raw type backed by type information has no serializable string
> representation. It needs to be resolved into a proper raw type.
> at
> org.apache.flink.table.types.logical.TypeInformationRawType.asSerializableString(TypeInformationRawType.java:101)
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$2.apply(TableSinkUtils.scala:92)
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$2.apply(TableSinkUtils.scala:92)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$.validateSchemaAndApplyImplicitCast(TableSinkUtils.scala:92)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:229)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:204)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:204)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:163)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:163)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:163)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1264)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:700)
> at
> org.apache.flink.table.api.internal.TableImpl.executeInsert(TableImpl.java:565)
> at
> org.apache.flink.table.api.internal.TableImpl.executeInsert(TableImpl.java:549)
> at my.TestMain.main(TestMain.java:62)


I found that two types do not match:
- queryLogicalType : ROW<`EXPR$0` RAW('java.util.List', ?)>
- sinkLogicalType : ROW<`a` ARRAY>

Why does the queryLogicalType contain 'RAW' instead of 'ARRAY'?
Is there no way for UDAF to return java.lang.List and store it as ARRAY?

Thanks in advance,

Dongwon

Re: Caching

[Dongwon Kim]

Oops, I forgot to mention that when doing bulk insert into Redis, you'd
better open a pipeline with a 'transaction' property set to False [1].

Otherwise, API calls from your Flink job will be timeout.

[1] https://github.com/andymccurdy/redis-py#pipelines

On Thu, Nov 26, 2020 at 11:09 PM Dongwon Kim  wrote:

> Hi Navneeth,
>
> I reported a similar issue to yours before [1] but I took the broadcasting
> approach at first.
>
> As you already anticipated, broadcasting is going to use more memory than
> your current approach based on a static object on each TM .
>
> And the broadcasted data will be treated as operator state and will be
> periodically checkpointed with serialization overhead & garbage collections.
> These are not negligible at all if you're not carefully choosing
> serialization strategy as explained in [2].
> Even with the proper one, I've experienced mild back pressure whenever
> - checkpoint is in progress (AFAIK, incremental checkpoint has nothing to
> do with operator states)
> - cache is being broadcasted
>
> For that reason, I decided to populate data on Redis but it also calls for
> design decisions:
> - which Java client to use? Jedis [3]? Lettuce [4]?
> - how to invoke APIs calls inside Flink? synchronously or asynchronously?
>
> Currently I'm very satisfied with Lettuce with Flink's async io [5] with
> very small memory footprint and without worrying about serialization
> overhead and garbage collections.
> Lettuce supports asynchronous communication so it works perfectly with
> Flink's async io.
> I bet you'll be very disappointed with invoking Jedis synchronously inside
> ProcessFunction.
>
> Best,
>
> Dongwon
>
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Better-way-to-share-large-data-across-task-managers-td38231.html
> [2]
> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
> [3] https://github.com/redis/jedis
> [4] https://lettuce.io/
> [5]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html
>
> On Thu, Nov 26, 2020 at 5:31 PM Navneeth Krishnan <
> reachnavnee...@gmail.com> wrote:
>
>> Hi All,
>>
>> We have a flink streaming job processing around 200k events per second.
>> The job requires a lot of less frequently changing data (sort of static but
>> there will be some changes over time, say 5% change once per day or so).
>> There are about 12 caches with some containing approximately 20k
>> entries whereas a few with about 2 million entries.
>>
>> In the current implementation we are using in-memory lazy loading static
>> cache to populate the data and the initialization happens in open function.
>> The reason to choose this approach is because we have allocated around 4GB
>> extra memory per TM for these caches and if a TM has 6 slots the cache can
>> be shared.
>>
>> Now the issue we have with this approach is everytime when a container is
>> restarted or a new job is deployed it has to populate the cache again.
>> Sometimes this lazy loading takes a while and it causes back pressure as
>> well. We were thinking to move this logic to the broadcast stream but since
>> the data has to be stored per slot it would increase the memory consumption
>> by a lot.
>>
>> Another option that we were thinking of is to replace the current near
>> far cache that uses rest api to load the data to redis based near far
>> cache. This will definitely reduce the overall loading time but still not
>> the perfect solution.
>>
>> Are there any recommendations on how this can be achieved effectively?
>> Also how is everyone overcoming this problem?
>>
>> Thanks,
>> Navneeth
>>
>>

Re: Caching

[Dongwon Kim]

Hi Navneeth,

I reported a similar issue to yours before [1] but I took the broadcasting
approach at first.

As you already anticipated, broadcasting is going to use more memory than
your current approach based on a static object on each TM .

And the broadcasted data will be treated as operator state and will be
periodically checkpointed with serialization overhead & garbage collections.
These are not negligible at all if you're not carefully choosing
serialization strategy as explained in [2].
Even with the proper one, I've experienced mild back pressure whenever
- checkpoint is in progress (AFAIK, incremental checkpoint has nothing to
do with operator states)
- cache is being broadcasted

For that reason, I decided to populate data on Redis but it also calls for
design decisions:
- which Java client to use? Jedis [3]? Lettuce [4]?
- how to invoke APIs calls inside Flink? synchronously or asynchronously?

Currently I'm very satisfied with Lettuce with Flink's async io [5] with
very small memory footprint and without worrying about serialization
overhead and garbage collections.
Lettuce supports asynchronous communication so it works perfectly with
Flink's async io.
I bet you'll be very disappointed with invoking Jedis synchronously inside
ProcessFunction.

Best,

Dongwon

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Better-way-to-share-large-data-across-task-managers-td38231.html
[2]
https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html
[3] https://github.com/redis/jedis
[4] https://lettuce.io/
[5]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html

On Thu, Nov 26, 2020 at 5:31 PM Navneeth Krishnan 
wrote:

> Hi All,
>
> We have a flink streaming job processing around 200k events per second.
> The job requires a lot of less frequently changing data (sort of static but
> there will be some changes over time, say 5% change once per day or so).
> There are about 12 caches with some containing approximately 20k
> entries whereas a few with about 2 million entries.
>
> In the current implementation we are using in-memory lazy loading static
> cache to populate the data and the initialization happens in open function.
> The reason to choose this approach is because we have allocated around 4GB
> extra memory per TM for these caches and if a TM has 6 slots the cache can
> be shared.
>
> Now the issue we have with this approach is everytime when a container is
> restarted or a new job is deployed it has to populate the cache again.
> Sometimes this lazy loading takes a while and it causes back pressure as
> well. We were thinking to move this logic to the broadcast stream but since
> the data has to be stored per slot it would increase the memory consumption
> by a lot.
>
> Another option that we were thinking of is to replace the current near far
> cache that uses rest api to load the data to redis based near far cache.
> This will definitely reduce the overall loading time but still not the
> perfect solution.
>
> Are there any recommendations on how this can be achieved effectively?
> Also how is everyone overcoming this problem?
>
> Thanks,
> Navneeth
>
>

Re: execution.runtime-mode=BATCH when reading from Hive

[Dongwon Kim]

Hi Aljoscha,

Unfortunately, it's not that easy right now because normal Sinks that
> rely on checkpointing to write out data, such as Kafka, don't work in
> BATCH execution mode because we don't have checkpoints there. It will
> work, however, if you use a source that doesn't rely on checkpointing it
> will work. The FlinkKafkaProducer with Semantic.NONE should work, for
> example.

As the output produced to Kafka is eventually stored on Cassandra, I might
use a different sink to write output directly to Cassandra for BATCH
execution.
In such a case, I have to replace both (A) source and (E) sink.

There is HiveSource, which is built on the new Source API that will work
> well with both BATCH and STREAMING. It's quite new and it was only added
> to be used by a Table/SQL connector but you might have some success with
> that one.

Oh, this one is a new one which will be introduced in the upcoming 1.12
release.
How I've missed this one.
I'm going to give it a try :-)

BTW, thanks a lot for the input and the nice presentation - it's very
helpful and insightful.

Best,

Dongwon

On Wed, Nov 18, 2020 at 9:44 PM Aljoscha Krettek 
wrote:

> Hi Dongwon,
>
> Unfortunately, it's not that easy right now because normal Sinks that
> rely on checkpointing to write out data, such as Kafka, don't work in
> BATCH execution mode because we don't have checkopoints there. It will
> work, however, if you use a source that doesn't rely on checkpointing it
> will work. The FlinkKafkaProducer with Semantic.NONE should work, for
> example.
>
> There is HiveSource, which is built on the new Source API that will work
> well with both BATCH and STREAMING. It's quite new and it was only added
> to be used by a Table/SQL connector but you might have some success with
> that one.
>
> Best,
> Aljoscha
>
> On 18.11.20 07:03, Dongwon Kim wrote:
> > Hi,
> >
> > Recently I've been working on a real-time data stream processing pipeline
> > with DataStream API while preparing for a new service to launch.
> > Now it's time to develop a back-fill job to produce the same result by
> > reading data stored on Hive which we use for long-term storage.
> >
> > Meanwhile, I watched Aljoscha's talk [1] and just wondered if I could
> reuse
> > major components of the pipeline written in DataStream API.
> > The pipeline conceptually looks as follows:
> > (A) reads input from Kafka
> > (B) performs AsyncIO to Redis in order to enrich the input data
> > (C) appends timestamps and emits watermarks before time-based window
> > (D) keyBy followed by a session window with a custom trigger for early
> > firing
> > (E) writes output to Kafka
> >
> > I have simple (maybe stupid) questions on reusing components of the
> > pipeline written in DataStream API.
> > (1) By replacing (A) with a bounded source, can I execute the pipeline
> with
> > a new BATCH execution mode without modifying (B)~(E)?
> > (2) Is there a bounded source for Hive available for DataStream API?
> >
> > Best,
> >
> > Dongwon
> >
> > [1] https://www.youtube.com/watch?v=z9ye4jzp4DQ
> >
>
>

execution.runtime-mode=BATCH when reading from Hive

[Dongwon Kim]

Hi,

Recently I've been working on a real-time data stream processing pipeline
with DataStream API while preparing for a new service to launch.
Now it's time to develop a back-fill job to produce the same result by
reading data stored on Hive which we use for long-term storage.

Meanwhile, I watched Aljoscha's talk [1] and just wondered if I could reuse
major components of the pipeline written in DataStream API.
The pipeline conceptually looks as follows:
(A) reads input from Kafka
(B) performs AsyncIO to Redis in order to enrich the input data
(C) appends timestamps and emits watermarks before time-based window
(D) keyBy followed by a session window with a custom trigger for early
firing
(E) writes output to Kafka

I have simple (maybe stupid) questions on reusing components of the
pipeline written in DataStream API.
(1) By replacing (A) with a bounded source, can I execute the pipeline with
a new BATCH execution mode without modifying (B)~(E)?
(2) Is there a bounded source for Hive available for DataStream API?

Best,

Dongwon

[1] https://www.youtube.com/watch?v=z9ye4jzp4DQ

Re: How to run a per-job cluster for a Beam pipeline w/ FlinkRunner on YARN

[Dongwon Kim]

Hi Aljoscha,

Thanks for the input.
The '-t' option seems to be available as of flink-1.11 while the latest
FlinkRunner is based on flink-1.10.
So I use '-e' option which is available in 1.10:

$ flink run -e yarn-per-job -d <...>


A short question here is that this command ignores *-yD* and *--yarnship*
options.
Are these options only for yarn session mode?

Best,

Dongwon




On Tue, Nov 17, 2020 at 5:16 PM Aljoscha Krettek 
wrote:

> Hi,
>
> to ensure that we really are using per-job mode, could you try and use
>
> $ flink run -t yarn-per-job -d <...>
>
> This will directly specify that we want to use the YARN per-job
> executor, which bypasses some of the logic in the older YARN code paths
> that differentiate between YARN session mode and YARN per-job mode.
>
> Best,
> Aljoscha
>
> On 17.11.20 07:02, Tzu-Li (Gordon) Tai wrote:
> > Hi,
> >
> > Not sure if this question would be more suitable for the Apache Beam
> > mailing lists, but I'm pulling in Aljoscha (CC'ed) who would know more
> > about Beam and whether or not this is an expected behaviour.
> >
> > Cheers,
> > Gordon
> >
> > On Mon, Nov 16, 2020 at 10:35 PM Dongwon Kim 
> wrote:
> >
> >> Hi,
> >>
> >> I'm trying to run a per-job cluster for a Beam pipeline w/ FlinkRunner
> on
> >> YARN as follows:
> >>
> >>> flink run -m yarn-cluster -d \
> >>
> >>  my-beam-pipeline.jar \
> >>>  --runner=FlinkRunner \
> >>>  --flinkMaster=[auto] \
> >>>  --parallelism=8
> >>
> >>
> >> Instead of creating a per-job cluster as wished, the above command seems
> >> to create a session cluster and then submit a job onto the cluster.
> >>
> >> I doubt it because
> >> (1) In the attached file, there's "Submit New Job" which is not shown in
> >> other per-job applications that are written in Flink APIs and submitted
> to
> >> YARN similar to the above command.
> >>
> >> [image: beam on yarn.png]
> >> (2) When the job is finished, the YARN application is still in its
> RUNNING
> >> state without being terminated. I had to kill the YARN application
> manually.
> >>
> >> FYI, I'm using
> >> - Beam v2.24.0 (Flink 1.10)
> >> - Hadoop v3.1.1
> >>
> >> Thanks in advance,
> >>
> >> Best,
> >>
> >> Dongwon
> >>
> >
>
>

Re: CREATE TABLE LIKE clause from different catalog or database

[Dongwon Kim]

Hi Danny~
Sorry for late reply,

Let's take a look at a running example:

> EnvironmentSettings settings = EnvironmentSettings.newInstance()
>   .inBatchMode()
>   .build();
>
> TableEnvironment tEnv = TableEnvironment.create(settings);
>
> HiveCatalog hiveCatalog = new HiveCatalog("hive",null, args[1]);
> tEnv.registerCatalog("hive", hiveCatalog);
>
> GenericInMemoryCatalog inmemCatalog = new GenericInMemoryCatalog("inmem");
> tEnv.registerCatalog("inmem", inmemCatalog);
> tEnv.useCatalog("inmem");
>
> TableResult result = tEnv.executeSql(
>   "CREATE TABLE copied LIKE hive.navi.gps"
> );
>

I've got the following log messages:

> 00:50:22,157 INFO  org.apache.flink.table.catalog.hive.HiveCatalog
>  [] - Setting hive conf dir as /Users/eastcirclek/hive-conf
> 00:50:22,503 INFO  org.apache.flink.table.catalog.hive.HiveCatalog
>  [] - Created HiveCatalog 'hive'
> 00:50:22,515 INFO  hive.metastore
>   [] - Trying to connect to metastore with URI thrift://...:9083
> 00:50:22,678 INFO  hive.metastore
>   [] - Connected to metastore.
> 00:50:22,678 INFO  org.apache.flink.table.catalog.hive.HiveCatalog
>  [] - Connected to Hive metastore
> 00:50:22,799 INFO  org.apache.flink.table.catalog.CatalogManager
>  [] - Set the current default catalog as [inmem] and the current
> default database as [default].
> *Exception in thread "main"
> org.apache.flink.table.api.ValidationException: Source table
> '`inmem`.`default`.`hive.navi.gps`' of the LIKE clause not found in the
> catalog, at line 1, column 26*
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.lambda$lookupLikeSourceTable$1(SqlCreateTableConverter.java:147)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.lookupLikeSourceTable(SqlCreateTableConverter.java:147)
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.createCatalogTable(SqlCreateTableConverter.java:96)
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.convertCreateTable(SqlCreateTableConverter.java:76)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:190)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:684)
> at Test.main(Test.java:53)
>

It seems like hive.navi.gps is recognized as a table name as a whole.
I currently declare such a table by specifying all fields without the LIKE
clause.

Do I miss something?

FYI, I'm working with Flink-1.11.2.

Thank you~

Best,

Dongwon


On Fri, Nov 13, 2020 at 5:19 PM Danny Chan  wrote:

> Hi Dongwon ~
>
> Table from different catalog/db is supported, you need to specify the full
> path of the source table:
>
> CREATE TABLE Orders_with_watermark (
> *...*) WITH (
> *...*)LIKE my_catalog.my_db.Orders;
>
>
> Dongwon Kim  于2020年11月11日周三 下午2:53写道：
>
>> Hi,
>>
>> Is it disallowed to refer to a table from different databases or catalogs
>> when someone creates a table?
>>
>> According to [1], there's no way to refer to tables belonging to
>> different databases or catalogs.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table
>>
>> Best,
>>
>> Dongwon
>>
>

CREATE TABLE LIKE clause from different catalog or database

[Dongwon Kim]

Hi,

Is it disallowed to refer to a table from different databases or catalogs
when someone creates a table?

According to [1], there's no way to refer to tables belonging to different
databases or catalogs.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/create.html#create-table

Best,

Dongwon

Re: Better way to share large data across task managers

[Dongwon Kim]

Hi Kostas,

Thanks for the input!

BTW, I guess you assume that the broadcasting occurs just once for
bootstrapping, huh?
My job needs not only bootstrapping but also periodically fetching a
new version of data from some external storage.

Thanks,

Dongwon

> 2020. 9. 23. 오전 4:59, Kostas Kloudas  작성:
>
> Hi Dongwon,





>
> If you know the data in advance, you can always use the Yarn options
> in [1] (e.g. the "yarn.ship-directories") to ship the directories with
> the data you want only once to each Yarn container (i.e. TM) and then
> write a udf which reads them in the open() method. This will allow the
> data to be shipped only once per TM but then each of the tasks will
> have its own copy in memory of course. By default the visibility of
> the files that you ship is set to APPLICATION [2], if I am not
> mistaken so if more than one TMs go to the same node, then you will
> have even less copies shipped.
>
> Does this help with your usecase?
>
> Cheers,
> Kostas
>
> [1] 
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html
> [2] 
> https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/api/records/LocalResourceVisibility.html
>
>> On Sun, Sep 20, 2020 at 6:05 PM Dongwon Kim  wrote:
>> Hi,
>> I'm using Flink broadcast state similar to what Fabian explained in [1]. One 
>> difference might be the size of the broadcasted data; the size is around 
>> 150MB.
>> I've launched 32 TMs by setting
>> - taskmanager.numberOfTaskSlots : 6
>> - parallelism of the non-broadcast side : 192
>> Here's some questions:
>> 1) AFAIK, the broadcasted data (150MB) is sent to all 192 tasks. Is it right?
>> 2) Any recommended way to broadcast data only to 32 TMs so that 6 tasks in 
>> each TM can read the broadcasted data? I'm considering implementing a static 
>> class for the non-broadcast side to directly load data only once on each 
>> TaskManager instead of the broadcast state (FYI, I'm using per-job clusters 
>> on YARN, so each TM is only for a single job). However, I'd like to use 
>> Flink native facilities if possible.
>> The type of broadcasted data is Map with around 600K entries, so 
>> every time the data is broadcasted a lot of GC is inevitable on each TM due 
>> to the (de)serialization cost.
>> Any advice would be much appreciated.
>> Best,
>> Dongwon
>> [1] https://flink.apache.org/2019/06/26/broadcast-state.html

Better way to share large data across task managers

[Dongwon Kim]

Hi,

I'm using Flink broadcast state similar to what Fabian explained in [1].
One difference might be the size of the broadcasted data; the size is
around 150MB.

I've launched 32 TMs by setting
- taskmanager.numberOfTaskSlots : 6
- parallelism of the non-broadcast side : 192

Here's some questions:
1) AFAIK, the broadcasted data (150MB) is sent to all 192 tasks. Is it
right?
2) Any recommended way to broadcast data only to 32 TMs so that 6 tasks in
each TM can read the broadcasted data? I'm considering implementing a
static class for the non-broadcast side to directly load data only once on
each TaskManager instead of the broadcast state (FYI, I'm using per-job
clusters on YARN, so each TM is only for a single job). However, I'd like
to use Flink native facilities if possible.

The type of broadcasted data is Map with around 600K entries, so
every time the data is broadcasted a lot of GC is inevitable on each TM due
to the (de)serialization cost.

Any advice would be much appreciated.

Best,

Dongwon

[1] https://flink.apache.org/2019/06/26/broadcast-state.html

Re: [SQL DDL] How to extract timestamps from Kafka message's metadata

[Dongwon Kim]

Hi Timo,

Thanks for your input.
We've been considering that as well, but this time I just wanted to solely
use TableEnvironment without DataStream APIs.

but that would be the most straightforward solution this time around.

Thanks and regards,

Dongwon


On Tue, Aug 11, 2020 at 4:50 PM Timo Walther  wrote:

> Hi Dongwon,
>
> another possibility is to use DataStream API before. There you can
> extract the metadata and use DataStream.assignTimestampsAndWatermarks
> before converting the stream to a table.
>
> Regards,
> Timo
>
>
> On 11.08.20 09:41, Dongwon Kim wrote:
> > Hi Dawid,
> >
> > I'll try your suggestion [2] and wait for [1] to be supported in next
> > versions.
> >
> > Thanks,
> >
> > p.s. It's not easy to insert the timestamp into the body because it will
> > affect other applications. In this regard, I hope [1] is going to be
> > available soon.
> >
> > Dongwon
> >
> >
> > On Tue, Aug 11, 2020 at 4:31 PM Dawid Wysakowicz  > <mailto:dwysakow...@apache.org>> wrote:
> >
> > I'm afraid it is not supported yet. The discussion[1] to support it
> > started in the past, but unfortunately it has not concluded yet.
> >
> > One approach I can think of, how you can work this limitation around
> > is to provide your own Format[2]. Unfortunately it is not the most
> > straightforward solution.
> >
> > Best,
> >
> > Dawid
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
> >
> > [2]
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sourceSinks.html#encoding--decoding-formats
> >
> > On 11/08/2020 09:20, Dongwon Kim wrote:
> >> Hi,
> >>
> >> I'm working on a Kafka topic where timestamps are not shown in the
> >> message body, instead in the message's metadata.
> >>
> >> I want to declare a table from the topic with DDL but
> >> "rowtime_column_name" in the below definition seems to accept only
> >> existing columns.
> >>
> >> WATERMARK FOR rowtime_column_name AS
> >> watermark_strategy_expression.
> >>
> >>
> >> Can I define watermarks in this situation where timestamps are
> >> shown only in the metadata?
> >>
> >> Thanks,
> >>
> >> Dongwon
> >
>
>

Re: [SQL DDL] How to extract timestamps from Kafka message's metadata

[Dongwon Kim]

Hi Dawid,

I'll try your suggestion [2] and wait for [1] to be supported in next
versions.

Thanks,

p.s. It's not easy to insert the timestamp into the body because it will
affect other applications. In this regard, I hope [1] is going to be
available soon.

Dongwon


On Tue, Aug 11, 2020 at 4:31 PM Dawid Wysakowicz 
wrote:

> I'm afraid it is not supported yet. The discussion[1] to support it
> started in the past, but unfortunately it has not concluded yet.
>
> One approach I can think of, how you can work this limitation around is to
> provide your own Format[2]. Unfortunately it is not the most
> straightforward solution.
>
> Best,
>
> Dawid
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sourceSinks.html#encoding--decoding-formats
> On 11/08/2020 09:20, Dongwon Kim wrote:
>
> Hi,
>
> I'm working on a Kafka topic where timestamps are not shown in the message
> body, instead in the message's metadata.
>
> I want to declare a table from the topic with DDL but
> "rowtime_column_name" in the below definition seems to accept only existing
> columns.
>
>> WATERMARK FOR rowtime_column_name AS watermark_strategy_expression.
>
>
> Can I define watermarks in this situation where timestamps are shown only
> in the metadata?
>
> Thanks,
>
> Dongwon
>
>

[SQL DDL] How to extract timestamps from Kafka message's metadata

[Dongwon Kim]

Hi,

I'm working on a Kafka topic where timestamps are not shown in the message
body, instead in the message's metadata.

I want to declare a table from the topic with DDL but "rowtime_column_name"
in the below definition seems to accept only existing columns.

> WATERMARK FOR rowtime_column_name AS watermark_strategy_expression.


Can I define watermarks in this situation where timestamps are shown only
in the metadata?

Thanks,

Dongwon

getting in an infinite loop while creating the dependency-reduced pom

[Dongwon Kim]

Hi,

I create a new maven project (I'm using Maven 3.6.3) w/ the below command

> curl https://flink.apache.org/q/quickstart-SNAPSHOT.sh | bash -s 1.11.1
>
> and add the following dependencies to dependencies

> 
>   org.apache.flink
>   flink-connector-kafka_${scala.binary.version}
>   ${flink.version}
> 
>
> 
>   org.apache.flink
>   flink-test-utils_${scala.binary.version}
>   ${flink.version}
>   test
> 
> 
>   org.apache.flink
>   flink-runtime_${scala.binary.version}
>   ${flink.version}
>   test
>   tests
> 
> 
>   org.apache.flink
>   flink-streaming-java_${scala.binary.version}
>   ${flink.version}
>   test
>   tests
> 
>
>
When executing "mvn clean package", I've got stuck in

> [INFO] Scanning for projects...
> [INFO]
> [INFO] --< org.myorg.quickstart:quickstart 
> >---
> [INFO] Building Flink Quickstart Job 0.1
> [INFO] [ jar 
> ]-
> [INFO]
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ quickstart ---
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> quickstart ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 1 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ quickstart 
> ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 2 source files to 
> /Users/east.12/tmp/quickstart/target/classes
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
> quickstart ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory 
> /Users/east.12/tmp/quickstart/src/test/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
> quickstart ---
> [INFO] No sources to compile
> [INFO]
> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ quickstart ---
> [INFO] No tests to run.
> [INFO]
> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ quickstart ---
> [INFO] Building jar: /Users/east.12/tmp/quickstart/target/quickstart-0.1.jar
> [INFO]
> [INFO] --- maven-shade-plugin:3.1.1:shade (default) @ quickstart ---
> [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.15 from the shaded jar.
> [INFO] Excluding org.apache.flink:force-shading:jar:1.11.1 from the shaded 
> jar.
> [INFO] Excluding org.apache.logging.log4j:log4j-slf4j-impl:jar:2.12.1 from 
> the shaded jar.
> [INFO] Excluding org.apache.logging.log4j:log4j-api:jar:2.12.1 from the 
> shaded jar.
> [INFO] Excluding org.apache.logging.log4j:log4j-core:jar:2.12.1 from the 
> shaded jar.
> [INFO] Including org.apache.flink:flink-connector-kafka_2.11:jar:1.11.1 in 
> the shaded jar.
> [INFO] Including org.apache.flink:flink-connector-kafka-base_2.11:jar:1.11.1 
> in the shaded jar.
> [INFO] Including org.apache.kafka:kafka-clients:jar:2.4.1 in the shaded jar.
> [INFO] Including com.github.luben:zstd-jni:jar:1.4.3-1 in the shaded jar.
> [INFO] Including org.xerial.snappy:snappy-java:jar:1.1.4 in the shaded jar.
> [INFO] Including org.lz4:lz4-java:jar:1.6.0 in the shaded jar.
> [INFO] Replacing original artifact with shaded artifact.
> [INFO] Replacing /Users/east.12/tmp/quickstart/target/quickstart-0.1.jar with 
> /Users/east.12/tmp/quickstart/target/quickstart-0.1-shaded.jar
> [INFO] Dependency-reduced POM written at: 
> /Users/east.12/tmp/quickstart/dependency-reduced-pom.xml
>
> Any help?

Thanks,

Dongwon

Re: How to use a nested column for CREATE TABLE PARTITIONED BY

[Dongwon Kim]

Thanks Jark for the update.

However, getting back to the original question, can I use a nested column
directly for CREATE TABLE PARTITIONED BY like below without declaring an
additional column?

CREATE TABLE output
> PARTITIONED BY (`location.transId`)
> WITH (
>   'connector' = 'filesystem',
>   'path' = 'east-out',
>   'format' = 'json'
> ) LIKE navi (EXCLUDING ALL)
>

I tried (`location`.transId) as well but it fails with an exception:

> Exception in thread "main" org.apache.flink.table.api.SqlParserException:
> SQL parse failed. Encountered "." at line 3, column 27.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
> org.apache.flink.table.planner.calcite.CalciteParser.parse(CalciteParser.java:56)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:76)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:678)
> at com.kakaomobility.SQLExecutor.main(SQLExecutor.java:28)
> Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered
> "." at line 3, column 27.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.convertException(FlinkSqlParserImpl.java:416)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.normalizeException(FlinkSqlParserImpl.java:201)
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:148)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:163)
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:188)
> at
> org.apache.flink.table.planner.calcite.CalciteParser.parse(CalciteParser.java:54)
> ... 3 more
> Caused by: org.apache.flink.sql.parser.impl.ParseException: Encountered
> "." at line 3, column 27.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.generateParseException(FlinkSqlParserImpl.java:36086)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_consume_token(FlinkSqlParserImpl.java:35900)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.ParenthesizedSimpleIdentifierList(FlinkSqlParserImpl.java:21398)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreateTable(FlinkSqlParserImpl.java:5292)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreateExtended(FlinkSqlParserImpl.java:6269)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreate(FlinkSqlParserImpl.java:19047)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmt(FlinkSqlParserImpl.java:3308)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmtEof(FlinkSqlParserImpl.java:3800)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.parseSqlStmtEof(FlinkSqlParserImpl.java:248)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:161)
> ... 5 more


Best,

Dongwon

On Wed, Jul 22, 2020 at 12:09 AM Jark Wu  wrote:

> Hi Dongwon,
>
> I think this is a bug in the Filesystem connector which doesn't exclude
> the computed columns when building the TableSource.
> I created an issue [1] to track this problem.
>
> Best,
> Jark
>
> [1]: https://issues.apache.org/jira/browse/FLINK-18665
>
> On Tue, 21 Jul 2020 at 17:31, Dongwon Kim  wrote:
>
>> Hi Danny,
>>
>>  Which version did you use
>>
>> I use Flink 1.11.0.
>>
>>
>>>  what SQL context throws the error ?
>>
>> I think the declaration itself is not a problem.
>> The exception occurs when I tried to execute the following which I didn't
>> show you in the previous email:
>>
>>> tEnv.sqlQuery("SELECT type, location FROM navi").executeInsert("output")
>>
>>
>> Thanks,
>>
>> Dongwon
>>
>> On Tue, Jul 21, 2020 at 6:16 PM Danny Chan  wrote:
>>
>>> Hi, I execute the sql below
>>>
>>>   """
>>> |create table navi (
>>> |  a STRING,
>>> |  location ROW
>>> |) with (
>>> |  'connector' = 'filesystem',
>>> |  'path' = 'east-out',
>>> |  'format' = 'json'
>>> |)
>>> |""".stripMargin
>>> tableEnv.executeSql(sql0)
>>> val sql =
>>>   """
>>> |CREATE TABLE output (
>>> |  `partition` AS location.transId
>>> |) PARTITIONED BY (`partition`)
>>> |WITH (
>>> |  'connector' = 'filesystem',
>>> |  'path' = 'east-out',
>

Re: How to use a nested column for CREATE TABLE PARTITIONED BY

[Dongwon Kim]

Hi Danny,

 Which version did you use

I use Flink 1.11.0.


>  what SQL context throws the error ?

I think the declaration itself is not a problem.
The exception occurs when I tried to execute the following which I didn't
show you in the previous email:

> tEnv.sqlQuery("SELECT type, location FROM navi").executeInsert("output")


Thanks,

Dongwon

On Tue, Jul 21, 2020 at 6:16 PM Danny Chan  wrote:

> Hi, I execute the sql below
>
>   """
> |create table navi (
> |  a STRING,
> |  location ROW
> |) with (
> |  'connector' = 'filesystem',
> |  'path' = 'east-out',
> |  'format' = 'json'
> |)
> |""".stripMargin
> tableEnv.executeSql(sql0)
> val sql =
>   """
> |CREATE TABLE output (
> |  `partition` AS location.transId
> |) PARTITIONED BY (`partition`)
> |WITH (
> |  'connector' = 'filesystem',
> |  'path' = 'east-out',
> |  'format' = 'json'
> |) LIKE navi (EXCLUDING ALL)
> |""".stripMargin
> tableEnv.executeSql(sql)
>
>
> In master branch, both are correct, can you share you stack trace detail ?
> Which version did you use and what SQL context throws the error ?
>
> Best,
> Danny Chan
> 在 2020年7月21日 +0800 PM4:55，Dongwon Kim ，写道：
>
> Hi,
>
> I want to create subdirectories named after values of a nested column,
> location.transId.
>
> This is my first attempt:
>
>> CREATE TABLE output
>> PARTITIONED BY (`location.transId`)
>> WITH (
>>   'connector' = 'filesystem',
>>   'path' = 'east-out',
>>   'format' = 'json'
>> ) LIKE navi (EXCLUDING ALL)
>>
>
> It fails with the following errors:
>
>> Exception in thread "main"
>> org.apache.flink.table.api.ValidationException: Partition column
>> 'location.transId' not defined in the table schema. Available columns:
>> ['type', 'location']
>> at
>> org.apache.flink.table.planner.operations.SqlCreateTableConverter.verifyPartitioningColumnsExist(SqlCreateTableConverter.java:164)
>> at
>> org.apache.flink.table.planner.operations.SqlCreateTableConverter.createCatalogTable(SqlCreateTableConverter.java:130)
>> at
>> org.apache.flink.table.planner.operations.SqlCreateTableConverter.convertCreateTable(SqlCreateTableConverter.java:76)
>> at
>> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:190)
>> at
>> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
>> at
>> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:678)
>> at com.kakaomobility.SQLExecutor.main(SQLExecutor.java:28)
>>
>
> As It seems like nested columns are not recognized as a eligible column
> for PARTITIONED BY, I tried the following:
>
>> CREATE TABLE output (
>>   `partition` AS location.transId
>> ) PARTITIONED BY (`partition`)
>> WITH (
>>   'connector' = 'filesystem',
>>   'path' = 'east-out',
>>   'format' = 'json'
>> ) LIKE navi (EXCLUDING ALL)
>>
> It also fails:
>
>>  Exception in thread "main"
>> org.apache.flink.table.api.ValidationException: The field count of logical
>> schema of the table does not match with the field count of physical schema
>
> . The logical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId`
> STRING>]
> The physical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId`
> STRING>,STRING].
>
> Thanks in advance,
>
> Dongwon
>
>

How to use a nested column for CREATE TABLE PARTITIONED BY

[Dongwon Kim]

Hi,

I want to create subdirectories named after values of a nested column,
location.transId.

This is my first attempt:

> CREATE TABLE output
> PARTITIONED BY (`location.transId`)
> WITH (
>   'connector' = 'filesystem',
>   'path' = 'east-out',
>   'format' = 'json'
> ) LIKE navi (EXCLUDING ALL)
>

It fails with the following errors:

> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Partition column 'location.transId' not defined in the table schema.
> Available columns: ['type', 'location']
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.verifyPartitioningColumnsExist(SqlCreateTableConverter.java:164)
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.createCatalogTable(SqlCreateTableConverter.java:130)
> at
> org.apache.flink.table.planner.operations.SqlCreateTableConverter.convertCreateTable(SqlCreateTableConverter.java:76)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:190)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:678)
> at com.kakaomobility.SQLExecutor.main(SQLExecutor.java:28)
>

As It seems like nested columns are not recognized as a eligible column for
PARTITIONED BY, I tried the following:

> CREATE TABLE output (
>   `partition` AS location.transId
> ) PARTITIONED BY (`partition`)
> WITH (
>   'connector' = 'filesystem',
>   'path' = 'east-out',
>   'format' = 'json'
> ) LIKE navi (EXCLUDING ALL)
>
It also fails:

>  Exception in thread "main"
> org.apache.flink.table.api.ValidationException: The field count of logical
> schema of the table does not match with the field count of physical schema

. The logical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId`
STRING>]
The physical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId`
STRING>,STRING].

Thanks in advance,

Dongwon

Re: Encoding problem in WHERE LIKE

[Dongwon Kim]

Hi Leonard,

You're right; I was missing a single quotation mark before the LIKE.

There's no encoding problem at all!
Sorry for the confusion.

Thanks,

Dongwon


On Tue, Jul 21, 2020 at 12:00 AM Leonard Xu  wrote:

> Hi, Kim
>
> The clause  ` LIKE '%양현마을%’ ` should work well, could you post the the
> entire query(or select clause) ?
>
> Best
> Leonard Xu
>
> 在 2020年7月20日，21:49，Dongwon Kim  写道：
>
> When I execute the following query in .sqlQuery(),
>
>> SELECT ...
>> FROM ...
>> WHERE location.goalName LIKE '%양현마을%'
>
>
>

Encoding problem in WHERE LIKE

[Dongwon Kim]

Hi,

When I execute the following query in .sqlQuery(),

> SELECT ...
> FROM ...
> WHERE location.goalName LIKE '%양현마을%'
>
I got the following error message

> Caused by: org.apache.calcite.sql.parser.SqlParseException: Lexical error
> at line 1, column 96.  Encountered: "\uc591" (50577), after : ""
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.convertException(FlinkSqlParserImpl.java:416)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.normalizeException(FlinkSqlParserImpl.java:201)
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:148)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:163)
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:188)
> at
> org.apache.flink.table.planner.calcite.CalciteParser.parse(CalciteParser.java:54)
> ... 3 more
> Caused by: org.apache.flink.sql.parser.impl.TokenMgrError: Lexical error
> at line 1, column 96.  Encountered: "\uc591" (50577), after : ""
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImplTokenManager.getNextToken(FlinkSqlParserImplTokenManager.java:16104)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_scan_token(FlinkSqlParserImpl.java:35909)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3R_430(FlinkSqlParserImpl.java:34086)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3R_349(FlinkSqlParserImpl.java:34072)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3R_206(FlinkSqlParserImpl.java:35321)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3R_111(FlinkSqlParserImpl.java:35344)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3_69(FlinkSqlParserImpl.java:34666)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3_70(FlinkSqlParserImpl.java:35107)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_3_71(FlinkSqlParserImpl.java:35188)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_2_71(FlinkSqlParserImpl.java:29567)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.Expression2(FlinkSqlParserImpl.java:16828)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.Expression2(FlinkSqlParserImpl.java:16992)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.Expression(FlinkSqlParserImpl.java:16758)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.WhereOpt(FlinkSqlParserImpl.java:12927)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlSelect(FlinkSqlParserImpl.java:6914)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.LeafQuery(FlinkSqlParserImpl.java:658)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.LeafQueryOrExpr(FlinkSqlParserImpl.java:16741)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.QueryOrExpr(FlinkSqlParserImpl.java:16204)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.OrderedQueryOrExpr(FlinkSqlParserImpl.java:532)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmt(FlinkSqlParserImpl.java:3761)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmtEof(FlinkSqlParserImpl.java:3800)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.parseSqlStmtEof(FlinkSqlParserImpl.java:248)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:161)
> ... 5 more


Can anyone help me with this problem?

Thanks,

Dongwon

Re: [Table API] how to configure a nested timestamp field

[Dongwon Kim]

Hi Leonard,

Unfortunately the answer is no, the YAML you defined will parse by Table
> API and then execute, the root cause of your post error is Table API does
> not support computed column now,
> there is a FLIP under discussion[1], this should be ready in 1.12.0. BTW,
> I think DDL is recommended way since FLINK 1.11.0.

Okay, thanks a lot for your input.

I just tried out Flink SQL client and wanted to store pre-defined YAML
files each declaring a source table from a Kafka topic.
As you advised, I have to manually enter DDL in the SQL client on FLINK
1.11.x

Best,

Dongwon


On Mon, Jul 20, 2020 at 3:59 PM Leonard Xu  wrote:

> Hi, Kim
>
> Hi Leonard,
>
> Can I have a YAML definition corresponding to the DDL you suggested?
>
>
> Unfortunately the answer is no, the YAML you defined will parse by Table
> API and then execute, the root cause of your post error is Table API does
> not support computed column now,
>
> there is a FLIP under discussion[1], this should be ready in 1.12.0. BTW,
> I think DDL is recommended way since FLINK 1.11.0.
>
> Best,
> Leonard Xu
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-129%3A+Refactor+Descriptor+API+to+register+connectors+in+Table+API
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-129:+Refactor+Descriptor+API+to+register+connectors+in+Table+API>
>
> 在 2020年7月20日，14:30，Dongwon Kim  写道：
>
>
> I tried below (Flink 1.11.0) but got some error:
>
>> tables:
>>   - name: test
>> type: source-table
>> update-mode: append
>> connector:
>>   property-version: 1
>>   type: kafka
>>   version: universal
>>   topic: ...
>>   properties:
>> bootstrap.servers: ...
>> group.id: ...
>> format:
>>   property-version: 1
>>   type: json
>> schema:
>>   - name: type
>> data-type: STRING
>>   - name: location
>> data-type: >
>>   ROW<
>> id STRING,
>> lastUpdateTime BIGINT
>>   >
>>   - name: timestampCol
>> data-type: TIMESTAMP(3)
>> rowtime:
>>   timestamps:
>> type: from-field
>> from:
>> TO_TIMESTAMP(FROM_UNIXTIME(location.lastUpdateTime/1000, '-MM-dd
>> HH:mm:ss'))
>>   watermarks:
>> type: periodic-bounded
>> delay: 5000
>>
>
> SQL client doesn't complain about the file but, when I execute "SELECT
> timestampCol from test", the job fails with the following error message:
>
>> Caused by: java.lang.NullPointerException
>> at
>> org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecLegacyTableSourceScan.scala:236)
>> at
>> org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecLegacyTableSourceScan.scala:228)
>> at
>> org.apache.flink.streaming.runtime.operators.TimestampsAndWatermarksOperator.processElement(TimestampsAndWatermarksOperator.java:94)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
>> at SourceConversion$4.processElement(Unknown Source)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
>> at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
>> at
>> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
>> at
>> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:310)
>> at
>> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.col

Re: [Table API] how to configure a nested timestamp field

[Dongwon Kim]

Hi Leonard,

Can I have a YAML definition corresponding to the DDL you suggested?

I tried below (Flink 1.11.0) but got some error:

> tables:
>   - name: test
> type: source-table
> update-mode: append
> connector:
>   property-version: 1
>   type: kafka
>   version: universal
>   topic: ...
>   properties:
> bootstrap.servers: ...
> group.id: ...
> format:
>   property-version: 1
>   type: json
> schema:
>   - name: type
> data-type: STRING
>   - name: location
> data-type: >
>   ROW<
> id STRING,
> lastUpdateTime BIGINT
>   >
>   - name: timestampCol
> data-type: TIMESTAMP(3)
> rowtime:
>   timestamps:
> type: from-field
> from: TO_TIMESTAMP(FROM_UNIXTIME(location.lastUpdateTime/1000,
> '-MM-dd HH:mm:ss'))
>   watermarks:
> type: periodic-bounded
> delay: 5000
>

SQL client doesn't complain about the file but, when I execute "SELECT
timestampCol from test", the job fails with the following error message:

> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecLegacyTableSourceScan.scala:236)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.PeriodicWatermarkAssignerWrapper.extractTimestamp(StreamExecLegacyTableSourceScan.scala:228)
> at
> org.apache.flink.streaming.runtime.operators.TimestampsAndWatermarksOperator.processElement(TimestampsAndWatermarksOperator.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
> at SourceConversion$4.processElement(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:310)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:409)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:352)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:185)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:141)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:755)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:201)


On Mon, Jul 6, 2020 at 3:09 PM Dongwon Kim  wrote:

> Hi Leonard,
>
> Wow, that's great! It works like a charm.
> I've never considered this approach at all.
> Thanks a lot.
>
> Best,
> Dongwon
>
> On Mon, Jul 6, 2020 at 11:26 AM Leonard Xu  wrote:
>
>> Hi, Kim
>>
>> The reason your attempts (2) and (3) failed is that the json format does
>> not support convert a BIGINT to TIMESTAMP, you can first define the BIGINT
>> field and then use a computed column to extract TIMESTAMP field, you can
>> also define the time attribute on TIMESTAMP filed for using time-based
>> operations in Flink 1.10.1. But the computed column only support in pure
>> DDL, the Table API lacks the support and should be aligned in 1.12 as I
>> know.
>> The DDL syntax  as following:
>>

Re: [Table API] how to configure a nested timestamp field

[Dongwon Kim]

Hi Leonard,

Wow, that's great! It works like a charm.
I've never considered this approach at all.
Thanks a lot.

Best,
Dongwon

On Mon, Jul 6, 2020 at 11:26 AM Leonard Xu  wrote:

> Hi, Kim
>
> The reason your attempts (2) and (3) failed is that the json format does
> not support convert a BIGINT to TIMESTAMP, you can first define the BIGINT
> field and then use a computed column to extract TIMESTAMP field, you can
> also define the time attribute on TIMESTAMP filed for using time-based
> operations in Flink 1.10.1. But the computed column only support in pure
> DDL, the Table API lacks the support and should be aligned in 1.12 as I
> know.
> The DDL syntax  as following:
>
> create table test (
>   `type` STRING,
>   `location` ROW<`id` STRING, lastUpdateTime BIGINT>,
>timestampCol as
> TO_TIMESTAMP(FROM_UNIXTIME(`location`.lastUpdateTime/1000, '-MM-dd
> HH:mm:ss')), —computed column
>WATERMARK FOR timestampCol AS timestampCol - INTERVAL '5' SECOND
> )   with (
>   'connector' = '...',
>   'format' = 'json',
>   ...
> );
>
>
> Best,
> Leonard Xu
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/create.html
>
>
> 在 2020年7月4日，21:21，Dongwon Kim  写道：
>
> Hi,
> I use Flink 1.10.1 and I want to use Table API to read JSON messages. The
> message looks like below.
>
>> {
>>"type":"Update",
>>"location":{
>>   "id":"123e4567-e89b-12d3-a456-42665234",
>>   "lastUpdateTime":1593866161436
>>}
>> }
>
>
> I wrote the following program just to see whether json messages are
> correctly parsed by Table API:
>
>> StreamExecutionEnvironment env =
>> StreamExecutionEnvironment.getExecutionEnvironment();
>> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
>> EnvironmentSettings envSettings =
>> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
>> StreamTableEnvironment tEnv = StreamTableEnvironment.create(env,
>> envSettings);
>> tEnv
>>   .connect(
>> new Kafka()
>>   .version("universal")
>>   .topic(consumerTopic)
>>   .startFromLatest()
>>   .properties(consumerProperties)
>>   )
>>   .withFormat(new Json())
>>   .withSchema(new Schema().schema(
>> TableSchema.builder()
>>   .field("type", STRING())
>>   .field("location",
>> ROW(
>>   FIELD("id", STRING()),
>>   // (1)
>>   FIELD("lastUpdateTime", BIGINT())
>>   // (2)
>>   FIELD("lastUpdateTime", TIMESTAMP())
>>   // (3)
>>   FIELD("lastUpdateTime",
>> TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class))
>> ))
>>   .build()
>>   ))
>>   .createTemporaryTable("message");
>> tEnv.toAppendStream(tEnv.from("message"), Row.class)
>>   .print();
>
>
> Note that I tried BIGINT(), TIMESTAMP(), and
> TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class).
> (1) it works fine but later I can't use time-based operations like
> windowing.
>
> (2) it causes the following exception
>
>> Exception in thread "main"
>> org.apache.flink.table.api.ValidationException: Type ROW<`id` STRING,
>> `lastUpdateTime` TIMESTAMP(6)> of table field 'location' does not match
>> with the physical type ROW<`id` STRING, `lastUpdateTime` TIMESTAMP(3)> of
>> the 'location' field of the TableSource return type.
>> at
>> org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:166)
>> at
>> org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:191)
>> at
>> org.apache.flink.table.utils.TypeMappingUtils.lambda$computeInCompositeType$8(TypeMappingUtils.java:252)
>> at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
>> at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
>> at
>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
>> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>> at
>> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>> at
>> java.util.

[Table API] how to configure a nested timestamp field

[Dongwon Kim]

Hi,
I use Flink 1.10.1 and I want to use Table API to read JSON messages. The
message looks like below.

> {
>"type":"Update",
>"location":{
>   "id":"123e4567-e89b-12d3-a456-42665234",
>   "lastUpdateTime":1593866161436
>}
> }


I wrote the following program just to see whether json messages are
correctly parsed by Table API:

> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
> EnvironmentSettings envSettings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(env,
> envSettings);
> tEnv
>   .connect(
> new Kafka()
>   .version("universal")
>   .topic(consumerTopic)
>   .startFromLatest()
>   .properties(consumerProperties)
>   )
>   .withFormat(new Json())
>   .withSchema(new Schema().schema(
> TableSchema.builder()
>   .field("type", STRING())
>   .field("location",
> ROW(
>   FIELD("id", STRING()),
>   // (1)
>   FIELD("lastUpdateTime", BIGINT())
>   // (2)
>   FIELD("lastUpdateTime", TIMESTAMP())
>   // (3)
>   FIELD("lastUpdateTime",
> TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class))
> ))
>   .build()
>   ))
>   .createTemporaryTable("message");
> tEnv.toAppendStream(tEnv.from("message"), Row.class)
>   .print();


Note that I tried BIGINT(), TIMESTAMP(), and
TIMESTAMP(3).bridgedTo(java.sql.Timestamp.class).
(1) it works fine but later I can't use time-based operations like
windowing.

(2) it causes the following exception

> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Type ROW<`id` STRING, `lastUpdateTime` TIMESTAMP(6)> of table field
> 'location' does not match with the physical type ROW<`id` STRING,
> `lastUpdateTime` TIMESTAMP(3)> of the 'location' field of the TableSource
> return type.
> at
> org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:166)
> at
> org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:191)
> at
> org.apache.flink.table.utils.TypeMappingUtils.lambda$computeInCompositeType$8(TypeMappingUtils.java:252)
> at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
> at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computeInCompositeType(TypeMappingUtils.java:234)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndices(TypeMappingUtils.java:212)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndicesOrTimeAttributeMarkers(TypeMappingUtils.java:116)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.computeIndexMapping(StreamExecTableSourceScan.scala:212)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:107)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:62)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlan(StreamExecTableSourceScan.scala:62)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToTransformation(StreamExecSink.scala:184)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:153)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:48)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlan(StreamExecSink.scala:48)
> at
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:60)
> at
> 

Re: Cannot start native K8s

[Dongwon Kim]

Hi Yang,

Oops, I forget to copy /etc/kube/admin.conf to $HOME/.kube/config so that
the current user account can access to K8s.
Now that I copied it, I found that kubernetes-session.sh is working fine.
Thanks very much!

Best,
Dongwon

[flink@DAC-E04-W06 ~]$ kubernetes-session.sh
2020-05-09 12:43:49,961 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.address, DAC-E04-W06
2020-05-09 12:43:49,962 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.port, 6123
2020-05-09 12:43:49,962 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.heap.size, 1024m
2020-05-09 12:43:49,962 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.memory.process.size, 24g
2020-05-09 12:43:49,963 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.numberOfTaskSlots, 24
2020-05-09 12:43:49,963 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: parallelism.default, 1
2020-05-09 12:43:49,963 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: high-availability, zookeeper
2020-05-09 12:43:49,963 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: high-availability.zookeeper.path.root, /flink
2020-05-09 12:43:49,964 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: high-availability.storageDir, hdfs:///user/flink/ha/
2020-05-09 12:43:49,964 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: high-availability.zookeeper.quorum, DAC-E04-W06:2181
2020-05-09 12:43:49,965 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.execution.failover-strategy, region
2020-05-09 12:43:49,965 INFO
 org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: rest.port, 8082
2020-05-09 12:43:51,122 INFO
 org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils  - The
derived from fraction jvm overhead memory (2.400gb (2576980416 bytes)) is
greater than its max value 1024.000mb (1073741824 bytes), max value will be
used instead
2020-05-09 12:43:51,123 INFO
 org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils  - The
derived from fraction network memory (2.291gb (2459539902 bytes)) is
greater than its max value 1024.000mb (1073741824 bytes), max value will be
used instead
2020-05-09 12:43:51,131 INFO
 org.apache.flink.kubernetes.utils.KubernetesUtils - Kubernetes
deployment requires a fixed port. Configuration blob.server.port will be
set to 6124
2020-05-09 12:43:51,131 INFO
 org.apache.flink.kubernetes.utils.KubernetesUtils - Kubernetes
deployment requires a fixed port. Configuration taskmanager.rpc.port will
be set to 6122
2020-05-09 12:43:51,134 INFO
 org.apache.flink.kubernetes.utils.KubernetesUtils - Kubernetes
deployment requires a fixed port. Configuration
high-availability.jobmanager.port will be set to 6123
2020-05-09 12:43:52,167 INFO
 org.apache.flink.kubernetes.KubernetesClusterDescriptor   - Create
flink session cluster flink-cluster-4a82d41b-af15-4205-8a44-62351e270242
successfully, JobManager Web Interface: http://cluster-endpoint:31513


On Sat, May 9, 2020 at 12:29 PM Yang Wang  wrote:

> Hi Dongwon Kim,
>
> Thanks a lot for your information. I will dig into this issue.
>
> I think the "UnknownHostException" is caused by incorrectly setting the
> Kubernetes
> ApiServer address. Maybe you are using "kubernetes.default.svc". However,
> it
> could not be accessed outside of the Kubernetes cluster. You need to
> configure
> a correct ip/hostname for ApiServer address, which could be accessed in
> your
> local environment. You could use `kubectl auth can-i create pods` to verify
> whether the kube config is correct.
>
> BTW, currently we only find the flink on native K8s could not work on
> 8u252. For
> 8u242 and lower version, it works well.
>
>
> Best,
> Yang
>
> Dongwon Kim  于2020年5月9日周六 上午10:43写道：
>
>> Hello Yang,
>>
>> I'm using K8s v1.18.2 installed by Kubeadm over a cluster of 5 nodes (not
>> a Minikube).
>> Previously, as you pointed out, openjdk version "1.8.0_252" was installed.
>> I bump up java version to openjdk 11.0.7 but got something different:
>>
>> [flink@DAC-E04-W06 bin]$ ./kubernetes-session.sh
>> 2020-05-09 11:39:36,737 INFO
>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property

Re: Cannot start native K8s

[Dongwon Kim]

.java:1298)
at org.apache.flink.kubernetes.shadded.okhttp3.Dns$1.lookup(Dns.java:40)
at
org.apache.flink.kubernetes.shadded.okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
at
org.apache.flink.kubernetes.shadded.okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:149)
at
org.apache.flink.kubernetes.shadded.okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
at
org.apache.flink.kubernetes.shadded.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:214)


Thanks a lot for your interest :-)

Best,

Dongwon

On Sat, May 9, 2020 at 11:15 AM Yang Wang  wrote:

> Hi Dongwon Kim,
>
> Are you running Flink on a minikube or a real Kubernetes cluster? I just
> could not
> reproduce it in a real Kubernetes cluster with java 8u252. For minikube, i
> get the
> exception with you.
>
>
> Best,
> Yang
>
> Yang Wang  于2020年5月6日周三 上午9:29写道：
>
>> Hi Dongwon Kim,
>>
>> I think it is a known issue. The native kubernetes integration could not
>> work with jdk 8u252
>> due to okhttp issue[1]. Currently, you could upgrade your jdk to a new
>> version to work around.
>>
>>
>> [1]. https://issues.apache.org/jira/browse/FLINK-17416
>>
>> Dongwon Kim  于2020年5月6日周三 上午7:15写道：
>>
>>> Hi,
>>>
>>> I'm using Flink-1.10 and tested everything [1] successfully.
>>> While trying [2], I got the following message.
>>> Can anyone help please?
>>>
>>> [root@DAC-E04-W06 bin]# ./kubernetes-session.sh
>>>> 2020-05-06 08:10:49,411 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: jobmanager.rpc.address, DAC-E04-W06
>>>> 2020-05-06 08:10:49,412 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: jobmanager.rpc.port, 6123
>>>> 2020-05-06 08:10:49,412 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: jobmanager.heap.size, 1024m
>>>> 2020-05-06 08:10:49,412 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: taskmanager.memory.process.size, 24g
>>>> 2020-05-06 08:10:49,413 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: taskmanager.numberOfTaskSlots, 24
>>>> 2020-05-06 08:10:49,413 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: parallelism.default, 1
>>>> 2020-05-06 08:10:49,413 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: high-availability, zookeeper
>>>> 2020-05-06 08:10:49,413 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: high-availability.zookeeper.path.root, /flink
>>>> 2020-05-06 08:10:49,414 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: high-availability.storageDir, 
>>>> hdfs:///user/flink/ha/
>>>> 2020-05-06 08:10:49,414 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: high-availability.zookeeper.quorum, 
>>>> DAC-E04-W06:2181
>>>> 2020-05-06 08:10:49,414 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: jobmanager.execution.failover-strategy, region
>>>> 2020-05-06 08:10:49,415 INFO
>>>>  org.apache.flink.configuration.GlobalConfiguration- Loading
>>>> configuration property: rest.port, 8082
>>>> 2020-05-06 08:10:50,386 ERROR
>>>> org.apache.flink.kubernetes.cli.KubernetesSessionCli  - Error while
>>>> running the Flink session.
>>>> io.fabric8.kubernetes.client.KubernetesClientException: Operation:
>>>> [get]  for kind: [Service]  with name:
>>>> [flink-cluster-5c12bd50-a540-4614-96d0-549785a8bc62]  in namespace:
>>>> [default]  failed.
>>>> at
>>>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
>>>> at
>>>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientEx

Cannot start native K8s

[Dongwon Kim]

Hi,

I'm using Flink-1.10 and tested everything [1] successfully.
While trying [2], I got the following message.
Can anyone help please?

[root@DAC-E04-W06 bin]# ./kubernetes-session.sh
> 2020-05-06 08:10:49,411 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: jobmanager.rpc.address, DAC-E04-W06
> 2020-05-06 08:10:49,412 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: jobmanager.rpc.port, 6123
> 2020-05-06 08:10:49,412 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: jobmanager.heap.size, 1024m
> 2020-05-06 08:10:49,412 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: taskmanager.memory.process.size, 24g
> 2020-05-06 08:10:49,413 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: taskmanager.numberOfTaskSlots, 24
> 2020-05-06 08:10:49,413 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: parallelism.default, 1
> 2020-05-06 08:10:49,413 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: high-availability, zookeeper
> 2020-05-06 08:10:49,413 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: high-availability.zookeeper.path.root, /flink
> 2020-05-06 08:10:49,414 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: high-availability.storageDir, hdfs:///user/flink/ha/
> 2020-05-06 08:10:49,414 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: high-availability.zookeeper.quorum, DAC-E04-W06:2181
> 2020-05-06 08:10:49,414 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: jobmanager.execution.failover-strategy, region
> 2020-05-06 08:10:49,415 INFO
>  org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8082
> 2020-05-06 08:10:50,386 ERROR
> org.apache.flink.kubernetes.cli.KubernetesSessionCli  - Error while
> running the Flink session.
> io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]
>  for kind: [Service]  with name:
> [flink-cluster-5c12bd50-a540-4614-96d0-549785a8bc62]  in namespace:
> [default]  failed.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:231)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:164)
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.getService(Fabric8FlinkKubeClient.java:334)
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.getInternalService(Fabric8FlinkKubeClient.java:246)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.run(KubernetesSessionCli.java:104)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.lambda$main$0(KubernetesSessionCli.java:185)
> at
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at
> org.apache.flink.kubernetes.cli.KubernetesSessionCli.main(KubernetesSessionCli.java:185)
> Caused by: java.net.SocketException: Broken pipe (Write failed)
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
> at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
> at
> sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
> at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
> at org.apache.flink.kubernetes.shadded.okio.Okio$1.write(Okio.java:79)
> at
> org.apache.flink.kubernetes.shadded.okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
> at
> org.apache.flink.kubernetes.shadded.okio.RealBufferedSink.flush(RealBufferedSink.java:224)
> at
> org.apache.flink.kubernetes.shadded.okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:203)
> at
> org.apache.flink.kubernetes.shadded.okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:515)
> at
> org.apache.flink.kubernetes.shadded.okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:505)
> at
> org.apache.flink.kubernetes.shadded.okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:298)
> at
> 

Re: Support of custom JDBC dialects in flink-jdbc

[Dongwon Kim]

Hi Jark,
Many thanks for creating the issue on Jira and nice summarization :-)

Best,
Dongwon

On Sat, Mar 28, 2020 at 12:37 AM Jark Wu  wrote:

> Hi Dongwon,
>
> I saw many requirements on this and I'm big +1 for this.
> I created https://issues.apache.org/jira/browse/FLINK-16833 to track this
> effort. Hope this can be done before 1.11 release.
>
> Best,
> Jark
>
> On Fri, 27 Mar 2020 at 22:22, Dongwon Kim  wrote:
>
>> Hi, I tried flink-jdbc [1] to read data from Druid because Druid
>> implements Calcite Avatica [2], but the connection string,
>> jdbc:avatica:remote:url=http://BROKER:8082/druid/v2/sql/avatica/, is not
>> supported by any of JDBCDialects [3].
>>
>> I implement custom JDBCDialect [4], custom StreamTableSourceFactory [5],
>> etc in my own repository [6], but it seems overkill.
>>
>> How about supporting custom JDBC dialects in flink-jdbc based on SPI?
>>
>> Best,
>> - Dongwon
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connect.html#jdbc-connector
>> [2] https://druid.apache.org/docs/latest/querying/sql.html#jdbc
>> [3]
>> https://github.com/apache/flink/blob/master/flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/dialect/JDBCDialects.java
>> [4]
>> https://github.com/eastcirclek/flink-avatica-source/blob/master/src/main/java/io/github/eastcirclek/AvaticaJDBCDialect.java
>> [5]
>> https://github.com/eastcirclek/flink-avatica-source/blob/master/src/main/java/io/github/eastcirclek/AvaticaJDBCSourceFactory.java
>> [6] https://github.com/eastcirclek/flink-avatica-source
>>
>

Support of custom JDBC dialects in flink-jdbc

[Dongwon Kim]

Hi, I tried flink-jdbc [1] to read data from Druid because Druid implements
Calcite Avatica [2], but the connection string, jdbc:avatica:remote:url=
http://BROKER:8082/druid/v2/sql/avatica/, is not supported by any of
JDBCDialects [3].

I implement custom JDBCDialect [4], custom StreamTableSourceFactory [5],
etc in my own repository [6], but it seems overkill.

How about supporting custom JDBC dialects in flink-jdbc based on SPI?

Best,
- Dongwon

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connect.html#jdbc-connector
[2] https://druid.apache.org/docs/latest/querying/sql.html#jdbc
[3]
https://github.com/apache/flink/blob/master/flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/dialect/JDBCDialects.java
[4]
https://github.com/eastcirclek/flink-avatica-source/blob/master/src/main/java/io/github/eastcirclek/AvaticaJDBCDialect.java
[5]
https://github.com/eastcirclek/flink-avatica-source/blob/master/src/main/java/io/github/eastcirclek/AvaticaJDBCSourceFactory.java
[6] https://github.com/eastcirclek/flink-avatica-source

Re: User program failures cause JobManager to be shutdown

[Dongwon Kim]

Hi Robert and Roman,
Yeah, letting users know System.exit() is called would be much more
appropriate than just intercepting and ignoring.

Best,
Dongwon

On Sat, Dec 7, 2019 at 11:29 PM Robert Metzger  wrote:

> I guess we could manage the security only when calling the user's main()
> method.
>
> This problem actually exists for all usercode in Flink: You can also kill
> TaskManagers like this.
> If we are going to add something like this to Flink, I would only log that
> System.exit() has been called by the user code, not intercept and ignore
> the call.
>
> On Fri, Dec 6, 2019 at 10:31 AM Khachatryan Roman <
> khachatryan.ro...@gmail.com> wrote:
>
>> Hi Dongwon,
>>
>> This should work but it could also interfere with Flink itself exiting in
>> case of a fatal error.
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Dec 6, 2019 at 2:54 AM Dongwon Kim  wrote:
>>
>>> FYI, we've launched a session cluster where multiple jobs are managed by
>>> a job manager. If that happens, all the other jobs also fail because the
>>> job manager is shut down and all the task managers get into chaos (failing
>>> to connect to the job manager).
>>>
>>> I just searched a way to prevent System.exit() calls from terminating
>>> JVMs and found [1]. Can it be a possible solution to the problem?
>>>
>>> [1]
>>> https://stackoverflow.com/questions/5549720/how-to-prevent-calls-to-system-exit-from-terminating-the-jvm
>>>
>>> Best,
>>> - Dongwon
>>>
>>> On Fri, Dec 6, 2019 at 10:39 AM Dongwon Kim 
>>> wrote:
>>>
>>>> Hi Robert and Roman,
>>>>
>>>> Thank you for taking a look at this.
>>>>
>>>> what is your main() method / client doing when it's receiving wrong
>>>>> program parameters? Does it call System.exit(), or something like that?
>>>>>
>>>>
>>>> I just found that our HTTP client is programmed to call System.exit(1).
>>>> I should guide not to call System.exit() in Flink applications.
>>>>
>>>> p.s. Just out of curiosity, is there no way for the web app to
>>>> intercept System.exit() and prevent the job manager from being shutting
>>>> down?
>>>>
>>>> Best,
>>>>
>>>> - Dongwon
>>>>
>>>> On Fri, Dec 6, 2019 at 3:59 AM Robert Metzger 
>>>> wrote:
>>>>
>>>>> Hi Dongwon,
>>>>>
>>>>> what is your main() method / client doing when it's receiving wrong
>>>>> program parameters? Does it call System.exit(), or something like that?
>>>>>
>>>>> By the way, the http address from the error message is
>>>>> publicly available. Not sure if this is internal data or not.
>>>>>
>>>>> On Thu, Dec 5, 2019 at 6:32 PM Khachatryan Roman <
>>>>> khachatryan.ro...@gmail.com> wrote:
>>>>>
>>>>>> Hi Dongwon,
>>>>>>
>>>>>> I wasn't able to reproduce your problem with Flink JobManager 1.9.1
>>>>>> with various kinds of errors in the job.
>>>>>> I suggest you try it on a fresh Flink installation without any other
>>>>>> jobs submitted.
>>>>>>
>>>>>> Regards,
>>>>>> Roman
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 5, 2019 at 3:48 PM Dongwon Kim 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Roman,
>>>>>>>
>>>>>>> We're using the latest version 1.9.1 and those two lines are all
>>>>>>> I've seen after executing the job on the web ui.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dongwon
>>>>>>>
>>>>>>> On Thu, Dec 5, 2019 at 11:36 PM r_khachatryan <
>>>>>>> khachatryan.ro...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dongwon,
>>>>>>>>
>>>>>>>> Could you please provide Flink version you are running and the job
>>>>>>>> manager
>>>>>>>> logs?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Roman
>>>>>>>>
>>>>>>>>
>>>>>>>> eastcirclek wrote
>>>>>>&g

Re: User program failures cause JobManager to be shutdown

[Dongwon Kim]

FYI, we've launched a session cluster where multiple jobs are managed by a
job manager. If that happens, all the other jobs also fail because the job
manager is shut down and all the task managers get into chaos (failing to
connect to the job manager).

I just searched a way to prevent System.exit() calls from terminating JVMs
and found [1]. Can it be a possible solution to the problem?

[1]
https://stackoverflow.com/questions/5549720/how-to-prevent-calls-to-system-exit-from-terminating-the-jvm

Best,
- Dongwon

On Fri, Dec 6, 2019 at 10:39 AM Dongwon Kim  wrote:

> Hi Robert and Roman,
>
> Thank you for taking a look at this.
>
> what is your main() method / client doing when it's receiving wrong
>> program parameters? Does it call System.exit(), or something like that?
>>
>
> I just found that our HTTP client is programmed to call System.exit(1). I
> should guide not to call System.exit() in Flink applications.
>
> p.s. Just out of curiosity, is there no way for the web app to intercept
> System.exit() and prevent the job manager from being shutting down?
>
> Best,
>
> - Dongwon
>
> On Fri, Dec 6, 2019 at 3:59 AM Robert Metzger  wrote:
>
>> Hi Dongwon,
>>
>> what is your main() method / client doing when it's receiving wrong
>> program parameters? Does it call System.exit(), or something like that?
>>
>> By the way, the http address from the error message is
>> publicly available. Not sure if this is internal data or not.
>>
>> On Thu, Dec 5, 2019 at 6:32 PM Khachatryan Roman <
>> khachatryan.ro...@gmail.com> wrote:
>>
>>> Hi Dongwon,
>>>
>>> I wasn't able to reproduce your problem with Flink JobManager 1.9.1 with
>>> various kinds of errors in the job.
>>> I suggest you try it on a fresh Flink installation without any other
>>> jobs submitted.
>>>
>>> Regards,
>>> Roman
>>>
>>>
>>> On Thu, Dec 5, 2019 at 3:48 PM Dongwon Kim 
>>> wrote:
>>>
>>>> Hi Roman,
>>>>
>>>> We're using the latest version 1.9.1 and those two lines are all I've
>>>> seen after executing the job on the web ui.
>>>>
>>>> Best,
>>>>
>>>> Dongwon
>>>>
>>>> On Thu, Dec 5, 2019 at 11:36 PM r_khachatryan <
>>>> khachatryan.ro...@gmail.com> wrote:
>>>>
>>>>> Hi Dongwon,
>>>>>
>>>>> Could you please provide Flink version you are running and the job
>>>>> manager
>>>>> logs?
>>>>>
>>>>> Regards,
>>>>> Roman
>>>>>
>>>>>
>>>>> eastcirclek wrote
>>>>> > Hi,
>>>>> >
>>>>> > I tried to run a program by uploading a jar on Flink UI. When I
>>>>> > intentionally enter a wrong parameter to my program, JobManager dies.
>>>>> > Below
>>>>> > is all log messages I can get from JobManager; JobManager dies as
>>>>> soon as
>>>>> > spitting the second line:
>>>>> >
>>>>> > 2019-12-05 04:47:58,623 WARN
>>>>> >>  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler-
>>>>> >> Configuring the job submission via query parameters is deprecated.
>>>>> Please
>>>>> >> migrate to submitting a JSON request instead.
>>>>> >>
>>>>> >>
>>>>> >> *2019-12-05 04:47:59,133 ERROR com.skt.apm.http.HTTPClient
>>>>> >>   - Cannot
>>>>> >> connect:
>>>>> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
>>>>> >> 
>>>>> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
>>>>> ;:
>>>>> >> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
>>>>> >> deserialize instance of `java.util.ArrayList` out of START_OBJECT
>>>>> token
>>>>> >> at
>>>>> >> [Source:
>>>>> >>
>>>>> (String)“{”code”:“GB0001”,“resource”:“msg.comm.unknown.error”,“details”:“NullPointerException:
>>>>> >> “}”; line: 1, column: 1]2019-12-05 04:47:59,166 INFO
>>>>> >>  org.apache.flink.runtime.blob.BlobServer  -
>>>>> Stopped
>>>>> >> BLOB server at 0.0.0.0:6124 http://0.0.0.0:6124*
>>>>> >
>>>>> >
>>>>> > The second line is obviously from my program and it shouldn't cause
>>>>> > JobManager to be shut down. Is it intended behavior?
>>>>> >
>>>>> > Best,
>>>>> >
>>>>> > Dongwon
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>>>
>>>>

Re: User program failures cause JobManager to be shutdown

[Dongwon Kim]

Hi Robert and Roman,

Thank you for taking a look at this.

what is your main() method / client doing when it's receiving wrong program
> parameters? Does it call System.exit(), or something like that?
>

I just found that our HTTP client is programmed to call System.exit(1). I
should guide not to call System.exit() in Flink applications.

p.s. Just out of curiosity, is there no way for the web app to intercept
System.exit() and prevent the job manager from being shutting down?

Best,

- Dongwon

On Fri, Dec 6, 2019 at 3:59 AM Robert Metzger  wrote:

> Hi Dongwon,
>
> what is your main() method / client doing when it's receiving wrong
> program parameters? Does it call System.exit(), or something like that?
>
> By the way, the http address from the error message is publicly available.
> Not sure if this is internal data or not.
>
> On Thu, Dec 5, 2019 at 6:32 PM Khachatryan Roman <
> khachatryan.ro...@gmail.com> wrote:
>
>> Hi Dongwon,
>>
>> I wasn't able to reproduce your problem with Flink JobManager 1.9.1 with
>> various kinds of errors in the job.
>> I suggest you try it on a fresh Flink installation without any other jobs
>> submitted.
>>
>> Regards,
>> Roman
>>
>>
>> On Thu, Dec 5, 2019 at 3:48 PM Dongwon Kim  wrote:
>>
>>> Hi Roman,
>>>
>>> We're using the latest version 1.9.1 and those two lines are all I've
>>> seen after executing the job on the web ui.
>>>
>>> Best,
>>>
>>> Dongwon
>>>
>>> On Thu, Dec 5, 2019 at 11:36 PM r_khachatryan <
>>> khachatryan.ro...@gmail.com> wrote:
>>>
>>>> Hi Dongwon,
>>>>
>>>> Could you please provide Flink version you are running and the job
>>>> manager
>>>> logs?
>>>>
>>>> Regards,
>>>> Roman
>>>>
>>>>
>>>> eastcirclek wrote
>>>> > Hi,
>>>> >
>>>> > I tried to run a program by uploading a jar on Flink UI. When I
>>>> > intentionally enter a wrong parameter to my program, JobManager dies.
>>>> > Below
>>>> > is all log messages I can get from JobManager; JobManager dies as
>>>> soon as
>>>> > spitting the second line:
>>>> >
>>>> > 2019-12-05 04:47:58,623 WARN
>>>> >>  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler-
>>>> >> Configuring the job submission via query parameters is deprecated.
>>>> Please
>>>> >> migrate to submitting a JSON request instead.
>>>> >>
>>>> >>
>>>> >> *2019-12-05 04:47:59,133 ERROR com.skt.apm.http.HTTPClient
>>>> >>   - Cannot
>>>> >> connect:
>>>> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
>>>> >> 
>>>> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
>>>> ;:
>>>> >> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
>>>> >> deserialize instance of `java.util.ArrayList` out of START_OBJECT
>>>> token
>>>> >> at
>>>> >> [Source:
>>>> >>
>>>> (String)“{”code”:“GB0001”,“resource”:“msg.comm.unknown.error”,“details”:“NullPointerException:
>>>> >> “}”; line: 1, column: 1]2019-12-05 04:47:59,166 INFO
>>>> >>  org.apache.flink.runtime.blob.BlobServer  -
>>>> Stopped
>>>> >> BLOB server at 0.0.0.0:6124 http://0.0.0.0:6124*
>>>> >
>>>> >
>>>> > The second line is obviously from my program and it shouldn't cause
>>>> > JobManager to be shut down. Is it intended behavior?
>>>> >
>>>> > Best,
>>>> >
>>>> > Dongwon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>>
>>>

Re: User program failures cause JobManager to be shutdown

[Dongwon Kim]

Hi Roman,

We're using the latest version 1.9.1 and those two lines are all I've seen
after executing the job on the web ui.

Best,

Dongwon

On Thu, Dec 5, 2019 at 11:36 PM r_khachatryan 
wrote:

> Hi Dongwon,
>
> Could you please provide Flink version you are running and the job manager
> logs?
>
> Regards,
> Roman
>
>
> eastcirclek wrote
> > Hi,
> >
> > I tried to run a program by uploading a jar on Flink UI. When I
> > intentionally enter a wrong parameter to my program, JobManager dies.
> > Below
> > is all log messages I can get from JobManager; JobManager dies as soon as
> > spitting the second line:
> >
> > 2019-12-05 04:47:58,623 WARN
> >>  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler-
> >> Configuring the job submission via query parameters is deprecated.
> Please
> >> migrate to submitting a JSON request instead.
> >>
> >>
> >> *2019-12-05 04:47:59,133 ERROR com.skt.apm.http.HTTPClient
> >>   - Cannot
> >> connect:
> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
> >> 
> http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
> ;:
> >> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
> >> deserialize instance of `java.util.ArrayList` out of START_OBJECT token
> >> at
> >> [Source:
> >>
> (String)“{”code”:“GB0001”,“resource”:“msg.comm.unknown.error”,“details”:“NullPointerException:
> >> “}”; line: 1, column: 1]2019-12-05 04:47:59,166 INFO
> >>  org.apache.flink.runtime.blob.BlobServer  - Stopped
> >> BLOB server at 0.0.0.0:6124 http://0.0.0.0:6124*
> >
> >
> > The second line is obviously from my program and it shouldn't cause
> > JobManager to be shut down. Is it intended behavior?
> >
> > Best,
> >
> > Dongwon
>
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

User program failures cause JobManager to be shutdown

[Dongwon Kim]

Hi,

I tried to run a program by uploading a jar on Flink UI. When I
intentionally enter a wrong parameter to my program, JobManager dies. Below
is all log messages I can get from JobManager; JobManager dies as soon as
spitting the second line:

2019-12-05 04:47:58,623 WARN
>  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler-
> Configuring the job submission via query parameters is deprecated. Please
> migrate to submitting a JSON request instead.
>
>
> *2019-12-05 04:47:59,133 ERROR com.skt.apm.http.HTTPClient
>   - Cannot
> connect:http://52.141.38.11:8380/api/spec/poc_asset_model_01/model/imbalance/models
> :
> com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot
> deserialize instance of `java.util.ArrayList` out of START_OBJECT token at
> [Source:
> (String)“{”code”:“GB0001”,“resource”:“msg.comm.unknown.error”,“details”:“NullPointerException:
> “}”; line: 1, column: 1]2019-12-05 04:47:59,166 INFO
>  org.apache.flink.runtime.blob.BlobServer  - Stopped
> BLOB server at 0.0.0.0:6124 *


The second line is obviously from my program and it shouldn't cause
JobManager to be shut down. Is it intended behavior?

Best,

Dongwon

Re: [ANNOUNCE] Weekly Community Update 2019/48

[Dongwon Kim]

Dear community,

As Konstantin mentioned, there will be the second Apache Flink meetup in
Seoul [1] and we just finalized the list of speakers. Here I share the talk
lineup with brief introduction.

1. "Mastering Flink's interval join" by Jihye Yeom (Kakao corp.)
: Kakao corp. recently adopts Apache Flink for processing marketing and
advertising data. In this talk, Jihye shares how she becomes able to make
full use of Flink's interval join over high volume of real-time data.

2. "Adopting Apache Flink to T map service" by Seungcheol Oh (T map, SK
telecom)
: This talk covers T map team's journey adopting and operating a real-time
driving score service [2] which is a mission-critical application written
in Flink DataStream API.

3. "Do Flink On Web with FLOW" by Dongwon Kim (Data Labs, SK telecom)
: Dongwon introduces a web service to build data stream processing
pipelines on GUI based on Flink Table API. It was presented at FlinkForward
2019 Europe [3].

4. "From source to sink", Hangu Kang (Kakao mobility)
: Hangu shares his experience of building and operating Flink applications
for various mobility services in Kakao mobility. This talk covers Kafka
connectors, event time processing, checkpoint/savepoint, state management,
custom windowing, etc.

[1] https://www.meetup.com/ko-KR/Seoul-Apache-Flink-Meetup/events/266824815/
[2] https://www.youtube.com/watch?v=wPQWFy5JENw
[3] https://www.youtube.com/watch?v=dxMn7cIqxG4

Best,

Dongwon

On Mon, Dec 2, 2019 at 1:47 AM Konstantin Knauf 
wrote:

> Dear community,
>
> happy to share a short community update this week. With one week to go to
> the planned feature freeze for Flink 1.10 and Flink Forward Asia in Beijing
> the dev@ mailing list pretty quiet these days.
>
> Flink Development
> ==
>
> * [releases] Hequn has started the vote on RC1 for Flink 1.8.3, which
> unfortunately has already received a -1 due to wrong/missing license
> information. Expecting a new RC soon. [1]
>
> * [sql] In the past timestamp fields in Flink SQL were internally
> represented as longs and it was recommended to use longs directly in
> user-defined functions. With the introduction of a new TimestampType the
> situation has changed and conversion between long and TIMESTAMP will be
> disabled. [2]
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-3-release-candidate-1-tp35401p35407.html
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Disable-conversion-between-TIMESTAMP-and-Long-in-parameters-and-results-of-UDXs-tp35269p35271.html
>
> Notable Bugs
> ==
>
> * [FLINK-14930] [1.9.1] The OSS filesystem did not allow the configuration
> of additional credentials providers due to a shading-related bug. Resolved
> for 1.9.2. [3]
>
> [3] https://issues.apache.org/jira/browse/FLINK-14930
>
> Events, Blog Posts, Misc
> ===
>
> * Flink Forward Asia took place this week at the National Congress Center
> in Beijing organized by Alibaba. Talks by Ververica, Tencent, Baidu,
> Alibaba, Dell, Lyft, Netflix, Cloudera and many other members of the
> Chinese Apache Flink community, and more than 1500 attendees as far as I
> heard. Impressive! [4]
>
> * At Flink Forward Asia Alibaba announced it has open sourced Alink, a
> machine learning library on top of Apache Flink[5,6]
>
> * Upcoming Meetups
> * The first meetup of the Apache Flink Meetup Chicago on 5th of
> December comes with four talks(!) highlighting different deployment methods
> of Apache Flink (AWS EMR, AWS Kinesis Analytics, Verveirca Platform, IBM
> Kubernetes). Talks by *Trevor Grant*, *Seth Wiesman*, *Joe Olson* and 
> *Margarita
> Uk*. [7]
> * On December 17th there will be the second Apache Flink meetup in
> Seoul. Maybe Dongwon can share the list of speakers in this thread, my
> Korean is a bit rusty. [8]
>
> [4] https://m.aliyun.com/markets/aliyun/developer/ffa2019
> [5]
> https://technode.com/2019/11/28/alibaba-cloud-machine-learning-platform-open-source/
> [6] https://github.com/alibaba/Alink/blob/master/README.en-US.md
> [7]
> https://www.meetup.com/Chicago-Apache-Flink-Meetup-CHAF/events/266609828/
> [8] https://www.meetup.com/Seoul-Apache-Flink-Meetup/events/266824815/
>
> Cheers,
>
> Konstantin (@snntrable)
>
> --
>
> Konstantin Knauf | Solutions Architect
>
> +49 160 91394525
>
>
> Follow us @VervericaData Ververica <https://www.ververica.com/>
>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Tony) Cheng
>

Re: Wrong result of MATCH_RECOGNIZE clause

[Dongwon Kim]

Oops, I think I explained something wrong in the previous email.
B means not A.
Therefore, after the completed match, there must be no new partial match
starting from there.
There's nothing wrong with the implementation, but the example in [2] is
wrong.

Am I right?

Best,
Dongwon

On Thu, Sep 5, 2019 at 9:15 PM Dongwon Kim  wrote:

> Hi,
> I'm using Flink 1.9 and testing MATCH_RECOGNIZE by following [1].
> While testing the query in [2] on myself, I've got the different result
> from [2]
> The query result from [2] is as follows:
>
>  symbol   start_tstamp   end_tstamp  avgPrice
> =  ==  ==  
> ACME   01-APR-11 10:00:00  01-APR-11 10:00:03 14.5
> ACME   01-APR-11 10:00:04  01-APR-11 10:00:09 13.5
>
> The other query result from the attached maven project (which only
> contains a sample program that executes the query in [2]) is as follows:
>
>> ACME,1970-01-01 00:00:01.0,1970-01-01 00:00:04.0,14.5
>>
> There's just one entry, not two.
> (As you might notice, the time of the first record in the attached maven
> project is 1970-01-01 00:00:01 for testing. The other numbers are the same.)
>
> I dug into the internal implementation of CepOperator and got the
> followings:
>
>1. INPUT : ACME,1000,12.0,1
>2. PARTIAL MATCH : [A*1]
>3.
>4. INPUT : ACME,2000,17.0,2
>5. PARTIAL MATCH : [A*2]
>6.
>7. INPUT : ACME,3000,13.0,1
>8. PARTIAL MATCH : [A*3]
>9. PARTIAL MATCH : [A*1]
>10.
>11. INPUT : ACME,4000,16.0,3
>12. PARTIAL MATCH : [A*4]
>13. PARTIAL MATCH : [A*2]
>14.
>15. *INPUT : ACME,5000,25.0,2*
>16. *COMPLETED MATCH : [A*4, B*1]*
>17.
>18. INPUT : ACME,6000,2.0,1
>19. PARTIAL MATCH : [A*1]
>20.
>21. INPUT : ACME,7000,4.0,1
>22. PARTIAL MATCH : [A*2]
>23. PARTIAL MATCH : [A*1]
>24.
>25. INPUT : ACME,8000,10.0,2
>26. PARTIAL MATCH : [A*3]
>27. PARTIAL MATCH : [A*2]
>28. PARTIAL MATCH : [A*1]
>29.
>30. INPUT : ACME,9000,15.0,2
>31. PARTIAL MATCH : [A*4]
>32. PARTIAL MATCH : [A*3]
>33. PARTIAL MATCH : [A*2]
>34.
>35. INPUT : ACME,1,25.0,2
>36. PARTIAL MATCH : [A*5]
>37. PARTIAL MATCH : [A*4]
>38.
>39. INPUT : ACME,11000,30.0,1
>40. PARTIAL MATCH : [A*6]
>
>
> My observation is that, when "ACME,5000,25.0,2" comes in (line 15), we get
> a completed match (line 16) but no partial match (which is [A*1] in my
> notation) starting from it.
> According to the definition of "AFTER MATCH SKIP TO FIRST B", as
> "ACME,5000,25,2" is B, a new match should start from "ACME,5000,25.0,2".
> However, a new match starts from the next one (line 18, 19) in the above
> trace.
> Therefore, when the last one "ACME,11000,30.0,1" comes in, the average at
> that point is 14.3(=2+4+10+15+25+30/6) which is less than 15
> so "ACME,11000,30.0,1" belongs to A, not B as shown in the example.
>
> Is it a bug? or did I miss something conceptually?
>
> p.s. how do you load rows from a local csv file with rowtime configured? I
> don't like the way I implemented my custom table source in the attached
> file which I use for testing.
>
> Best,
> Dongwon
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#aggregations
>

Wrong result of MATCH_RECOGNIZE clause

[Dongwon Kim]

Hi,
I'm using Flink 1.9 and testing MATCH_RECOGNIZE by following [1].
While testing the query in [2] on myself, I've got the different result
from [2]
The query result from [2] is as follows:

 symbol   start_tstamp   end_tstamp  avgPrice
=  ==  ==  
ACME   01-APR-11 10:00:00  01-APR-11 10:00:03 14.5
ACME   01-APR-11 10:00:04  01-APR-11 10:00:09 13.5

The other query result from the attached maven project (which only contains
a sample program that executes the query in [2]) is as follows:

> ACME,1970-01-01 00:00:01.0,1970-01-01 00:00:04.0,14.5
>
There's just one entry, not two.
(As you might notice, the time of the first record in the attached maven
project is 1970-01-01 00:00:01 for testing. The other numbers are the same.)

I dug into the internal implementation of CepOperator and got the
followings:

   1. INPUT : ACME,1000,12.0,1
   2. PARTIAL MATCH : [A*1]
   3.
   4. INPUT : ACME,2000,17.0,2
   5. PARTIAL MATCH : [A*2]
   6.
   7. INPUT : ACME,3000,13.0,1
   8. PARTIAL MATCH : [A*3]
   9. PARTIAL MATCH : [A*1]
   10.
   11. INPUT : ACME,4000,16.0,3
   12. PARTIAL MATCH : [A*4]
   13. PARTIAL MATCH : [A*2]
   14.
   15. *INPUT : ACME,5000,25.0,2*
   16. *COMPLETED MATCH : [A*4, B*1]*
   17.
   18. INPUT : ACME,6000,2.0,1
   19. PARTIAL MATCH : [A*1]
   20.
   21. INPUT : ACME,7000,4.0,1
   22. PARTIAL MATCH : [A*2]
   23. PARTIAL MATCH : [A*1]
   24.
   25. INPUT : ACME,8000,10.0,2
   26. PARTIAL MATCH : [A*3]
   27. PARTIAL MATCH : [A*2]
   28. PARTIAL MATCH : [A*1]
   29.
   30. INPUT : ACME,9000,15.0,2
   31. PARTIAL MATCH : [A*4]
   32. PARTIAL MATCH : [A*3]
   33. PARTIAL MATCH : [A*2]
   34.
   35. INPUT : ACME,1,25.0,2
   36. PARTIAL MATCH : [A*5]
   37. PARTIAL MATCH : [A*4]
   38.
   39. INPUT : ACME,11000,30.0,1
   40. PARTIAL MATCH : [A*6]


My observation is that, when "ACME,5000,25.0,2" comes in (line 15), we get
a completed match (line 16) but no partial match (which is [A*1] in my
notation) starting from it.
According to the definition of "AFTER MATCH SKIP TO FIRST B", as
"ACME,5000,25,2" is B, a new match should start from "ACME,5000,25.0,2".
However, a new match starts from the next one (line 18, 19) in the above
trace.
Therefore, when the last one "ACME,11000,30.0,1" comes in, the average at
that point is 14.3(=2+4+10+15+25+30/6) which is less than 15
so "ACME,11000,30.0,1" belongs to A, not B as shown in the example.

Is it a bug? or did I miss something conceptually?

p.s. how do you load rows from a local csv file with rowtime configured? I
don't like the way I implemented my custom table source in the attached
file which I use for testing.

Best,
Dongwon

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#aggregations


cepTest.tar.gz
Description: application/gzip

How to shorten MATCH_RECOGNIZE's DEFINE clause

[Dongwon Kim]

Hi,

Flink relational apis with MATCH_RECOGNITION looks very attractive and
promising but it's currently not easy to use at all.

While looking into [1],
especially the following DEFINE clause,

> DEFINE
> PRICE_DOWN AS
> (LAST(PRICE_DOWN.price, 1) IS NULL AND PRICE_DOWN.price < START_ROW.price) OR
> PRICE_DOWN.price < LAST(PRICE_DOWN.price, 1),
> PRICE_UP AS
> PRICE_UP.price > LAST(PRICE_DOWN.price, 1)

what came up to my mind is that the condition is quite verbose.

Meanwhile, I found [2] in which Oracle uses the similar data and the
MATCH_RECOGNIZE clause as follows:

> SELECT *
> FROM Ticker MATCH_RECOGNIZE (
>  PARTITION BY symbol
>  ORDER BY tstamp
>  MEASURES  STRT.tstamp AS start_tstamp,
>LAST(DOWN.tstamp) AS bottom_tstamp,
>LAST(UP.tstamp) AS end_tstamp
>  ONE ROW PER MATCH
>  AFTER MATCH SKIP TO LAST UP
>  PATTERN (STRT DOWN+ UP+)
>  DEFINE
> DOWN AS DOWN.price < PREV(DOWN.price),
> UP AS UP.price > PREV(UP.price)
>  ) MR
> ORDER BY MR.symbol, MR.start_tstamp;
>
> Oracle's way of defining pattern variables looks much simple and neat
thanks to
1) its use of PREV which is not available in Flink's MATCH_RECOGNIZE clause
(mentioned in [3])
2) no need to check the initial case or when there is no last row that met
the PRICE_DOWN condition

Is there a way to shorten the DEFINE cluase in MATCH_RECOGNIZE at Flink 1.8
or 1.9?
If there's any plan regarding this, could anyone let me know related Jira
issues?

Best,
Dongwon

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#introduction-and-examples
[2]  https://docs.oracle.com/database/121/DWHSG/pattern.htm
[3]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html#known-limitations

Re: [DataStream API] Best practice to handle custom session window - time gap based but handles event for ending session

[Dongwon Kim]

Hi Jungtaek,

I've faced a similar problem in the past; we need to calculate an aggregate
upon receiving an end message from each user.

While you're trying to solve problem by defining a custom window assigner,
I took a different approach to the problem by implementing a custom trigger.

You can see my implementation in the following link (but I'm not quite sure
if my implementation could solve your case):
https://github.com/eastcirclek/flink-examples/blob/master/src/main/scala/com/github/eastcirclek/examples/customtrigger/trigger/TrackingEarlyResultEventTimeTrigger.scala

Best,
Dongwon

p.s. FYI, I presented the background of the problem and the general idea
last year at FlinkForward 2017 Berlin. Hope this presentation helps you:
https://www.youtube.com/watch?v=wPQWFy5JENw



On Sun, Aug 4, 2019 at 10:57 PM Jungtaek Lim  wrote:

> Hi Flink users,
>
> I've been spending time to learn and play with Flink DataStream API, not
> an expert level but as a beginner. :)
>
> To play with custom window API, I just created a small example, session
> window based on fixed time gap, but indicate the type of event which may
> contain "end of session". I guess it's not unusual to see this kind of
> things (like manual logout and login) though I don't have concrete real use
> case.
>
> This is an implementation based on Flink DataStream API:
> https://gist.github.com/HeartSaVioR/1d865b1a444af1ef7cae201bbdb196b0
>
> Custom window works pretty well and I could leverage side output very
> easily. One thing leading the code a bit longer was new definition of
> TimeWindow (to deal with event type of "close session"). Even though I
> tried to leverage TimeWindow via inheritance, the code goes a bit verbose
> as I need to implement a new Serializer as well.
> (Actually it required to implement a new Trigger as well, but took
> workaround to leverage existing EventTimeTrigger.)
>
> Assuming this pattern is not unusual (it would be pretty OK if it's
> unusual), could someone point out some points to improve or simplify the
> code? That would be really nice if there's something I could contribute in
> this case.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> ps. This is an implementation based on Spark Structured Streaming (no
> custom window API, so had to put everything in state function of
> flatMapGroupsWithState):
> https://gist.github.com/HeartSaVioR/133c3bdc163f1fd5332397c5cd4b8b29
>

Re: [Table API] ClassCastException when converting a table to DataStream

[Dongwon Kim]

Hi Fabian,

Thanks for clarification :-)
I could convert back and forth without worrying about it as I keep using
Row type during the conversion (even though fields are added).

Best,

Dongwon



On Tue, Jul 23, 2019 at 8:15 PM Fabian Hueske  wrote:

> Hi Dongwon,
>
> regarding the question about the conversion: If you keep using the Row
> type and not adding/removing fields, the conversion is pretty much for free
> right now.
> It will be a MapFunction (sometimes even not function at all) that should
> be chained with the other operators. Hence, it should boil down to a
> function call.
>
> Best, Fabian
>
> Am Sa., 20. Juli 2019 um 03:58 Uhr schrieb Dongwon Kim <
> eastcirc...@gmail.com>:
>
>> Hi Rong,
>>
>> I have to dig deeper into the code to reproduce this error. This seems to
>>> be a bug to me and will update once I find anything.
>>
>> Thanks a lot for spending your time on this.
>>
>> However from what you explained, if I understand correctly you can do all
>>> of your processing within the TableAPI scope without converting it back and
>>> forth to DataStream.
>>> E.g. if your "map(a -> a)" placeholder represents some sort of map
>>> function that's simple enough, you can implement and connect with the table
>>> API via UserDefinedFunction[1].
>>> As TableAPI becoming the first class citizen [2,3,4], this would be much
>>> cleaner implementation from my perspective.
>>
>> I also agree with you in that the first class citizen Table API will make
>> everything not only easier but also a lot cleaner.
>> We however contain some corner cases that force us to covert Table from
>> and to DataStream.
>> One such case is to append to Table a column showing the current
>> watermark of each record; there's no other way but to do that as
>> ScalarFunction doesn't allow us to get the runtime context information as
>> ProcessFunction does.
>>
>> I have a question regarding the conversion.
>> Do I have to worry about runtime performance penalty in case that I
>> cannot help but convert back and fourth to DataStream?
>>
>> Best,
>>
>> Dongwon
>>
>> On Sat, Jul 20, 2019 at 12:41 AM Rong Rong  wrote:
>>
>>> Hi Dongwon,
>>>
>>> I have to dig deeper into the code to reproduce this error. This seems
>>> to be a bug to me and will update once I find anything.
>>>
>>> However from what you explained, if I understand correctly you can do
>>> all of your processing within the TableAPI scope without converting it back
>>> and forth to DataStream.
>>> E.g. if your "map(a -> a)" placeholder represents some sort of map
>>> function that's simple enough, you can implement and connect with the table
>>> API via UserDefinedFunction[1].
>>> As TableAPI becoming the first class citizen [2,3,4], this would be much
>>> cleaner implementation from my perspective.
>>>
>>> --
>>> Rong
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html#scalar-functions
>>> [2]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Table-API-Enhancement-Outline-td25070.html
>>> [3]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td27658.html
>>> [4]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Embracing-Table-API-in-Flink-ML-td25368.html
>>>
>>>
>>> On Thu, Jul 18, 2019 at 8:42 PM Dongwon Kim 
>>> wrote:
>>>
>>>> Hi Rong,
>>>>
>>>> Thank you for reply :-)
>>>>
>>>> which Flink version are you using?
>>>>
>>>> I'm using Flink-1.8.0.
>>>>
>>>> what is the "sourceTable.getSchema().toRowType()" return?
>>>>
>>>> Row(time1: TimeIndicatorTypeInfo(rowtime))
>>>>
>>>> what is the line *".map(a -> a)" *do and can you remove it?
>>>>
>>>> *".map(a->a)"* is just to illustrate a problem.
>>>> My actual code contains a process function (instead of .map() in the
>>>> snippet) which appends a new field containing watermark to a row.
>>>> If there were ways to get watermark inside a scalar UDF, I wouldn't
>>>> convert table to datastream and vice versa.
>>>>
>>>> if I am understanding correctly, you are also using "time1" as the

Re: [Table API] ClassCastException when converting a table to DataStream

[Dongwon Kim]

Hi Rong,

I have to dig deeper into the code to reproduce this error. This seems to
> be a bug to me and will update once I find anything.

Thanks a lot for spending your time on this.

However from what you explained, if I understand correctly you can do all
> of your processing within the TableAPI scope without converting it back and
> forth to DataStream.
> E.g. if your "map(a -> a)" placeholder represents some sort of map
> function that's simple enough, you can implement and connect with the table
> API via UserDefinedFunction[1].
> As TableAPI becoming the first class citizen [2,3,4], this would be much
> cleaner implementation from my perspective.

I also agree with you in that the first class citizen Table API will make
everything not only easier but also a lot cleaner.
We however contain some corner cases that force us to covert Table from and
to DataStream.
One such case is to append to Table a column showing the current watermark
of each record; there's no other way but to do that as ScalarFunction
doesn't allow us to get the runtime context information as ProcessFunction
does.

I have a question regarding the conversion.
Do I have to worry about runtime performance penalty in case that I cannot
help but convert back and fourth to DataStream?

Best,

Dongwon

On Sat, Jul 20, 2019 at 12:41 AM Rong Rong  wrote:

> Hi Dongwon,
>
> I have to dig deeper into the code to reproduce this error. This seems to
> be a bug to me and will update once I find anything.
>
> However from what you explained, if I understand correctly you can do all
> of your processing within the TableAPI scope without converting it back and
> forth to DataStream.
> E.g. if your "map(a -> a)" placeholder represents some sort of map
> function that's simple enough, you can implement and connect with the table
> API via UserDefinedFunction[1].
> As TableAPI becoming the first class citizen [2,3,4], this would be much
> cleaner implementation from my perspective.
>
> --
> Rong
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html#scalar-functions
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Table-API-Enhancement-Outline-td25070.html
> [3]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td27658.html
> [4]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Embracing-Table-API-in-Flink-ML-td25368.html
>
>
> On Thu, Jul 18, 2019 at 8:42 PM Dongwon Kim  wrote:
>
>> Hi Rong,
>>
>> Thank you for reply :-)
>>
>> which Flink version are you using?
>>
>> I'm using Flink-1.8.0.
>>
>> what is the "sourceTable.getSchema().toRowType()" return?
>>
>> Row(time1: TimeIndicatorTypeInfo(rowtime))
>>
>> what is the line *".map(a -> a)" *do and can you remove it?
>>
>> *".map(a->a)"* is just to illustrate a problem.
>> My actual code contains a process function (instead of .map() in the
>> snippet) which appends a new field containing watermark to a row.
>> If there were ways to get watermark inside a scalar UDF, I wouldn't
>> convert table to datastream and vice versa.
>>
>> if I am understanding correctly, you are also using "time1" as the
>>> rowtime, is that want your intension is to use it later as well?
>>
>> yup :-)
>>
>> As far as I know *".returns(sourceTable.getSchema().toRowType());"* only
>>> adds a type information hint about the return type of this operator. It is
>>> used in cases where Flink cannot determine automatically[1].
>>
>> The reason why I specify
>> *".returns(sourceTable.getSchema().toRowType());"* is to give a type
>> information hint as you said.
>> That is needed later when I need to make another table like
>>"*Table anotherTable = tEnv.fromDataStream(stream);"*,
>> Without the type information hint, I've got an error
>>"*An input of GenericTypeInfo cannot be converted to Table.
>> Please specify the type of the input with a RowTypeInfo."*
>> That's why I give a type information hint in that way.
>>
>> Best,
>>
>> Dongwon
>>
>> On Fri, Jul 19, 2019 at 12:39 AM Rong Rong  wrote:
>>
>>> Hi Dongwon,
>>>
>>> Can you provide a bit more information:
>>> which Flink version are you using?
>>> what is the "sourceTable.getSchema().toRowType()" return?
>>> what is the line *".map(a -> a)" *do and can you remove it?
>>> if I am understanding correctly, you are also usin

Re: [Table API] ClassCastException when converting a table to DataStream

[Dongwon Kim]

Hi Rong,

Thank you for reply :-)

which Flink version are you using?

I'm using Flink-1.8.0.

what is the "sourceTable.getSchema().toRowType()" return?

Row(time1: TimeIndicatorTypeInfo(rowtime))

what is the line *".map(a -> a)" *do and can you remove it?

*".map(a->a)"* is just to illustrate a problem.
My actual code contains a process function (instead of .map() in the
snippet) which appends a new field containing watermark to a row.
If there were ways to get watermark inside a scalar UDF, I wouldn't convert
table to datastream and vice versa.

if I am understanding correctly, you are also using "time1" as the rowtime,
> is that want your intension is to use it later as well?

yup :-)

As far as I know *".returns(sourceTable.getSchema().toRowType());"* only
> adds a type information hint about the return type of this operator. It is
> used in cases where Flink cannot determine automatically[1].

The reason why I specify *".returns(sourceTable.getSchema().toRowType());"* is
to give a type information hint as you said.
That is needed later when I need to make another table like
   "*Table anotherTable = tEnv.fromDataStream(stream);"*,
Without the type information hint, I've got an error
   "*An input of GenericTypeInfo cannot be converted to Table. Please
specify the type of the input with a RowTypeInfo."*
That's why I give a type information hint in that way.

Best,

Dongwon

On Fri, Jul 19, 2019 at 12:39 AM Rong Rong  wrote:

> Hi Dongwon,
>
> Can you provide a bit more information:
> which Flink version are you using?
> what is the "sourceTable.getSchema().toRowType()" return?
> what is the line *".map(a -> a)" *do and can you remove it?
> if I am understanding correctly, you are also using "time1" as the
> rowtime, is that want your intension is to use it later as well?
>
> As far as I know *".returns(sourceTable.getSchema().toRowType());"* only
> adds a type information hint about the return type of this operator. It is
> used in cases where Flink cannot determine automatically[1].
>
> Thanks,
> Rong
>
> --
> [1]
> https://github.com/apache/flink/blob/release-1.8/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/SingleOutputStreamOperator.java#L351
>
>
> On Wed, Jul 17, 2019 at 1:29 AM Dongwon Kim  wrote:
>
>> Hello,
>>
>> Consider the following snippet:
>>
>>> Table sourceTable = getKafkaSource0(tEnv);
>>> DataStream stream = tEnv.toAppendStream(sourceTable, Row.class)
>>>
>>> *  .map(a -> a)  .returns(sourceTable.getSchema().toRowType());*
>>> stream.print();
>>>
>> where sourceTable.printSchema() shows:
>>
>>> root
>>>  |-- time1: TimeIndicatorTypeInfo(rowtime)
>>
>>
>>
>>  This program returns the following exception:
>>
>>> Exception in thread "main"
>>> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>> at
>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>> at
>>> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
>>> at
>>> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
>>> at
>>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
>>> at app.metatron.test.Main2.main(Main2.java:231)
>>> *Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be
>>> cast to java.lang.Long*
>>> * at
>>> org.apache.flink.api.common.typeutils.base.LongSerializer.copy(LongSerializer.java:32)*
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:93)
>>> at
>>> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:44)
>>> ...
>>
>>
>> The row serializer seems to try to deep-copy an instance of
>> java.sql.Timestamp using LongSerializer instead of SqlTimestampSerializer.
>> Could anybody help me?
>>
>> Best,
>>
>> - Dongwon
>>
>> p.s. though removing .returns() makes everything okay, I need to do that
>> as I want to convert DataStream into another table later.
>> p.s. the source table is created as follows:
>>
>> private static final Table getKafkaSource0(StreamTableEnvironment tEnv) {
>>> ConnectorDescriptor connectorDescriptor = new Kafka()
>>>   .version("universal")
>>>   .topic("myt

[Table API] ClassCastException when converting a table to DataStream

[Dongwon Kim]

Hello,

Consider the following snippet:

> Table sourceTable = getKafkaSource0(tEnv);
> DataStream stream = tEnv.toAppendStream(sourceTable, Row.class)
>
> *  .map(a -> a)  .returns(sourceTable.getSchema().toRowType());*
> stream.print();
>
where sourceTable.printSchema() shows:

> root
>  |-- time1: TimeIndicatorTypeInfo(rowtime)



 This program returns the following exception:

> Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
> at app.metatron.test.Main2.main(Main2.java:231)
> *Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be
> cast to java.lang.Long*
> * at
> org.apache.flink.api.common.typeutils.base.LongSerializer.copy(LongSerializer.java:32)*
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:93)
> at
> org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:44)
> ...


The row serializer seems to try to deep-copy an instance of
java.sql.Timestamp using LongSerializer instead of SqlTimestampSerializer.
Could anybody help me?

Best,

- Dongwon

p.s. though removing .returns() makes everything okay, I need to do that as
I want to convert DataStream into another table later.
p.s. the source table is created as follows:

private static final Table getKafkaSource0(StreamTableEnvironment tEnv) {
> ConnectorDescriptor connectorDescriptor = new Kafka()
>   .version("universal")
>   .topic("mytopic")
>   .property("bootstrap.servers", "localhost:9092")
>   .property("group.id", "mygroup")
>   .startFromEarliest();
> FormatDescriptor formatDescriptor = new Csv()
>   .deriveSchema()
>   .ignoreParseErrors()
>   .fieldDelimiter(',');
> Schema schemaDescriptor = new Schema()
>   .field("time1", SQL_TIMESTAMP())
>   .rowtime(
> new Rowtime()
>   .timestampsFromField("rowTime")
>   .watermarksPeriodicBounded(100)
>   );
> tEnv.connect(connectorDescriptor)
>   .withFormat(formatDescriptor)
>   .withSchema(schemaDescriptor)
>   .inAppendMode()
>   .registerTableSource("mysrc");
> return tEnv.scan("mysrc");
>   }

Re: How to split tuple2 returned by UDAF into two columns in a result table

[Dongwon Kim]

Another, yet related question:

Is there something like aggregate table function?
In the above scenario, I have to apply an aggregate function and then apply
a table function solely to flatten tuples, which seems quite inefficient.



On Wed, Mar 20, 2019 at 1:09 PM Dongwon Kim  wrote:

> Hi Kurt,
> You're right; It is table function like "mytablefunc(col1, col2, col3) as
> (col4, col5)".
> I've got to define a custom UDTF for that purpose.
> Thanks,
>
> - Dongwon
>
> On Wed, Mar 20, 2019 at 12:04 PM Kurt Young  wrote:
>
>> Hi Dongwon,
>>
>> AFAIK, Flink doesn't support the usage like "myscalar(col1, col2, col3)
>> as (col4, col5)". Am I missing something?
>>
>> If you want to split Tuple2 into two different columns, you can use UDTF.
>>
>> Best,
>> Kurt
>>
>>
>> On Wed, Mar 20, 2019 at 9:59 AM Dongwon Kim 
>> wrote:
>>
>>> Hi,
>>>
>>> I want to split Tuple2 returned by AggregateFunction.getValue into two
>>> different columns in a resultant table.
>>>
>>> Let's consider the following example where myudaf returns
>>> Tuple2:
>>>
>>>   Table table2 = table1
>>>   .window(Slide.over("3.rows").every("1.rows").on("time").as("w"))
>>>   .groupBy("w, name")
>>>   .select("name, myudaf(col1, col2, col3) as (col4, col5)")
>>>
>>> Then table2.printSchema() returns (w/ Flink 1.7.2)
>>>
>>>   root
>>> |-- name: String
>>> |-- col4: Java Tuple2
>>>
>>> whereas my expectation is
>>>
>>>   root
>>> |-- name: String
>>> |-- col4: Boolean
>>> |-- col5: Boolean
>>>
>>> When I define a scalar function which returns Tuple2 and use like
>>> "myscalar(col1, col2, col3) as (col4, col5)", it works as expected.
>>>
>>> Is there a possible way of splitting the tuple into two different
>>> columns in Flink-1.7.2?
>>> If not, do I have to define an additional UDF in order to flatten the
>>> tuple? or there's already one I can make use of?
>>>
>>> - Dongwon
>>
>>

Re: How to split tuple2 returned by UDAF into two columns in a result table

[Dongwon Kim]

Hi Kurt,
You're right; It is table function like "mytablefunc(col1, col2, col3) as
(col4, col5)".
I've got to define a custom UDTF for that purpose.
Thanks,

- Dongwon

On Wed, Mar 20, 2019 at 12:04 PM Kurt Young  wrote:

> Hi Dongwon,
>
> AFAIK, Flink doesn't support the usage like "myscalar(col1, col2, col3) as
> (col4, col5)". Am I missing something?
>
> If you want to split Tuple2 into two different columns, you can use UDTF.
>
> Best,
> Kurt
>
>
> On Wed, Mar 20, 2019 at 9:59 AM Dongwon Kim  wrote:
>
>> Hi,
>>
>> I want to split Tuple2 returned by AggregateFunction.getValue into two
>> different columns in a resultant table.
>>
>> Let's consider the following example where myudaf returns Tuple2> Boolean>:
>>
>>   Table table2 = table1
>>   .window(Slide.over("3.rows").every("1.rows").on("time").as("w"))
>>   .groupBy("w, name")
>>   .select("name, myudaf(col1, col2, col3) as (col4, col5)")
>>
>> Then table2.printSchema() returns (w/ Flink 1.7.2)
>>
>>   root
>> |-- name: String
>> |-- col4: Java Tuple2
>>
>> whereas my expectation is
>>
>>   root
>> |-- name: String
>> |-- col4: Boolean
>> |-- col5: Boolean
>>
>> When I define a scalar function which returns Tuple2 and use like
>> "myscalar(col1, col2, col3) as (col4, col5)", it works as expected.
>>
>> Is there a possible way of splitting the tuple into two different columns
>> in Flink-1.7.2?
>> If not, do I have to define an additional UDF in order to flatten the
>> tuple? or there's already one I can make use of?
>>
>> - Dongwon
>
>

How to split tuple2 returned by UDAF into two columns in a result table

[Dongwon Kim]

Hi,

I want to split Tuple2 returned by AggregateFunction.getValue into two 
different columns in a resultant table.

Let's consider the following example where myudaf returns Tuple2:

  Table table2 = table1
  .window(Slide.over("3.rows").every("1.rows").on("time").as("w"))
  .groupBy("w, name")
  .select("name, myudaf(col1, col2, col3) as (col4, col5)")

Then table2.printSchema() returns (w/ Flink 1.7.2)

  root
|-- name: String
|-- col4: Java Tuple2

whereas my expectation is

  root
|-- name: String
|-- col4: Boolean
|-- col5: Boolean

When I define a scalar function which returns Tuple2 and use like 
"myscalar(col1, col2, col3) as (col4, col5)", it works as expected.

Is there a possible way of splitting the tuple into two different columns in 
Flink-1.7.2?
If not, do I have to define an additional UDF in order to flatten the tuple? or 
there's already one I can make use of?

- Dongwon

RocksDB "shared" directory on HDFS getting bigger

[Dongwon Kim]

Hi,

I'm using RocksDB state backend w/ incremental checkpoint.
hdfs:shared directory is getting bigger
as time goes on.

Does it mean that I forget to clean up state somewhere?

- Dongwon

Re: Service discovery for flink-metrics-prometheus

[Dongwon Kim]

Hi all,

I also suffer from the lack of service discovery for flink-metrics-prometheus 
while using YARN for deployment, Prometheus for instrumentation, and Flink for 
stream processing.
I just upload a Python script for the purpose here: 
https://github.com/eastcirclek/flink-service-discovery 

Hope it can be helpful for your use case.

Best,

- Dongwon


https://github.com/eastcirclek/flink-service-discovery 


> 2018. 1. 8. 오후 7:27, Chesnay Schepler  작성:
> 
> Yes, the logs are the only way to find out which port the reporter is bound 
> to. 
> 
> We may be able to display this information in the web-UI, but it isn't very 
> high on my list and will probably require 
> modifications to the reporter interface. 
> 
> On 06.01.2018 04:24, Kien Truong wrote:
>> Hi,
>> 
>> We are using YARN for deployment, so the combination of host for the 
>> Prometheus reporters can be really random depending on how the containers 
>> are co-located.  
>> 
>> One option we thought of was scrapping the log for this information, but it 
>> can be really messy in the long run.
>> 
>> Regards,
>> Kien
>> 
>> Sent from TypeApp 
>> On Jan 5, 2018, at 03:53, Stephan Ewen > > wrote:
>> How are you running deploying your Flink processes? For Service Discovery 
>> for Prometheus on Kubernetes, there are a few articles out there...
>> 
>> On Thu, Jan 4, 2018 at 3:52 PM, Aljoscha Krettek > > wrote: 
>> I'm not aware of how this is typically done but maybe Chesnay (cc'ed) has an 
>> idea. 
>> 
>> > On 14. Dec 2017, at 16:55, Kien Truong < duckientru...@gmail.com 
>> > > wrote: 
>> > 
>> > Hi, 
>> > 
>> > Does anyone have recommendations about integrating 
>> > flink-metrics-prometheus with some SD mechanism 
>> > 
>> > so that Prometheus can pick up the Task Manager's location dynamically ? 
>> > 
>> > Best regards, 
>> > 
>> > Kien 
>> > 
>> 
>> 
> 

Service discovery for Prometheus on YARN

[Dongwon Kim]

Hi community,

Prior to my presentation [1], I'd like to share a Python script [2] to discover 
Flink clusters on YARN and let Prometheus know via its file-based service 
discovery mechanism [3].

Prometheus needs to pull metrics from Prometheus exporters running inside 
TaskManagers.
The problem is that, as also discussed in [4], we cannot know in advance hosts 
and ports to which Prometheus exporters are bound.
As Chesnay mentioned in [4], unless the information as to which the exporters 
are bound is exposed by Flink's REST APIs, the only way is to manually scrape 
logs of JM and TMs.
Then you need to let Prometheus know where to pull metrics from.

The script is intended to automate the manual process.
Please visit the git repository for the detailed information.

Best,

- Dongwon

[1] 
https://berlin-2018.flink-forward.org/conference-program/#real-time-driving-score-service-using-flink
 

[2] https://github.com/eastcirclek/flink-service-discovery 

[3] 
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E
 

[4] 
http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3cb9b8868a-3026-689a-941f-ac20b42e6...@apache.org%3E
 

Modify EventTimeTrigger only for event-time session windows

[Dongwon Kim]

Hi all,

I want to be sure about when EventTimeTrigger.onEventTime() method is
called with event-time session windows.
It returns TriggerResult.FIRE only when the timestamp of the registered
timer equals to the max timestamp of the current window:

@Override
public TriggerResult onEventTime(long time, TimeWindow window,
TriggerContext ctx) {
return time == window.maxTimestamp() ?
TriggerResult.FIRE :
TriggerResult.CONTINUE;
}

As far as I understand, when EventTimeTrigger is used with event-time
session window, there's no chance of EventTimeTrigger.onEventTime being
called with time != window.maxTimestamp.
Is it true? If not, could anyone let me know the corner case?

The reason why I'm asking is because I want to register an event-time timer
when my custom trigger receives a special event which signifies the end of
a session.
The timestamp of the registered timer is not going to be equal to
window.maxTimestamp and I want to return Trigger.Result.FIRE_AND_PURGE in
such a case.
As I also want to purge the content of a window when it expires,
onEventTime should look like this:

@Override
public TriggerResult onEventTime(long time, TimeWindow window,
TriggerContext ctx) {
return TriggerResult.FIRE_AND_PURGE;
}

It will return FIRE_AND_PURGE every time MyTrigger.onEventTime is called.
It looks quite dangerous and I'm not quite sure about that.

- Dongwon

Re: # of active session windows of a streaming job

[Dongwon Kim]

Hi Fabian and Chesnay,

Thank you guys.

Fabian : Unfortunately, as Chesnay said, MetricGroup doesn't allow for
ProcessWindowFunction to access to a counter defined in Trigger.
Chesnay : I'm going to follow your advice on how to modify Flink. Thank you
very much!

Best,

- Dongwon

On Thu, Jun 21, 2018 at 10:26 PM, Chesnay Schepler 
wrote:

> Without modifications to Flink? No. By design nothing can intercept or
> retrieve metrics with the metrics API.
> For this pattern the usual recommendation is to explicitly pass the metric
> to components that require it.
>
> If modifications are an option, what you could do is
> * define a counter in the OperatorIOMetricGroup
> * have the operator checkpoint/restore the counter,
> * access it in the trigger by casting your way through the MetricGroups to
> an OperatorMetricGroup from which you can retrieve the
> OperatorIOMetricGroup.
>
>
>
> On 21.06.2018 11:16, Fabian Hueske wrote:
>
> Hi Dongwon,
>
> Yes, the counter state should be stored in operator state which is not
> available on Triggers.
> Chesnay: Can a window function (like ProcessWindowFunction) access (read,
> write) the counter of its associated Trigger to checkpoint and restore it?
>
> Best, Fabian
>
> 2018-06-20 16:59 GMT+02:00 Dongwon Kim :
>
>> Hi Fabian and Chesnay,
>>
>> As Chesnay pointed out, it seems that I need to write the current counter
>> (which is defined inside Trigger) into state which I think should be the
>> operator state of the window operator.
>> However, as I previously said, TriggerContext allows for users to access
>> only the partitioned state that are scoped to *the key and* *the window* of
>> the current Trigger invocation.
>> There's no way for me to access to the operator state of the window
>> operator through TriggerContext.
>> The partitioned state doesn't seem suitable as we have more than *ten
>> million keys*.
>> This amount of keys could possibly break down the metric system and the
>> external metric systems like Ganglia and Prometheus.
>>
>> What I want the most is to achieve the goal using the current API (I'm
>> using Flink-1.4.2) without modification.
>> But a change in TriggerContext seems unavoidable because it has to expose
>> an additional method for users like me to access to the operator state of
>> the window operator.
>>
>> Thank you guys for the useful discussion.
>>
>> p.s. Fabian, yes you're right. It is Trigger.clear(), not
>> Trigger.onClose().
>>
>> Best,
>> - Dongwon
>>
>>
>> On Wed, Jun 20, 2018 at 7:30 PM, Chesnay Schepler 
>> wrote:
>>
>>> Checkpointing of metrics is a manual process.
>>> The operator must write the current value into state, retrieve it on
>>> restore and restore the counter's count.
>>>
>>>
>>> On 20.06.2018 12:10, Fabian Hueske wrote:
>>>
>>> Hi Dongwon,
>>>
>>> You are of course right! We need to decrement the counter when the
>>> window is closed.
>>>
>>> The idea of using Trigger.clear() (the clean up method is called clear()
>>> instead of onClose()) method is great!
>>> It will be called when the window is closed but also when it is merged.
>>> So, I think you are right and we only need to increment the counter in
>>> Trigger.onElement() and decrement in Trigger.clear().
>>>
>>> I'm not 100% sure, but I doubt that metrics can checkpointed. Chesnay
>>> (in CC) would know that.
>>> Not sure what would be the best approach if you need a fault tolerant
>>> solution.
>>>
>>> Best, Fabian
>>>
>>>
>>>
>>>
>>> 2018-06-19 16:38 GMT+02:00 Dongwon Kim :
>>>
>>>> Hi Fabian,
>>>> Thanks a lot for your reply.
>>>>
>>>> Do you need to number of active session windows as a DataStream or
>>>>> would you like to have it as a metric that you can expose.
>>>>> I possible, I would recommend to expose it as a metric because they
>>>>> are usually easier to collect.
>>>>
>>>> I want to have it as a metric and it doesn't look difficult thanks to
>>>> the metric system exposed by TriggerContext.
>>>>
>>>> In order to track how many session windows exist, we would need to
>>>>> increment a counter by one when a new window is created (or an element is
>>>>> assigned to a window, which is equivalent for session windows)
>>>>
>>>> I agree with you that we need to increment a counter when
>>>&

Re: # of active session windows of a streaming job

[Dongwon Kim]

Hi Fabian and Chesnay,

As Chesnay pointed out, it seems that I need to write the current counter
(which is defined inside Trigger) into state which I think should be the
operator state of the window operator.
However, as I previously said, TriggerContext allows for users to access
only the partitioned state that are scoped to *the key and* *the window* of
the current Trigger invocation.
There's no way for me to access to the operator state of the window
operator through TriggerContext.
The partitioned state doesn't seem suitable as we have more than *ten
million keys*.
This amount of keys could possibly break down the metric system and the
external metric systems like Ganglia and Prometheus.

What I want the most is to achieve the goal using the current API (I'm
using Flink-1.4.2) without modification.
But a change in TriggerContext seems unavoidable because it has to expose
an additional method for users like me to access to the operator state of
the window operator.

Thank you guys for the useful discussion.

p.s. Fabian, yes you're right. It is Trigger.clear(), not Trigger.onClose().

Best,
- Dongwon


On Wed, Jun 20, 2018 at 7:30 PM, Chesnay Schepler 
wrote:

> Checkpointing of metrics is a manual process.
> The operator must write the current value into state, retrieve it on
> restore and restore the counter's count.
>
>
> On 20.06.2018 12:10, Fabian Hueske wrote:
>
> Hi Dongwon,
>
> You are of course right! We need to decrement the counter when the window
> is closed.
>
> The idea of using Trigger.clear() (the clean up method is called clear()
> instead of onClose()) method is great!
> It will be called when the window is closed but also when it is merged.
> So, I think you are right and we only need to increment the counter in
> Trigger.onElement() and decrement in Trigger.clear().
>
> I'm not 100% sure, but I doubt that metrics can checkpointed. Chesnay (in
> CC) would know that.
> Not sure what would be the best approach if you need a fault tolerant
> solution.
>
> Best, Fabian
>
>
>
>
> 2018-06-19 16:38 GMT+02:00 Dongwon Kim :
>
>> Hi Fabian,
>> Thanks a lot for your reply.
>>
>> Do you need to number of active session windows as a DataStream or would
>>> you like to have it as a metric that you can expose.
>>> I possible, I would recommend to expose it as a metric because they are
>>> usually easier to collect.
>>
>> I want to have it as a metric and it doesn't look difficult thanks to the
>> metric system exposed by TriggerContext.
>>
>> In order to track how many session windows exist, we would need to
>>> increment a counter by one when a new window is created (or an element is
>>> assigned to a window, which is equivalent for session windows)
>>
>> I agree with you that we need to increment a counter when
>> Trigger.onElement() is called due to the characteristic of session windows.
>>
>> and decrement the counter when windows are merged by the number of merged
>>> windows minus one.
>>
>> You decrement the counter when windows are merged, but I think we need
>> to decrement the counter when a window is expired as well.
>>
>> However, decrementing the counter is difficult. Although the
>>> Trigger.onMerge() method is called, it does not know how many windows were
>>> merged (which is done by the WindowAssigner) and only sees the merged
>>> window.
>>
>> We assume that timestamps of records from a user are in ascending order,
>> so only one window is closed at a time which simplifies the problem of how
>> to decrement the counter.
>> Nevertheless, I think I need to decrement the counter in
>> Trigger.onClose(), not Trigger.onMerge().
>> By doing that in Trigger.onClose(), we can take care of both cases: when
>> a window is merged and when a window is expired.
>> How do you think about it?
>>
>> The reason I mention state is to calculate the exact number of active
>> sessions even after my Flink application is restarted from checkpoints or
>> savepoints.
>> If we restore from a savepoint and the counter is initialized to 0, we'll
>> see an incorrect value from a dashboard.
>> This is the biggest concern of mine at this point.
>>
>> Best,
>>
>> - Dongwon
>>
>>
>> On Tue, Jun 19, 2018 at 7:14 PM, Fabian Hueske  wrote:
>>
>>> Hi Dongwon,
>>>
>>> Do you need to number of active session windows as a DataStream or would
>>> you like to have it as a metric that you can expose.
>>> I possible, I would recommend to expose it as a metric because they are
>>> usually easier to collect.
>>>
>>> SessionWind

Re: # of active session windows of a streaming job

[Dongwon Kim]

Hi Fabian,
Thanks a lot for your reply.

Do you need to number of active session windows as a DataStream or would
> you like to have it as a metric that you can expose.
> I possible, I would recommend to expose it as a metric because they are
> usually easier to collect.

I want to have it as a metric and it doesn't look difficult thanks to the
metric system exposed by TriggerContext.

In order to track how many session windows exist, we would need to
> increment a counter by one when a new window is created (or an element is
> assigned to a window, which is equivalent for session windows)

I agree with you that we need to increment a counter when
Trigger.onElement() is called due to the characteristic of session windows.

and decrement the counter when windows are merged by the number of merged
> windows minus one.

You decrement the counter when windows are merged, but I think we need to
decrement the counter when a window is expired as well.

However, decrementing the counter is difficult. Although the
> Trigger.onMerge() method is called, it does not know how many windows were
> merged (which is done by the WindowAssigner) and only sees the merged
> window.

We assume that timestamps of records from a user are in ascending order, so
only one window is closed at a time which simplifies the problem of how to
decrement the counter.
Nevertheless, I think I need to decrement the counter in Trigger.onClose(),
not Trigger.onMerge().
By doing that in Trigger.onClose(), we can take care of both cases: when a
window is merged and when a window is expired.
How do you think about it?

The reason I mention state is to calculate the exact number of active
sessions even after my Flink application is restarted from checkpoints or
savepoints.
If we restore from a savepoint and the counter is initialized to 0, we'll
see an incorrect value from a dashboard.
This is the biggest concern of mine at this point.

Best,

- Dongwon


On Tue, Jun 19, 2018 at 7:14 PM, Fabian Hueske  wrote:

> Hi Dongwon,
>
> Do you need to number of active session windows as a DataStream or would
> you like to have it as a metric that you can expose.
> I possible, I would recommend to expose it as a metric because they are
> usually easier to collect.
>
> SessionWindows work internally as follows:
> - every new record is added to a new window that starts at the timestamp
> of the record and ends at timestamp + gap size. When a record is added to a
> window, Trigger.onElement() is called.
> - after a window was created, the session window assigner tries to merge
> window with overlapping ranges. When windows are merged, Trigger.onMerge()
> is called.
>
> In order to track how many session windows exist, we would need to
> increment a counter by one when a new window is created (or an element is
> assigned to a window, which is equivalent for session windows) and
> decrement the counter when windows are merged by the number of merged
> windows minus one.
>
> Incrementing the counter is rather easy and can be done in
> Trigger.onElement(), either by using state or a Counter metric (Triggers
> have access to the metric system).
> However, decrementing the counter is difficult. Although the
> Trigger.onMerge() method is called, it does not know how many windows were
> merged (which is done by the WindowAssigner) and only sees the merged
> window. There might be a way to maintain state in a Trigger that allows to
> infer how many windows were merged.
>
> Best, Fabian
>
> 2018-06-16 16:39 GMT+02:00 Dongwon Kim :
>
>> Hi Fabian,
>>
>> I'm still eager to expose # of active sessions as a key metric of our
>> service but I haven’t figured it out yet.
>>
>> First of all, I want to ask you some questions regarding your suggestion.
>>
>> You could implement a Trigger that fires when a new window is created and
>> when the window is closed. A ProcessWindowFunction would emit a +1 if the
>> window was created and a -1 when the window is closes.
>> Session windows are a bit special, because you also need to handle the
>> case of merging windows, i.e., two opened windows can be merged and only
>> one (the merged) window is closed. So would need to emit a -2 if a merged
>> window was closes (assuming only two windows were merged).
>>
>> Q1)
>> How to fire when a new window is created and when the window is closed?
>> AFAIK, we can return TriggerResult only through the three functions:
>> onElement, onEventTime, and onProcessingTime.
>> Q2)
>> Firing is to emit elements in windows down to the window function, not
>> emitting values like +1, -1 and -2 which are not in windows.
>> Or do I miss something that you meant?
>>
>> In order to do that, you'd need to carry the merging information 

Re: # of active session windows of a streaming job

[Dongwon Kim]

Hi Fabian,

I'm still eager to expose # of active sessions as a key metric of our service 
but I haven’t figured it out yet.

First of all, I want to ask you some questions regarding your suggestion.
> You could implement a Trigger that fires when a new window is created and 
> when the window is closed. A ProcessWindowFunction would emit a +1 if the 
> window was created and a -1 when the window is closes.
> Session windows are a bit special, because you also need to handle the case 
> of merging windows, i.e., two opened windows can be merged and only one (the 
> merged) window is closed. So would need to emit a -2 if a merged window was 
> closes (assuming only two windows were merged).

Q1) 
How to fire when a new window is created and when the window is closed?
AFAIK, we can return TriggerResult only through the three functions: onElement, 
onEventTime, and onProcessingTime.
Q2)
Firing is to emit elements in windows down to the window function, not emitting 
values like +1, -1 and -2 which are not in windows.
Or do I miss something that you meant?
> In order to do that, you'd need to carry the merging information forward. The 
> Trigger.onMerge method cannot trigger the window function, but it could store 
> the merging information in state that is later accessed.

Q3) 
I didn't understand what you mean at all. What do you mean by carrying the 
merging information?

Besides your suggestion, I implemented a custom trigger which is almost the 
same as EventTimeTrigger except the followings:
- it maintains a variable to count sessions in an instance of a window operator
- it increases the variable by 1 when onElement is invoked
- it decreases the variable by 1 when onClose is invoked
Considering the logic of Flink’s session window, it correctly counts sessions 
in an instance of a window operator. 

As you might have already noticed, this approach has a critical problem: 
there's no way to maintain an operator state inside a trigger. 
TriggerContext only allows to interact with state that is scoped to the window 
and the key of the current trigger invocation (as shown in 
Trigger#TriggerContext)

Now I've come to a conclusion that it might not be possible using DataStream 
API.
Otherwise, do I need to think in a totally different way to achieve the goal?

Best,

- Dongwon



> 2018. 2. 20. 오후 6:53, Fabian Hueske  작성:
> 
> Hi Dongwon Kim,
> 
> That's an interesting question. 
> 
> I don't have a solution blueprint for you, but a few ideas that should help 
> to solve the problem.
> 
> I would start with a separate job first and later try to integrate it with 
> the other job.
> You could implement a Trigger that fires when a new window is created and 
> when the window is closed. A ProcessWindowFunction would emit a +1 if the 
> window was created and a -1 when the window is closes.
> Session windows are a bit special, because you also need to handle the case 
> of merging windows, i.e., two opened windows can be merged and only one (the 
> merged) window is closed. So would need to emit a -2 if a merged window was 
> closes (assuming only two windows were merged).
> In order to do that, you'd need to carry the merging information forward. The 
> Trigger.onMerge method cannot trigger the window function, but it could store 
> the merging information in state that is later accessed.
> 
> Hope this helps,
> Fabian
> 
> 2018-02-20 9:54 GMT+01:00 Dongwon Kim :
>> Hi,
>> 
>> It could be a totally stupid question but I currently have no idea how to 
>> get the number of active session windows from a running job.
>> 
>> Our traffic trajectory application (which handles up to 10,000 tps) uses 
>> event-time session window on KeyedStream (keyed by userID).
>> 
>> Should I write another Flink job for the purpose?
>> 
>> Cheers,
>> 
>> Dongwon Kim

Wrong endpoints to cancel a job

[Dongwon Kim]

Hi,

1.5.0 needs to update its web page for rest APIs.
I'm testing YARN dispatcher and had difficulty canceling jobs today.
I've been sending DELETE requests to dispatcher according to 
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html#cancel-job
 

 but I've got only 404 errors for all day long.
I finally print all end points with their handlers by setting log level to 
DEBUG and it turns out that I have to GET /jobs/:jobid/yarn-cancel to cancel a 
job.

Below is what I summarize based on DEBUG messages of dispatcher.

Method   Endpoint   
  Handler
GET  /blobserver/port   
  o.a.f.runtime.rest.handler.job.BlobServerPortHandler
GET  /config
  o.a.f.runtime.rest.handler.cluster.DashboardConfigHandler
GET  /jars  
  o.a.f.runtime.webmonitor.handlers.JarListHandler
POST /jars/upload   
  o.a.f.runtime.webmonitor.handlers.JarUploadHandler
DELETE   /jars/:jarid   
  o.a.f.runtime.webmonitor.handlers.JarDeleteHandler
GET  /jars/:jarid/plan  
  o.a.f.runtime.webmonitor.handlers.JarPlanHandler
POST /jars/:jarid/run   
  o.a.f.runtime.webmonitor.handlers.JarRunHandler
GET  /jobmanager/config 
  o.a.f.runtime.rest.handler.cluster.ClusterConfigHandler
GET  /jobmanager/log
  
o.a.f.runtime.rest.handler.legacy.files.StaticFileServerHandler
GET  /jobmanager/metrics
  
o.a.f.runtime.rest.handler.job.metrics.JobManagerMetricsHandler
GET  /jobmanager/stdout 
  
o.a.f.runtime.rest.handler.legacy.files.StaticFileServerHandler
GET  /jobs  
  o.a.f.runtime.rest.handler.job.JobIdsHandler
POST /jobs  
  o.a.f.runtime.rest.handler.job.JobSubmitHandler
GET  /jobs/metrics  
  
o.a.f.runtime.rest.handler.job.metrics.AggregatingJobsMetricsHandler
GET  /jobs/overview 
  o.a.f.runtime.rest.handler.job.JobsOverviewHandler
GET  /jobs/:jobid   
  o.a.f.runtime.rest.handler.job.JobDetailsHandler
PATCH/jobs/:jobid   
  o.a.f.runtime.rest.handler.job.JobTerminationHandler
GET  /jobs/:jobid/accumulators  
  o.a.f.runtime.rest.handler.job.JobAccumulatorsHandler
GET  /jobs/:jobid/checkpoints   
  
o.a.f.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler
GET  /jobs/:jobid/checkpoints/config
  
o.a.f.runtime.rest.handler.job.checkpoints.CheckpointConfigHandler
GET  /jobs/:jobid/checkpoints/details/:checkpointid/subtasks/:vertexid  
  
o.a.f.runtime.rest.handler.job.checkpoints.TaskCheckpointStatisticDetailsHandler
GET  /jobs/:jobid/checkpoints/:checkpointid 
  
o.a.f.runtime.rest.handler.job.checkpoints.CheckpointStatisticDetailsHandler
GET  /jobs/:jobid/config
  o.a.f.runtime.rest.handler.job.JobConfigHandler
GET  /jobs/:jobid/exceptions
  o.a.f.runtime.rest.handler.job.JobExceptionsHandler
GET  /jobs/:jobid/execution-result  
  o.a.f.runtime.rest.handler.job.JobExecutionResultHandler
GET  /jobs/:jobid/metrics   
  o.a.f.runtime.rest.handler.job.metrics.JobMetricsHandler
GET  /jobs/:jobid/plan  
  o.a.f.runtime.rest.handler.job.JobPlanHandler
PATCH

Re: Run programs w/ params including comma via REST api

[Dongwon Kim]

Hi Chesnay,

I already modified our application to use semicolon as delimiter and now I can 
run a job using rest API via dispatcher.
Nevertheless, though I understand the difficulty of modifying API, it needs to 
be adjusted as many users are highly likely to use a list of brokers as an 
argument of their streaming applications and use comma to parse the list.
Otherwise, it must be written on the document to warn users to use other 
characters as delimiter for program arguments.

Best,

- Dongwon

> 2018. 4. 25. 오후 5:28, Chesnay Schepler <ches...@apache.org> 작성:
> 
> Currently I don't see a way to circumvent the splitting. You will have to use 
> a different delimiter, I guess a semi-colon could work?
> 
> The code is rather optimistic in that it assumes commas to not occur within a 
> parameter value, and doesn't support any kind of escaping or quoting. (And 
> this is a rabbit hole I'd rather avoid)
> 
> Ultimately I would love to change this call to send the parameters as JSON 
> instead (then you wouldn't have to deal with escaping characters...), but we 
> can't do that until the API versioning is in place (no ETA).
> 
> On 20.04.2018 12:37, Dongwon Kim wrote:
>> Hi,
>> 
>> I'm trying to run a program by sending POST requests.
>> I've already spawned dispatcher in YARN and uploaded a jar file to the 
>> dispatcher.
>> 
>> I want to execute my application using the following arguments (--topic and 
>> --kafkaParams):
>>  --topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245,group.id 
>> <http://group.id/>=trajectory-tracker
>> As you can see, there's a comma in the value of kafkaParams.
>> 
>> When I'm sending my application using the following command from Bash (%20 
>> is a space and $2C is a comma), 
>> bash> curl -X POST 
>> 'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run
>>  
>> <http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run>?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id
>>  <http://group.id/>=trajectory-tracker'
>> 
>> I get the following response from the dispatcher:
>> {
>> "errors": [
>> "Expected only one value [--topic gps-topic --kafkaParams 
>> bootstrap.servers=dacoe2:20245, group.id 
>> <http://group.id/>=trajectory-tracker]."
>> ]
>> }
>> 
>> What I found from the source code is that 
>> org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of 
>> program-ages using comma as a delimiter.
>> 
>> I think I could modify my program to get arguments in a different way but it 
>> is not going to be intuitive to use different characters for a delimiter 
>> instead of comma.
>> 
>> How you guys think?
>> 
>> Or there's a way to avoid this behavior of splitting the value of 
>> program-args into multiple pieces?
>> 
>> best,
>> 
>> - Dongwon
> 

Run programs w/ params including comma via REST api

[Dongwon Kim]

Hi,

I'm trying to run a program by sending POST requests.
I've already spawned dispatcher in YARN and uploaded a jar file to the 
dispatcher.

I want to execute my application using the following arguments (--topic and 
--kafkaParams):
--topic gps-topic --kafkaParams 
bootstrap.servers=dacoe2:20245,group.id=trajectory-tracker
As you can see, there's a comma in the value of kafkaParams.

When I'm sending my application using the following command from Bash (%20 is a 
space and $2C is a comma), 
bash> curl -X POST 
'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id=trajectory-tracker'

I get the following response from the dispatcher:
{
"errors": [
"Expected only one value [--topic gps-topic --kafkaParams 
bootstrap.servers=dacoe2:20245, group.id=trajectory-tracker]."
]
}

What I found from the source code is that 
org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of 
program-ages using comma as a delimiter.

I think I could modify my program to get arguments in a different way but it is 
not going to be intuitive to use different characters for a delimiter instead 
of comma.

How you guys think?

Or there's a way to avoid this behavior of splitting the value of program-args 
into multiple pieces?

best,

- Dongwon

Re: rest.port is reset to 0 by YarnEntrypointUtils

[Dongwon Kim]

Hi Gary,

Thanks a lot for replay.
Hope the issue is resolved soon.

I have a suggestion regarding the rest port.
Considering the role of dispatcher, it needs to have its own port range that is 
not shared by job managers spawned by dispatcher.
If I understand FLIP-6 correctly, only a few dispatchers are going to be 
launched as a kind of gateway and they should be visible outside the cluster.
it seems quite reasonable to give them a range of ports that are only consumed 
by dispatcher.

Best,

- Dongwon



> 2018. 4. 18. 오후 6:40, Gary Yao <g...@data-artisans.com> 작성:
> 
> Hi Dongwon,
> 
> I think the rationale was to avoid conflicts between multiple Flink instances
> running on the same YARN cluster. There is a ticket that proposes to allow
> configuring a port range instead [1]. 
>   
> Best,
> Gary
> 
> [1] https://issues.apache.org/jira/browse/FLINK-5758 
> <https://issues.apache.org/jira/browse/FLINK-5758>
> 
> On Tue, Apr 17, 2018 at 9:56 AM, Dongwon Kim <eastcirc...@gmail.com 
> <mailto:eastcirc...@gmail.com>> wrote:
> Hi,
> 
> I'm trying to launch a dispatcher on top of YARN by executing 
> "yarn-session.sh" on the command line.
> 
> To access the rest endpoint outside the cluster, I need to assign a port from 
> an allowed range.
> 
> YarnEntrypointUtils, however, sets rest.port to 0 for random binding.
> 
> Is there any reason on it?
> 
> 
> Best,
> 
> - Dongwon
> 

rest.port is reset to 0 by YarnEntrypointUtils

[Dongwon Kim]

Hi,

I'm trying to launch a dispatcher on top of YARN by executing "yarn-session.sh" 
on the command line.

To access the rest endpoint outside the cluster, I need to assign a port from 
an allowed range.

YarnEntrypointUtils, however, sets rest.port to 0 for random binding.

Is there any reason on it?


Best,

- Dongwon

Re: Restore from a savepoint is very slow

[Dongwon Kim]

It was due to too low parallelism.
I increase parallelism large enough (actually set it to the total number of 
task slots on the cluster) and it makes restore from a savepoint much faster.

This is somewhat related to the previous discussion I had with Robert and 
Aljoscha.
Having a standalone cluster consisting of 7 taskmanagers, I wanted to schedule 
session window tasks (each of which holds large state in memory) evenly over 7 
taskmanagers.
To that end, I had to set the number of tasks for session window to 7 as a 
single core seems enough for the simple computation logic of our application.
Luckily enough 7 is smaller than 
ExecutionVertex.MAX_DISTINCT_LOCATIONS_TO_CONSIDER which is hard-coded to 8, so 
7 tasks happen to be evenly scheduled on 7 taskmanagers.
However, I found that, when restoring from a savepoint, each of HDFS clients, 
which are 7 session window tasks, reads a large file (operator state) from HDFS 
for a long period of time at 4MB per second.
The slow speed seems to result from a single thread reads and de-seriealize 
each state entry from state stored on HDFS.
So I use up the total number of task slots for a single streaming job on the 
standalone cluster.
Note that, if I set the number of session window tasks to somewhere between 7 
and the total number of task slots, tasks are scheduled on few taskmanagers and 
the taskmanagers are dead due to lack of memory. And I do not have SSDs so I 
prefer FsStateBackend over RocksDBStateBackend.
Of course the standalone cluster cannot be shared across multiple jobs as we 
don't have free slots anymore.

As it seems like GA release of flink-1.5.0 is around the corner, I divert my 
attention to Mesos and Flip-6 for per-job clusters.
One concern is that multiple taskmanagers can be scheduled on the same node on 
Mesos AFAIK. 

@Eron Is it still a not-yet-solved issue? 
If so, do you think it requires a lot of work to add Fenzo's uniqueness 
constraint to Flink's Mesos ResourceManager?
I want to open an issue and figure it out (hopefully with your kind advice).
p.s. this time my Flink application has nothing to do with GPUs.

Best,

- Dongwon

> 2018. 4. 2. 오후 3:33, Dongwon Kim <eastcirc...@gmail.com> 작성:
> 
> Attached is a log file from a taskmanager.
> Please take a look at the log file considering the below events:
> - Around 01:10:47 : the job is submitted to the job manager.
> - Around 01:16:30 : suddenly source starts to read from and sink starts to 
> write data to Kafka
> 
> Any help would be greatly appreciated! T.T
> 
> Best,
> - Dongwon
> 
> 
> 
>> 2018. 4. 2. 오후 2:30, Dongwon Kim <eastcirc...@gmail.com> 작성:
>> 
>> Hi,
>> 
>> While restoring from the latest checkpoint starts immediately after the job 
>> is restarted, restoring from a savepoint takes more than five minutes until 
>> the job makes progress.
>> During the blackout, I cannot observe any resource usage over the cluster.
>> After that period of time, I observe that Flink tries to catch up with the 
>> progress in the source topic via various metrics including 
>> flink_taskmanager_job_task_currentLowWatermark.
>> 
>> FYI, I'm using
>> - Flink-1.4.2
>> - FsStateBackend configured with HDFS
>> - EventTime with BoundedOutOfOrdernessTimestampExtractor
>> 
>> The size of an instance of checkpoint/savepoint is ~50GB and we have 7 
>> servers for taskmanagers.
>> 
>> Best,
>> 
>> - Dongwon
> 

Re: Restore from a savepoint is very slow

[Dongwon Kim]

Attached is a log file from a taskmanager.
Please take a look at the log file considering the below events:
- Around 01:10:47 : the job is submitted to the job manager.
- Around 01:16:30 : suddenly source starts to read from and sink starts to 
write data to Kafka

Any help would be greatly appreciated! T.T

Best,
- Dongwon



tm.log
Description: Binary data


> 2018. 4. 2. 오후 2:30, Dongwon Kim <eastcirc...@gmail.com> 작성:
> 
> Hi,
> 
> While restoring from the latest checkpoint starts immediately after the job 
> is restarted, restoring from a savepoint takes more than five minutes until 
> the job makes progress.
> During the blackout, I cannot observe any resource usage over the cluster.
> After that period of time, I observe that Flink tries to catch up with the 
> progress in the source topic via various metrics including 
> flink_taskmanager_job_task_currentLowWatermark.
> 
> FYI, I'm using
> - Flink-1.4.2
> - FsStateBackend configured with HDFS
> - EventTime with BoundedOutOfOrdernessTimestampExtractor
> 
> The size of an instance of checkpoint/savepoint is ~50GB and we have 7 
> servers for taskmanagers.
> 
> Best,
> 
> - Dongwon

Restore from a savepoint is very slow

[Dongwon Kim]

Hi,

While restoring from the latest checkpoint starts immediately after the job is 
restarted, restoring from a savepoint takes more than five minutes until the 
job makes progress.
During the blackout, I cannot observe any resource usage over the cluster.
After that period of time, I observe that Flink tries to catch up with the 
progress in the source topic via various metrics including 
flink_taskmanager_job_task_currentLowWatermark.

FYI, I'm using
- Flink-1.4.2
- FsStateBackend configured with HDFS
- EventTime with BoundedOutOfOrdernessTimestampExtractor

The size of an instance of checkpoint/savepoint is ~50GB and we have 7 servers 
for taskmanagers.

Best,

- Dongwon