Restarting of flink master does not change the jobId if one yarn
application.
To be simple, in a yarn application that runs a flink cluster, the job id
of a job does not change once the job is submitted.
You can even submit a flink application multiples times to that cluster (if
it is session mode)
Thanks Till - in the case of restart of flink master - I believe the jobid
will be different. Thanks
On Thursday, May 28, 2020, 11:33:38 AM EDT, Till Rohrmann
wrote:
Hi,
Yarn won't resubmit the job. In case of a process failure where Yarn restarts
the Flink Master, the Master will re
Hi,
Yarn won't resubmit the job. In case of a process failure where Yarn
restarts the Flink Master, the Master will recover the submitted jobs from
a persistent storage system.
Cheers,
Till
On Thu, May 28, 2020 at 4:05 PM M Singh wrote:
> Hi Till/Zhu/Yang: Thanks for your replies.
>
> So just
Hi Till/Zhu/Yang: Thanks for your replies.
So just to clarify - the job id remains same if the job restarts have not been
exhausted. Does Yarn also resubmit the job in case of failures and if so, then
is the job id different.
ThanksOn Wednesday, May 27, 2020, 10:05:40 AM EDT, Till Rohrmann
Hi M,
Sorry I missed your message.
JobID will not change for a generated JobGraph. However, a new JobGraph
will be generated each time a job is submitted.
So that multiple submissions will have multiple JobGraphs. This is because
different submissions are considered as different jobs, as Till ment
Hi,
if you submit the same job multiple times, then it will get every time a
different JobID assigned. For Flink, different job submissions are
considered to be different jobs. Once a job has been submitted, it will
keep the same JobID which is important in order to retrieve the checkpoints
associ
Hi Zhu Zhu:
I have another clafication - it looks like if I run the same app multiple times
- it's job id changes. So it looks like even though the graph is the same the
job id is not dependent on the job graph only since with different runs of the
same app it is not the same.
Please let me kn
Hi Zhu Zhu:
Just to clarify - from what I understand, EMR also has by default restart times
(I think it is 3). So if the EMR restarts the job - the job id is the same
since the job graph is the same.
Thanks for the clarification.
On Monday, May 25, 2020, 04:01:17 AM EDT, Yang Wang
wrote:
Just share some additional information.
When deploying Flink application on Yarn and it exhausted restart policy,
then
the whole application will failed. If you start another instance(Yarn
application),
even the high availability is configured, we could not recover from the
latest
checkpoint becau
Hi M,
Regarding your questions:
1. yes. The id is fixed once the job graph is generated.
2. yes
Regarding yarn mode:
1. the job id keeps the same because the job graph will be generated once
at client side and persist in DFS for reuse
2. yes if high availability is enabled
Thanks,
Zhu Zhu
M Sin
Hi Flink Folks:
If I have a Flink Application with 10 restarts, if it fails and restarts, then:
1. Does the job have the same id ?2. Does the automatically restarting
application, pickup from the last checkpoint ? I am assuming it does but just
want to confirm.
Also, if it is running on AWS EMR I
I have four physical nodes with docker installed on each of them. I configured
Mesos,Flink,Zookeeper,Hadoop and Marathon on docker of each one. I had already
had three nodes,one slave and two masters, that I had run Flink on Marathon and
its UI had been run without any problems. After that, I ch
, 2019 2:13
*To:* User
*Subject:* Apache Flink - Question about metric registry and reporter
and context information
Hi:
I wanted to find if the metric reporter and registry are instantiated
per task manager (which is a single JVM process) or per slot. I
believe it per task manager (JVM process
, June 15, 2019 2:13
To: User
Subject: Apache Flink - Question about metric registry and reporter and context
information
Hi:
I wanted to find if the metric reporter and registry are instantiated per task
manager (which is a single JVM process) or per slot. I believe it per task
manager (JVM
Hi:
I wanted to find if the metric reporter and registry are instantiated per task
manager (which is a single JVM process) or per slot. I believe it per task
manager (JVM process) but just wanted to confirm.
Also, is there a way to access context information (eg: task manager name etc)
in the m
Sounds great Fabian.
I was just trying to see if I can use higher level datastream apis.
I appreciate your advice and help.
Mans
On Monday, April 29, 2019, 5:41:36 AM EDT, Fabian Hueske
wrote:
Hi Mans,
I don't know if that would work or not. Would need to dig into the source c
Hi Mans,
I don't know if that would work or not. Would need to dig into the source
code for that.
TBH, I would recommend to check if you can implement the logic using a
(Keyed-)ProcessFunction.
IMO, process functions are a lot easier to reason about than Flink's
windowing framework.
You can manag
Thanks Sameer/Rong:
As Fabian and you have mentioned, the window still sticks around forever for
global window, so I am trying avoid that scenario.
Fabian & Flink team - do you have any insights into what would happen if I
create a window and the later change it's end time during the stream proc
Hi Mans,
Sameer is correct. if you would like to control window triggering based on
other elements that does not belong to this window (in a keyed stream
context) then this is probably the best way to approach.
I think you've also posted in another thread that describes what will be
left after fi
Global Windows is fine for this use case. I have used the same strategy.
You just define custom evictors and triggers and you are all good. Windows
are managed by keys, so as such as long as events are evicted from the
window, that counts towards reclaiming memory for the key+window
combination. Pl
Hi Rong:
Thanks for your answer.
>From what I understand the dynamic gap session windows are also created when
>the event is encountered. I need to be able to change the window end time at
>a later time based on what other events are in that window. One way to do
>this is to use GlobalWindows
Hi Mans,
I am not sure what you meant by "dynamically change the end-time of a
window. If you are referring to dynamically determines the firing time of
the window, then it fits into the description of session window [1]:
If you want to handle window end time dynamically, one way of which I can
th
Hi:
I am working on a project and need to change the end time of the window
dynamically. I want to find out if the end time of the window is used
internally (for sorting windows/etc) except for handling watermarks that would
cause problems if the end time was changed during run time after the w
Hi,
I think your understanding is correct.
Having multiple map states for a broadcasted stream gives more flexibility.
You can have states of different key and value types and store completely
different information in them.
Fabian
Am Fr., 12. Apr. 2019 um 20:30 Uhr schrieb M Singh :
> Hi Fabi
Hi Fabian: Thanks for your answer.
>From my understanding (please correct me), in the example above, we are
>passing map descriptors to the same broadcast stream. So, the elements/items
>in that stream will be the same. The only difference would be that in the
>processBroadcastElement meth
Hi,
you would simply pass multiple MapStateDescriptors to the broadcast method:
MapStateDescriptor bcState1 = ...
MapStateDescriptor bcState2 = ...
DataStream stream = ...
BroadcastStream bcStream = stream.broadcast(bcState1, bcState2);
Best,
Fabian
Am Mi., 10. Apr. 2019 um 19:44 Uhr schrieb
Hi Guowei;
Thanks for your answer.
Do you have any example which illustrates using broadcast is used with multiple
descriptors ?
Thanks
On Sunday, April 7, 2019, 10:10:15 PM EDT, Guowei Ma
wrote:
Hi1. I think you could use "Using Managed Operator State"[1]
(context.getOperatorState
Hi
1. I think you could use "Using Managed Operator State"[1]
(context.getOperatorStateStore().getBroadcastState()) to use the
BroadCastState. But you must use it very carefully and guarantee the
semantics of broadcast state yourself. I think "The Broadcast State
Pattern"[2] is some best practice
Hi,
I’m not if I understand your question/concerns.
As Rong Rong explained, key selector is used to assign records to window
operators.
Within key context, you do not have access to other keys/values in your
operator/functions, so your reduce/process/… functions when processing key:1
won’t b
Sorry, I still don't understand. Can I ask for help again?
For example, the degree of parallelism is 2, which will produce two Window
threads.
).setParallelism(2)
).These two windows are how to read their own partition data.
).input data
1 2 3 4 5 6 7 8 9 10
).source -> operator -> Rec
Hi
I am not sure if I understand your question correctly, so will try to
explain the flow how elements gets into window operators.
Flink makes the partition assignment before invoking the operator to
process element. For the word count example, WindowOperator is invoked by
StreamInputProcessor[1]
). Environment Flink1.7.2 WordCount local, stream processing
).source RecordWriter.emit(), for each element by key, divided into different
partitions, the partition location of each element has been determined, the
number of partitions is determined by DataStream.setParallelism(2)
). By copyFrom
Hi Steve,
I’m not sure what you mean by “replacing addSource with CSV String data”. Are
your Kinesis records CSV and you want to parse them into Events?
If so, you should be able to do that in the provided DeserializationSchema.
Cheers,
Gordon
On 9 November 2018 at 10:54:22 PM, Steve Bistline
I am having problems with the Flink Kinesis adapter. I have some native KCL
code that works fine. I want to replace the .addSource with the CSV String
data that is coming in from my KCL code. How can I do that?
// Consume the data streams from AWS Kinesis stream
DataStream dataStream = env.addSou
Hi Fabian:
Thanks for your answer - it is starting to make sense to me now.
On Thursday, January 4, 2018 12:58 AM, Fabian Hueske
wrote:
Hi,
the ReduceFunction holds the last emitted record as state. When a new record
arrives, it reduces the new record and last emitted record, updates
Hi,
the ReduceFunction holds the last emitted record as state. When a new
record arrives, it reduces the new record and last emitted record, updates
its state, and emits the new result.
Therefore, a ReduceFunction emits one output record for each input record,
i.e., it is triggered for each input
Hi Stefan:
Thanks for your response.
A follow up question - In a streaming environment, we invoke the operation
reduce and then output results to the sink. Does this mean reduce will be
executed once on every trigger per partition with all the items in each
partition ?
Thanks
On Wednesday
Hi,
I would interpret this as: the reduce produces an output for every new reduce
call, emitting the updated value. There is no need for a window because it
kicks in on every single invocation.
Best,
Stefan
> Am 31.12.2017 um 22:28 schrieb M Singh :
>
> Hi:
>
> Apache Flink documentation
>
Hi:
Apache Flink documentation
(https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html)
indicates that a reduce function on a KeyedStream as follows:
A "rolling" reduce on a keyed data stream. Combines the current element with
the last reduced value and emit
Thanks Aljoscha.
On Sunday, November 26, 2017 11:21 PM, Aljoscha Krettek
wrote:
Hi,
Yes, after the watermark (or processing-time) pass "end-of-window +
allowed-lateness" everything that is stored for a window is deleted.
Best,Aljoscha
On 25. Nov 2017, at 18:07, M Singh wrote:
Hi:
An
Hi,
Yes, after the watermark (or processing-time) pass "end-of-window +
allowed-lateness" everything that is stored for a window is deleted.
Best,
Aljoscha
> On 25. Nov 2017, at 18:07, M Singh wrote:
>
> Hi:
>
> Another question - what happens if the trigger never calls PURGE or
> FIRE_AND_
Hi:
Another question - what happens if the trigger never calls PURGE or
FIRE_AND_PURGE and only calls FIRE ? Are the window and it's contents removed
after the end time + lateness are exceeded ?
Thanks
On Monday, November 20, 2017 2:18 AM, Stefan Richter
wrote:
Hi,
"In the first ca
Hi Mans,
For understanding the difference between FIRE and FIRE_AND_PURGE it's helpful
to look at the cases where it really makes a difference. In my opinion this
only makes a difference when you have event-time windowing and when you have
multiple firing for the same window (i.e. multiple firi
Hi,
>
> "In the first case, it is a new window without the previous elements, in the
> second case the window reflects the old contents plus all changes since the
> last trigger."
>
> I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I
> was thinking that in the first c
Also, Stefan - You mentioned
"In the first case, it is a new window without the previous elements, in the
second case the window reflects the old contents plus all changes since the
last trigger."
I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was
thinking that in th
Thanks Stefan and Aljoscha for your responses.
Stefan - When I mentioned "new window" - I meant the next window being created.
Eg: if the event was in w1 based processing time and the trigger returned FIRE
- then after the window function is computed, what happens to the events in
that window (
Yes, all of this is correct. Sliding windows in fact look like completely
separate windows to the windowing system.
Best,
Aljoscha
> On 16. Nov 2017, at 10:15, Stefan Richter wrote:
>
> Hi,
>
> I think the effect is pretty straight forward, the elements in a window are
> not purged if the tr
Hi,
I think the effect is pretty straight forward, the elements in a window are not
purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your
question is a bit unclear about what exactly you mean by „new window“: a truly
„new“ window or another triggering of the previous (n
Hi Guys
Is there any insight into this ?
Thanks
Mans
On Monday, November 13, 2017 11:19 AM, M Singh wrote:
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method -
the window computation is triggered but element
Hi Xingcan: Thanks for your response.
So to summarize - global windows can be applied to keyed and non keyed windows
- we only have to specify trigger with it to invoke the computation function.
Thanks
Mans
On Wednesday, November 15, 2017 5:43 AM, Xingcan Cui
wrote:
Hi Mans,
the "glob
Hi Mans,
the "global" here indicates the "horizontal" (count, time, etc.) dimension
instead of the "vertical" (keyBy) dimension, i.e., all the received data
will be placed into a single huge window. Actually, it's an orthogonal
concept with the *KeyBy* operations since both *DataStream* and
*Keyed
Hi:
I am reading about global windows and the documentation indicates:
'A global windows assigner assigns all elements with the same key to the same
single global window'
>From my understanding if we have a keyed stream - then all elements with the
>same key are also assigned to a single window.
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method -
the window computation is triggered but elements are kept in the window. If
there a second invocation of the onProcessingTime method will the elements from
the pr
Hello M Singh,
If you check the comment at the beginning of the file, it said
Forwarding elements, watermarks, or status status elements must be protected
by synchronizing
* on the given lock object. This ensures that we don't call methods on a
* {@link OneInputStreamOperator} conc
Thanks Kien/Jorn:
I see the code for processElement being called with a lock but did not see that
the timer based invocation is synchronized by the same lock. Does that mean
that I should use synchonization in my code and how will that impact the
performance ? Please let me know if you have
Be careful though with racing conditions .
> On 12. Nov 2017, at 02:47, Kien Truong wrote:
>
> Hi Mans,
>
> They're not executed in the same thread, but the methods that called them are
> synchronized[1] and therefore thread-safe.
>
> Best regards,
>
> Kien
>
> [1]
> https://github.com/apa
Hi Mans,
They're not executed in the same thread, but the methods that called
them are synchronized[1] and therefore thread-safe.
Best regards,
Kien
[1]
https://github.com/apache/flink/blob/1cd3ba3f2af454bc33f2c880163c014d1738/flink-streaming-java/src/main/java/org/apache/flink/streamin
Hi:
I am working on a project and need to save MapState in a process function and
register a timer to check for updates. I wanted to find out if it is safe to
access and modify the state in the processElement function as well as the time
onTimer methods.
The example
https://ci.apache.org/pro
58 matches
Mail list logo
