/ Each job has 3 asynch operators
with Executors with thread counts of 20,20,100/
Flink handles parallelisms for you. If you want a higher parallelism of a
operator, you can call setParallelism()
for example,
flatMap(new Mapper1()).setParallelism(20)
flatMap(new Mapper2()).setParallelism(20)
fla
What about scaling up with #task slots left?
You can obtain this information from Flink's endpoint.
Cheers,
Sendoh
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
it depends on how you partition your file. in my case I write file per hour,
so I'm sure that file is ready after that hour period, in processing time.
Here, read to be ready means this file contains all the data in that hour
period.
If the downstream runs in a batch way, you may want to ensure th
after you keyBy() each of your window has its group of events.
or what you want is a global window?
Best,
Sendoh
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
So there are three ways.
1. make your model as stream source
2. let master read the model once, distribute it via constructor, and update
it periodically
3. let worker read the model and update it periodically(you mentioned)
option 3 would be problematic if you scale a lot and use many parallelis
"Flink will only commit the kafka offsets when the data has been saved to S3"
-> no, you can check the BucketingSink code, and it would mean BucketingSink
depends on Kafka which is not reasonable.
Flink stores checkpoint in disk of each worker, not Kafka.
(KafkaStream, the other streaming API prov
in my case I usually check the tests they write for each function I want to
use.
Take CountTrigger as an example, if I want to customize my own way of
counting, I will have a look at
the test the write
https://github.com/apache/flink/blob/8dfb9d00653271ea4adbeb752da8f62d7647b6d8/flink-streaming
Hi guys,
I found the testing part mentioned
make sources and sinks pluggable in your production code and inject special
test sources and test sinks in your tests.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs
I think it would be useful to have
Hi guys,
I found the testing part mentioned
make sources and sinks pluggable in your production code and inject special
test sources and test sinks in your tests.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs
I think it would be useful to have
After reading the document and configuring to test failure strategy,
it seems to me Flink restarts the job once any failures (e.g.
exception thrown, etc.) occur.
https://ci.apache.org/projects/flink/flink-docs-master/internals/stream_checkpointing.html
My question:
Is it possible to configure in
10 matches
Mail list logo