Awesome, thanks! Will open a new thread. But yes the inprogress file was
helpful.
On Thu, Feb 14, 2019, 7:50 AM Kostas Kloudas Hi Vishal,
>
> For the StreamingFileSink vs Rolling/BucketingSink:
> - you can use the StreamingFileSink instead of the Rolling/BucketingSink.
> You can see the Streami
Hi Vishal,
For the StreamingFileSink vs Rolling/BucketingSink:
- you can use the StreamingFileSink instead of the Rolling/BucketingSink.
You can see the StreamingFileSink as an evolution of the previous two.
In the StreamingFileSink the files in Pending state are not renamed, but
they keep their
Thanks Fabian,
more questions
1. I had on k8s standlone job
env.getCheckpointConfig().setFailOnCheckpointingErrors(true)// the default.
The job failed on chkpoint and I would have imagined that under HA the job
would restore from the last checkpoint but it did not ( The UI showed the
job had res
Hi Vishal,
Kostas (in CC) should be able to help here.
Best, Fabian
Am Mo., 11. Feb. 2019 um 00:05 Uhr schrieb Vishal Santoshi <
vishal.santo...@gmail.com>:
> Any one ?
>
> On Sun, Feb 10, 2019 at 2:07 PM Vishal Santoshi
> wrote:
>
>> You don't have to. Thank you for the input.
>>
>> On Sun, F
Any one ?
On Sun, Feb 10, 2019 at 2:07 PM Vishal Santoshi
wrote:
> You don't have to. Thank you for the input.
>
> On Sun, Feb 10, 2019 at 1:56 PM Timothy Victor wrote:
>
>> My apologies for not seeing your use case properly. The constraint on
>> rolling policy is only applicable for bulk for
You don't have to. Thank you for the input.
On Sun, Feb 10, 2019 at 1:56 PM Timothy Victor wrote:
> My apologies for not seeing your use case properly. The constraint on
> rolling policy is only applicable for bulk formats such as Parquet as
> highlighted in the docs.
>
> As for your questions
My apologies for not seeing your use case properly. The constraint on
rolling policy is only applicable for bulk formats such as Parquet as
highlighted in the docs.
As for your questions, I'll have to defer to others more familiar with it.
I mostly just use bulk formats such as avro and parquet
That said the in the DefaultRollingPolicy it seems the check is on the file
size ( mimics the check shouldRollOnEVent()).
I guess the question is
Is the call to shouldRollOnCheckPoint. done by the checkpointing thread ?
Are the calls to the other 2 methods shouldRollOnEVent and
shouldRollOnPro
Thanks for the quick reply.
I am confused. If this was a more full featured BucketingSink ,I would
imagine that based on shouldRollOnEvent and shouldRollOnEvent, an in
progress file could go into pending phase and on checkpoint the pending
part file would be finalized. For exactly once any files
I think the only rolling policy that can be used is CheckpointRollingPolicy
to ensure exactly once.
Tim
On Sun, Feb 10, 2019, 9:13 AM Vishal Santoshi Can StreamingFileSink be used instead of
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/filesystem_sink.html,
> ev
Can StreamingFileSink be used instead of
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/filesystem_sink.html,
even though it looks it could.
This code for example
StreamingFileSink
.forRowFormat(new Path(PATH),
new Simp
11 matches
Mail list logo