FileStream sink, where a metadata directory is no longer needed. It may
>>>> require the implementation to go back to the old way of atomic renaming,
>>>> but it will also get rid of the necessity of a metadata directory, so
>>>> someone might find it useful. F
way of atomic renaming,
>>> but it will also get rid of the necessity of a metadata directory, so
>>> someone might find it useful. For end-to-end exactly once, people can
>>> either use a limited current FileStream sink or use Data Lake products. I
>>> don
might find it useful. For end-to-end exactly once, people can
>> either use a limited current FileStream sink or use Data Lake products. I
>> don't see the value in making improvements to the current FileStream sink.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>&
in making improvements to the current FileStream sink.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> On Sun, Apr 16, 2023 at 2:52 AM Wojciech Indyk
> wrote:
>
>> Hi!
>> I raised a ticket on parametrisable output metadata path
>> https://issues.apache.org/j
16, 2023 at 2:52 AM Wojciech Indyk
wrote:
> Hi!
> I raised a ticket on parametrisable output metadata path
> https://issues.apache.org/jira/browse/SPARK-43152.
> I am going to raise a PR against it and I realised, that this relatively
> simple change impacts on method hasMetadata(path)
Hi!
I raised a ticket on parametrisable output metadata path
https://issues.apache.org/jira/browse/SPARK-43152.
I am going to raise a PR against it and I realised, that this relatively
simple change impacts on method hasMetadata(path), that would have a new
meaning if we can define custom path for
