Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen Thanks! Looking forward~
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
@sihuazhou I got caught up in some other tasks - will try to get back to
this here soon, I would like to have this feature in as a base for "search for
completed checkpoint".
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Could anybody have a look at this?
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen I've addressed your comments, could you please have a look
again?
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Yes, @StephanEwen thanks for the continuous suggestions, will follow your
suggestion.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
I think we need to have a special output stream type
(`AtomicCreatingFsDataOutputStream` or similar) as the return type of
`FileSystem.createAtomic()`. Otherwise, how can a user actually create a
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen I guess this PR is already for an another look now...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
My gut feeling is that we don't need `WriteMode.OVERWRITE` in cases where
one wants such an atomic file creation...
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen Thanks for your good suggestions! Will update PR like it, and
what about the problem related to WriteMode.OVERWRITE, do you against if we
don't support it in `createAtomically()`?
--
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
Good point about the renaming on `close()` in case close is called for
cleanup, rather than success.
We could follow the same semantics as in
[CheckpointStateOutputStream](https://github
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi, I met a problem here, for the Hadoop FileSystem when the
`schema='hdfs'`, we can't rename a file to a existing file. This means that it
hard(I'm not sure whether it's possible) to support the `
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi @StephanEwen, I added more tests for the
`FileSystem#createAtomically()`, concerning the `TwoPhaseFsDataoutputStream`,
can we introduce a `commit_on_close` option for it to make it easier to use
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen After thinking your comments again, I think I misunderstood
the `Also, we need this method to be implemented in all FileSystem subclasses.`
ð , I will address that. But the questio
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi @StephanEwen Thank you very much for your reply, I'm not sure whether
just overriding `close()` to do `super.close()` + `rename()` is enough, for
example.
```
try (outputStream = new Two
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
Thanks for preparing this.
I looked at the `TwoPhraseFSDatautputStream` - maybe we can make this
simpler. Do we need the distinction between phases? Is it not enough to behave
as a regul
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi @StephanEwen I have update the PR according to the above comments, it's
ready for an another review.
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi @StephanEwen,
- I prefer to introduce the `create(Path, WriteMode)`, because I feel this
is more extensible (it could allow us to introduce other WriteMode's in the
future).
- I would ch
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
How about adding the method `createAtomically` or so, with otherwise the
same signature as the `create(Path, WriteMode)` method?
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
@StephanEwen Thank you very much for your great suggestion! I will address
this that way.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
I think this fix here might not work for S3, because a rename() with the S3
file systems will actually trigger a copy (or even a download and upload), so
it is not a cheap operation.
The
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi @StephanEwen Could you please have a look at this?
---
21 matches
Mail list logo
