pabloem commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-767757877
Hi Reuven - I ran some experiments on various GCS buckets. Here's what I
found
The experiments were done like so:
- A file of 1.6 MB was copied with `rewrite`
- A total of 1MB was copied on every call to `rewrite` (this is configurable
in the call)
- Source bucket: My bucket, without any special policies.
**Experiment 1** - Normal, successful workflow
- Destination bucket: Bucket in different region. No retention policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls
gs://bucket/file`
**Experiment 2** - Call has to be retried
- Destination bucket: Bucket in different region. No retention policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- After running first `rewrite` call again
- Call is successful
- Response contains a **new `rewriteToken`**
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls
gs://bucket/file`
**Experiment 3** - Call has to be retried - bucket has retention policy
- Destination bucket: Bucket in different region. Bucket has retention
policy.
- After first `rewrite` call
- Call is successful
- Response contains a `rewriteToken`
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- After running first `rewrite` call again
- Call is successful
- Response contains a **new `rewriteToken`**
- `done: False` in response
- File does **not** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- After second `rewrite` call
- Call is successful
- `done: True` in response
- File **does** appear in the bucket when performing `gsutil ls
gs://bucket/file`
- Trying to delete file
- Call fails. Unable to delete file.
- Trying to overwrite file with new `rewrite` call
- Call fails. Unable to delete file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
