its-amann commented on code in PR #76887:
URL: https://github.com/apache/airflow/pull/76887#discussion_r4537278300
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/transfers/s3_to_wasb.py:
##########
@@ -257,9 +257,16 @@ def get_files_to_move(self) -> list[str]:
azure_blob_files.append(self.blob_name.split("/")[-1])
elif self.blob_prefix:
- azure_blob_files += self.wasb_hook.get_blobs_list_recursive(
- container_name=self.container_name, prefix=self.blob_prefix
- )
+ clean_prefix = self.blob_prefix.rstrip("/")
+ prefix_to_strip = f"{clean_prefix}/" if clean_prefix else ""
+ azure_blob_files += [
+ blob.replace(prefix_to_strip, "", 1)
+ if prefix_to_strip and blob.startswith(prefix_to_strip)
+ else blob
+ for blob in self.wasb_hook.get_blobs_list_recursive(
+ container_name=self.container_name,
prefix=self.blob_prefix
+ )
+ ]
Review Comment:
move_file writes `f"{blob_prefix}/{file}"`, so with `blob_prefix="TEST/"`
the blobs this operator uploads itself are `TEST//TEST1.csv`. stripping `TEST/`
leaves `/TEST1.csv`, which never matches, so the next `replace=False` run still
overwrites them. stripping the exact written prefix first covers both
```suggestion
# move_file writes f"{blob_prefix}/{file}", so a prefix
ending in "/" gives "prefix//file"
written_prefix = f"{self.blob_prefix}/"
clean_prefix = f"{self.blob_prefix.rstrip('/')}/"
for blob in self.wasb_hook.get_blobs_list_recursive(
container_name=self.container_name,
prefix=self.blob_prefix
):
name = blob
for prefix in (written_prefix, clean_prefix):
if prefix != "/" and blob.startswith(prefix):
name = blob[len(prefix) :]
break
azure_blob_files.append(name)
```
fixed this plus a test in this commit, you can just cherry pick it instead
of redoing it:
https://github.com/its-amann/airflow/commit/54040362e4d5d195e79756c0fa76856ea70aba6b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
