Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
eladkal closed issue #52920: S3KeySensor in deferrable mode splits key into individual chars URL: https://github.com/apache/airflow/issues/52920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
jroachgolf84 commented on issue #52920: URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045877938 I've created a PR for this issue: https://github.com/apache/airflow/pull/52983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
jroachgolf84 commented on issue #52920: URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045719465 @dominikhei, would you like this assigned? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
vincbeck commented on issue #52920: URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045394274 As far as I know this is not intentional and most likely a bug, please feel free to create a PR :) Thanks for catching this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
dominikhei commented on issue #52920: URL: https://github.com/apache/airflow/issues/52920#issuecomment-3045358491 @vincbeck is the splitting of the key in deferrable mode somehow intentional? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
dominikhei opened a new issue, #52920:
URL: https://github.com/apache/airflow/issues/52920
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
_No response_
### Apache Airflow version
3.0.2
### Operating System
linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
I was trying to replicate the behavior in #52869 and noticed something
interesting.
When executed with `deferable = True` no files are passed to the S3KeySensor
by a check_fn, even though they exist in S3. If `deferable = False`, the
correct files in the bucket will be passed. I used the following function:
```
def check_key_not_present(files: list, **kwargs) -> bool:
print(f"Files received by check_fn: {files}")
return len(files) == 0
```
These are the two tasks:
```
deferrable_sensor = S3KeySensor(
task_id='deferrable_sensor',
bucket_name='#',
bucket_key='scripts/extract-raw-data-job.py',
poke_interval=10,
timeout=60,
check_fn=check_key_not_present,
deferrable=True,
mode="poke",
aws_conn_id='aws_default',
dag=dag,
trigger_rule='none_skipped'
)
non_deferrable_sensor = S3KeySensor(
task_id='non_deferrable_sensor',
bucket_name='##',
bucket_key='scripts/extract-raw-data-job.py',
poke_interval=10,
timeout=60,
check_fn=check_key_not_present,
deferrable=False,
mode="poke",
aws_conn_id='aws_default',
dag=dag,
trigger_rule='none_skipped'
)
```
Logs of non_deferrable_sensor:
`[2025-07-04, 20:24:09] INFO - Files received by check_fn: [{'Size': 363,
'Key': None}]: chan="stdout": source="task"`
Logs of the deferrable_sensor:
`[2025-07-04, 20:23:53] INFO - Files received by check_fn: []:
chan="stdout": source="task"`
After my adjustments:
```INFO - Files received by check_fn: [{'Key': 'xxx',
'LastModified': DateTime(2025, 7, 5, 8, 59, 14, tzinfo=Timezone('UTC')),
'ETag': '"9fe1b02003ad4a0a062ff7df14b36d54"', 'ChecksumAlgorithm':
['CRC64NVME'], 'ChecksumType': 'FULL_OBJECT', 'Size': 209, 'StorageClass':
'STANDARD'}]
```
The problem lies here:
```
async def get_files_async(
self,
client: AioBaseClient,
bucket: str,
bucket_keys: str | list[str],
wildcard_match: bool,
delimiter: str | None = "/",
) -> list[Any]:
"""Get a list of files in the bucket."""
keys: list[Any] = []
for key in bucket_keys:
prefix = key
if wildcard_match:
prefix = re.split(r"[\[*?]", key, 1)[0]
paginator = client.get_paginator("list_objects_v2")
params = {
"Bucket": bucket,
"Prefix": prefix,
"Delimiter": delimiter,
}
```
Why dont we check whether `bucket_keys` is a single string or a list? Is
this somehow intentional? If `bucket_keys` is a single string: `for key in
bucket_keys` will split it into individual chars.
I am opening this issue instead of a direct PR, to validate that this is not
actually intended and I am missing something. Because [this
test](https://github.com/apache/airflow/blob/main/providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py#L718)
makes me question whether this is intended behavior?
### What you think should happen instead
_No response_
### How to reproduce
Create a Dag with two S3 KeySensors, one deferrable and one not and pass a
custom check_fn.
### Anything else
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] S3KeySensor in deferrable mode splits key into individual chars [airflow]
boring-cyborg[bot] commented on issue #52920: URL: https://github.com/apache/airflow/issues/52920#issuecomment-3039810302 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
