sunank200 commented on code in PR #37101:
URL: https://github.com/apache/airflow/pull/37101#discussion_r1500260607
##########
airflow/datasets/__init__.py:
##########
@@ -106,8 +112,90 @@ class DatasetAny(_DatasetBooleanCondition):
agg_func = any
+ def __init__(self, *objects: Dataset | DatasetAny | DatasetAll) -> None:
+ """Initialize with one or more Dataset, DatasetAny, or DatasetAll
instances."""
+ super().__init__(*objects)
+
+ def __or__(self, other):
+ if isinstance(other, (Dataset, DatasetAny, DatasetAll)):
+ return DatasetAny(*self.objects, other)
+ return NotImplemented
+
+ def __and__(self, other):
+ if isinstance(other, (Dataset, DatasetAny, DatasetAll)):
+ return DatasetAll(self, other)
Review Comment:
I think the current logic holds. Please correct me if I am wrong
@jedcunningham. The `__or__` and `__and__` methods in the `DatasetAny` and
`DatasetAll` classes are based on the intended semantics of "any" (logical OR)
and "all" (logical AND) in the context of dataset dependencies.
### DatasetAny Class
- **__or__ Method**: When using the `__or__` method, if one of the operands
is already a `DatasetAny` instance, we want to combine all its contained
datasets with the new one, effectively extending the logical OR condition.
Hence, `*self.objects, other` is correct because it takes all the existing
datasets in the `DatasetAny` instance and combines them with the `other`
dataset or condition, maintaining the OR relationship across all of them.
- **__and__ Method**: Conversely, when using the `__and__` method on a
`DatasetAny`, combining it with another dataset or condition (regardless of its
type) means we are transitioning from a logical OR to a logical AND
relationship. This is a fundamental shift in logic: we're saying the new
condition requires all previous conditions (represented by `self` as an
existing `DatasetAny` instance) AND the new one to be true. Hence,
`DatasetAll(self, other)` is correct because it constructs a new AND condition
that groups all conditions encapsulated by `self` (which were previously in an
OR relationship) with the `other`.
### DatasetAll Class
- **__or__ Method**: Similarly, for the `DatasetAll` class, when we use
`__or__`, we are shifting from an AND relationship to an OR relationship. If
`self` is a `DatasetAll` instance, it represents a set of datasets that were
all required (AND condition). By using `DatasetAny(self, other)`, we create a
new condition where either all datasets in `self` or the `other` dataset (or
condition) need to be true, effectively transitioning from AND to OR.
- **__and__ Method**: The `__and__` method in `DatasetAll` is used to extend
the existing AND condition with a new dataset or condition. Thus,
`*self.objects, other` is correct because it adds the `other` condition to the
existing set of conditions already within `DatasetAll`, preserving the AND
relationship across all combined datasets.
WDYT? I will add the test for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]