jmcarp commented on a change in pull request #4685: [AIRFLOW-3862] Check types
with mypy.
URL: https://github.com/apache/airflow/pull/4685#discussion_r262799908
##
File path: airflow/contrib/operators/adls_list_operator.py
##
@@ -46,7 +48,7 @@ class AzureDataLakeStorageListOperator(BaseOperator):
azure_data_lake_conn_id='azure_data_lake_default'
)
"""
-template_fields = ('path',)
+template_fields = ('path',) # type: Sequence[str]
Review comment:
We don't have to add type annotations for every operator--we have to add
annotations on operators that are later subclassed. Changing the annotation to
`Iterable[str]` doesn't change this. The issue is that a subclass defining
`template_fields` as e.g. `("path", )` overrides its type from `Sequence[str]`
to `Tuple[str]`; when the subclass of the subclass overrides the value to e.g.
`("src_adls", "dest_gcs")`, its new type of `Tuple[str, str]` is incompatible
with `Tuple[str]`. We could also set the types in `BaseOperator` to be
`List[str]` like we talked about earlier, but that would mean changing all
operators to use lists, which I think we agree we shouldn't do.
Here's a simplified example, using python3-style annotations for convenience:
```python
from typing import Iterable
class A:
a: Iterable[str] = ("a", )
class B(A):
a = ("a", "b")
class C(B):
a = ("a", "b", "c")
```
This isn't valid, because the class variable gets typed as `Tuple[str, str]`
in `B` but `Tuple[str, str, str]` in `C`:
```
error: Incompatible types in assignment (expression has type "Tuple[str,
str, str]", base class "B" defined the type as "Tuple[str, str]")
```
If we annotate `B.a` with `Iterable[str]`, then mypy passes.
Altogether, the annotations we have so far are the simplest solution that I
can think of without a larger refactor: we only have to annotate variables in a
handful of classes (`BaseOperator` and other operators that are later
subclassed), and we don't have to change tuples to lists for all operators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
