MatrixManAtYrService opened a new issue #15789:
URL: https://github.com/apache/airflow/issues/15789
**Apache Airflow version**
13faa6912f7cd927737a1dc15630d3bbaf2f5d4d
**Environment**
- **Configuration**: Local Executor
- **OS** (e.g. from /etc/os-release): Mac OS 11.3
- **Kernel**: Darwin Kernel Version 20.4.0
- **Install tools**: `pip install -e .`
**The DAG**
```python
def callable():
print("hi")
with DAG(dag_id="two_virtualenv") as dag:
a = PythonOperator(
task_id="a",
python_callable=callable,
)
# b = PythonOperator( # works
b = PythonVirtualenvOperator( # doesn't work
task_id="b",
python_callable=callable,
)
a >> b
```
**What happened**:
Failure somewhere between first task and second:
```
INFO - Marking task as SUCCESS. dag_id=two_virtualenv, task_id=a
ERROR - Failed to execute task: cannot pickle 'module' object.
Traceback (most recent call last):
File "/Users/matt/src/airflow/airflow/executors/debug_executor.py", line
79, in _run_task
ti._run_raw_task(job_id=ti.job_id, **params) # pylint:
disable=protected-access
File "/Users/matt/src/airflow/airflow/utils/session.py", line 70, in
wrapper
return func(*args, session=session, **kwargs)
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 1201,
in _run_raw_task
self._run_mini_scheduler_on_child_tasks(session)
File "/Users/matt/src/airflow/airflow/utils/session.py", line 67, in
wrapper
return func(*args, **kwargs)
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 1223,
in _run_mini_scheduler_on_child_tasks
partial_dag = self.task.dag.partial_subset(
File "/Users/matt/src/airflow/airflow/models/dag.py", line 1490, in
partial_subset
dag.task_dict = {
File "/Users/matt/src/airflow/airflow/models/dag.py", line 1491, in
<dictcomp>
t.task_id: copy.deepcopy(t, {id(t.dag): dag}) # type: ignore
File
"/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py",
line 153, in deepcopy
y = copier(memo)
File "/Users/matt/src/airflow/airflow/models/baseoperator.py", line 961,
in __deepcopy__
setattr(result, k, copy.deepcopy(v, memo)) # noqa
File
"/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py",
line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle 'module' object
ERROR - Task instance <TaskInstance: two_virtualenv.a 2021-05-11
00:00:00+00:00 [failed]> failed
```
**What you expected to happen**:
Both tasks say "hi" and succeed
**To Replicate**
The DAG and output above are shortened for brevity. A more complete story:
https://gist.github.com/MatrixManAtYrService/6b27378776470491eb20b60e01cfb675
Ran it like this:
```
$ airflow dags test two_virtualenv $(date "+%Y-%m-%d")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
