[GitHub] [airflow] jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator
jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8256#discussion_r408517234 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self, input_filename): # serialize args to file +if self.use_dill: +serializer = dill +else: +serializer = pickle +# some args from context can't be loaded in virtual env +invalid_args = set(['dag', 'task', 'ti']) Review comment: Maybe I am using it wrong but I don't get the idea why it is an anti-pattern to exchange data between tasks. I am using this feature really often. I will make the changes on my side but I would love to see your PR merged soon. Thanks a lot for your work :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator
jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8256#discussion_r408456448 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self, input_filename): # serialize args to file +if self.use_dill: +serializer = dill +else: +serializer = pickle +# some args from context can't be loaded in virtual env +invalid_args = set(['dag', 'task', 'ti']) Review comment: I see, it's tricky. Thanks for the detailed answers. I'll try to retrieve the TaskInstance or Xcom directly as a workaround. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator
jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8256#discussion_r408434493 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self, input_filename): # serialize args to file +if self.use_dill: +serializer = dill +else: +serializer = pickle +# some args from context can't be loaded in virtual env +invalid_args = set(['dag', 'task', 'ti']) Review comment: Thanks mik-laj. This means that xcoms are still not working for PythonVirtualenvOperators, right? Can I inject the system site-packages into the virtualenv to get it working? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator
jloehel commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8256#discussion_r408422100 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self, input_filename): # serialize args to file +if self.use_dill: +serializer = dill +else: +serializer = pickle +# some args from context can't be loaded in virtual env +invalid_args = set(['dag', 'task', 'ti']) Review comment: Why is it not possible to load `ti`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
