Comments inline On Thu, Apr 18, 2019, 11:10 Kamil Breguła <kamil.breg...@polidea.com> wrote:
> On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer <arthur.wied...@gmail.com> > wrote: > > > Hi Kamil, > > > > Thank you for this doc, it was a lot of good work to put all of this > > together. Could you open the doc for comment? > > May I recommend that once comments have been added to eventually port the > > doc to the wiki and keep trace of the history there as well. > > > > I would prefer not to turn on comments in this document. This will make > it > difficult to manage this document. Turned on comments are also the ability > to add your own suggestions. History of comments is the second reason why I > did turn on it. Saving the discussion history in the comments will be very > hard. The official communication channel is a mailing list. > Sure. But then why not send the entire proposal via email as well to preserve history? If you feel it is too much for email, maybe the wiki would be a better place. In particular, comments in the mailing list might refer to a version of the doc which is not available anymore. We will be missing context eventually. > > > Case #2 A, but I do not think it is backwards compatible as is claimed at > > the top of the document. People will have to update their DAGs. > > > > It is possible to move/rename classes and keep backward compatibility. > Proff: > https://github.com/apache/airflow/pull/4667 > > https://github.com/apache/airflow/blob/master/airflow/contrib/operators/gcs_to_gcs_transfer_operator.py > > https://github.com/apache/airflow/blob/master/airflow/contrib/operators/gcp_transfer_operator.py > Do you have a specific case for which this is not possible? > I started working on this document when I found a way to preserve backwards > compatibility. A detailed description of my observations is included in the > mentioned PR. > I will take a closer look. Thanks for the pointers! > Case #3 I agree about some form of submodules for large cloud providers. It > > sounds helpful. One question: where would I contribute an > > S3_to_GCS_operator in some of the new models? > > > > > "Operators that integrate with two services will not change." > They will be in the package with the remaining operators. The specific > place depends on the adoption of other solutions. > I am mostly worried about dependencies. What if the single integration breaks the multiple integration operators. We do not have this issue yet, but it might crop up once we split them more evenly. > > > Case #5 Are we talking about the class name or the file name? The class > > name is fine to me, but we could remove _operator from the file name. > > > > Case #2 describes the change of file names > Case #5 describes the change of class names > > I added examples to two cases to better describe the changes. > Thank you. I guess I see it for file names, but I am wondering about the operators and name collisions. Say I need both the HiveOperator and I inline a custom operator for which I need a hive hook. # Would you recommmend the following? Or does case #5 only apply to Operators? from airflow.operators.hive import Hive as HiveOperator from airfow.hooks.hive import Hive as HiveHook I was just thinking it might be nice to be able to import what I need without renaming and not worry too much about names shadowing others. Especially if I am new-ish to Apache Airflow. Best, Arthur
