Comments inline On Thu, Apr 18, 2019 at 10:18 PM Arthur Wiedmer <[email protected]> wrote:
> Comments inline > > On Thu, Apr 18, 2019, 11:10 Kamil Breguła <[email protected]> > wrote: > > > On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer <[email protected] > > > > wrote: > > > > > Hi Kamil, > > > > > > Thank you for this doc, it was a lot of good work to put all of this > > > together. Could you open the doc for comment? > > > May I recommend that once comments have been added to eventually port > the > > > doc to the wiki and keep trace of the history there as well. > > > > > > I would prefer not to turn on comments in this document. This will make > > it > > difficult to manage this document. Turned on comments are also the > ability > > to add your own suggestions. History of comments is the second reason > why I > > did turn on it. Saving the discussion history in the comments will be > very > > hard. The official communication channel is a mailing list. > > > > Sure. But then why not send the entire proposal via email as well to > preserve history? If you feel it is too much for email, maybe the wiki > would be a better place. > > In particular, comments in the mailing list might refer to a version of the > doc which is not available anymore. We will be missing context eventually. > > You can not put this document on the mailing list because the document contains advanced formatting. I will try to put this document in the wiki in the near future to the archival purpose. I would like to point out that this is not the final document and contains the final recommendations. I wanted to publish final recommendations as AIP on the wiki in the future. I treat this document as an intermediate step to develop the final document. > > Case #3 I agree about some form of submodules for large cloud providers. > It > > > sounds helpful. One question: where would I contribute an > > > S3_to_GCS_operator in some of the new models? > > > > > > > > "Operators that integrate with two services will not change." > > They will be in the package with the remaining operators. The specific > > place depends on the adoption of other solutions. > > > > I am mostly worried about dependencies. What if the single integration > breaks the multiple integration operators. We do not have this issue yet, > but it might crop up once we split them more evenly. > > I am not sure if I understood you correctly. Changes in imports should not cause pip dependency problems. The changes presented in the document are not intended to divide Airflow into many pip packages. > > > > Case #5 Are we talking about the class name or the file name? The class > > > name is fine to me, but we could remove _operator from the file name. > > > > > > > Case #2 describes the change of file names > > Case #5 describes the change of class names > > > > I added examples to two cases to better describe the changes. > > > > Thank you. I guess I see it for file names, but I am wondering about the > operators and name collisions. > Say I need both the HiveOperator and I inline a custom operator for which I > need a hive hook. > # Would you recommmend the following? Or does case #5 only apply to > Operators? > from airflow.operators.hive import Hive as HiveOperator > from airfow.hooks.hive import Hive as HiveHook > > I was just thinking it might be nice to be able to import what I need > without renaming and not worry too much about names shadowing others. > Especially if I am new-ish to Apache Airflow. > > I think that the operator should describe the behavior(verb), hook should describe a service(noun). These are other parts of speech. In my opinion, HiveOperator should be named HiveExecuteQuery. Hook will be called Hive. Then there will be no name conflicts. > Best, > Arthur > -- Kamil Breguła Polidea <https://www.polidea.com/> | Software Engineer M: +48 505 458 451 <+48505458451> E: [email protected] [image: Polidea] <https://www.polidea.com/> We create human & business stories through technology. Check out our projects! <https://www.polidea.com/our-work> [image: Github] <https://github.com/Polidea> [image: Facebook] <https://www.facebook.com/Polidea.Software> [image: Twitter] <https://twitter.com/polidea> [image: Linkedin] <https://www.linkedin.com/company/polidea> [image: Instagram] <https://instagram.com/polidea> [image: Behance] <https://www.behance.net/polidea>
