Hi Kamil, Thank you for this doc, it was a lot of good work to put all of this together. Could you open the doc for comment? May I recommend that once comments have been added to eventually port the doc to the wiki and keep trace of the history there as well.
Here are my choices Case #1 B or A with some mechanism to "certify" the level of care on an operator (code coverage, whatever). I think that when people come to airflow, they need to know which parts are battle tested versus which parts are more beta status. Case #2 A, but I do not think it is backwards compatible as is claimed at the top of the document. People will have to update their DAGs. Case #3 I agree about some form of submodules for large cloud providers. It sounds helpful. One question: where would I contribute an S3_to_GCS_operator in some of the new models? Case #4 A sounds good, but the transition will need to be done carefully. We have had a hard time splitting up models.py, there will be some interesting pieces there as well. Finally, same question with my S3_to_GCS_operator, where does it fit? Case #5 Are we talking about the class name or the file name? The class name is fine to me, but we could remove _operator from the file name. Case #6 Sure, those look fine to me Case #7 yes to the python native solution. The fewer the number of external dependencies, the best. Another minor point that I would like to correct, the airflow vs airflow/contrib structure was always meant to be based on the amount of testing that the operators had received. It was the case when Airbnb open sourced it. Some Airbnb contributed operators are in airflow/contrib, and some externally contributed operators are in the main airflow folder. The main reason why in the end, operators and hooks did not move was backwards compatibility. We have not had a major version release since the project has been open sourced, as such we were trying (and sometimes inadvertently failing) not to break backwards compatibility. Best, Arthur On Thu, Apr 18, 2019 at 6:58 AM Kamil Breguła <kamil.breg...@polidea.com> wrote: > Hello all, > > During the 4-year project development, the community made many > decisions about the structure and principles of module creation. Some > decisions have been made by Airbnb and they no longer apply. Other > recommendations were rejected because of the low recognition of the > community. > > I have created a document that collects several issues related to > import paths, because I see that this appears in the discussion many > times. > > https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# > However, it is not possible to solve this problem at the level of > individual contributions, but it is necessary to organize at the > project level > > I invite everyone to discuss the topic presented in this document. > > Greetings, > > -- > Kamil Breguła > Polidea | Software Engineer > > M: +48 505 458 451 > E: kamil.breg...@polidea.com > > We create human & business stories through technology. > Check out our projects! >
