Re: [D] SAP HANA Provider Package [airflow]
GitHub user TsubasaBE added a comment to the discussion: SAP HANA Provider Package @j6takish Your provider works perfectly! Thanks for the package. You are a life saviour. I'll star your repo. Thanks again. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-12186791 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user TsubasaBE added a comment to the discussion: SAP HANA Provider Package @j6takish Oh. Great! I'll check it out. Thanks. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11925907 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish added a comment to the discussion: SAP HANA Provider Package I published mine. It's linked on the Airflow ecosystem page. Very interested in feedback about missing features etc. Works for everything I need to do internally but I'm sure there are use cases I didn't consider. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11903507 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package > I'm interested in an official HanaDb provider. I'm at the verge of having to > implement one myself for our Airflow instance. Feel free. As explained above we encourage people who are interested in new providers to manage and maintain them outside of Airflow official repo, because we have no expertise and we do not want to carry the burden of maintaining such providers unless they prove to be highly popular. So "at the verge of having to implement one myself" is the way to go. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11902742 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user TsubasaBE added a comment to the discussion: SAP HANA Provider Package I'm interested in an official HanaDb provider. I'm at the verge of having to implement one myself for our Airflow instance. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11901555 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish deleted a comment on the discussion: SAP HANA Provider Package I went the third party provider route. Is it a faux pas to ask in slack for help with integration testing? The hook works the way I expect it to and the way it works internally at my company but I am looking for feedback on whether it works the way anyone other than myself would expect it to. Looking for other people who use HANA in prod All the contributor docs are written as if you were making official contributions, I assume everything would be the same for third party but I want to be sure. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11652922 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish added a comment to the discussion: SAP HANA Provider Package I went the third party provider route. Is it a faux pas to ask in slack for help with integration testing? The hook works the way I expect it to and the way it works internally at my company but I am looking for feedback on whether it works the way anyone other than myself would expect it to. Looking for other people who use HANA in prod All the contributor docs are written as if you were making official contributions, I assume everything would be the same for third party but I want to be sure. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11652922 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package Ideally SAP should maintain their own provider :D GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521960 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk edited a comment on the discussion: SAP HANA Provider Package Posted it in a wrong thread :). Moved the comment to where it belongs GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521948 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package Yeah. I think 3rd-party is the way to go too GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521955 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk deleted a comment on the discussion: SAP HANA Provider Package And this is one of the reason we in Airflow community are reluctant to accept providers that are 3rd-party and the 3rd-party is not committed to supporting it. We do not want to deal with things like that and promise our users things we cannot promise. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11517968 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package Posted it in a wrong thread :) GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521948 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package And this is one of the reason we in Airflow community are reluctant to accept providers that are 3rd-party and the 3rd-party is not committed to supporting it. We do not want to deal with things like that and promise our users things we cannot promise. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521937 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish added a comment to the discussion: SAP HANA Provider Package I have mostly moved away from sqlalchemy for HANA as hdbcli works much better but there are a few things (reflection, inspection) from sqlalchemy that are incredibly convenient. Interestingly, the SAP documentation does direct you over to sqlalchemy, I do not recall them ever acknowledging SA in the past. I feel like they used to push hana-ml, which was their attempt at making their own proprietary pandas, which is very on-brand for SAP. https://help.sap.com/docs/HANA_SERVICE_CF/1efad1691c1f496b8b580064a6536c2d/01e93e584e524747b570cd9083b08d2b.html GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11521014 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package And this is one of the reason we in Airflow community are reluctant to accept providers that are 3rd-party and the 3rd-party is not committed to supporting it. We do not want to deal with things like that and promise our users things we cannot promise. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11517968 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user Joffreybvn added a comment to the discussion: SAP HANA Provider Package Side note: If you use [`sqlalchemy-hana`](https://github.com/SAP/sqlalchemy-hana), beware that **this not an officially supported SAP product** (as stated in their README). Which means SAP can decide to yank all the releases on a friday afternoon (like they did for [`pyrfc`](https://pypi.org/project/pyrfc/#history)). GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11517542 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish added a comment to the discussion: SAP HANA Provider Package Third party is definitely the way to go as I can't imagine there being a ton of HANA/airflow users, I just wanted to gauge the demand for a HANA provider package. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11513188 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user dabla added a comment to the discussion: SAP HANA Provider Package Yes you can and we actually do it ar our company, I’ve done a PR in the past so you can use sqlalchemy with SAP Hana through jdbc GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11512557 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user potiuk added a comment to the discussion: SAP HANA Provider Package @j6takish -> generally approach we have in the community that we are reluctant to maintain something that is used by a handful of our users (or maybe even few handfulls - but not really massive numbers) - and where we - the apache community built around Airlfow - do not have a lot of experience with. We have very high expectations on when we accept new providers for new technologies - simply because we do not want to increase maintenance burden on us - Airflow is developed by most people in their free time, and maintainers are also volunteers and the maintenanc effort to review, merge, fix etc. is on them (while some of them are paid to do other stuff, they are not necessarily paid to maintain technoloigy X or Y integration with Airflow - so they do it effectively in their own free time). That's why we have not only guidelines but also very strict expectations on accepting new providers to be maintained here in the community - I recommend you to read them https://github.com/apache/airflow/blob/main/PROVIDERS.rst#community-managed-providers - there are also some examples of discussions and previous attempts (both successful and unsuccessful) of submitting providers to the community here. It also explains what are the limitations of having comunity provider. One of the important limitations is that the community has to have an easy way to test such provider when things change - if that requires external service like SAP/HANA, that someone must also provide "system test dashboard" where system tests are run against such external service (and pay for it) - and provide public dashboard to the community so that release manager can see that the system tests passed. You can look at the past discussions and decide yourself if you want to go that road - eventually any provider will have to be accepted by formal VOTE on the devlist, after initiall discussion there. And you are also absolutely free to go another route - providers do not have to live here - in airlfow repo, in fact many of them don't https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers - you can easily package and release your own provider in the way and manner you think it is best - you can also add link to it at our "ecosystem" page - we will merge such change gladly. So it's really up to you to see what road you think is best for such SAP/HANA provider. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11512363 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user j6takish added a comment to the discussion: SAP HANA Provider Package It appears that you are interacting with HANA using JDBC and not hdbcli, correct? hdbcli is the official SAP cursor package and only dialect for sqlalchemy. Without using hdbcli, you can't create a sqlalchemy engine, or an inspector, and many operations will be executed sub-optimally GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11511616 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user dabla added a comment to the discussion: SAP HANA Provider Package But we managed to use SAP Hana in Airflow without any modifications, except the ones we already merged into Airflow regarding DbApiHook and GenericTransfer operator. GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11511108 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user dabla added a comment to the discussion: SAP HANA Provider Package There is also another [PR](https://github.com/apache/airflow/pull/41327/) in progress, which introduces the notion of dialects which is independant of the connection type you want to use (native/odbc/jdbc/...) GitHub link: https://github.com/apache/airflow/discussions/44768#discussioncomment-11511091 This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
Re: [D] SAP HANA Provider Package [airflow]
GitHub user dabla added a comment to the discussion: SAP HANA Provider Package
We actually use SAP Hana, but will ditch it very soon, with the SQL hooks and
operators without any issues. For the replace/upsert statements you only need
to pass the 'replace_statement_format' hook parameter, and it should work.
```
transfer_table_task = GenericTransfer(
task_id="transfer_table",
source_conn_id="jdbc_hana_conn",
destination_conn_id="jdbc_hana_conn",
destination_hook_params={"replace_statement_format": "UPSERT {} {}
VALUES ({}) WITH PRIMARY KEY"},
insert_args={
"commit_every": "5000",
"executemany": True,
"fast_executemany": True,
"replace": True,
},
dag=dag,
sql="SELECT * FROM STG_USERS",
destination_table="USERS",
)
```
GitHub link:
https://github.com/apache/airflow/discussions/44768#discussioncomment-11511070
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
