tirkarthi commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-2296062775
I just want to add that we have been using yarn rest API to fetch
application status in our custom deferrable operators related to spark on yarn
and it has been working fine. We
tokoko commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1781312438
Depends on what you mean exactly. YARN always exposes rest api, it just
might or might not be accessible from airflow worker because of network
security policies. The address can be
vchiapaikeo commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1779390194
Ooo @tokoko , I just recently learned that Spark master exposes a rest API
with `spark.master.rest.enabled true`. Can you check if that's the case on your
yarn deployment? If p
tokoko commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1776846004
@vchiapaikeo that's right. Full solution is really tricky here, that's why I
chose to keep the implementation in-house, it was mostly just suited to our
environment only :) Running
vchiapaikeo commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1776796804
That'd be great @tokoko - I'm familiar with turning http processes into
deferrable. My concern here is that the
[SparkSubmitHook](https://github.com/apache/airflow/blob/dcb72b5
tokoko commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1775814934
@vchiapaikeo Hey, I went through several iterations of this since this
issue. The version I'm trying to adopt right now is deferrable and uses yarn
rest api rather than yarn cli, be
vchiapaikeo commented on issue #24171:
URL: https://github.com/apache/airflow/issues/24171#issuecomment-1775487585
Happy to give this a shot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
