hussein-awala commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1939739020
> Hey Hussein, Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same. You can take a look at this blog post though: https://cloud.google.com/blog/products/infrastructure-modernization/google-distributed-cloud-new-ai-and-data-services It will be a bit complicated to review this PR without a doc, I will try to review the syntax and check if our conventions are respected. (cc: @eladkal could you take a look?) > The CRD for DPGDC is completely different than the sparkoperator.k8s.io/v1beta2, which stops us from leveraging the same operator/sensor. using KRM APIs is one of the ways to interact with a DPGDC cluster. In future, we plan to add the operator similar to dataproc.py(ex: DataprocSubmitSparkJobOperator) which will leverage Google's internal One Platform API mechanism. No problem with that, my goal was to reduce code duplication if possible, but since they are different, we can keep building your operator from scratch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
