akashsriv07 commented on PR #37223: URL: https://github.com/apache/airflow/pull/37223#issuecomment-1935231856
Hey Hussein, Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same. You can take a look at this blog post though: https://cloud.google.com/blog/products/infrastructure-modernization/google-distributed-cloud-new-ai-and-data-services The CRD for DPGDC is completely different than the sparkoperator.k8s.io/v1beta2, which stops us from leveraging the same operator/sensor. using KRM APIs is one of the ways to interact with a DPGDC cluster. In future, we plan to add the operator similar to dataproc.py(ex: DataprocSubmitSparkJobOperator) which will leverage Google's internal One Platform API mechanism. > I didn't find anything about this in the GCP documentation, could you please add the documentation link? > > Is the CRD `dataprocgdc.cloud.google.com/v1alpha1` based on on `sparkoperator.k8s.io/v1beta2`? I'm asking because we already have two operators for the `spark-on-k8s-operator`, so maybe we can use one of them as a superclass to your operator to avoid code duplication and implementing everything from scratch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
