[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324480637 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324476450 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324467572 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324429558 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
