[
https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801459#comment-16801459
]
jack commented on AIRFLOW-3489:
-------------------------------
The Operator has the code that convert the Json to
{color:#032f62}newline-delimited{color}
[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py#L160]
However the PostgreSQL operator doesn't have this documented. In says:
{code:java}
Copy data from Postgres to Google Cloud Storage in JSON format.{code}
While for example the MySQL operator says:
{code:java}
Copy data from MySQL to Google cloud storage in JSON or CSV format.
The JSON data files generated are newline-delimited to enable them to be loaded
into BigQuery.{code}
[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py#L42]
Please check if this is a code issue (bug) or a documentation issue.
> PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-3489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3489
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Duan Shiqiang
> Priority: Major
>
> PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb)
> as native python types (i.e. dictionary) to gcs new line separated json data.
> But it generates bigquery schema for that field as data type string which
> won't work if user want to import the gcs data into bigquery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
