[ https://issues.apache.org/jira/browse/AIRFLOW-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621941#comment-16621941 ]
Christopher Fei commented on AIRFLOW-2902: ------------------------------------------ I do S3 to BigQuery in production and would be happy to help contribute my learnings. However in my implementation, I use two operators and I do rely on GCS as an intermediate store. I found this to be a more suitable solution for large datasets because you can leverage Google's managed [Storage Transfer|https://cloud.google.com/storage-transfer/docs/overview] service. > Add S3ToBigQuery operator > ------------------------- > > Key: AIRFLOW-2902 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2902 > Project: Apache Airflow > Issue Type: Wish > Components: operators > Affects Versions: 1.10.0 > Reporter: jack > Priority: Major > > Please add operators that allow to move data between Amazon and Google > services. > I saw there is > [S3ToHiveTransfer|https://airflow.apache.org/integration.html#s3tohivetransfer] > operator... It described as: > "Moves data from S3 to Hive. The operator downloads a file from S3, stores > the file locally before loading it into a Hive table." > > So what I'm asking here is very similar. -- This message was sent by Atlassian JIRA (v7.6.3#76005)