Good point. I often forget that Amazon do things other than AWS and selling things :)
-a > On 31 Oct 2019, at 12:15, Kaxil Naik <kaxiln...@gmail.com> wrote: > > I think amazon/aws is better for 2 reasons: > > > - Consistent with what we are doing with google > - There can be API services for Amazon platform like > https://developer.amazon.com/apps-and-games/incentives-api | > https://developer.amazon.com/docs/app-submission-api/overview.html etc > > > On Thu, Oct 31, 2019 at 10:44 AM Ash Berlin-Taylor <a...@apache.org> wrote: > >> Personally I would prefer not amazon. - it feels unnecessary as there's >> unlikely to be anything else but amazon.aws >> >> -a >> >> On 31 October 2019 09:29:18 GMT, MinJae Kwon <mingram...@gmail.com> wrote: >>> I love all those rules too :) >>> >>> On 2019/10/30 08:13:27, Bas Harenslak <basharens...@godatadriven.com> >>> wrote: >>>> Hi, >>>> >>>> How about the following conventions? >>>> >>>> * No AWS in whatever notation in the class name, since this is >>> already in the module (airflow.providers.aws….) >>>> * Only exception to above is the AwsHook, which is a “base” hook >>> which most other hooks inherit from >>>> * No “Transfer” in class names, since all transfer operators >>> already have “to” in the name, e.g. RedshiftToS3Transfer becomes >>> RedshiftToS3Operator >>>> * For all class names, the naming convention is >>> {service}{component}, e.g. “AthenaHook" >>>> * For all “transfer” names, the naming convention is >>> {service}To{service}{component}, e.g. “RedshiftToS3Operator" >>>> * Migrate all AWS related components to /airflow/providers/aws/… >>>> * Sidenote: lambda is a reserved Python keyword, so name the >>> Lambda service files "lambda_.py", according to Python conventions to >>> avoid name clashes >>>> >>>> Applying these rules, I get the following changes (done some regex, >>> might have missed a few): >>>> >>>> ./contrib/hooks/aws_dynamodb_hook.py:AwsDynamoDBHook -> >>> ./providers/aws/hooks/dynamodb.py:DynamoDBHook >>>> ./contrib/hooks/aws_firehose_hook.py:AwsFirehoseHook -> >>> ./providers/aws/hooks/firehose.py:FirehoseHook >>>> ./contrib/hooks/aws_glue_catalog_hook.py:AwsGlueCatalogHook -> >>> ./providers/aws/hooks/glue.py:GlueHook >>>> ./contrib/hooks/aws_hook.py:AwsHook -> >>> ./providers/aws/hooks/base/base.py:AwsHook >>>> ./contrib/hooks/aws_lambda_hook.py:AwsLambdaHook -> >>> ./providers/aws/hooks/lambda_.py:LambdaHook >>>> ./contrib/hooks/aws_logs_hook.py:AwsLogsHook -> >>> ./providers/aws/hooks/cloudwatch.py:CloudwatchLogsHook >>>> ./contrib/hooks/aws_sns_hook.py:AwsSnsHook -> >>> ./providers/aws/hooks/sns.py:SnsHook >>>> ./contrib/hooks/aws_sqs_hook.py:SQSHook -> >>> ./providers/aws/hooks/sqs.py:SqsHook >>>> ./contrib/hooks/emr_hook.py:EmrHook -> >>> ./providers/aws/hooks/emr.py:EmrHook >>>> ./contrib/hooks/redshift_hook.py:RedshiftHook -> >>> ./providers/aws/hooks/redshift.py:RedshiftHook >>>> ./contrib/hooks/sagemaker_hook.py:SageMakerHook -> >>> ./providers/aws/hooks/sagemaker.py:SagemakerHook >>>> ./hooks/S3_hook.py:S3Hook -> ./providers/aws/hooks/s3.py:S3Hook >>>> ./providers/aws/hooks/athena.py:AWSAthenaHook -> >>> ./providers/aws/hooks/athena.py:AthenaHook >>>> >>>> ./contrib/operators/aws_sqs_publish_operator.py:SQSPublishOperator -> >>> ./providers/aws/operators/sqs.py:SQSPublishOperator >>>> ./contrib/operators/dynamodb_to_s3.py:DynamoDBToS3Operator -> >>> ./providers/aws/operators/s3.py:DynamoDBToS3Operator >>>> >>> ./contrib/operators/hive_to_dynamodb.py:HiveToDynamoDBTransferOperator >>> -> ./providers/aws/operators/dynamodb.py:HiveToDynamoDBOperator >>>> >> >>> ./contrib/operators/imap_attachment_to_s3_operator.py:ImapAttachmentToS3Operator >>> -> ./providers/aws/operators/s3.py:ImapAttachmentToS3Operator >>>> ./contrib/operators/mongo_to_s3.py:MongoToS3Operator -> >>> ./providers/aws/operators/s3.py:MongoToS3Operator >>>> ./contrib/operators/s3_copy_object_operator.py:S3CopyObjectOperator >>> -> ./providers/aws/operators/s3.py:S3CopyObjectOperator >>>> >>> ./contrib/operators/s3_delete_objects_operator.py:S3DeleteObjectsOperator >>> -> ./providers/aws/operators/s3.py:S3DeleteObjectsOperator >>>> ./contrib/operators/s3_list_operator.py:S3ListOperator -> >>> ./providers/aws/operators/s3.py:S3ListOperator >>>> >>> ./contrib/operators/s3_to_gcs_operator.py:S3ToGoogleCloudStorageOperator >>> -> ./providers/aws/operators/s3.py:S3ToGoogleCloudStorageOperator >>>> ./contrib/operators/s3_to_sftp_operator.py:S3ToSFTPOperator -> >>> ./providers/aws/operators/s3.py:S3ToSFTPOperator >>>> ./contrib/operators/sftp_to_s3_operator.py:SFTPToS3Operator -> >>> ./providers/aws/operators/s3.py:SFTPToS3Operator >>>> ./operators/gcs_to_s3.py:GoogleCloudStorageToS3Operator -> >>> ./providers/aws/operators/s3.py:GoogleCloudStorageToS3Operator >>>> ./operators/google_api_to_s3_transfer.py:GoogleApiToS3Transfer -> >>> ./providers/aws/operators/s3.py:GoogleApiToS3Operator >>>> ./operators/redshift_to_s3_operator.py:RedshiftToS3Transfer -> >>> ./providers/aws/operators/s3.py:RedshiftToS3Operator >>>> ./operators/s3_file_transform_operator.py:S3FileTransformOperator -> >>> ./providers/aws/operators/s3.py:S3FileTransformOperator >>>> ./operators/s3_to_hive_operator.py:S3ToHiveTransfer -> >>> ./providers/aws/operators/s3.py:S3ToHiveOperator >>>> ./operators/s3_to_redshift_operator.py:S3ToRedshiftTransfer -> >>> ./providers/aws/operators/s3.py:S3ToRedshiftOperator >>>> ./providers/aws/operators/athena.py:AWSAthenaOperator -> >>> ./providers/aws/operators/athena.py:AthenaOperator >>>> ./providers/aws/operators/batch.py:AWSBatchOperator -> >>> ./providers/aws/operators/batch.py:BatchOperator >>>> >>>> >> >>> ./contrib/sensors/aws_glue_catalog_partition_sensor.py:AwsGlueCatalogPartitionSensor >>> -> ./providers/aws/sensors/glue.py:GlueCatalogPartitionSensor >>>> >>> ./contrib/sensors/aws_redshift_cluster_sensor.py:AwsRedshiftClusterSensor >>> -> ./providers/aws/sensors/redshift.py:RedshiftClusterSensor >>>> ./contrib/sensors/aws_sqs_sensor.py:SQSSensor -> >>> ./providers/aws/sensors/sqs.py:SQSSensor >>>> ./providers/aws/sensors/athena.py:AthenaSensor -> >>> ./providers/aws/sensors/athena.py:AthenaSensor >>>> ./sensors/s3_key_sensor.py:S3KeySensor -> >>> ./providers/aws/sensors/s3.py:S3KeySensor >>>> ./sensors/s3_prefix_sensor.py:S3PrefixSensor -> >>> ./providers/aws/sensors/s3.py:S3PrefixSensor >>>> >>>> WDYT? >>>> >>>> Bas >>>> >>>> On 30 Oct 2019, at 08:52, Tomasz Urbaszek >>> <tomasz.urbas...@polidea.com<mailto:tomasz.urbas...@polidea.com>> >>> wrote: >>>> >>>> I think it's a great idea and I am willing to help. >>>> >>>> Bests, >>>> Tomek >>>> >>>> On Wed, Oct 30, 2019 at 3:17 AM MinJae Kwon >>> <mingram...@gmail.com<mailto:mingram...@gmail.com>> wrote: >>>> >>>> Hi all. >>>> >>>> While I was working on AIRFLOW-5803, I found there are >>> inconsistencies >>>> between class names for each AWS operators/hooks. >>>> >>>> For example, Athena hook is named as 'AWSAthenaHook', but Dynamodb >>> hook is >>>> named as 'AwsDynamoDBHook', and S3 hook is just 'S3Hook' (So, >>> AIRFLOW-5803 >>>> is trying to rename S3Hook to AWSS3Hook) >>>> >>>> So we might need to have some conventions or guidelines for AWS >>>> integrations like GCP ( >>>> >>> >> https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb72dfd# >>>> ) >>>> >>>> What do you think about it? >>>> >>>> Thank you. >>>> >>>> >>>> >>>> -- >>>> >>>> Tomasz Urbaszek >>>> Polidea <https://www.polidea.com/> | Junior Software Engineer >>>> >>>> M: +48 505 628 493 <+48505628493> >>>> E: tomasz.urbas...@polidea.com<mailto:tomasz.urbas...@polidea.com> >>> <tomasz.urbasz...@polidea.com<mailto:tomasz.urbasz...@polidea.com>> >>>> >>>> Unique Tech >>>> Check out our projects! <https://www.polidea.com/our-work> >>>> >>>> >>
