Hello Sachin, I am trying to write data to Snowflake using SnowflakeIO Connector. As per https://docs.snowflake.com/en/user-guide/data-load-s3-config, there are three options to configure the access to S3 buckets in Snowflake. 1) Storage integration which avoids the need to provide access key and secret; 2) IAM role; 3) IAM user which needs us to provide the access key and secret. The option of IAM role is deprecated by Snowflake.
As per https://beam.apache.org/documentation/io/built-in/snowflake/, it mentions the usage of storage integration. [image: image.png] In the meantime, it also mentions that we need to provide both keys, which should be related to the option of IAM user. [image: image.png] I googled the usage of SnowflakeIO connector and S3 bucket and then found examples using Python with both keys (access key and secret key). https://github.com/dennisbachmann/beam-s3-reader/tree/master https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592 https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14 Since we are not allowed to create an IAM user in AWS due to security policy, we would like to go with the option of storage integration, I did the test with that and got the error below. Not sure if it was caused by the environment or something else. org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The provided token has expired. (Service: Amazon S3; Status Code: 400; Error Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID: foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=; Proxy: null), S3 Extended Request ID: foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4= I would appreciate it if you could share with me the steps that you followed to do the test using the SnowflakeIO connector and S3 bucket. Thanks. Regards, Xinmin On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <[email protected]> wrote: > Hi, > What are you trying to do ? > Write to snowflake or read from it ? > > Thanks > Sachin > > > On Fri, Dec 1, 2023 at 10:43 AM Xinmin <[email protected]> wrote: > >> Hello Sachin, >> >> You said that you were able to configure SnowflakeIO with S3 bucket. Can >> you please share with me the steps to configure and test it? I would really >> appreciate it. Thanks. >> >> >> Regards, >> Xinmin >> >> >> On Thu, Oct 26, 2023 at 9:42 AM mybeam <[email protected]> wrote: >> >>> Hi Sachin, >>> >>> Thanks for your information. >>> >>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <[email protected]> >>> wrote: >>> >>>> I think AWS is supported and I was able to configure snowflake io with >>>> S3 buckets. >>>> >>>> >>>> >>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> As per the javadoc below, only the GCP bucket is supported currently >>>>> by SnowflakeIO connector? Can you please confirm that AWS and Azure are >>>>> supported now? Thanks. >>>>> >>>>> >>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html >>>>> , >>>>> >>>>
