Re: Java based AWS IO connector

[Alexey Romanenko]

Hi Sahayaraj,

Yes, there is a module “beam-sdks-java-io-amazon-web-services” which can help 
with this. 

Also, I’d suggest you to take a look on this example which reads data from S3 
bucket:
https://github.com/jbonofre/beam-samples/blob/master/amazon-web-services/src/main/java/org/apache/beam/samples/ingest/amazon/IngestToS3.java
 

WBR,
Alexey

> On 29 May 2018, at 15:26, S. Sahayaraj  wrote:
> 
> Hello All,
> The data source for our Beam pipleline is in S3 bucket, Is 
> there any built-in I/O Connector available with Java samples? If so, can you 
> please guide me how to integrate with them?.
>  
> I am using Bean SDK for Java version 2.4.0 and Spark runner 
> in clustered deployment.
>  
> 
>   org.apache.beam
>   beam-sdks-java-core
>   2.4.0
> 
>  
> Cheers,
> S. Sahayaraj

Re: Java based AWS IO connector

[Eugene Kirpichov]

Beam works with S3 out of the box, you can provide s3://... paths to
anything that works with files. Don't remember if it's already available in
2.4 or only starting 2.5 - does it currently not work for you?

On Tue, May 29, 2018, 2:26 PM S. Sahayaraj  wrote:

> Hello All,
>
> The data source for our Beam pipleline is in S3 bucket, Is
> there any built-in I/O Connector available with Java samples? If so, can
> you please guide me how to integrate with them?.
>
>
>
> I am using Bean SDK for Java version 2.4.0 and Spark
> runner in clustered deployment.
>
>
>
> 
>
>   org.apache.beam
>
>   beam-*sdks*-java-core
>
>   2.4.0
>
> 
>
>
>
> Cheers,
>
> S. Sahayaraj
>

Java based AWS IO connector

[S. Sahayaraj]

Hello All,
The data source for our Beam pipleline is in S3 bucket, Is 
there any built-in I/O Connector available with Java samples? If so, can you 
please guide me how to integrate with them?.

I am using Bean SDK for Java version 2.4.0 and Spark runner in 
clustered deployment.


  org.apache.beam
  beam-sdks-java-core
  2.4.0


Cheers,
S. Sahayaraj