hmm, i do not believe security group can control s3 bucket access... is this something new? or you mean IAM role?
@Daniel, using spark on EMR, you should be able to use IAM role to access AWS resources, you do not need to specify fs.s3a.access.key or fs.s3a.secret.key at all. S3A is able to use IAM role for the EC2 instances of EMR cluster. then, for accessing "S3 buckets in multiple accounts", you need following two steps: 1) define your policies of IAM role with Get/Put permissions for all of your s3 bucket's ARN uri, such as something like this: https://github.com/zalando-incubator/ro2key/blob/master/policy_bucket_readonly.json 2) you need to add this IAM role's ARN with Get/Put permissions in all the "s3 bucket policy" in your other accounts. refer to "Granting cross-account bucket access to a specific IAM role" from https://blogs.aws.amazon.com/security/post/TxK5WUJK3DG9G8/How-to-Restrict-Amazon-S3-Bucket-Access-to-a-Specific-IAM-Role Then your cross account s3 access should work. and nice to read this part: When to use IAM policies vs. S3 policies from https://blogs.aws.amazon.com/security/post/TxPOJBY6FE360K/IAM-policies-and-Bucket-Policies-and-ACLs-Oh-My-Controlling-Access-to-S3-Resourc 2016-09-28 10:33 GMT+02:00 Eike von Seggern <eike.segg...@sevenval.com>: > Hi Daniel, > > you can start your EMR Cluster in a dedicated security group and configure > the foreign bucket's policy to allow read-write access from that SG. > > Best > > Eike > > 2016-09-27 16:53 GMT+02:00 Daniel Siegmann <dsiegm...@securityscorecard.io > >: > >> I am running Spark on Amazon EMR and writing data to an S3 bucket. >> However, the data is read from an S3 bucket in a separate AWS account. >> Setting the fs.s3a.access.key and fs.s3a.secret.key values is sufficient to >> get access to the other account (using the s3a protocol), however I then >> won't have access to the S3 bucket in the EMR cluster's AWS account. >> >> Is there any way for Spark to access S3 buckets in multiple accounts? If >> not, is there any best practice for how to work around this? >> >> -- >> Daniel Siegmann >> Senior Software Engineer >> *SecurityScorecard Inc.* >> 214 W 29th Street, 5th Floor >> New York, NY 10001 >> >> > > > -- > ------------------------------------------------ > *Jan Eike von Seggern* > Data Scientist > ------------------------------------------------ > *Sevenval Technologies GmbH * > > FRONT-END-EXPERTS SINCE 1999 > > Köpenicker Straße 154 | 10997 Berlin > > office +49 30 707 190 - 229 > mail eike.segg...@sevenval.com > > www.sevenval.com > > Sitz: Köln, HRB 79823 > Geschäftsführung: Jan Webering (CEO), Thorsten May, Sascha Langfus, > Joern-Carlos Kuntze > > *Wir erhöhen den Return On Investment bei Ihren Mobile und Web-Projekten. > Sprechen Sie uns an:*http://roi.sevenval.com/ > ------------------------------------------------------------ > ------------------------------------------------------------ > ----------------------- > FOLLOW US on > > [image: Sevenval blog] > <http://sevenval.us11.list-manage1.com/track/click?u=5f2d34577b3182d6f029ebe63&id=ff955ef848&e=b789cc1a5f> > > [image: sevenval on twitter] > <http://sevenval.us11.list-manage.com/track/click?u=5f2d34577b3182d6f029ebe63&id=998e8f655c&e=b789cc1a5f> > [image: sevenval on linkedin] > <http://sevenval.us11.list-manage.com/track/click?u=5f2d34577b3182d6f029ebe63&id=7ae7d93d42&e=b789cc1a5f>[image: > sevenval on pinterest] > <http://sevenval.us11.list-manage2.com/track/click?u=5f2d34577b3182d6f029ebe63&id=f8c66fb950&e=b789cc1a5f> >
