[ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036418#comment-15036418 ]
Christoph Pirkl commented on SPARK-10969: ----------------------------------------- This issue is about adding additional (optional) credentials for dynamodb and cloudwatch. In my opinion it would be better to introduce a parameter object containing all credentials than to have four additional String parameters. I don't think it is possible to use the same mechanism as in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L98 (i.e. reading the credentials from System.getenv()) as this would mean defining environment variables on each node. A generic configuration using a map would mean a potential error source when you mistype a parameter name. > Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis > and DynamoDB > --------------------------------------------------------------------------------------- > > Key: SPARK-10969 > URL: https://issues.apache.org/jira/browse/SPARK-10969 > Project: Spark > Issue Type: Improvement > Components: Streaming > Affects Versions: 1.5.1 > Reporter: Christoph Pirkl > Priority: Critical > > {{KinesisUtils.createStream()}} allows specifying only one set of AWS > credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB > and CloudWatch. > h5. Motivation > In a scenario where one needs to read from a Kinesis Stream owned by a > different AWS account the user usually has minimal rights (i.e. only read > from the stream). In this case creating the DynamoDB table in KCL will fail. > h5. Proposal > My proposed solution would be to allow specifying multiple credentials in > {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The > additional credentials could then be passed to the constructor of > {{KinesisClientLibConfiguration}} or method > {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org