[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266430#comment-16266430
]
Christoph Pirkl commented on SPARK-10969:
-
Thanks for the information, this solves my problem. You can close the ticket.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266374#comment-16266374
]
Grega Kespret commented on SPARK-10969:
---
I believe this is now possible through the {{KinesisInputDStream.Builder}}
class as:
{code}
KinesisInputDStream.builder
...
.kinesisCredentials(creds1)
.dynamoDBCredentials(creds2)
.cloudWatchCredentials(creds3)
.build
{code}
Close the ticket?
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148366#comment-15148366
]
Christoph Pirkl commented on SPARK-10969:
-
I created PR https://github.com/apache/spark/pull/11215 that implements this.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148365#comment-15148365
]
Apache Spark commented on SPARK-10969:
--
User 'kaklakariada' has created a pull request for this issue:
https://github.com/apache/spark/pull/11215
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035873#comment-15035873
]
Christoph Pirkl commented on SPARK-10969:
-
While this commit is useful it does not fix this issue. In order to fix this,
it would be best to introduce a serializable parameter object that contains aws
credentials for kinesis, dynamodb and cloudwatch (and maybe other values). This
would make it easier to add more parameters later.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035963#comment-15035963
]
Brian London commented on SPARK-10969:
--
I'm not sure why it needs to be an object as opposed to two strings, or is the
issue that this change only adds custom credentials to kinesis and not dynamodb
and cloudwatch? There are already the `fs.s3.awsAccessKeyId` and
`fs.s3.awsSecretAccessKey` parameters that are used for the `.saveToTextFile`
RDD method. Perhaps something like that could be used for the other services
as well.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036418#comment-15036418
]
Christoph Pirkl commented on SPARK-10969:
-
This issue is about adding additional (optional) credentials for dynamodb and
cloudwatch. In my opinion it would be better to introduce a parameter object
containing all credentials than to have four additional String parameters.
I don't think it is possible to use the same mechanism as in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L98
(i.e. reading the credentials from System.getenv()) as this would mean
defining environment variables on each node.
A generic configuration using a map would mean a potential error source when
you mistype a parameter name.
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
[
https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034820#comment-15034820
]
Brian London commented on SPARK-10969:
--
Was this fixed with
https://github.com/apache/spark/commit/63accc79625d8a03d0624717af5e1d81b18a6da3
?
> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis
> and DynamoDB
> ---
>
> Key: SPARK-10969
> URL: https://issues.apache.org/jira/browse/SPARK-10969
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.5.1
>Reporter: Christoph Pirkl
>Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS
> credentials that will be used by Amazon KCL for accessing Kinesis, DynamoDB
> and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a
> different AWS account the user usually has minimal rights (i.e. only read
> from the stream). In this case creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in
> {{KinesisUtils.createStream()}} for Kinesis, DynamoDB and CloudWatch. The
> additional credentials could then be passed to the constructor of
> {{KinesisClientLibConfiguration}} or method
> {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
