[jira] [Created] (SPARK-29677) Upgrade Kinesis Client

2019-10-30 Thread Eric S Meisel (Jira) 
Eric S Meisel created SPARK-29677:
-

 Summary: Upgrade Kinesis Client
 Key: SPARK-29677
 URL: https://issues.apache.org/jira/browse/SPARK-29677
 Project: Spark
  Issue Type: Improvement
  Components: DStreams
Affects Versions: 2.4.4
Reporter: Eric S Meisel


The current amazon-kinesis-client version is 1.8.10. This version depends on 
the use of `describeStream`, which has a hard limit on an AWS account (10 reqs 
/ second). Versions 1.9.0 and up leverage `listShards`, which has no such 
limit. For large customers, this can be a major problem. 

 

Upgrading the amazon-kinesis-client version should resolve this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27950) Additional DynamoDB and CloudWatch config

2019-06-04 Thread Eric S Meisel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-27950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric S Meisel updated SPARK-27950:
--
Priority: Minor  (was: Major)

> Additional DynamoDB and CloudWatch config
> -
>
> Key: SPARK-27950
> URL: https://issues.apache.org/jira/browse/SPARK-27950
> Project: Spark
>  Issue Type: Improvement
>  Components: Structured Streaming
>Affects Versions: 2.4.3
>Reporter: Eric S Meisel
>Priority: Minor
>
> I was researching getting Spark’s Kinesis integration running locally against 
> {{localstack}}. We found this issue, and it creates a complication: 
> [localstack/localstack#677|https://github.com/localstack/localstack/issues/677]
> Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and 
> Cloudwatch in order for the KCL to properly use the {{localstack}} 
> infrastructure. We have successfully done this with the KCL (both 1.x and 
> 2.x), but with Spark’s integration we are unable to configure DynamoDB and 
> Cloudwatch’s endpoints:
> [https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-27950) Additional DynamoDB and CloudWatch config

2019-06-04 Thread Eric S Meisel (JIRA) 
Eric S Meisel created SPARK-27950:
-

 Summary: Additional DynamoDB and CloudWatch config
 Key: SPARK-27950
 URL: https://issues.apache.org/jira/browse/SPARK-27950
 Project: Spark
  Issue Type: Improvement
  Components: Structured Streaming
Affects Versions: 2.4.3
Reporter: Eric S Meisel


I was researching getting Spark’s Kinesis integration running locally against 
{{localstack}}. We found this issue, and it creates a complication: 
[localstack/localstack#677|https://github.com/localstack/localstack/issues/677]

Effectively, we need to be able to redirect calls for Kinesis, DynamoDB and 
Cloudwatch in order for the KCL to properly use the {{localstack}} 
infrastructure. We have successfully done this with the KCL (both 1.x and 2.x), 
but with Spark’s integration we are unable to configure DynamoDB and 
Cloudwatch’s endpoints:

[https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala#L162]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org