Hisao Furuichi created CAMEL-12179: -------------------------------------- Summary: Implement Camel Aws-Kinesis Using Kinesis Client Library(KCL) Key: CAMEL-12179 URL: https://issues.apache.org/jira/browse/CAMEL-12179 Project: Camel Issue Type: New Feature Components: camel-aws Affects Versions: 2.17.0 Environment: * JBoss Fuse 6.3 R5 ** camel 2.17.0 Reporter: Hisao Furuichi
Amazon provides 2 options for Kinesis Java Client: * *Kinesis Client Library(KCL)* * *Kinesis Streams API* "*Kinesis Streams API*" is a low level API set, and "*KCL*" is a rich library. In the official document, "*KCL*" is described like this: {quote} The KCL is compiled into your application to enable fault-tolerant consumption of data from the stream. The KCL ensures that for every shard there is a record processor running and processing that shard. The library also simplifies reading data from the stream. The KCL uses an Amazon DynamoDB table to store control data. It creates one table per application that is processing data. The KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to resharding. The KCL enables you to focus on writing record processing logic. {quote} Since "*Kinesis Streams API*" is used to implement current aws-kinesis component, usability and scalability are not provided by default. * User needs to specify *shardID*(ENTESB-7517) which might be changed dynamically * If a Stream uses multiple shards, user needs to create multiple endpoints for it. (Increasing number of shards are tips to scale out. If there are 200 shards, user needs to maintain 200 endpoints) * If user wants to consume a same shard from multiple Camel runtimes, user needs to implement load-balancing By using "*KCL*" to implement *aws-kinesis*, above current difficulties/concern can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)