Thanks Aniket , clears a lot of confusion.
On Dec 14, 2014 7:11 PM, Aniket Bhatnagar aniket.bhatna...@gmail.com
wrote:
The reason is because of the following code:
val numStreams = numShards
val kinesisStreams = (0 until numStreams).map { i =
KinesisUtils.createStream(ssc, streamName,
The reason is because of the following code:
val numStreams = numShards
val kinesisStreams = (0 until numStreams).map { i =
KinesisUtils.createStream(ssc, streamName, endpointUrl,
kinesisCheckpointInterval,
InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2)
}
In the above
Thanks Aniket,
The trick is to have the #workers = #shards + 1. But I don’t know why is that.
http://spark.apache.org/docs/latest/streaming-kinesis-integration.html
Here in the figure[spark streaming kinesis architecture], it seems like one
node should be able to take on more than one shards.
Guys,
In my local machine it consumes a stream of Kinesis with 3 shards. But in EC2
it does not consume from the stream. Later we found that the EC2 machine was of
2 cores and my local machine was of 4 cores. I am using a single machine and in
spark standalone mode. And we got a larger machine
Hi guys,
When we are using Kinesis with 1 shard then it works fine. But when we use more
that 1 then it falls into an infinite loop and no data is processed by the
spark streaming. In the kinesis dynamo DB, I can see that it keeps increasing
the leaseCounter. But it do start processing.
I am
I have it working without any issues (tried with 5 shrads), except my java
version was 1.7.
Here's the piece of code that i used.
System.setProperty(AWS_ACCESS_KEY_ID,
this.kConf.getOrElse(access_key, ))
System.setProperty(AWS_SECRET_KEY, this.kConf.getOrElse(secret,
)) val streamName
What's your cluster size? For streamig to work, it needs shards + 1
executors.
On Wed, Nov 26, 2014, 5:53 PM A.K.M. Ashrafuzzaman
ashrafuzzaman...@gmail.com wrote:
Hi guys,
When we are using Kinesis with 1 shard then it works fine. But when we use
more that 1 then it falls into an infinite
Did you set spark master as local[*]? If so, then it means that nunber of
executors is equal to number of cores of the machine. Perhaps your mac
machine has more cores (certainly more than number of kinesis shards +1).
Try explicitly setting master as local[N] where N is number of kinesis
shards
