[Spark 2.0 state store] Streaming wordcount using spark state store

2016-05-18 Thread Shekhar Bansal
Hi What is the right way of using spark2.0 state store feature in spark streaming??I referred test cases in this(https://github.com/apache/spark/pull/11645/files) pull request and implemented word count using state store.My source is kafka(1 topic, 10 partitions). My data pump is pushing

Spark Streaming share state between two streams

2016-04-08 Thread Shekhar Bansal
HiCan we share spark streaming state between two DStreams??Basically I want to create state using first stream and enrich second stream using state.Example: I have modified StatefulNetworkWordCount example. I am creating state using first stream and enriching second stream with count of first

Re: --executor-cores cannot change vcores in yarn?

2014-11-03 Thread Shekhar Bansal
If you are using capacity scheduler in yarn: By default yarn capacity scheduler uses DefaultResourceCalculator. DefaultResourceCalculator considerĀ¹s only memory while allocating contains. You can use DominantResourceCalculator, it considers memory and cpu. In capacity-scheduler.xml set