Hi All, Here is a problem and I am wondering if Spark Streaming is the right tool for this ?
I have stream of messages m1, m2, m3....and each of those messages can be in state s1, s2, s3,....sn (you can imagine the number of states are about 100) and I want to compute some metrics that visit all the states from s1 to sn but these state transitions can happen at indefinite amount of time. A simple example of that would be count all messages that visited state s1, s2, s3. Other words, the transition function should know that say message m1 had visited state s1 and s2 but not s3 yet and once the message m1 visits s3 increment the counter +=1 . If it makes anything easier I can say a message has to visit s1 before visiting s2 and s2 before visiting s3 and so on but would like to know both with and without order. Thanks!
