Hallo, Only because you receive the log files hourly it means that you have to use Spark Streaming. Spark streaming is often used if you receive new events each minute /second potentially at an irregular frequency. Of course your analysis window can be larger.
I think your use case justifies standard Spark or MR. If you are not restricted to them you may check a key/value or column store, such as redis or apache cassandra. They have also some nice mechanisms for storage /performance optimal counting of unique users (hyperloglog) etc. In any case you can join the data with historical data. Best regards Le 5 avr. 2015 12:44, "Bahubali Jain" <bahub...@gmail.com> a écrit : > Hi, > I have a requirement in which I plan to use the SPARK Streaming. > I am supposed to calculate the access count to certain webpages.I receive > the webpage access information thru log files. > By Access count I mean "how many times was the page accessed *till now* " > I have the log files for past 2 years and everyday we keep receiving > almost 6 GB of access logs(on an hourly basis). > Since we receive these logs on an hourly basis I feel that I should use > the SPARK Streaming. > But the problem is that the access counts have to be cumulative , i.e even > the older access(past 2 years) counts for a webpage should also be > considered for the final value. > > How to achieve this thru streaming, since streaming picks only new files. > I don't want to use DB to store the access counts since it would > considerably slow down the processing. > > Thanks, > Baahu > -- > Twitter:http://twitter.com/Baahu > >
