2015-02-06 17:28 GMT+00:00 King sami <kgsam...@gmail.com>:
> The purpose is to build a data processing system for door events. An event > will describe a door unlocking > with a badge system. This event will differentiate unlocking by somebody > from the inside and by somebody > from the outside. > > *Producing the events*: > You will need a simulator capable of producing events at random intervals. > Simulating 200 doors seems like > a good number, but adapt it as you see fit to get relevant results. Make > sure different doors have different > patterns to make the analysis interesting. > > *Processing the events:* > After having accumulated a certain amount of events (for example: a day), > you will calculate statistics. To do > this, you will use spark for your batch processing. You will extract: > • most used door, less used door, door with most exits, door with most > entrances > • most and less busy moment (when people entered and exited a lot, or not > at all) > • less busy moment of the day > > *Hints:* > • Spark is required: http://spark.apache.org > • Coding in Scala is required. > • Using HDFS for file storage is a plus. > > 2015-02-06 17:00 GMT+00:00 Nagesh sarvepalli <sarvepalli.nag...@gmail.com> > : > >> Hi, >> >> Here is the sequence I suggest. Feel free if you need further help. >> >> 1) You need to decide if you want to go with any particular distribution >> of Hadoop (Cloudera / Hortonworks / MapR) or want to go for apache version >> . Downloading Hadoop from Apache and integrating with various projects is >> laborious (compared to distributions). Also, you need to take care of >> maintenance including version compatibility of various projects. Cloudera >> Manager is the best when it comes to cluster installation and maintenance >> but it is memory intensive. Cloud offerings (ex: from Microsoft) are even >> much more simpler and hassle free when it comes to installation and >> maintenance. >> >> 2) Depending on the server resources and the data size, you need to >> decide on the HDFS cluster size (number of nodes). Ensure you have the >> right JDK version installed if you are installing Hadoop on your own. >> >> 3) Once Hadoop is installed, you need to download Scala from >> scala-lang.org and then >> >> 4) Download and install spark from http://spark.apache.org/downloads.html >> >> Hope this helps to kick-start. >> >> Thanks & Regards >> Nagesh >> Cloudera Certified Hadoop Developer >> >> On Fri, Feb 6, 2015 at 4:09 PM, King sami <kgsam...@gmail.com> wrote: >> >>> Hi, >>> >>> I'm new in Spark, I'd like to install Spark with Scala. The aim is to >>> build a data processing system foor door events. >>> >>> the first step is install spark, scala, hdfs and other required tools. >>> the second is build the algorithm programm in Scala which can treat a >>> file of my data logs (events). >>> >>> Could you please help me to install the required tools: Spark, Scala, >>> HDF and tell me how can I execute my programm treating the entry file. >>> >>> >>> Best regards, >>> >>> >>> >>> >>> >> >
