RE: IOT in Spark
Hi Gaurav, You can process IoT data using Spark. But where will you store the raw/processed data - Cassandra, Hive, HBase? You might want to look at the Hadoop cluster for data storage and processing (Spark using Yarn). For processing streaming data, you might also explore Apache Storm and Apache Flink. I suggest it is better to do a POC in each of them and then decide on what works best for you. Best regards / Mit freundlichen Grüßen / Sincères salutations M. Lohith Samaga -Original Message- From: Gaurav1809 [mailto:gauravhpan...@gmail.com] Sent: Friday, May 19, 2017 09.28 To: user@spark.apache.org Subject: IOT in Spark Hello gurus, How exactly it works in real world scenarios when it come to read data from IOT devices (say for example censors at in/out gate in huge mall)? Can we do it in Spark? Do we need to use any other tool/utility (kafka???) to read data from those censors and then process them in Spark? Please share your thoughts on this and it will give me headstart to my work. I am completely unaware of the technology stack that can be used here, so any pointers to this will be so much helpful. Thanks. -Gaurav -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/IOT-in-Spark-tp28698.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at mailmas...@mphasis.com and delete this mail from your records. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: IOT in Spark
Hi Gaurav, You might want to look for Lambda Architecture with Spark. https://www.youtube.com/watch?v=xHa7pA94DbA Regards, Kuchekar, Nilesh On Thu, May 18, 2017 at 8:58 PM, Gaurav1809 wrote: > Hello gurus, > > How exactly it works in real world scenarios when it come to read data from > IOT devices (say for example censors at in/out gate in huge mall)? Can we > do > it in Spark? Do we need to use any other tool/utility (kafka???) to read > data from those censors and then process them in Spark? Please share your > thoughts on this and it will give me headstart to my work. I am completely > unaware of the technology stack that can be used here, so any pointers to > this will be so much helpful. Thanks. > > -Gaurav > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/IOT-in-Spark-tp28698.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: IOT in Spark
Hi Gaurav, Answer is - yes; you can do it on Spark. Note that, first you need to understand what Spark is used for. For the problem statement you mentioned, you need many more technology components - Kafka, Spark Streaming, and Spark - in addition to IoT related software at the edge and gateway. My suggestion would be to read up on these technologies first, and design your solution to your problem. -Bharath On Thu, May 18, 2017 at 8:58 PM, Gaurav1809 wrote: > Hello gurus, > > How exactly it works in real world scenarios when it come to read data from > IOT devices (say for example censors at in/out gate in huge mall)? Can we > do > it in Spark? Do we need to use any other tool/utility (kafka???) to read > data from those censors and then process them in Spark? Please share your > thoughts on this and it will give me headstart to my work. I am completely > unaware of the technology stack that can be used here, so any pointers to > this will be so much helpful. Thanks. > > -Gaurav > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/IOT-in-Spark-tp28698.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
IOT in Spark
Hello gurus, How exactly it works in real world scenarios when it come to read data from IOT devices (say for example censors at in/out gate in huge mall)? Can we do it in Spark? Do we need to use any other tool/utility (kafka???) to read data from those censors and then process them in Spark? Please share your thoughts on this and it will give me headstart to my work. I am completely unaware of the technology stack that can be used here, so any pointers to this will be so much helpful. Thanks. -Gaurav -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/IOT-in-Spark-tp28698.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
