Thanks Mitch, i will check it. Cheers
Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2016-08-30 9:52 GMT+02:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > You can use Hbase for building real time dashboards > > Check this link > <https://www.sigmoid.com/integrating-spark-kafka-hbase-to-power-a-real-time-dashboard/> > > HTH > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 30 August 2016 at 08:33, Alonso Isidoro Roman <alons...@gmail.com> > wrote: > >> HBase for real time queries? HBase was designed with the batch in mind. >> Impala should be a best choice, but i do not know what Druid can do.... >> >> >> Cheers >> >> Alonso Isidoro Roman >> [image: https://]about.me/alonso.isidoro.roman >> >> <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> >> >> 2016-08-30 8:56 GMT+02:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >> >>> Hi Chanh, >>> >>> Druid sounds like a good choice. >>> >>> But again the point being is that what else Druid brings on top of >>> Hbase. >>> >>> Unless one decides to use Druid for both historical data and real time >>> data in place of Hbase! >>> >>> It is easier to write API against Druid that Hbase? You still want a UI >>> dashboard? >>> >>> Cheers >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 30 August 2016 at 03:19, Chanh Le <giaosu...@gmail.com> wrote: >>> >>>> Hi everyone, >>>> >>>> Seems a lot people using Druid for realtime Dashboard. >>>> I’m just wondering of using Druid for main storage engine because Druid >>>> can store the raw data and can integrate with Spark also (theoretical). >>>> In that case do we need to store 2 separate storage Druid (store >>>> segment in HDFS) and HDFS?. >>>> BTW did anyone try this one https://github.com/Sparkli >>>> neData/spark-druid-olap? >>>> >>>> >>>> Regards, >>>> Chanh >>>> >>>> >>>> On Aug 30, 2016, at 3:23 AM, Mich Talebzadeh <mich.talebza...@gmail.com> >>>> wrote: >>>> >>>> Thanks Bhaarat and everyone. >>>> >>>> This is an updated version of the same diagram >>>> >>>> <LambdaArchitecture.png> >>>> >>>> The frequency of Recent data is defined by the Windows length in Spark >>>> Streaming. It can vary between 0.5 seconds to an hour. ( Don't think we can >>>> move any Spark granularity below 0.5 seconds in anger. For some >>>> applications like Credit card transactions and fraud detection. Data is >>>> stored real time by Spark in Hbase tables. Hbase tables will be on HDFS as >>>> well. The same Spark Streaming will write asynchronously to HDFS Hive >>>> tables. >>>> One school of thought is never write to Hive from Spark, write >>>> straight to Hbase and then read Hbase tables into Hive periodically? >>>> >>>> Now the third component in this layer is Serving Layer that can combine >>>> data from the current (Hbase) and the historical (Hive tables) to give the >>>> user visual analytics. Now that visual analytics can be Real time dashboard >>>> on top of Serving Layer. That Serving layer could be an in-memory NoSQL >>>> offering or Data from Hbase (Red Box) combined with Hive tables. >>>> >>>> I am not aware of any industrial strength Real time Dashboard. The >>>> idea is that one uses such dashboard in real time. Dashboard in this sense >>>> meaning a general purpose API to data store of some type like on Serving >>>> layer to provide visual analytics real time on demand, combining real time >>>> data and aggregate views. As usual the devil in the detail. >>>> >>>> >>>> >>>> Let me know your thoughts. Anyway this is first cut pattern. >>>> >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 29 August 2016 at 18:53, Bhaarat Sharma <bhaara...@gmail.com> wrote: >>>> >>>>> Hi Mich >>>>> >>>>> This is really helpful. I'm trying to wrap my head around the last >>>>> diagram you shared (the one with kafka). In this diagram spark streaming >>>>> is >>>>> pushing data to HDFS and NoSql. However, I'm confused by the "Real Time >>>>> Queries, Dashboards" annotation. Based on this diagram, will real time >>>>> queries be running on Spark or HBase? >>>>> >>>>> PS: My intention was not to steer the conversation away from what >>>>> Ashok asked but I found the diagrams shared by Mich very insightful. >>>>> >>>>> On Sun, Aug 28, 2016 at 7:18 PM, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> In terms of positioning, Spark is really the first Big Data platform >>>>>> to integrate batch, streaming and interactive computations in a unified >>>>>> framework. What this boils down to is the fact that whichever way one >>>>>> look >>>>>> at it there is somewhere that Spark can make a contribution to. In >>>>>> general, >>>>>> there are few design patterns common to Big Data >>>>>> >>>>>> >>>>>> >>>>>> - *ETL & Batch* >>>>>> >>>>>> The first one is the most common one with Established tools like >>>>>> Sqoop, Talend for ETL and HDFS for storage of some kind. Spark can be >>>>>> used >>>>>> as the execution engine for Hive at the storage level which actually >>>>>> makes it a true vendor independent (BTW, Impala and Tez and LLAP) are >>>>>> offered by vendors) processing engine. Personally I use Spark at ETL >>>>>> layer >>>>>> by extracting data from sources through plug ins (JDBC and others) and >>>>>> storing in on HDFS in some kind >>>>>> >>>>>> >>>>>> >>>>>> - *Batch, real time plus Analytics* >>>>>> >>>>>> In this pattern you have data coming in real time and you want to >>>>>> query them real time through real time dashboard. HDFS is not ideal for >>>>>> updating data in real time and neither for random access of data. Source >>>>>> could be all sorts of Web Servers and need Flume Agent with Flume. At the >>>>>> storage layer we are probably looking at something like Hbase. The >>>>>> crucial >>>>>> point being that saved data needs to be ready for queries immediately The >>>>>> dashboards requires Hbase APIs. The Analytics can be done through Hive >>>>>> again running on Spark engine. Again note here that we ideally should >>>>>> process batch and real time separately. >>>>>> >>>>>> >>>>>> >>>>>> - *Real time / Streaming* >>>>>> >>>>>> This is most relevant to Spark as we are moving to near real time. >>>>>> Where Spark excels. We need to capture the incoming events (logs, sensor >>>>>> data, pricing, emails) through interfaces like Kafka, Message Queues etc. >>>>>> Need to process these events with minimum latency. Again Spark is a >>>>>> very good candidate here with its Spark Streaming and micro-batching >>>>>> capabilities. There are others like Storm, Flink etc. that are event >>>>>> based >>>>>> but you don’t hear much. Again for streaming architecture you need to >>>>>> sync >>>>>> data in real time using something like Hbase, Cassandra (?) and others as >>>>>> real time store or forever storage HDFS or Hive etc. >>>>>> >>>>>> >>>>>> In general there is also *Lambda Architecture* that is >>>>>> designed for streaming analytics. The streaming data ends up in both >>>>>> batch >>>>>> layer and speed layer. Batch layer is used to answer batch queries. On >>>>>> the >>>>>> other hand speed later is used ti handle fast/real time queries. This >>>>>> model >>>>>> is really cool as Spark Streaming can feed both the batch layer and >>>>>> the speed layer. >>>>>> >>>>>> >>>>>> At a high level this looks like this, from >>>>>> http://lambda-architecture.net/ >>>>>> >>>>>> <image.png> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> My favourite would be something like below with Spark playing a major >>>>>> role >>>>>> >>>>>> >>>>>> <LambdaArchitecture.png> >>>>>> >>>>>> >>>>>> HTH >>>>>> >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> On 28 August 2016 at 19:43, Sivakumaran S <siva.kuma...@me.com> >>>>>> wrote: >>>>>> >>>>>>> Spark best fits for processing. But depending on the use case, you >>>>>>> could expand the scope of Spark to moving data using the native >>>>>>> connectors. >>>>>>> The only that Spark is not, is Storage. Connectors are available for >>>>>>> most >>>>>>> storage options though. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Sivakumaran S >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 28-Aug-2016, at 6:04 PM, Ashok Kumar < >>>>>>> ashok34...@yahoo.com.INVALID <ashok34...@yahoo.com.invalid>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> There are design patterns that use Spark extensively. I am new to >>>>>>> this area so I would appreciate if someone explains where Spark fits in >>>>>>> especially within faster or streaming use case. >>>>>>> >>>>>>> What are the best practices involving Spark. Is it always best to >>>>>>> deploy it for processing engine, >>>>>>> >>>>>>> For example when we have a pattern >>>>>>> >>>>>>> Input Data -> Data in Motion -> Processing -> Storage >>>>>>> >>>>>>> Where does Spark best fit in. >>>>>>> >>>>>>> Thanking you >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >
