Re: How best we can store streaming data on dashboards for real time user experience?

[Pierce Lamb]

SnappyData should work well for what you want, it deeply integrates an
in-memory database with Spark which supports ingesting streaming data and
concurrently querying it from a dashboard. SnappyData currently has an
integration with Apache Zeppelin (notebook visualization) and soon it will
have one with Tableau. Finally, the database and Spark executors share the
same JVM in SnappyData, so you will get better performance than any
database that works with Spark over a connector.

Repo: https://github.com/SnappyDataInc/snappydata

Using it with Zeppelin:
https://github.com/SnappyDataInc/snappydata/blob/master/docs/aqp_aws.md#using-apache-zeppelin

Hope this helps

On Thu, Mar 30, 2017 at 7:56 AM, Alonso Isidoro Roman 
wrote:

> you can check if you want this link
> 
>
>
> elastic, kibana and spark working together.
>
> Alonso Isidoro Roman
> [image: https://]about.me/alonso.isidoro.roman
>
> 
>
> 2017-03-30 16:25 GMT+02:00 Miel Hostens :
>
>> We're doing exactly same thing over here!
>>
>> Spark + ELK
>>
>> Met vriendelijk groeten,
>>
>> *Miel Hostens*,
>>
>> *B**uitenpraktijk+*
>> Department of Reproduction, Obstetrics and Herd Health
>> 
>>
>> Ambulatory Clinic
>> Faculty of Veterinary Medicine 
>> Ghent University  
>> Salisburylaan 133
>> 9820 Merelbeke
>> Belgium
>>
>>
>>
>> Tel:   +32 9 264 75 28 <+32%209%20264%2075%2028>
>>
>> Fax:  +32 9 264 77 97 <+32%209%20264%2077%2097>
>>
>> Mob: +32 478 59 37 03 <+32%20478%2059%2037%2003>
>>
>>  e-mail: miel.host...@ugent.be
>>
>> On 30 March 2017 at 15:01, Szuromi Tamás  wrote:
>>
>>> For us, after some Spark Streaming transformation, Elasticsearch +
>>> Kibana is a great combination to store and visualize data.
>>> An alternative solution that we use is Spark Streaming put some data
>>> back to Kafka and we consume it with nodejs.
>>>
>>> Cheers,
>>> Tamas
>>>
>>> 2017-03-30 9:25 GMT+02:00 Alonso Isidoro Roman :
>>>
 Read this first:

 http://www.oreilly.com/data/free/big-data-analytics-emerging
 -architecture.csp

 https://www.ijircce.com/upload/2015/august/97_A%20Study.pdf

 http://www.pentaho.com/assets/pdf/CqPxTROXtCpfoLrUi4Bj.pdf

 http://www.gartner.com/smarterwithgartner/six-best-practices
 -for-real-time-analytics/

 https://speakerdeck.com/elasticsearch/using-elasticsearch-lo
 gstash-and-kibana-to-create-realtime-dashboards

 https://www.youtube.com/watch?v=PuvHINcU9DI

 then take a look to

 https://kudu.apache.org/

 Tell us later what you think.




 Alonso Isidoro Roman
 [image: https://]about.me/alonso.isidoro.roman

 

 2017-03-30 7:14 GMT+02:00 Gaurav Pandya :

> Hi Noorul,
>
> Thanks for the reply.
> But then how to build the dashboard report? Don't we need to store
> data anywhere?
> Please suggest.
>
> Thanks.
> Gaurav
>
> On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
> noo...@noorul.com> wrote:
>
>> I think better place would be a in memory cache for real time.
>>
>> Regards,
>> Noorul
>>
>> On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
>> wrote:
>> > I am getting streaming data and want to show them onto dashboards
>> in real
>> > time?
>> > May I know how best we can handle these streaming data? where to
>> store? (DB
>> > or HDFS or ???)
>> > I want to give users a real time analytics experience.
>> >
>> > Please suggest possible ways. Thanks.
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/How-best-we-can-store-streaming-data-o
>> n-dashboards-for-real-time-user-experience-tp28548.html
>> > Sent from the Apache Spark User List mailing list archive at
>> Nabble.com.
>> >
>> > 
>> -
>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>> >
>>
>
>

>>>
>>
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Alonso Isidoro Roman]

you can check if you want this link



elastic, kibana and spark working together.

Alonso Isidoro Roman
[image: https://]about.me/alonso.isidoro.roman


2017-03-30 16:25 GMT+02:00 Miel Hostens :

> We're doing exactly same thing over here!
>
> Spark + ELK
>
> Met vriendelijk groeten,
>
> *Miel Hostens*,
>
> *B**uitenpraktijk+*
> Department of Reproduction, Obstetrics and Herd Health
> 
>
> Ambulatory Clinic
> Faculty of Veterinary Medicine 
> Ghent University  
> Salisburylaan 133
> 9820 Merelbeke
> Belgium
>
>
>
> Tel:   +32 9 264 75 28 <+32%209%20264%2075%2028>
>
> Fax:  +32 9 264 77 97 <+32%209%20264%2077%2097>
>
> Mob: +32 478 59 37 03 <+32%20478%2059%2037%2003>
>
>  e-mail: miel.host...@ugent.be
>
> On 30 March 2017 at 15:01, Szuromi Tamás  wrote:
>
>> For us, after some Spark Streaming transformation, Elasticsearch + Kibana
>> is a great combination to store and visualize data.
>> An alternative solution that we use is Spark Streaming put some data back
>> to Kafka and we consume it with nodejs.
>>
>> Cheers,
>> Tamas
>>
>> 2017-03-30 9:25 GMT+02:00 Alonso Isidoro Roman :
>>
>>> Read this first:
>>>
>>> http://www.oreilly.com/data/free/big-data-analytics-emerging
>>> -architecture.csp
>>>
>>> https://www.ijircce.com/upload/2015/august/97_A%20Study.pdf
>>>
>>> http://www.pentaho.com/assets/pdf/CqPxTROXtCpfoLrUi4Bj.pdf
>>>
>>> http://www.gartner.com/smarterwithgartner/six-best-practices
>>> -for-real-time-analytics/
>>>
>>> https://speakerdeck.com/elasticsearch/using-elasticsearch-lo
>>> gstash-and-kibana-to-create-realtime-dashboards
>>>
>>> https://www.youtube.com/watch?v=PuvHINcU9DI
>>>
>>> then take a look to
>>>
>>> https://kudu.apache.org/
>>>
>>> Tell us later what you think.
>>>
>>>
>>>
>>>
>>> Alonso Isidoro Roman
>>> [image: https://]about.me/alonso.isidoro.roman
>>>
>>> 
>>>
>>> 2017-03-30 7:14 GMT+02:00 Gaurav Pandya :
>>>
 Hi Noorul,

 Thanks for the reply.
 But then how to build the dashboard report? Don't we need to store data
 anywhere?
 Please suggest.

 Thanks.
 Gaurav

 On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
 noo...@noorul.com> wrote:

> I think better place would be a in memory cache for real time.
>
> Regards,
> Noorul
>
> On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
> wrote:
> > I am getting streaming data and want to show them onto dashboards in
> real
> > time?
> > May I know how best we can handle these streaming data? where to
> store? (DB
> > or HDFS or ???)
> > I want to give users a real time analytics experience.
> >
> > Please suggest possible ways. Thanks.
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/How-best-we-can-store-streaming-data-o
> n-dashboards-for-real-time-user-experience-tp28548.html
> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >
> > 
> -
> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> >
>


>>>
>>
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Miel Hostens]

We're doing exactly same thing over here!

Spark + ELK

Met vriendelijk groeten,

*Miel Hostens*,

*B**uitenpraktijk+*
Department of Reproduction, Obstetrics and Herd Health


Ambulatory Clinic
Faculty of Veterinary Medicine 
Ghent University  
Salisburylaan 133
9820 Merelbeke
Belgium



Tel:   +32 9 264 75 28

Fax:  +32 9 264 77 97

Mob: +32 478 59 37 03

 e-mail: miel.host...@ugent.be

On 30 March 2017 at 15:01, Szuromi Tamás  wrote:

> For us, after some Spark Streaming transformation, Elasticsearch + Kibana
> is a great combination to store and visualize data.
> An alternative solution that we use is Spark Streaming put some data back
> to Kafka and we consume it with nodejs.
>
> Cheers,
> Tamas
>
> 2017-03-30 9:25 GMT+02:00 Alonso Isidoro Roman :
>
>> Read this first:
>>
>> http://www.oreilly.com/data/free/big-data-analytics-emerging
>> -architecture.csp
>>
>> https://www.ijircce.com/upload/2015/august/97_A%20Study.pdf
>>
>> http://www.pentaho.com/assets/pdf/CqPxTROXtCpfoLrUi4Bj.pdf
>>
>> http://www.gartner.com/smarterwithgartner/six-best-practices
>> -for-real-time-analytics/
>>
>> https://speakerdeck.com/elasticsearch/using-elasticsearch-
>> logstash-and-kibana-to-create-realtime-dashboards
>>
>> https://www.youtube.com/watch?v=PuvHINcU9DI
>>
>> then take a look to
>>
>> https://kudu.apache.org/
>>
>> Tell us later what you think.
>>
>>
>>
>>
>> Alonso Isidoro Roman
>> [image: https://]about.me/alonso.isidoro.roman
>>
>> 
>>
>> 2017-03-30 7:14 GMT+02:00 Gaurav Pandya :
>>
>>> Hi Noorul,
>>>
>>> Thanks for the reply.
>>> But then how to build the dashboard report? Don't we need to store data
>>> anywhere?
>>> Please suggest.
>>>
>>> Thanks.
>>> Gaurav
>>>
>>> On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
>>> noo...@noorul.com> wrote:
>>>
 I think better place would be a in memory cache for real time.

 Regards,
 Noorul

 On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
 wrote:
 > I am getting streaming data and want to show them onto dashboards in
 real
 > time?
 > May I know how best we can handle these streaming data? where to
 store? (DB
 > or HDFS or ???)
 > I want to give users a real time analytics experience.
 >
 > Please suggest possible ways. Thanks.
 >
 >
 >
 >
 > --
 > View this message in context: http://apache-spark-user-list.
 1001560.n3.nabble.com/How-best-we-can-store-streaming-data-o
 n-dashboards-for-real-time-user-experience-tp28548.html
 > Sent from the Apache Spark User List mailing list archive at
 Nabble.com.
 >
 > -
 > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
 >

>>>
>>>
>>
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Szuromi Tamás]

For us, after some Spark Streaming transformation, Elasticsearch + Kibana
is a great combination to store and visualize data.
An alternative solution that we use is Spark Streaming put some data back
to Kafka and we consume it with nodejs.

Cheers,
Tamas

2017-03-30 9:25 GMT+02:00 Alonso Isidoro Roman :

> Read this first:
>
> http://www.oreilly.com/data/free/big-data-analytics-
> emerging-architecture.csp
>
> https://www.ijircce.com/upload/2015/august/97_A%20Study.pdf
>
> http://www.pentaho.com/assets/pdf/CqPxTROXtCpfoLrUi4Bj.pdf
>
> http://www.gartner.com/smarterwithgartner/six-best-
> practices-for-real-time-analytics/
>
> https://speakerdeck.com/elasticsearch/using-elasticsearch-logstash-and-
> kibana-to-create-realtime-dashboards
>
> https://www.youtube.com/watch?v=PuvHINcU9DI
>
> then take a look to
>
> https://kudu.apache.org/
>
> Tell us later what you think.
>
>
>
>
> Alonso Isidoro Roman
> [image: https://]about.me/alonso.isidoro.roman
>
> 
>
> 2017-03-30 7:14 GMT+02:00 Gaurav Pandya :
>
>> Hi Noorul,
>>
>> Thanks for the reply.
>> But then how to build the dashboard report? Don't we need to store data
>> anywhere?
>> Please suggest.
>>
>> Thanks.
>> Gaurav
>>
>> On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
>> noo...@noorul.com> wrote:
>>
>>> I think better place would be a in memory cache for real time.
>>>
>>> Regards,
>>> Noorul
>>>
>>> On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
>>> wrote:
>>> > I am getting streaming data and want to show them onto dashboards in
>>> real
>>> > time?
>>> > May I know how best we can handle these streaming data? where to
>>> store? (DB
>>> > or HDFS or ???)
>>> > I want to give users a real time analytics experience.
>>> >
>>> > Please suggest possible ways. Thanks.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context: http://apache-spark-user-list.
>>> 1001560.n3.nabble.com/How-best-we-can-store-streaming-data-o
>>> n-dashboards-for-real-time-user-experience-tp28548.html
>>> > Sent from the Apache Spark User List mailing list archive at
>>> Nabble.com.
>>> >
>>> > -
>>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>> >
>>>
>>
>>
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Alonso Isidoro Roman]

Read this first:

http://www.oreilly.com/data/free/big-data-analytics-emerging-architecture.csp

https://www.ijircce.com/upload/2015/august/97_A%20Study.pdf

http://www.pentaho.com/assets/pdf/CqPxTROXtCpfoLrUi4Bj.pdf

http://www.gartner.com/smarterwithgartner/six-best-practices-for-real-time-analytics/

https://speakerdeck.com/elasticsearch/using-elasticsearch-logstash-and-kibana-to-create-realtime-dashboards

https://www.youtube.com/watch?v=PuvHINcU9DI

then take a look to

https://kudu.apache.org/

Tell us later what you think.




Alonso Isidoro Roman
[image: https://]about.me/alonso.isidoro.roman


2017-03-30 7:14 GMT+02:00 Gaurav Pandya :

> Hi Noorul,
>
> Thanks for the reply.
> But then how to build the dashboard report? Don't we need to store data
> anywhere?
> Please suggest.
>
> Thanks.
> Gaurav
>
> On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
> noo...@noorul.com> wrote:
>
>> I think better place would be a in memory cache for real time.
>>
>> Regards,
>> Noorul
>>
>> On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
>> wrote:
>> > I am getting streaming data and want to show them onto dashboards in
>> real
>> > time?
>> > May I know how best we can handle these streaming data? where to store?
>> (DB
>> > or HDFS or ???)
>> > I want to give users a real time analytics experience.
>> >
>> > Please suggest possible ways. Thanks.
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/How-best-we-can-store-streaming-data-
>> on-dashboards-for-real-time-user-experience-tp28548.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > -
>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>> >
>>
>
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Gaurav Pandya]

Hi Noorul,

Thanks for the reply.
But then how to build the dashboard report? Don't we need to store data
anywhere?
Please suggest.

Thanks.
Gaurav

On Thu, Mar 30, 2017 at 10:32 AM, Noorul Islam Kamal Malmiyoda <
noo...@noorul.com> wrote:

> I think better place would be a in memory cache for real time.
>
> Regards,
> Noorul
>
> On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809 
> wrote:
> > I am getting streaming data and want to show them onto dashboards in real
> > time?
> > May I know how best we can handle these streaming data? where to store?
> (DB
> > or HDFS or ???)
> > I want to give users a real time analytics experience.
> >
> > Please suggest possible ways. Thanks.
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/How-best-we-can-store-streaming-
> data-on-dashboards-for-real-time-user-experience-tp28548.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > -
> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> >
>

Re: How best we can store streaming data on dashboards for real time user experience?

[Noorul Islam Kamal Malmiyoda]

I think better place would be a in memory cache for real time.

Regards,
Noorul

On Thu, Mar 30, 2017 at 10:31 AM, Gaurav1809  wrote:
> I am getting streaming data and want to show them onto dashboards in real
> time?
> May I know how best we can handle these streaming data? where to store? (DB
> or HDFS or ???)
> I want to give users a real time analytics experience.
>
> Please suggest possible ways. Thanks.
>
>
>
>
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/How-best-we-can-store-streaming-data-on-dashboards-for-real-time-user-experience-tp28548.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org