Re: Using Ignite Native Persistence as a "temporary durable" cache

[Alexander Korenshteyn]

origonal question:
ferhadcebi...@gmail.com
5:00 AM (5 hours ago)
 to me


 Thanks for answer. But, how Ignite will store that datas to store? Will
append to the end of WAL? If, then it is sure faster than storing cache
operations in some 3rd part database.


Ignite does store data to WAL first, but per my benchmarks, storing 100,000
records took a few seconds, and deleting took a few seconds more.

 You can experiment using :

https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java


Here is sample code: (modified section in above example)
[image: image.png]

On Thu, Sep 5, 2019 at 12:17 PM Alexander Korenshteyn <
alexanderko...@gmail.com> wrote:

> Hello,
>Ignite native persistence has a good track record, is fast and
> reliable, you can use it in your application.
>
>Take a look at the following example of how to use a streamer to
> quickly insert data:
> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java
>
>Use cache.removeAll() -- to quickly remove all entries once you are
> done.
> Thanks, Alex
>
> On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev 
> wrote:
>
>> Hi all,
>>
>> We have MS SQL database server which contains all data. Our application
>> will fetch some datas from database server and put them to the cache. And
>> then, during 5-10 seconds we will do some updates to that objects and push
>> that changes to ignite in-memory cache. And then, after 5-10 seconds we
>> will take that changes and sync them will database as a bulk operation and
>> then will clear that permanent storage. So, we need some permanent storage
>> to store those updates for 5-10 seconds.
>> And actually, we will work at most with 100.000 entity.
>>
>> The idea behind that flow is that we can't now scale DB and users already
>> putting load to the database.
>>
>> We can't decide whether we have to use Native Persistent or some another
>> 3rd party database like Maria or PostgreSQL for storing that cache
>> operations for 5-10 seconds.
>>
>> Will Ignite works fast if we will use native persistent and clear the
>> cache periodically?
>>
>

Re: Using Ignite Native Persistence as a "temporary durable" cache

[Alexander Korenshteyn]

Hello,
   Ignite native persistence has a good track record, is fast and reliable,
you can use it in your application.

   Take a look at the following example of how to use a streamer to quickly
insert data:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java

   Use cache.removeAll() -- to quickly remove all entries once you are done.
Thanks, Alex

On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev 
wrote:

> Hi all,
>
> We have MS SQL database server which contains all data. Our application
> will fetch some datas from database server and put them to the cache. And
> then, during 5-10 seconds we will do some updates to that objects and push
> that changes to ignite in-memory cache. And then, after 5-10 seconds we
> will take that changes and sync them will database as a bulk operation and
> then will clear that permanent storage. So, we need some permanent storage
> to store those updates for 5-10 seconds.
> And actually, we will work at most with 100.000 entity.
>
> The idea behind that flow is that we can't now scale DB and users already
> putting load to the database.
>
> We can't decide whether we have to use Native Persistent or some another
> 3rd party database like Maria or PostgreSQL for storing that cache
> operations for 5-10 seconds.
>
> Will Ignite works fast if we will use native persistent and clear the
> cache periodically?
>

Re: Using Ignite as blob store?

[colinc]

>From anecdotal experience of storing larger objects (up to say 10MB) in
Ignite, I find that the overall access performance is significantly better
than storing lots of small objects. The main thing to watch out for is that
very large objects can cause unbalanced data distribution. Similar to
over-use of affinity.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Using Ignite as blob store?

[Pavel Kovalenko]

Denis,

You can't set page size greater than 16Kb due to our page memory
limitations.

чт, 22 авг. 2019 г. в 22:34, Denis Magda :

> How about setting page size to more KBs or MBs based on the average value?
> That should work perfectly fine.
>
> -
> Denis
>
>
> On Thu, Aug 22, 2019 at 8:11 AM Shane Duan  wrote:
>
>> Thanks, Ilya. The blob size varies from a few KBs to a few MBs.
>>
>> Cheers,
>> Shane
>>
>>
>> On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> How large are these blobs? Ignite is going to divide blobs into <4k
>>> chunks. We have no special optimizations for storing large key-value pairs.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> чт, 22 авг. 2019 г. в 02:53, Shane Duan :
>>>
 Hi Igniters, is it a good idea to use Ignite(with persistence) as a
 blob store? I did run some testing with a small dataset, and it looks
 performing okay, even with a small off-heap mem for the data region.

 Thanks!

 Shane

>>>

Re: Using Ignite as blob store?

[colinc]

I understand from this post:
https://stackoverflow.com/questions/50116444/unable-to-increase-pagesize/50121410#50121410

that the maximum page size is 16K. Is that still true?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Using Ignite as blob store?

[Denis Magda]

How about setting page size to more KBs or MBs based on the average value?
That should work perfectly fine.

-
Denis


On Thu, Aug 22, 2019 at 8:11 AM Shane Duan  wrote:

> Thanks, Ilya. The blob size varies from a few KBs to a few MBs.
>
> Cheers,
> Shane
>
>
> On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> How large are these blobs? Ignite is going to divide blobs into <4k
>> chunks. We have no special optimizations for storing large key-value pairs.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 22 авг. 2019 г. в 02:53, Shane Duan :
>>
>>> Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob
>>> store? I did run some testing with a small dataset, and it looks performing
>>> okay, even with a small off-heap mem for the data region.
>>>
>>> Thanks!
>>>
>>> Shane
>>>
>>

Re: Using Ignite as blob store?

[Shane Duan]

Thanks, Ilya. The blob size varies from a few KBs to a few MBs.

Cheers,
Shane


On Thu, Aug 22, 2019 at 5:02 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> How large are these blobs? Ignite is going to divide blobs into <4k
> chunks. We have no special optimizations for storing large key-value pairs.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 22 авг. 2019 г. в 02:53, Shane Duan :
>
>> Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob
>> store? I did run some testing with a small dataset, and it looks performing
>> okay, even with a small off-heap mem for the data region.
>>
>> Thanks!
>>
>> Shane
>>
>

Re: Using Ignite as blob store?

[Ilya Kasnacheev]

Hello!

How large are these blobs? Ignite is going to divide blobs into <4k chunks.
We have no special optimizations for storing large key-value pairs.

Regards,
-- 
Ilya Kasnacheev


чт, 22 авг. 2019 г. в 02:53, Shane Duan :

> Hi Igniters, is it a good idea to use Ignite(with persistence) as a blob
> store? I did run some testing with a small dataset, and it looks performing
> okay, even with a small off-heap mem for the data region.
>
> Thanks!
>
> Shane
>

Re: Using Ignite as a IMDG Write Through Cache in Azure POC

[Denis Magda]

Hello,

Feel free to use Ignite .NET for that. Moreover, you have 2 options here:

   1. Use .NET standard client (supports most of the APIs but connects to
   the cluster via a JVM process started internally). Here is how you can
   define its config for entries eviction:
   https://apacheignite-net.readme.io/docs/eviction-policies
   2. Use .NET thin client (lightweight, connects via a proxy server - not
   that fast but will be addressed in next releases and doesn't start the
   JVM): https://apacheignite-net.readme.io/docs/thin-client

Btw, keep in mind that SQL engine requires all the data to be in Ignite
cluster - it won't go to SQL Server if anything is missing in RAM. Our SQL
engine can go to disk only if native persistence is enabled.
-
Denis


On Tue, Apr 9, 2019 at 6:57 AM Asadikhan  wrote:

> I want to setup a basic Ignite cluster to start playing around with. What I
> am trying to achieve is setup 3 VMs in Azure that will make up the Ignite
> Cluster. On the back-end I want to use either SQL (for my POC) which I will
> later replace with Cassandra (for POC again - I think SQL would be easier
> to
> start with given my knowledge).
>
> The key thing I want to achieve is use Ignite as a write through cache. So
> if I ingest 1TB of data, I want to retain say most recent 100GB of that in
> Ignite while the rest of the it passes through to the backing SQL Server.
>
> I am more familiar with .net but I can use Java too if needed. Should I use
> Apache Ignite.net for this or Apache Ignite. Also, can someone either point
> me to resources/documentation or give me a high level breakdown of what I
> need to do to achieve this?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Using ignite with spark

[Patrick Brunmayr]

Hello Val

First of all thx for this answer. Let me explain our use case.

*What we are doing*

Our company is a monitoring solution for machines in the manufacturing
industry. We have a hardware logger attached to each machine wich collects
up to 6 different metrics (  like power, piece count ). These metrics are
sampled on a per second basis and sent to our cloud every minute. Data is
currently stored in a cassandra cluster.

*For the math of that *

One metric will generate about 33 million data points per year meaning all
six metrics will cause a total of 100 million data points per machine /
year. Lets say we have about 2000 machines out there its very obvious that
we are talking about terra bytes of metric data.

*The goal*

We need to do some analytics on this data to provide reports for our
customers. Therefore we need to do all kind of transformations, filtering
and joining on that data. We also need support for secondary indexes and
grouping! This was the reason we chose spark for this kind of job. We want
to speed up the spark calculations with Ignite to provide a better
experience for our customers.

My idea was to use Ignite as a read through cache to our cassandra cluster
and combining this with Spark SQL. The data for the calculation should only
stay in the cache during the calculations and can easily be discared
afterwards.


Now i need some information how to setup my cluster correctly for that use
case. I don't know how many nodes i need and how much GB of RAM and if i
should put my ignite nodes on the spark workers or create a separate
cluster. I need this information for cost estimates.

Hope that helps a bit

Thx










2017-09-22 5:12 GMT+02:00 Valentin Kulichenko :

> Hello Patrick,
>
> See my comments below.
>
> Most of your questions don't have a generic answer and would heavily
> depend on your use case. Would you mind giving some more details about it
> so that I can give more specific suggestions?
>
> -Val
>
> On Thu, Sep 21, 2017 at 8:24 AM, Patrick Brunmayr <
> patrick.brunm...@kpibench.com> wrote:
>
>> Hello
>>
>>
>>- What is currently the best practice of deploying Ignite with Spark ?
>>
>>
>>- Should the Ignite node sit on the same machine as the Spark
>>executor ?
>>
>>
> Ignite can run either on same boxes where Spark runs, or as a separate
> cluster, and both approaches have their pros and cons.
>
>
>> According to this documentation
>>  Spark
>> should be given 75% of machine memory but what is left for Ignite then ?
>>
>> In general, Spark can run well with anywhere from *8 GB to hundreds of
>>> gigabytes* of memory per machine. In all cases, we recommend allocating
>>> only at most 75% of the memory for Spark; leave the rest for the operating
>>> system and buffer cache.
>>
>>
> Documentation states that you should give *at most* 75% to make sure OS
> has a safe cushion for its own purposes. If Ignite runs along with Spark,
> amount of memory allocated to Spark should be less then that maximum of
> course.
>
>
>>
>>- Don't they battle for memory ?
>>
>>
> You should configure both Spark and Ignite so that they never try to
> consume more memory than physically available, also leaving some for OS.
> This way there will be no conflict.
>
>>
>>-
>>- Should i give the memory to Ignite or Spark ?
>>
>>
> Again, this heavily depends on use case and on how heavily you use both
> Spark and Ignite.
>
>
>>-
>>- Would Spark even benefit from Ignite if the Ignite nodes would be
>>hostet on other machines ?
>>
>>
> There are definitely use cases when this can be useful. Although in others
> it is better to run Ignite separately.
>
>
>>-
>>
>>
>> We are currently having hundress of GB for analytics and we want to use
>> ignite to speed up things up.
>>
>> Thank you
>>
>>
>>
>>
>>
>>
>>
>>
>

Re: Using Ignite as the SQL Engine for Cassandra

[Igor Rudyak]

Hi Roger,

You can use Ignite-Cassandra integration module. In case you need to load
specific portion of you Cassandra dataset into Ignite you can use
*loadCache(...)* method of Ignite cache API, providing it appropriate CQL
query.

Igor

On Tue, Jul 11, 2017 at 8:29 PM, Roger Fischer (CW) 
wrote:

> Hello,
>
>
>
> I have seen the page on using Cassandra as the persistent store for
> Ignite. Are the same concepts / classes applicable when using Cassandra as
> the backing database?
>
>
>
> I have a large data set in Cassandra. At system start I want to load the
> most recent (ie. most used) data into Ignite. The application then performs
> SQL queries to Ignite. My persistence API will check if the data set is
> already in Ignite before submitting the query. When part of the data set is
> not in Ignite, the persistence API loads the needed data from Cassandra
> into Ignite first.
>
>
>
> Is there a plugin for Ignite that gets me started on this? Or are there
> interfaces or classes that I need to implement / extend?
>
>
>
> I am looking for some initial pointers (ie. where to get started).
>
>
>
> Thanks…
>
>
>
> Roger
>
>
>

Re: using ignite web console

[Andrey Novikov]

Hi,

Note your are trying to use development mode in production!
To bind development web server to all interfaces on your linux machine you
need fix following file:
modules/web-console/frontend/gulpfile.babel.js/webpack/environments/development.js
like this
https://github.com/apache/ignite/blob/master/modules/web-console/frontend/gulpfile.babel.js/webpack/environments/development.js
or use sources from master branch.


Also web console may be started from prepared docker image:
https://hub.docker.com/r/apacheignite/web-console-standalone/



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/using-ignite-web-console-tp12847p12964.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: using ignite web console

[Neeraj Bhatt]

Hi Denis

Thanks for information, ignite web agent default. properties details are
documented in the link you shared
Can you please give name of property files of back end /frontend which need
to be changed as this is not documented anywhere.

Thanks



On Tue, May 16, 2017 at 5:31 AM, Denis Magda  wrote:

> Yes, you need to give the machine where the console is to be deployed a
> unique IP address so that it’s accessible from remote machines. Also you
> might need to tweak some of web agent’s configuration parameters if the
> console should be linked with a remote Ignite cluster:
> https://apacheignite-tools.readme.io/v2.0/docs/getting-
> started#section-ignite-web-agent
>
> —
> Denis
>
> > On May 15, 2017, at 6:11 AM, neerajbhatt 
> wrote:
> >
> > I am trying to install ignite web console in one of our servers (linux)as
> > given in https://apacheignite-tools.readme.io/v1.9/docs/build-and-deploy
> >
> > We will be accessing the web console from different windows machine
> browser
> >
> > Do we need server ip address while building front end or back end as we
> > won't be hitting  http://localhost:9000
> > but  http://:9000
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/using-ignite-web-console-tp12847.html
> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>

Re: using ignite web console

[Denis Magda]

Yes, you need to give the machine where the console is to be deployed a unique 
IP address so that it’s accessible from remote machines. Also you might need to 
tweak some of web agent’s configuration parameters if the console should be 
linked with a remote Ignite cluster:
https://apacheignite-tools.readme.io/v2.0/docs/getting-started#section-ignite-web-agent

—
Denis

> On May 15, 2017, at 6:11 AM, neerajbhatt  wrote:
> 
> I am trying to install ignite web console in one of our servers (linux)as
> given in https://apacheignite-tools.readme.io/v1.9/docs/build-and-deploy
> 
> We will be accessing the web console from different windows machine browser
> 
> Do we need server ip address while building front end or back end as we
> won't be hitting  http://localhost:9000 
> but  http://:9000
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/using-ignite-web-console-tp12847.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Using Ignite within within networks without internet egress

[Nikolai Tikhonov]

Hi, Haithem Turki!

If your yarn cluster running in network without internet, you can use
IGNITE_PATH property. The property allows to use apache ignite build from
hdfs. Also error message isn't clear, I've created ticket [1] for this
issue.

1. https://issues.apache.org/jira/browse/IGNITE-3268

On Tue, May 31, 2016 at 11:58 PM, Haithem Turki 
wrote:

> Hello,
>
> When deploying Ignite on YARN in a network without outbound internet
> access, I run into the following issue on startup:
>
> Exception in thread "main" java.lang.RuntimeException: Failed update
> ignite.
> at
> org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:243)
> at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
> at
> org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:194)
> at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.net.ConnectException: Connection timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:220)
> ... 9 more
>
> Is there a supported way of disabling the download?
>
> As a separate bug, even with internet access, you run into the following
> issue during startup:
>
> Exception in thread "main" java.io.FileNotFoundException: File
> ignite-releases/gridgain-community-fabric-1.5.22.zip does not exist
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
>
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:424)
>
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
>
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1944)
>
> at
> org.apache.ignite.yarn.utils.IgniteYarnUtils.copyLocalToHdfs(IgniteYarnUtils.java:86)
>
> at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:105)
>
> at
> org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:194)
>
> at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:84)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> Renaming the downloaded jar from
> ignite-releases/gridgain-professional-fabric-1.5.22.zip to
> gnite-releases/gridgain-community-fabric-1.5.22.zip seems to solve the
> issue.
>
> - Haithem
>

Re: Using Ignite

[Murthy Kakarlamudi]

Thanks all for your responses. Will look into these alternatives.

On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke  wrote:

> You seem to look for streaming solutions , such as Spark Streaming or
> Flink Streaming or Storm
>
> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
> >
> > Kafka may suit your needs as a "queue" with producer/consumer and
> persistence capabilities also.
> >
> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
> >> Hi,
> >>   We have a scenario where in we have a c++ application that pumps out
> data ticks multiple times in a second. These data ticks needs to be
> displayed in the web front end. We need some middleware platform to hold
> those events temporarily before being consumed by web-front end. Can Ignite
> work as a viable middleware option in this scenario? In future we are
> expecting that there will be multiple consumers for these data ticks. Can
> Ignite Messaging be used in this scenario?
> >>
> >> Thanks,
> >> Satya.
> >
>

Re: Using Ignite

[Dood]

Kafka may suit your needs as a "queue" with producer/consumer and 
persistence capabilities also.


On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:

Hi,
   We have a scenario where in we have a c++ application that pumps 
out data ticks multiple times in a second. These data ticks needs to 
be displayed in the web front end. We need some middleware platform to 
hold those events temporarily before being consumed by web-front end. 
Can Ignite work as a viable middleware option in this scenario? In 
future we are expecting that there will be multiple consumers for 
these data ticks. Can Ignite Messaging be used in this scenario?


Thanks,
Satya.

Re: Using Ignite

[Denis Magda]

Hi Murthy,

It looks like that you can use Ignite Streaming feature [1] with
pre-configured sliding window [2] basing on what you mean under
"temporarily". One of the advantages of Ignite's sliding windows is that
you can query particular events using advanced SQL, text or other queries.

[1] https://apacheignite.readme.io/docs/streaming--cep
[2] https://apacheignite.readme.io/docs/sliding-windows

--
Denis

On Mon, Jan 18, 2016 at 8:50 PM, Murthy Kakarlamudi 
wrote:

> Thanks all for your responses. Will look into these alternatives.
>
> On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke 
> wrote:
>
>> You seem to look for streaming solutions , such as Spark Streaming or
>> Flink Streaming or Storm
>>
>> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
>> >
>> > Kafka may suit your needs as a "queue" with producer/consumer and
>> persistence capabilities also.
>> >
>> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
>> >> Hi,
>> >>   We have a scenario where in we have a c++ application that pumps out
>> data ticks multiple times in a second. These data ticks needs to be
>> displayed in the web front end. We need some middleware platform to hold
>> those events temporarily before being consumed by web-front end. Can Ignite
>> work as a viable middleware option in this scenario? In future we are
>> expecting that there will be multiple consumers for these data ticks. Can
>> Ignite Messaging be used in this scenario?
>> >>
>> >> Thanks,
>> >> Satya.
>> >
>>
>
>

Re: Using Ignite

[Murthy Kakarlamudi]

Hi Denis,
 That kind of querying will be extremely helpful if it is supported.
Because in our use case, events sitting in the cache need to be queried
based on time as the criteria before it can be published to the clients.
Looks like the sliding window feature fits perfectly for this. Thanks for
sharing.



On Mon, Jan 18, 2016 at 2:27 PM, Denis Magda  wrote:

> Hi Murthy,
>
> It looks like that you can use Ignite Streaming feature [1] with
> pre-configured sliding window [2] basing on what you mean under
> "temporarily". One of the advantages of Ignite's sliding windows is that
> you can query particular events using advanced SQL, text or other queries.
>
> [1] https://apacheignite.readme.io/docs/streaming--cep
> [2] https://apacheignite.readme.io/docs/sliding-windows
>
> --
> Denis
>
> On Mon, Jan 18, 2016 at 8:50 PM, Murthy Kakarlamudi 
> wrote:
>
>> Thanks all for your responses. Will look into these alternatives.
>>
>> On Mon, Jan 18, 2016 at 11:24 AM, Jörn Franke 
>> wrote:
>>
>>> You seem to look for streaming solutions , such as Spark Streaming or
>>> Flink Streaming or Storm
>>>
>>> > On 18 Jan 2016, at 17:19, Dood@ODDO  wrote:
>>> >
>>> > Kafka may suit your needs as a "queue" with producer/consumer and
>>> persistence capabilities also.
>>> >
>>> >> On 1/18/2016 9:52 AM, Murthy Kakarlamudi wrote:
>>> >> Hi,
>>> >>   We have a scenario where in we have a c++ application that pumps
>>> out data ticks multiple times in a second. These data ticks needs to be
>>> displayed in the web front end. We need some middleware platform to hold
>>> those events temporarily before being consumed by web-front end. Can Ignite
>>> work as a viable middleware option in this scenario? In future we are
>>> expecting that there will be multiple consumers for these data ticks. Can
>>> Ignite Messaging be used in this scenario?
>>> >>
>>> >> Thanks,
>>> >> Satya.
>>> >
>>>
>>
>>
>

Re: Using ignite configuration with JCache api

[Denis Magda]

Sridhar,

Please check where Ignite home directory refers to in your case. Just 
executed the following code


System.out.println(System.getProperty(IgniteSystemProperties.IGNITE_HOME))

Then create 'config' folder there and put default-config. xml in this 
folder.


You can always change Ignite home directory by passing the following VM 
options upon your apllication launch:

 -DIGNITE_HOME=

--
Denis

On 11/18/2015 1:06 PM, edwardk wrote:

Hi,

Thanks for your response.

But, I am still not clear on this.

I have ignite-config.xml in my project folder (src/main/resources)

I have used JCache API to create and read from the cache.

In ignite-config.xml I have added cacheConfiguration bean's for each of the
caches I have created using JCache api to configure expiry and eviction
times respectively .

But, it seems the ignite-config.xml has not be referred at all , as expiry
and eviction is not happening as per the configuration.

I even tried adding a config folder(src/main/resources/config) with
default-config.xml in it with same configuration as in igntie-config.xml,
but still it is not working as expected.

There should be some way to refer to my cache configuration xml from JCache.
But, not sure of how to achieve it.

Thanks,
Sridhar







--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Using-ignite-configuration-with-JCache-api-tp2000p2006.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Using ignite configuration with JCache api

[edwardk]

Hi,

Thanks for your response.

But, I am still not clear on this.

I have ignite-config.xml in my project folder (src/main/resources)

I have used JCache API to create and read from the cache.

In ignite-config.xml I have added cacheConfiguration bean's for each of the
caches I have created using JCache api to configure expiry and eviction
times respectively .

But, it seems the ignite-config.xml has not be referred at all , as expiry
and eviction is not happening as per the configuration.

I even tried adding a config folder(src/main/resources/config) with
default-config.xml in it with same configuration as in igntie-config.xml, 
but still it is not working as expected.

There should be some way to refer to my cache configuration xml from JCache.
But, not sure of how to achieve it.

Thanks,
Sridhar







--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Using-ignite-configuration-with-JCache-api-tp2000p2006.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.