Re: Using a cache as an affinity co-located processing buffer in Ignite.Net
There will be two instances, and these instances will have different
lifetimes.
> it seems weird that there are two different interfaces for essentially
the same purpose
They are a bit different, one operates on ICacheEntry, another on
ICacheEntryEvent (which carries additional data)
On Wed, Apr 25, 2018 at 8:40 AM, Raymond Wilson
wrote:
> Let’s say I do this, where I define a single class FileFilter that
> implements both remote filter interfaces (BTW, it seems weird that there
> are two different interfaces for essentially the same purpose):
>
>
>
> public class RemoteFileFilter :
>
> ICacheEntryFilter,
>
> ICacheEntryEventFilter,
>
>
>
>
>
> RemoteFileFilter FileFilter = new RemoteFileFilter();
>
>
>
> // Construct the continuous query machinery
>
> // Set the initial query to return all elements in the cache
>
> // Instantiate the queryHandle and start the continous query
> on the remote nodes
>
> // Note: Only cache items held on this local node will be
> handled here
>
> IContinuousQueryHandle BufferQueueItem>> queryHandle = queueCache.QueryContinuous
>
> (qry: new ContinuousQuery BufferQueueItem>(new LocalFileListener())
>
> {
>
> Filter = FileFilter
>
> },
>
> initialQry: new ScanQuery
>
> {
>
> Filter = FileFilter
>
> }))
>
>
>
> Here, the same filter is supplied to both the continuous query and initial
> scan query aspects. When this continuous query is serialized, send to the
> remote node, then deserialised, will the continuous query on the remote
> node retain two references to the same remote filter, or two instances of
> the remote filter?
>
>
>
> Thanks,
>
> Raymond.
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
> *Sent:* Wednesday, April 25, 2018 5:19 PM
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Using a cache as an affinity co-located processing buffer
> in Ignite.Net
>
>
>
> What do you mean by "instance"? In terms of CLR that would be a different
> instance on every node.
>
>
>
> On Wed, Apr 25, 2018 at 2:50 AM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> I’m using a Continuous Query in both options (grid deployed service using
> a CQ versus an independent context using a CQ). I was curious which context
> using a CQ would be seen as desirable.
>
>
>
> In the case where a filter is provided to a CQ for both the initial query
> and for newly items arriving in the cache I would need to supply the same
> filter instance for both as the processing logic has state that will need
> to be shared between the two. Once the CQ has been serialized to the remote
> nodes, will that filter be two separate instances or will is retain the
> same singular instance?
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
> *Sent:* Wednesday, April 25, 2018 6:08 AM
>
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Using a cache as an affinity co-located processing buffer
> in Ignite.Net
>
>
>
> ContinuousQuery is the best practice for most kinds of streaming use
> cases. I think it fits your use case as well.
>
>
>
> On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> Thanks, that makes sense.
>
>
>
> From a best practices perspective, is better to have a grid deployed
> service on each node executing local continuous queries against the cache
> and orchestrating the processing from within the service, versus having
> some singular context in the grid that uses the continuous query by placing
> processing orchestration logic in the filter sent to the remote nodes?
>
> Sent from my iPhone
>
>
> On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
>
> Sorry, looks like I have misunderstood you.
>
>
>
> If you need initial scan, of course you can have it by using ScanQuery as
> initialQuery.
>
> Place all the processing logic into the ScanQuery filter, and return false
> from there.
>
> This way you can process all existing entries in a co-located fashion
> without sending them to the initiator node.
>
>
>
> Thanks,
>
> Pavel
>
>
>
> On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> Not being able to do an initial scan of elements on the remote nodes is a
> bit of a problem (possibly a bug?)
>
>
>
> Something that’s occurred to me is to wrap this behaviour into an Ignite
> service deployed onto all of the server nodes, and use a local mode
> continuous query from within each service to perform an initial scan of
> elements and then steady state handling as new elements arrive.
>
>
>
> The reason the initial scan is important is I need to handle cases where
> there may be a non-trivial queue of items waiting for processing and there
> is either a shutdown/restart of the grid, or there is a topology change
> event that triggers rebalancing
>
>
>
> *From:* Pavel
RE: Using a cache as an affinity co-located processing buffer in Ignite.Net
Let’s say I do this, where I define a single class FileFilter that
implements both remote filter interfaces (BTW, it seems weird that there
are two different interfaces for essentially the same purpose):
public class RemoteFileFilter :
ICacheEntryFilter,
ICacheEntryEventFilter,
RemoteFileFilter FileFilter = new RemoteFileFilter();
// Construct the continuous query machinery
// Set the initial query to return all elements in the cache
// Instantiate the queryHandle and start the continous query on
the remote nodes
// Note: Only cache items held on this local node will be
handled here
IContinuousQueryHandle> queryHandle = queueCache.QueryContinuous
(qry: new ContinuousQuery(
new LocalFileListener())
{
Filter = FileFilter
},
initialQry: new ScanQuery
{
Filter = FileFilter
}))
Here, the same filter is supplied to both the continuous query and initial
scan query aspects. When this continuous query is serialized, send to the
remote node, then deserialised, will the continuous query on the remote
node retain two references to the same remote filter, or two instances of
the remote filter?
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Wednesday, April 25, 2018 5:19 PM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
What do you mean by "instance"? In terms of CLR that would be a different
instance on every node.
On Wed, Apr 25, 2018 at 2:50 AM, Raymond Wilson
wrote:
I’m using a Continuous Query in both options (grid deployed service using a
CQ versus an independent context using a CQ). I was curious which context
using a CQ would be seen as desirable.
In the case where a filter is provided to a CQ for both the initial query
and for newly items arriving in the cache I would need to supply the same
filter instance for both as the processing logic has state that will need
to be shared between the two. Once the CQ has been serialized to the remote
nodes, will that filter be two separate instances or will is retain the
same singular instance?
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Wednesday, April 25, 2018 6:08 AM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
ContinuousQuery is the best practice for most kinds of streaming use cases.
I think it fits your use case as well.
On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson
wrote:
Thanks, that makes sense.
>From a best practices perspective, is better to have a grid deployed
service on each node executing local continuous queries against the cache
and orchestrating the processing from within the service, versus having
some singular context in the grid that uses the continuous query by placing
processing orchestration logic in the filter sent to the remote nodes?
Sent from my iPhone
On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
Sorry, looks like I have misunderstood you.
If you need initial scan, of course you can have it by using ScanQuery as
initialQuery.
Place all the processing logic into the ScanQuery filter, and return false
from there.
This way you can process all existing entries in a co-located fashion
without sending them to the initiator node.
Thanks,
Pavel
On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson
wrote:
Not being able to do an initial scan of elements on the remote nodes is a
bit of a problem (possibly a bug?)
Something that’s occurred to me is to wrap this behaviour into an Ignite
service deployed onto all of the server nodes, and use a local mode
continuous query from within each service to perform an initial scan of
elements and then steady state handling as new elements arrive.
The reason the initial scan is important is I need to handle cases where
there may be a non-trivial queue of items waiting for processing and there
is either a shutdown/restart of the grid, or there is a topology change
event that triggers rebalancing
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Tuesday, April 24, 2018 5:54 AM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
> Is the initial query also run in the context of the remote node and the
remote filter?
No, it is just a query (can be SQL or Scan) which allows you to get a "full
picture" on the calling node:
all existing data and all future data.
So in your scenario it is not very useful.
> return false from the filter so the element is not sent to the local
listener
Yes, exactly
On Mon, Apr 23, 2018 at 11:18 AM, Raymond Wilson
wrote:
OK – I see how that works.
In the page https://apacheignite-net.readme.io/docs
Re: Service Grid and .net core 2
Антон, здравствуйте,
Да, services работают в полном объёме под .NET Core, тесты под Linux все
бегают.
Можете прислать минимальный проект, где это воспроизводится?
Спасибо,
Павел
On Wed, Apr 25, 2018 at 1:30 AM, Anton Levkovich wrote:
> Здравствуйте Павел,
>
> подскажите, работает ли в Apache Ignite.net Service Grid для .net core 2.
>
> В примере /examples/dotnetcore/... отсутсует пример для Service.
> Компиляции кода под .net framework все работает, а .net core 2 ошибка на
> этапе получения сервиса (ignite.GetServices().GetServiceProxy string>>("service", true)):
>
> System.TypeInitializationException: 'The type initializer for
> 'Apache.Ignite.Core.Impl.Services.ServiceProxyFactory`1' threw an
> exception.'
>
> MissingMethodException: Method not found:
> 'System.Reflection.Emit.AssemblyBuilder
> System.AppDomain.DefineDynamicAssembly(System.Reflection.AssemblyName,
> System.Reflection.Emit.AssemblyBuilderAccess)'.
>
>
>
> --
> С уважением,
> Левкович Антон
>
Re: Using a cache as an affinity co-located processing buffer in Ignite.Net
What do you mean by "instance"? In terms of CLR that would be a different
instance on every node.
On Wed, Apr 25, 2018 at 2:50 AM, Raymond Wilson
wrote:
> I’m using a Continuous Query in both options (grid deployed service using
> a CQ versus an independent context using a CQ). I was curious which context
> using a CQ would be seen as desirable.
>
>
>
> In the case where a filter is provided to a CQ for both the initial query
> and for newly items arriving in the cache I would need to supply the same
> filter instance for both as the processing logic has state that will need
> to be shared between the two. Once the CQ has been serialized to the remote
> nodes, will that filter be two separate instances or will is retain the
> same singular instance?
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
> *Sent:* Wednesday, April 25, 2018 6:08 AM
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Using a cache as an affinity co-located processing buffer
> in Ignite.Net
>
>
>
> ContinuousQuery is the best practice for most kinds of streaming use
> cases. I think it fits your use case as well.
>
>
>
> On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> Thanks, that makes sense.
>
>
>
> From a best practices perspective, is better to have a grid deployed
> service on each node executing local continuous queries against the cache
> and orchestrating the processing from within the service, versus having
> some singular context in the grid that uses the continuous query by placing
> processing orchestration logic in the filter sent to the remote nodes?
>
> Sent from my iPhone
>
>
> On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
>
> Sorry, looks like I have misunderstood you.
>
>
>
> If you need initial scan, of course you can have it by using ScanQuery as
> initialQuery.
>
> Place all the processing logic into the ScanQuery filter, and return false
> from there.
>
> This way you can process all existing entries in a co-located fashion
> without sending them to the initiator node.
>
>
>
> Thanks,
>
> Pavel
>
>
>
> On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> Not being able to do an initial scan of elements on the remote nodes is a
> bit of a problem (possibly a bug?)
>
>
>
> Something that’s occurred to me is to wrap this behaviour into an Ignite
> service deployed onto all of the server nodes, and use a local mode
> continuous query from within each service to perform an initial scan of
> elements and then steady state handling as new elements arrive.
>
>
>
> The reason the initial scan is important is I need to handle cases where
> there may be a non-trivial queue of items waiting for processing and there
> is either a shutdown/restart of the grid, or there is a topology change
> event that triggers rebalancing
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
> *Sent:* Tuesday, April 24, 2018 5:54 AM
>
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Using a cache as an affinity co-located processing buffer
> in Ignite.Net
>
>
>
> > Is the initial query also run in the context of the remote node and
> the remote filter?
>
> No, it is just a query (can be SQL or Scan) which allows you to get a
> "full picture" on the calling node:
>
> all existing data and all future data.
>
>
>
> So in your scenario it is not very useful.
>
>
>
> > return false from the filter so the element is not sent to the local
> listener
>
> Yes, exactly
>
>
>
> On Mon, Apr 23, 2018 at 11:18 AM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
> OK – I see how that works.
>
>
>
> In the page https://apacheignite-net.readme.io/docs/continuous-queries ,
> there is this code:
>
>
>
> using (var queryHandle = cache.QueryContinuous(qry, initialQry))
>
> {
>
> // Iterate through existing data stored in cache.
>
> foreach (var entry in queryHandle.GetInitialQueryCursor())
>
> Console.WriteLine("key={0}, val={1}", entry.Key, entry.Value);
>
>
>
> // Add a few more keys and watch a few more query notifications.
>
> for (int i = 5; i < 15; i++)
>
> cache.Put(i, i.ToString());
>
> }
>
>
>
> Is the initial query also run in the context of the remote node and the
> remote filter?
>
>
>
> Construction of the ContinuousQuery also requires provision of
> LocalListener to receive the cache update items. Is the approach here to
> processing the element in the remote filter context then return false from
> the filter so the element is not sent to the local listener?
>
>
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
> *Sent:* Monday, April 23, 2018 7:50 PM
>
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Using a cache as an affinity co-located processing buffer
> in Ignite.Net
>
>
>
> Remote Listener is deployed on every cache node and is invoked only on a
> primary node for that key.
>
> In other words, for each key there is only one invocation of the remote
> filter, and that invocation i
Re: Apache Ignite application deploy without rebalancing
Unfortunately not work T^T
I did 2way like below
1. I set region name in cache configuration -> not work
2. I set DefaultDataRegionConfiguration except regionName -> not work
below log is when I set DefaultDataRegionConfiguration except any other
region
**) when loading*
2018-04-25 13:32:44.887 WARN 5837 --- [ main]
o.a.i.s.c.tcp.TcpCommunicationSpi: Message queue limit is set to 0
which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[13:32:44] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
2018-04-25 13:32:45.076 WARN 5837 --- [ main]
o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to
enable configure any GridCheckpointSpi implementation)
"/data/log/catalina/localhost/catalina.out" 193L, 26597C
[13:37:39] Data Regions Configured:
[13:37:39] ^-- default [initSize=2.0 GiB, maxSize=2.0 GiB,
persistenceEnabled=true]
2018-04-25 13:37:39.744 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped
"{[/ready/deploy/group],methods=[POST]}" onto public java.lang.Object
cc.platform.expresso.storage.controller.ManageController.clusterBaselineTopologying()
2018-04-25 13:37:39.745 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped
"{[/cluster/activation],methods=[POST]}" onto public void
cc.platform.expresso.storage.controller.ManageController.activeCluster()
2018-04-25 13:37:39.746 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped
"{[/version],methods=[GET]}" onto public java.lang.Object
cc.platform.expresso.storage.controller.ManageController.version()
2018-04-25 13:37:39.768 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped
"{[/error],produces=[text/html]}" onto public
org.springframework.web.servlet.ModelAndView
org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2018-04-25 13:37:39.782 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public
org.springframework.http.ResponseEntity>
org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2018-04-25 13:37:40.035 INFO 6006 --- [ main]
s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice:
org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@5ecddf8f:
startup date [Wed Apr 25 13:37:28 KST 2018]; root of context hierarchy
2018-04-25 13:37:40.681 INFO 6006 --- [ main]
o.s.j.d.e.EmbeddedDatabaseFactory: Starting embedded database:
url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false',
username='sa'
2018-04-25 13:37:42.827 INFO 6006 --- [ main]
o.s.j.e.a.AnnotationMBeanExporter: Registering beans for JMX
exposure on startup
2018-04-25 13:37:42.856 INFO 6006 --- [ main]
o.s.j.e.a.AnnotationMBeanExporter: Bean with name 'startNode' has
been autodetected for JMX exposure
2018-04-25 13:37:42.864 INFO 6006 --- [ main]
o.s.j.e.a.AnnotationMBeanExporter: Located MBean 'startNode':
registering with JMX server as MBean
[org.apache.ignite.internal:name=startNode,type=IgniteKernal]
2018-04-25 13:37:43.261 INFO 6006 --- [ main]
o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080
(http) with context path ''
2018-04-25 13:37:43.277 INFO 6006 --- [ main]
c.p.expresso.storage.StorageApplication : Started StorageApplication in
16.334 seconds (JVM running for 18.209)
[13:37:52] Topology snapshot [ver=2, servers=2, clients=0, CPUs=2,
offheap=4.0GB, heap=2.0GB]
[13:37:52] Data Regions Configured:
[13:37:52] ^-- default [initSize=2.0 GiB, maxSize=2.0 GiB,
persistenceEnabled=true]
2018-04-25 13:37:52.044 INFO 6006 --- [vent-worker-#35]
c.p.e.s.c.c.EventListenerConfigurator: Event - time :
2018-04-25T13:37:52.039, name : NODE_JOINED, type : 10, msg: Node joined:
TcpDiscoveryNode [id=6a61cd48-e111-456f-9376-e3198323fa83,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.135.35],
sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
/172.16.135.35:47500], discPort=47500, order=2, intOrder=2,
lastExchangeTime=1524631071855, loc=false, ver=2.4.0#20180305-sha1:aa342270,
isClient=false], nodeId : 48ee771f-2b52-425c-b0bf-70985508e34b
[13:37:58] Topology snapshot [ver=3, servers=3, clients=0, CPUs=3,
offheap=6.0GB, heap=3.0GB]
[13:37:58] Data Regions Configured:
[13:37:58] ^-- default [initSize=2.0 GiB, maxSize=2.0 GiB,
persistenceEnabled=true]
2018-04-25 13:37:58.584 INFO 6006 --- [vent-worker-#35]
c.p.e.s.c.c.EventListenerConfigurator:
RE: Using a cache as an affinity co-located processing buffer in Ignite.Net
I’m using a Continuous Query in both options (grid deployed service using a
CQ versus an independent context using a CQ). I was curious which context
using a CQ would be seen as desirable.
In the case where a filter is provided to a CQ for both the initial query
and for newly items arriving in the cache I would need to supply the same
filter instance for both as the processing logic has state that will need
to be shared between the two. Once the CQ has been serialized to the remote
nodes, will that filter be two separate instances or will is retain the
same singular instance?
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Wednesday, April 25, 2018 6:08 AM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
ContinuousQuery is the best practice for most kinds of streaming use cases.
I think it fits your use case as well.
On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson
wrote:
Thanks, that makes sense.
>From a best practices perspective, is better to have a grid deployed
service on each node executing local continuous queries against the cache
and orchestrating the processing from within the service, versus having
some singular context in the grid that uses the continuous query by placing
processing orchestration logic in the filter sent to the remote nodes?
Sent from my iPhone
On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
Sorry, looks like I have misunderstood you.
If you need initial scan, of course you can have it by using ScanQuery as
initialQuery.
Place all the processing logic into the ScanQuery filter, and return false
from there.
This way you can process all existing entries in a co-located fashion
without sending them to the initiator node.
Thanks,
Pavel
On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson
wrote:
Not being able to do an initial scan of elements on the remote nodes is a
bit of a problem (possibly a bug?)
Something that’s occurred to me is to wrap this behaviour into an Ignite
service deployed onto all of the server nodes, and use a local mode
continuous query from within each service to perform an initial scan of
elements and then steady state handling as new elements arrive.
The reason the initial scan is important is I need to handle cases where
there may be a non-trivial queue of items waiting for processing and there
is either a shutdown/restart of the grid, or there is a topology change
event that triggers rebalancing
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Tuesday, April 24, 2018 5:54 AM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
> Is the initial query also run in the context of the remote node and the
remote filter?
No, it is just a query (can be SQL or Scan) which allows you to get a "full
picture" on the calling node:
all existing data and all future data.
So in your scenario it is not very useful.
> return false from the filter so the element is not sent to the local
listener
Yes, exactly
On Mon, Apr 23, 2018 at 11:18 AM, Raymond Wilson
wrote:
OK – I see how that works.
In the page https://apacheignite-net.readme.io/docs/continuous-queries ,
there is this code:
using (var queryHandle = cache.QueryContinuous(qry, initialQry))
{
// Iterate through existing data stored in cache.
foreach (var entry in queryHandle.GetInitialQueryCursor())
Console.WriteLine("key={0}, val={1}", entry.Key, entry.Value);
// Add a few more keys and watch a few more query notifications.
for (int i = 5; i < 15; i++)
cache.Put(i, i.ToString());
}
Is the initial query also run in the context of the remote node and the
remote filter?
Construction of the ContinuousQuery also requires provision of
LocalListener to receive the cache update items. Is the approach here to
processing the element in the remote filter context then return false from
the filter so the element is not sent to the local listener?
*From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
*Sent:* Monday, April 23, 2018 7:50 PM
*To:* user@ignite.apache.org
*Subject:* Re: Using a cache as an affinity co-located processing buffer in
Ignite.Net
Remote Listener is deployed on every cache node and is invoked only on a
primary node for that key.
In other words, for each key there is only one invocation of the remote
filter, and that invocation is local to that key.
So you can place your processing logic into the Remote Filter.
On Mon, Apr 23, 2018 at 10:42 AM, Raymond Wilson
wrote:
Hi Pavel,
Yes, I looked at continuous queries. They appear to be oriented toward a
single context being sent the newly arrived elements in the cache from all
primary nodes hosting the cache involved in the query.
In the use case I outlined below, I would like to have the items processed
in co-located contexts (ie: the data does not move and is processed in situ
Re: Ignite client to connect only to cache owners
Hello Subash, The client node directly communicates with a primary node [1] from the cluster (via TcpComminicationSpi [2]) in order to get a value from a cache. There are no intermediate nodes. The same fact is correct for server nodes as well. [1] https://apacheignite.readme.io/docs/primary-and-backup-copies [2] https://apacheignite.readme.io/docs/network-config#tcpcommunicationspi Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Effective Data through DataStream
Hi, Someone could help me? Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using a cache as an affinity co-located processing buffer in Ignite.Net
ContinuousQuery is the best practice for most kinds of streaming use cases.
I think it fits your use case as well.
On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson wrote:
> Thanks, that makes sense.
>
> From a best practices perspective, is better to have a grid deployed
> service on each node executing local continuous queries against the cache
> and orchestrating the processing from within the service, versus having
> some singular context in the grid that uses the continuous query by placing
> processing orchestration logic in the filter sent to the remote nodes?
>
> Sent from my iPhone
>
> On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
>
> Sorry, looks like I have misunderstood you.
>
> If you need initial scan, of course you can have it by using ScanQuery as
> initialQuery.
> Place all the processing logic into the ScanQuery filter, and return false
> from there.
> This way you can process all existing entries in a co-located fashion
> without sending them to the initiator node.
>
> Thanks,
> Pavel
>
> On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson <
> raymond_wil...@trimble.com> wrote:
>
>> Not being able to do an initial scan of elements on the remote nodes is a
>> bit of a problem (possibly a bug?)
>>
>>
>>
>> Something that’s occurred to me is to wrap this behaviour into an Ignite
>> service deployed onto all of the server nodes, and use a local mode
>> continuous query from within each service to perform an initial scan of
>> elements and then steady state handling as new elements arrive.
>>
>>
>>
>> The reason the initial scan is important is I need to handle cases where
>> there may be a non-trivial queue of items waiting for processing and there
>> is either a shutdown/restart of the grid, or there is a topology change
>> event that triggers rebalancing
>>
>>
>>
>> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> *Sent:* Tuesday, April 24, 2018 5:54 AM
>>
>> *To:* user@ignite.apache.org
>> *Subject:* Re: Using a cache as an affinity co-located processing buffer
>> in Ignite.Net
>>
>>
>>
>> > Is the initial query also run in the context of the remote node and
>> the remote filter?
>>
>> No, it is just a query (can be SQL or Scan) which allows you to get a
>> "full picture" on the calling node:
>>
>> all existing data and all future data.
>>
>>
>>
>> So in your scenario it is not very useful.
>>
>>
>>
>> > return false from the filter so the element is not sent to the local
>> listener
>>
>> Yes, exactly
>>
>>
>>
>> On Mon, Apr 23, 2018 at 11:18 AM, Raymond Wilson <
>> raymond_wil...@trimble.com> wrote:
>>
>> OK – I see how that works.
>>
>>
>>
>> In the page https://apacheignite-net.readme.io/docs/continuous-queries ,
>> there is this code:
>>
>>
>>
>> using (var queryHandle = cache.QueryContinuous(qry, initialQry))
>>
>> {
>>
>> // Iterate through existing data stored in cache.
>>
>> foreach (var entry in queryHandle.GetInitialQueryCursor())
>>
>> Console.WriteLine("key={0}, val={1}", entry.Key, entry.Value);
>>
>>
>>
>> // Add a few more keys and watch a few more query notifications.
>>
>> for (int i = 5; i < 15; i++)
>>
>> cache.Put(i, i.ToString());
>>
>> }
>>
>>
>>
>> Is the initial query also run in the context of the remote node and the
>> remote filter?
>>
>>
>>
>> Construction of the ContinuousQuery also requires provision of
>> LocalListener to receive the cache update items. Is the approach here to
>> processing the element in the remote filter context then return false from
>> the filter so the element is not sent to the local listener?
>>
>>
>>
>>
>>
>> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> *Sent:* Monday, April 23, 2018 7:50 PM
>>
>>
>> *To:* user@ignite.apache.org
>> *Subject:* Re: Using a cache as an affinity co-located processing buffer
>> in Ignite.Net
>>
>>
>>
>> Remote Listener is deployed on every cache node and is invoked only on a
>> primary node for that key.
>>
>> In other words, for each key there is only one invocation of the remote
>> filter, and that invocation is local to that key.
>>
>>
>>
>> So you can place your processing logic into the Remote Filter.
>>
>>
>>
>> On Mon, Apr 23, 2018 at 10:42 AM, Raymond Wilson <
>> raymond_wil...@trimble.com> wrote:
>>
>> Hi Pavel,
>>
>>
>>
>> Yes, I looked at continuous queries. They appear to be oriented toward a
>> single context being sent the newly arrived elements in the cache from all
>> primary nodes hosting the cache involved in the query.
>>
>>
>>
>> In the use case I outlined below, I would like to have the items
>> processed in co-located contexts (ie: the data does not move and is
>> processed in situ on the primary node). How do you do that with a
>> continuous query?
>>
>>
>>
>> Thanks,
>>
>> Raymond.
>>
>>
>>
>> *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> *Sent:* Monday, April 23, 2018 7:18 PM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: Using a cache as an affinity co-located processing buffer
>> in Ignite.Net
>>
>>
>>
>> Hi R
Re: Ignite client to connect only to cache owners
Hi, Subash, By default, a cache is partitioned and partitions spread across the cluster, so you can't avoid network activity. On Tue, Apr 24, 2018 at 7:22 PM, Subash Chaturanga wrote: > Hi everyone, > Is there a way for a ignite client to connect only to the set of servers > who holds the cache or its backups. So that it will not requests from a > server which then has to talk to another node and return the values. > > I know near cache is there, but that again increase allocated cache size > to the node and I wouldn’t be able to do a true partitioned cache. > -- > /subash >
Ignite client to connect only to cache owners
Hi everyone, Is there a way for a ignite client to connect only to the set of servers who holds the cache or its backups. So that it will not requests from a server which then has to talk to another node and return the values. I know near cache is there, but that again increase allocated cache size to the node and I wouldn’t be able to do a true partitioned cache. -- /subash
Re: Why Ignite SQL can not support transactional mode transaction
hi vkulichenko I may not express myself clearly，What I mean is that since KV storage supports transactions, why doesn't SQL support transactional transactions? Dose Transaction impact other session‘s select SQL？ -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Apache Ignite application deploy without rebalancing
Hello,
It seems that you have to assign your cacheы to "myName" data region as
follows:
config.setCacheConfiguration(
new CacheConfiguration()
.setName(name1)
.*setDataRegionName("myName")*
.setBackups(1)
.setCacheMode(PARTITIONED)
.setWriteSynchronizationMode(PRIMARY_SYNC)
.setCopyOnRead(false));
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite configuration for starting in client mode
Hello Prasad, I think it is not necessary to setup cache configurations and/or data storage config on the client node until you don't use local caches. The most important things that you have to configure are IgniteConfiguration#clientMode(true) [1] and Discovery SPI [2]. [1] https://apacheignite.readme.io/docs/clients-vs-servers#section-configuring-clients-and-servers [2] https://apacheignite.readme.io/docs/cluster-config Thanks! 2018-04-24 10:40 GMT+03:00 Prasad Bhalerao : > Hi, > > I want to start ignite in tomcat container but in client mode. > > Is it necessary to set cache configuration CacheConfiguration and > DataStorageConfiguration in IgniteConfiguration while starting the in > client mode? > > Thanks, > Prasad >
Re: Ignite configuration for starting in client mode
Hi,
You can omit cache configuration, in this case Ignite will pick default
configuration for caches you will create using
getOrCreateCache("CACHE_NAME").
If you want to enable Ignite Persistence, you have to set
DataStorageConfiguration at least to enable it with default configuration:
https://apacheignite-net.readme.io/v2.4/docs/ignite-persistent-store
Best regards,
Anton
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: When I use Ignite create table ddl，can I set the db schema instead of the schema name "PUBLIC"？
Hello, As for now, all tables created through DDL 'CREATE TABLE' are bounded to the 'PUBLIC' schema. There is a JIRA ticket that will allow assigning a table to a custom schema. [1] https://apacheignite-sql.readme.io/docs/create-table [2] https://issues.apache.org/jira/browse/IGNITE-5780 Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Apache Ignite application deploy without rebalancing
Hi, I asked this question at stackoverflow.
In my case, Spring Rest API embed ignite, that API is act as cache. There
are only Put/Get API, and they put to /get from Ignite caches.
My total Configuration like below(Java Code)
IgniteConfiguration config = new IgniteConfiguration();
//discovery : multicast, I set consistentId manually for easily
Identifying me, actually I set named node1, node2, node3
config.setRebalanceThreadPoolSize(3);
config.setConsistentId(uniqueNodeId);
//3caches, Partitioned,
config.setCacheConfiguration(
new CacheConfiguration()
.setName(name1)
.setBackups(1)
.setCacheMode(PARTITIONED)
.setWriteSynchronizationMode(PRIMARY_SYNC)
.setCopyOnRead(false),
new CacheConfiguration()
.setName(name2)
.setBackups(1)
.setCacheMode(PARTITIONED)
.setWriteSynchronizationMode(PRIMARY_SYNC)
.setCopyOnRead(false),
new CacheConfiguration()
.setName(name3)
.setBackups(1)
.setCacheMode(PARTITIONED)
.setWriteSynchronizationMode(PRIMARY_SYNC)
.setCopyOnRead(false))
//pageSize : 4096, enable persistent, initRegionSize/maxRegionSize : 2G
config.setDataStorageConfiguration(new DataStorageConfiguration()
.setPageSize(4096)
.setMetricsEnabled(false)
.setDataRegionConfigurations(new DataRegionConfiguration()
.setPersistenceEnabled(true)
.setName("myName")
.setInitialSize(2147483648)
.setMaxSize(2147483648)
.setPageEvictionMode(DataPageEvictionMode.DISABLED)
.setMetricsEnabled(false)));
//event listening, for monitoring
int [] listenedEvent = {
EventType.EVT_CACHE_NODES_LEFT,
EventType.EVT_CACHE_OBJECT_PUT,
EventType.EVT_CACHE_REBALANCE_STARTED,
EventType.EVT_NODE_JOINED
};
config.setIncludeEventTypes(listenedEvent);
config.setLocalEventListeners(new HashMap, int[]>() {{
put(event -> {
LOGGER.info("Event - time : {}, name : {}, type : {}, msg: {},
nodeId : {}",
LocalDateTime.ofInstant(Instant.ofEpochMilli(event.timestamp()),
ZoneId.systemDefault()),
event.name(), event.type(), event.message(), event.node().id());
return true;
}, listenedEvent);
}});
then, when I started tomcat, below is my ignite log/I checked here,
persistentEnabled true and 3 nodes are all joined.
[18:17:19] (wrn) Failed to resolve IGNITE_HOME automatically for class
codebase [class=class o.a.i.i.util.IgniteUtils, e=URI is not hierarchical]
2018-04-23 18:17:19.086 ERROR 9519 --- [ main]
: Failed to resolve default logging config file:
config/java.util.logging.properties
Console logging handler is not configured.
[18:17:19]__
[18:17:19] / _/ ___/ |/ / _/_ __/ __/
[18:17:19] _/ // (7 7// / / / / _/
[18:17:19] /___/\___/_/|_/___/ /_/ /___/
[18:17:19]
[18:17:19] ver. 2.4.0#20180305-sha1:aa342270
[18:17:19] 2018 Copyright(C) Apache Software Foundation
[18:17:19]
[18:17:19] Ignite documentation: http://ignite.apache.org
[18:17:19]
[18:17:19] Quiet mode.
[18:17:19] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[18:17:19] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[18:17:19]
[18:17:19] OS: Linux 3.10.0-327.36.3.el7.x86_64 amd64
[18:17:19] VM information: Java(TM) SE Runtime Environment 1.8.0_171-b11
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.171-b11
[18:17:19] Configured plugins:
[18:17:19] ^-- None
[18:17:19]
2018-04-23 18:17:19.923 WARN 9519 --- [ main]
o.a.i.s.c.tcp.TcpCommunicationSpi: Message queue limit is set to 0
which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[18:17:19] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
2018-04-23 18:17:19.964 WARN 9519 --- [ main]
o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to
enable configure any GridCheckpointSpi implementation)
2018-04-23 18:17:20.031 WARN 9519 --- [ main]
o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled
(all jobs will be activated upon arrival).
[18:17:20] Security status [authentication=off, tls/ssl=off]
2018-04-23 18:17:21.046 WARN 9519 --- [ main]
.s.d.t.i.m.TcpDiscoveryMulticastIpFinder : TcpDiscoveryMulticastIpFinder has
no pre-configured addresses (it is recommended in production to specify at
least one address in TcpDiscoveryMulticastIpFinder.getAddresses()
configuration property)
2018-04-23 18:17:22.803 WARN 9519 --- [ main]
org.apache.ignite.internal.IgniteKernal : Nodes started on local machine
require more than 20% of phy
Re: Inconsistency reading cache from code and via REST?
ok that's great; wasn't too clear from the docs that I needed the addQueryField -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite configuration for starting in client mode
Hi, I want to start ignite in tomcat container but in client mode. Is it necessary to set cache configuration CacheConfiguration and DataStorageConfiguration in IgniteConfiguration while starting the in client mode? Thanks, Prasad
Re: Using a cache as an affinity co-located processing buffer in Ignite.Net
Thanks, that makes sense.
From a best practices perspective, is better to have a grid deployed service on
each node executing local continuous queries against the cache and
orchestrating the processing from within the service, versus having some
singular context in the grid that uses the continuous query by placing
processing orchestration logic in the filter sent to the remote nodes?
Sent from my iPhone
> On 24/04/2018, at 6:53 PM, Pavel Tupitsyn wrote:
>
> Sorry, looks like I have misunderstood you.
>
> If you need initial scan, of course you can have it by using ScanQuery as
> initialQuery.
> Place all the processing logic into the ScanQuery filter, and return false
> from there.
> This way you can process all existing entries in a co-located fashion without
> sending them to the initiator node.
>
> Thanks,
> Pavel
>
>> On Mon, Apr 23, 2018 at 11:50 PM, Raymond Wilson
>> wrote:
>> Not being able to do an initial scan of elements on the remote nodes is a
>> bit of a problem (possibly a bug?)
>>
>>
>>
>> Something that’s occurred to me is to wrap this behaviour into an Ignite
>> service deployed onto all of the server nodes, and use a local mode
>> continuous query from within each service to perform an initial scan of
>> elements and then steady state handling as new elements arrive.
>>
>>
>>
>> The reason the initial scan is important is I need to handle cases where
>> there may be a non-trivial queue of items waiting for processing and there
>> is either a shutdown/restart of the grid, or there is a topology change
>> event that triggers rebalancing
>>
>>
>>
>> From: Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> Sent: Tuesday, April 24, 2018 5:54 AM
>>
>>
>> To: user@ignite.apache.org
>> Subject: Re: Using a cache as an affinity co-located processing buffer in
>> Ignite.Net
>>
>>
>> > Is the initial query also run in the context of the remote node and the
>> > remote filter?
>>
>> No, it is just a query (can be SQL or Scan) which allows you to get a "full
>> picture" on the calling node:
>>
>> all existing data and all future data.
>>
>>
>>
>> So in your scenario it is not very useful.
>>
>>
>>
>> > return false from the filter so the element is not sent to the local
>> > listener
>>
>> Yes, exactly
>>
>>
>>
>> On Mon, Apr 23, 2018 at 11:18 AM, Raymond Wilson
>> wrote:
>>
>> OK – I see how that works.
>>
>>
>>
>> In the page https://apacheignite-net.readme.io/docs/continuous-queries ,
>> there is this code:
>>
>>
>>
>> using (var queryHandle = cache.QueryContinuous(qry, initialQry))
>>
>> {
>>
>> // Iterate through existing data stored in cache.
>>
>> foreach (var entry in queryHandle.GetInitialQueryCursor())
>>
>> Console.WriteLine("key={0}, val={1}", entry.Key, entry.Value);
>>
>>
>>
>> // Add a few more keys and watch a few more query notifications.
>>
>> for (int i = 5; i < 15; i++)
>>
>> cache.Put(i, i.ToString());
>>
>> }
>>
>>
>>
>> Is the initial query also run in the context of the remote node and the
>> remote filter?
>>
>>
>>
>> Construction of the ContinuousQuery also requires provision of LocalListener
>> to receive the cache update items. Is the approach here to processing the
>> element in the remote filter context then return false from the filter so
>> the element is not sent to the local listener?
>>
>>
>>
>>
>>
>> From: Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> Sent: Monday, April 23, 2018 7:50 PM
>>
>>
>> To: user@ignite.apache.org
>> Subject: Re: Using a cache as an affinity co-located processing buffer in
>> Ignite.Net
>>
>>
>>
>> Remote Listener is deployed on every cache node and is invoked only on a
>> primary node for that key.
>>
>> In other words, for each key there is only one invocation of the remote
>> filter, and that invocation is local to that key.
>>
>>
>>
>> So you can place your processing logic into the Remote Filter.
>>
>>
>>
>> On Mon, Apr 23, 2018 at 10:42 AM, Raymond Wilson
>> wrote:
>>
>> Hi Pavel,
>>
>>
>>
>> Yes, I looked at continuous queries. They appear to be oriented toward a
>> single context being sent the newly arrived elements in the cache from all
>> primary nodes hosting the cache involved in the query.
>>
>>
>>
>> In the use case I outlined below, I would like to have the items processed
>> in co-located contexts (ie: the data does not move and is processed in situ
>> on the primary node). How do you do that with a continuous query?
>>
>>
>>
>> Thanks,
>>
>> Raymond.
>>
>>
>>
>> From: Pavel Tupitsyn [mailto:ptupit...@apache.org]
>> Sent: Monday, April 23, 2018 7:18 PM
>> To: user@ignite.apache.org
>> Subject: Re: Using a cache as an affinity co-located processing buffer in
>> Ignite.Net
>>
>>
>>
>> Hi Raymond,
>>
>>
>>
>> To process incoming data in a co-located fashion there is a Continuous Query
>> feature [1].
>>
>> Looks like it fits your use ca
