Re: Server Nodes Stopped Unexpectedly
Can someone please help me on this issue ? On Wed, Jul 24, 2019 at 12:04 PM Akash Shinde wrote: > Hi, > Please find attached logs from all server and client nodes.Also attached > gc logs for each node. > > Thanks, > Akash > > > On Tue, Jul 23, 2019 at 8:59 PM Evgenii Zhuravlev < > e.zhuravlev...@gmail.com> wrote: > >> Hi, >> >> Can you please share full logs from the node start from all nodes in the >> cluster? >> >> Thanks, >> Evgenii >> >> вт, 23 июл. 2019 г. в 16:51, Akash Shinde : >> >>> I am using Ignite 2.6 version. I have created a cluster of 7 server >>> nodes and three client nodes. Out of seven nodes five nodes stopped >>> unexpectedly with below error logs lines. >>> I have attached logs of two such server nodes. >>> >>> FailureDetectionTimeout is set to 3 ms in Ignite configuration. >>> Network time out is default. >>> ClientFailureDetectionTimeout is set to 3 ms. >>> >>> I check gc logs but it does not seem to be GC pause issue. I have >>> attached GC logs too. >>> >>> 1) Can someone please help me to identify the reason for this issue? >>> 2) Are there any specific reasons which causes this issue or it is a bug >>> in Ignite 2.6 version? >>> >>> >>> *ERROR LOGS LINES* >>> 2019-07-22 09:22:47,281 19417675 [tcp-disco-srvr-#3%springDataNode%] >>> ERROR - Critical system error detected. Will be handled accordingly to >>> configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, >>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, >>> err=java.lang.IllegalStateException: Thread >>> tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly.]] >>> java.lang.IllegalStateException: Thread >>> tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly. >>> at >>> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5686) >>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) >>> 2019-07-22 09:22:47,281 19417675 [tcp-disco-srvr-#3%springDataNode%] >>> ERROR - JVM will be halted immediately due to the failure: >>> [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, >>> err=java.lang.IllegalStateException: Thread >>> tcp-disco-srvr-#3%springDataNode% is terminated unexpectedly.]] >>> >>> >>> Thanks, >>> Akash >>> >>
Re: Threadpools and .WithExecute() for C# clients
Thanks Pavel! Does the priority on the Jira ticket suggest this will target IA 2.8? Thanks, Raymond. On Wed, Jul 24, 2019 at 8:21 PM Pavel Tupitsyn wrote: > Denis, yes, looks like a simple thing to add. > > On Tue, Jul 23, 2019 at 10:38 PM Denis Magda wrote: > >> Looping in the dev list. >> >> Pavel, Igor and other C# maintainers, this looks like a valuable extension >> of our C# APIs. Shouldn't this be a quick addition to Ignite? >> >> - >> Denis >> >> >> On Mon, Jul 22, 2019 at 3:22 PM Raymond Wilson < >> raymond_wil...@trimble.com> >> wrote: >> >> > Alexandr, >> > >> > If .WithExecute is not planned to be made available in the C# client, >> what >> > is the plan to support custom thread pools from the C# side of things? >> > >> > Thanks, >> > Raymond. >> > >> > >> > On Thu, Jul 18, 2019 at 9:28 AM Raymond Wilson < >> raymond_wil...@trimble.com> >> > wrote: >> > >> >> The source of inbound requests into Server A is from client >> applications. >> >> >> >> Server B is really a cluster of servers that are performing clustered >> >> transformations and computations across a data set. >> >> >> >> I originally used IComputeJob and similar functions which work very >> well >> >> but have the restriction that they return the entire result set from a >> >> Server B node in a single response. These result sets can be large >> (100's >> >> of megabytes and larger), which makes life pretty hard for Server A if >> it >> >> has to field multiple incoming responses of this size. So, these types >> of >> >> requests progressively send responses back (using Ignite messaging) to >> >> Server A using the Ignite messaging fabric. As Server A receives each >> part >> >> of the overall response it processes it according the business rules >> >> relevant to the request. >> >> >> >> The cluster config and numbers of nodes are not really material to >> this. >> >> >> >> Raymond. >> >> >> >> On Thu, Jul 18, 2019 at 12:26 AM Alexandr Shapkin >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> >> >>> >> >>> Can you share a more detailed use case, please? >> >>> >> >>> >> >>> >> >>> Right now it’s not clear why do you need a messaging fabric. >> >>> >> >>> If you are interesting in a progress tracking, then you could try a >> >>> CacheAPI or QueryContinious, for example. >> >>> >> >>> >> >>> >> >>> What are the sources of inbound requests? Is it a client requests? >> >>> >> >>> >> >>> >> >>> What is your cluster config? How many nodes do you have for your >> >>> distributed computations? >> >>> >> >>> >> >>> >> >>> *From: *Raymond Wilson >> >>> *Sent: *Wednesday, July 17, 2019 1:49 PM >> >>> *To: *user >> >>> *Subject: *Re: Threadpools and .WithExecute() for C# clients >> >>> >> >>> >> >>> >> >>> Hi Alexandr, >> >>> >> >>> >> >>> >> >>> To summarise from the original thread, say I have server A that >> accepts >> >>> requests. It contacts server B in order to help processing those >> requests. >> >>> Server B sends in-progress results to server A using the Ignite >> messaging >> >>> fabric. If the thread pool in server A is saturated with inbound >> requests, >> >>> then there are no available threads to service the messaging fabric >> traffic >> >>> from server B to server A resulting in a deadlock condition. >> >>> >> >>> >> >>> >> >>> In the original discussion it was suggested creating a custom thread >> >>> pool to handle the Server B to Server A traffic would resolve it. >> >>> >> >>> >> >>> >> >>> Thanks, >> >>> >> >>> Raymond. >> >>> >> >>> >> >>> >> >>> On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin >> >>> wrote: >> >>> >> >>> Hi, Raymond! >> >>> >> >>> >> >>> >> >>> As far as I can see, there are no plans for porting custom executors >> >>> configuration in .NET client right now [1]. >> >>> >> >>> >> >>> >> >>> Please, remind, why do you need a separate pool instead of a default >> >>> PublicPool? >> >>> >> >>> >> >>> >> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-6566 >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> *From: *Raymond Wilson >> >>> *Sent: *Wednesday, July 17, 2019 10:58 AM >> >>> *To: *user >> >>> *Subject: *Threadpools and .WithExecute() for C# clients >> >>> >> >>> >> >>> >> >>> Some time ago I ran into and issue with thread pool exhaustion and >> >>> deadlocking in AI 2.2. >> >>> >> >>> >> >>> >> >>> This is the original thread: >> >>> >> http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html >> >>> >> >>> >> >>> >> >>> >> >>> At the time .WithExecutor() was not implemented in the C# client so >> >>> there was little option but to expand the size of the public thread >> pool >> >>> sufficiently to prevent the deadlocking. >> >>> >> >>> >> >>> >> >>> We have been revisiting this issue and see that .WithExecutor() is not >> >>> supported in the AI 2.7.5 client. >> >>> >> >>> >> >>> >> >>> Can this be supported in the C# client, or is there a workaround in >> the >> >>> .Net environment? that does not require this capabi
Re: Invoking an entry processor from a CacheInterceptorAdapter
This message means that you are trying to start two ignite instances in
the same JVM with the same
igniteInstanceName in your configuration.
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/IgniteConfiguration.html#setIgniteInstanceName-java.lang.String-
Check your configuration file for this property.
On Thu, Jul 25, 2019 at 12:41 PM Jean-Philippe Laroche <
jplaroche.2...@gmail.com> wrote:
> I am trying to reference a cache from within a CacheInterceptorAdapter
> (onAfterPut method) to invoke an entry processor to another cache after a
> put.
>
> How to get a refrence to a cache?
>
> I tried the followings:
>
>
> Ignite ignite = Ignition.start("default-config.xml");
> IgniteCache cache =
> ignite.getOrCreateCache("MyOtherCache");
>
> or
>
> IgniteCache cache =
> ignite.start().getOrCreateCache("MyOtherCache");
>
> But it fails:
>
>
> class org.apache.ignite.IgniteCheckedException: Ignite instance with this
> name has already been started: gridgain-cluster
>
Invoking an entry processor from a CacheInterceptorAdapter
I am trying to reference a cache from within a CacheInterceptorAdapter
(onAfterPut method) to invoke an entry processor to another cache after a
put.
How to get a refrence to a cache?
I tried the followings:
Ignite ignite = Ignition.start("default-config.xml");
IgniteCache cache =
ignite.getOrCreateCache("MyOtherCache");
or
IgniteCache cache =
ignite.start().getOrCreateCache("MyOtherCache");
But it fails:
class org.apache.ignite.IgniteCheckedException: Ignite instance with this
name has already been started: gridgain-cluster
Ignite Client Affinity Questions
Hello, I plan on using an Ignite node topology as follows: 1) Application configured as an Ignite Client that handles distributing requests to the correct server node. 2) 3+ Partitioned Server nodes that handle cache storage and processing of requests. 3) Each server node has two caches which use AffinityKey for pinning entries in the two caches to one node for a given key. Questions: 1) Will the client have up to date access to the AffinityFunction for the two caches? 2) What sort of overhead is associated with running as client mode? I want to make it as dumb as possible, so it would only just be aware of topology changes and always have the latest AffinityFunction for the partitioned server node topology. Thank you, Nick. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: log4J Configuration Ignored
Hi,
I've tried to reproduce the issue by using your examples with Ignite 2.7.5
and seems like it works correctly for me.
On node start I see next information regarding log level:
log4j: Level value for root is [INFO].
log4j: root level set to INFO
Did you specify IGNITE_HOME property in your environment? If yes then
Ignite will try to load config file from ignite home folder at first, and
maybe it contains configuration file with different settings.
Regards,
Igor Belyakov
On Wed, Jul 24, 2019 at 5:25 PM wrote:
> Hello,
>
>
>
> I'm new to Ignite and am trying to setup log4j logging for an Ignite node.
>
>
>
> I'm adding the dependency as follows:
>
>
>
>
>
> org.apache.ignite
>
> ignite-log4j
>
> 2.7.5
>
>
>
>
>
> My log 4J is as follows:
>
>
>
>
>
>
>
>
> xmlns:log4j='http://jakarta.apache.org/log4j/'>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And I'm starting up the node as follows:
>
>
>
> public class SampleServerCache {
>
> public static void main(String[] args) throws IgniteException,
> IgniteCheckedException {
>
> CacheConfiguration cacheConfiguration =
> new CacheConfiguration();
>
> cacheConfiguration.setName("sampleCache");
>
> cacheConfiguration.setStatisticsEnabled(true);
>
>
>
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
>
>
>
> Log4JLogger logger = new Log4JLogger("config/log4j.xml");
>
> igniteConfiguration.setGridLogger(logger);
>
>
>
> Ignition.start(igniteConfiguration);
>
> }
>
> }
>
>
>
> However, when I run this the log level is set to DEBUG instead of INFO. I
> believe the code is finding my log4j.xml because if I rename it an
> exception is thrown. It seems the configuration is being ignored.
>
>
>
> Any ideas?
>
> This e-mail and any attachments are confidential and intended solely for
> the addressee and may also be privileged or exempt from disclosure under
> applicable law. If you are not the addressee, or have received this e-mail
> in error, please notify the sender immediately, delete it from your system
> and do not copy, disclose or otherwise act upon any part of this e-mail or
> its attachments.
>
> Internet communications are not guaranteed to be secure or virus-free. The
> Barclays Group does not accept responsibility for any loss arising from
> unauthorised access to, or interference with, any Internet communications
> by any third party, or from the transmission of any viruses. Replies to
> this e-mail may be monitored by the Barclays Group for operational or
> business reasons.
>
> Any opinion or other information in this e-mail or its attachments that
> does not relate to the business of the Barclays Group is personal to the
> sender and is not given or endorsed by the Barclays Group.
>
> Barclays Execution Services Limited provides support and administrative
> services across Barclays group. Barclays Execution Services Limited is an
> appointed representative of Barclays Bank UK plc, Barclays Bank plc and
> Clydesdale Financial Services Limited. Barclays Bank UK plc and Barclays
> Bank plc are authorised by the Prudential Regulation Authority and
> regulated by the Financial Conduct Authority and the Prudential Regulation
> Authority. Clydesdale Financial Services Limited is authorised and
> regulated by the Financial Conduct Authority.
>
RE: Ignite .NET 2.7.5 - IEvents.RemoteQuery - Failed to unmarshal eventsquery response
Hello Simon, Can you, please, share a full log? There should be an internal Java exception with that message. It would be perfect if you can share a little reproducer as well. From: simonatelier Sent: Thursday, July 25, 2019 3:20 PM To: user@ignite.apache.org Subject: Ignite .NET 2.7.5 - IEvents.RemoteQuery - Failed to unmarshal eventsquery response Hi there We have a Client node-Server node setup where the Client node is running a RemoteQuery on the cluster for CacheEvents. When the IEventFilter runs for the RemoteQuery it returns true but once that happens, the following error is raised in the Ignite logs: Failed to unmarshal events query response: GridEventStorageMessage [ex=null, clsLdrId=null, depMode=null, filterClsName=null, userVer=null, ldrParties=null] We are able to query the cache objects normally and they are returned fine, and the above isn't intuitively giving a pointer to what we need to resolve to fix the problem. Any help appreciated. Simon -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite .NET 2.7.5 - IEvents.RemoteQuery - Failed to unmarshal events query response
Hi there We have a Client node-Server node setup where the Client node is running a RemoteQuery on the cluster for CacheEvents. When the IEventFilter runs for the RemoteQuery it returns true but once that happens, the following error is raised in the Ignite logs: Failed to unmarshal events query response: GridEventStorageMessage [ex=null, clsLdrId=null, depMode=null, filterClsName=null, userVer=null, ldrParties=null] We are able to query the cache objects normally and they are returned fine, and the above isn't intuitively giving a pointer to what we need to resolve to fix the problem. Any help appreciated. Simon -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite 2.7.0 : server node: null pointer exception
IGNITE_DISCOVERY_HISTORY_SIZE=700 Does this go on the server side or the thick client side ?
Re: Ignite 2.7.0 : server node: null pointer exception
Hi Mahesh, It definitely looks like a bug. I have created this ticket in order to track the issue https://issues.apache.org/jira/browse/IGNITE-12013 As a temporary workaround, I would propose increasing discovery history size https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteSystemProperties.html#IGNITE_DISCOVERY_HISTORY_SIZE The default value is 500, so let's try to use 700, for instance. (You need to pass it to your java process -D IGNITE_DISCOVERY_HISTORY_SIZE=700) Thanks, S. чт, 25 июл. 2019 г. в 08:12, Mahesh Renduchintala < mahesh.renduchint...@aline-consulting.com>: > The clients come in and get disconnected from the cluster for many reasons > - some intentionally and some due to poor network. > Cant have Ignite nodes crashing with null pointer exception. > > >
Re: Threadpools and .WithExecute() for C# clients
Denis, I've just created one: https://issues.apache.org/jira/browse/IGNITE-12012 Thanks On Thu, Jul 25, 2019 at 2:25 AM Denis Magda wrote: > Pavel, > > Do we already have a ticket or do you want me to create one? > > - > Denis > > > On Wed, Jul 24, 2019 at 1:21 AM Pavel Tupitsyn > wrote: > > > Denis, yes, looks like a simple thing to add. > > > > On Tue, Jul 23, 2019 at 10:38 PM Denis Magda wrote: > > > >> Looping in the dev list. > >> > >> Pavel, Igor and other C# maintainers, this looks like a valuable > extension > >> of our C# APIs. Shouldn't this be a quick addition to Ignite? > >> > >> - > >> Denis > >> > >> > >> On Mon, Jul 22, 2019 at 3:22 PM Raymond Wilson < > >> raymond_wil...@trimble.com> > >> wrote: > >> > >> > Alexandr, > >> > > >> > If .WithExecute is not planned to be made available in the C# client, > >> what > >> > is the plan to support custom thread pools from the C# side of things? > >> > > >> > Thanks, > >> > Raymond. > >> > > >> > > >> > On Thu, Jul 18, 2019 at 9:28 AM Raymond Wilson < > >> raymond_wil...@trimble.com> > >> > wrote: > >> > > >> >> The source of inbound requests into Server A is from client > >> applications. > >> >> > >> >> Server B is really a cluster of servers that are performing clustered > >> >> transformations and computations across a data set. > >> >> > >> >> I originally used IComputeJob and similar functions which work very > >> well > >> >> but have the restriction that they return the entire result set from > a > >> >> Server B node in a single response. These result sets can be large > >> (100's > >> >> of megabytes and larger), which makes life pretty hard for Server A > if > >> it > >> >> has to field multiple incoming responses of this size. So, these > types > >> of > >> >> requests progressively send responses back (using Ignite messaging) > to > >> >> Server A using the Ignite messaging fabric. As Server A receives each > >> part > >> >> of the overall response it processes it according the business rules > >> >> relevant to the request. > >> >> > >> >> The cluster config and numbers of nodes are not really material to > >> this. > >> >> > >> >> Raymond. > >> >> > >> >> On Thu, Jul 18, 2019 at 12:26 AM Alexandr Shapkin > > >> >> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> > >> >>> > >> >>> Can you share a more detailed use case, please? > >> >>> > >> >>> > >> >>> > >> >>> Right now it’s not clear why do you need a messaging fabric. > >> >>> > >> >>> If you are interesting in a progress tracking, then you could try a > >> >>> CacheAPI or QueryContinious, for example. > >> >>> > >> >>> > >> >>> > >> >>> What are the sources of inbound requests? Is it a client requests? > >> >>> > >> >>> > >> >>> > >> >>> What is your cluster config? How many nodes do you have for your > >> >>> distributed computations? > >> >>> > >> >>> > >> >>> > >> >>> *From: *Raymond Wilson > >> >>> *Sent: *Wednesday, July 17, 2019 1:49 PM > >> >>> *To: *user > >> >>> *Subject: *Re: Threadpools and .WithExecute() for C# clients > >> >>> > >> >>> > >> >>> > >> >>> Hi Alexandr, > >> >>> > >> >>> > >> >>> > >> >>> To summarise from the original thread, say I have server A that > >> accepts > >> >>> requests. It contacts server B in order to help processing those > >> requests. > >> >>> Server B sends in-progress results to server A using the Ignite > >> messaging > >> >>> fabric. If the thread pool in server A is saturated with inbound > >> requests, > >> >>> then there are no available threads to service the messaging fabric > >> traffic > >> >>> from server B to server A resulting in a deadlock condition. > >> >>> > >> >>> > >> >>> > >> >>> In the original discussion it was suggested creating a custom thread > >> >>> pool to handle the Server B to Server A traffic would resolve it. > >> >>> > >> >>> > >> >>> > >> >>> Thanks, > >> >>> > >> >>> Raymond. > >> >>> > >> >>> > >> >>> > >> >>> On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin > > >> >>> wrote: > >> >>> > >> >>> Hi, Raymond! > >> >>> > >> >>> > >> >>> > >> >>> As far as I can see, there are no plans for porting custom executors > >> >>> configuration in .NET client right now [1]. > >> >>> > >> >>> > >> >>> > >> >>> Please, remind, why do you need a separate pool instead of a default > >> >>> PublicPool? > >> >>> > >> >>> > >> >>> > >> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-6566 > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> *From: *Raymond Wilson > >> >>> *Sent: *Wednesday, July 17, 2019 10:58 AM > >> >>> *To: *user > >> >>> *Subject: *Threadpools and .WithExecute() for C# clients > >> >>> > >> >>> > >> >>> > >> >>> Some time ago I ran into and issue with thread pool exhaustion and > >> >>> deadlocking in AI 2.2. > >> >>> > >> >>> > >> >>> > >> >>> This is the original thread: > >> >>> > >> > http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> At the time .WithE
