Scaling with SQL query
Hi, I have a cluster of 10 nodes, and a cache with replication factor 3 and no persistency enabled. The SQL query is pretty simple -- "SELECT * FROM Logs ORDER by time DESC LIMIT 100". I have checked the index for "time" attribute is applied. When I increase the number of nodes, throughput drops and latency increases. Can you please explain why and how Ignite processes this SQL request?
Re: Ignite on RBAC enabled K8s cluster
Hi Roman, Thanks for the information. I was able to get ignite working with RBAC enabled K8s cluster. The issue was with K8s calico networking, I had set it up wrongly because of which I was getting errors. Regards, Vishwas -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Distributed Closures Apply method/c#
Very strange. By default, there is no any timeout. I will take a look more carefully. Also is it possible that you cancel the closure somehow? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Hello, > So when we need to add a new field, say in "addr", we can use current > ALTER TABLE to add a normal column, > but now the problem is how to map the new field to the column. Yes, that is a problem. ALTER TABLE command does not support adding new columns to nested objects. So, if you need to update SQL schema dynamically, then I would suggest avoiding nested objects. > I think the problem is to support dynamic update of fields in QueryEntity. I am afraid that there is no plan to support that. At least, I didn't find a ticket in JIRA. Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Running Node removal from baseline
Hi, Now in case of the node was left the BLT cluster will wait for it. In case if it can't return for a long time user should manually remove it from BTL and it will do rebalance like in this example: https://apacheignite.readme.io/docs/baseline-topology#section-triggering-rebalancing-programmatically However, you can see that the discussions about automatic triggering rebalancing on apache ignite dev list: for example next - http://apache-ignite-developers.2346864.n4.nabble.com/Triggering-rebalancing-on-timeout-or-manually-if-the-baseline-topology-is-not-reassembled-td29299.html. I think that you can ask about it there. BR, Andrei -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Distributed Closures Apply method/c#
Some of the code is proprietary, but this is the exception. No, I have not set a manual timeout System.Threading.ThreadAbortException: Thread was being aborted. at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj) at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at Apache.Ignite.Core.Impl.Compute.Compute.Apply[TArg,TJobRes](IComputeFunc`2 clo, TArg arg) On Mon, Jun 25, 2018 at 2:37 PM, aealexsandrov wrote: > Hi, > > Could you please provide the exception? > > As I see from the code if you didn't set some timeout manual using > writeTimeout method then Long.MAX_VALUE should be used. > > Long timeout = (Long)map.get(TC_TIMEOUT); > > long timeout0 = timeout == null || timeout == 0 ? Long.MAX_VALUE : > timeout; > > BR, > Andrei > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: ClassCastException in Hibernate QueryCache
Hi, Could you please provide some more details: cache configuration, an example of code that was failed and logs? BR, Andrei -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Distributed Closures Apply method/c#
Hi, Could you please provide the exception? As I see from the code if you didn't set some timeout manual using writeTimeout method then Long.MAX_VALUE should be used. Long timeout = (Long)map.get(TC_TIMEOUT); long timeout0 = timeout == null || timeout == 0 ? Long.MAX_VALUE : timeout; BR, Andrei -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Ignite Node failure - Node out of topology (SEGMENTED)
Hi Dmitry, We are again seeing segmentation failure in one of the node of our prod env. This time we did not run jmap, but still node failed. -> CPU, memory utilization and network are in optimal state. We observed that there are page faults in memory at the same time of segmentation failure, as reported by dynatrace agent (attached screenshot). Can you please confirm if page faults could result in network segmentation in a node? I think, we see page faults in a node, but not always result in segmentation failure. Logs from Failed Agent: INFO: FreeList [name=delivery, buckets=256, dataPages=4, reusePages=0] Jun 23, 2018 8:40:00 PM org.apache.ignite.logger.java.JavaLogger info INFO: Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=3f568bb8, name=delivery, uptime=24:31:12.859] ^-- H/N/C [hosts=9, nodes=9, CPUs=18] ^-- CPU [cur=7%, avg=9.06%, GC=0%] ^-- PageMemory [pages=30244] ^-- Heap [used=3184MB, free=22.09%, comm=4087MB] ^-- Non heap [used=213MB, free=-1%, comm=222MB] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=5, qSize=0] ^-- Outbound messages queue [size=0] Jun 23, 2018 8:40:00 PM org.apache.ignite.logger.java.JavaLogger info INFO: FreeList [name=delivery, buckets=256, dataPages=4879, reusePages=0] Jun 23, 2018 8:40:00 PM org.apache.ignite.logger.java.JavaLogger info INFO: FreeList [name=delivery, buckets=256, dataPages=4, reusePages=0] Jun 23, 2018 8:40:34 PM org.apache.ignite.logger.java.JavaLogger info INFO: TCP discovery accepted incoming connection [rmtAddr=/10.40.173.14, rmtPort=33762] Jun 23, 2018 8:40:34 PM org.apache.ignite.logger.java.JavaLogger info INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.40.173.14, rmtPort=33762] Jun 23, 2018 8:40:34 PM org.apache.ignite.logger.java.JavaLogger info INFO: Started serving remote node connection [rmtAddr=/10.40.173.14:33762, rmtPort=33762] Jun 23, 2018 8:40:34 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Node is out of topology (probably, due to short-time network problems). Jun 23, 2018 8:40:34 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Local node SEGMENTED: TcpDiscoveryNode [id=3f568bb8-813d-47f7-b8da-4ecbff3e9753, addrs=[10.40.173.78, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.40.173.78:47500], discPort=47500, order=54, intOrder=32, lastExchangeTime=152978 6434361, loc=true, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Finished serving remote node connection [rmtAddr=/10.40.173.14:33762, rmtPort=33762 Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Finished serving remote node connection [rmtAddr=/10.40.173.41:52584, rmtPort=52584 Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Stopping local node according to configured segmentation policy. Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Node FAILED: TcpDiscoveryNode [id=9165f32c-9765-49d7-8856-5b77b0bded6d, addrs=[10.40.173.14, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.40.173.14:47500], discPort=47500, order=22, intOrder=15, lastExchangeTime=1529050123714, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Command protocol successfully stopped: TCP binary Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Topology snapshot [ver=56, servers=8, clients=0, CPUs=16, heap=28.0GB] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Node FAILED: TcpDiscoveryNode [id=a26de809-dde1-41b8-87a3-d5576851a0be, addrs=[10.40.173.56, 127.0.0.1], sockAddrs=[/10.40.173.56:47500, /127.0.0.1:47500], discPort=47500, order=23, intOrder=16, lastExchangeTime=1529050123735, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Topology snapshot [ver=57, servers=7, clients=0, CPUs=14, heap=26.0GB] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Node FAILED: TcpDiscoveryNode [id=910ea19f-af5c-4745-a035-b24a3bb48206, addrs=[10.40.173.88, 127.0.0.1], sockAddrs=[/10.40.173.88:47500, /127.0.0.1:47500], discPort=47500, order=25, intOrder=17, lastExchangeTime=1529050123735, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=false] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger info INFO: Topology snapshot [ver=58, servers=6, clients=0, CPUs=12, heap=24.0GB] Jun 23, 2018 8:40:35 PM org.apache.ignite.logger.java.JavaLogger warning WARNING: Node FAILED: TcpDiscoveryNode [id=17f3ba9c-e32e-47e4-9ca2-136338d8c4ac, addrs=[10.40.173.39, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, /10.40.173.39:47500], discPort=47500, order=30, intOrder=19, lastExchangeTime=1529050123735, loc=false,
Distributed Closures Apply method/c#
Hi all Is there a timeout implied or otherwise with Apply() method for a Distributed closure? Seems to kick out a exception after roughly two mins. Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: A series of Apache Ignite meetups in New York
Hi Dennis, Would you please consider writing a blog post on 'Ignite + kubernetes' for us folks who are not in NY? Also if you can record this session and post links later, it would be very useful. Thanks, Arun -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Deadlock during cache loading
"When receiver is invoked for key K, it’s holding the lock for K." is not correct, at least in the 2.4 code. When a custom stream receiver is called, the data streamer thread has a read-lock preventing termination, and there is a real-lock on the topology, but DataStreamerUpdateJob.call() does not get any per entry locks. Since the DataStreamer threads are in a separate pool, a custom stream receiver should be able to make any calls that a client can w/o fear of deadlock. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re-post: java.io.IOException: Too many open files
Hi, In case dial timeout, client is nil so that client.Close() can't work. Thanks.
ClassCastException in Hibernate QueryCache
Using Ignite 2.4 for Hibernate query cache and getting an exception: Caused by: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.io.Serializable; at org.hibernate.cache.internal.StandardQueryCache.get(StandardQueryCache.java:189) at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2587) The problematic line in Hibernate reads: TypeHelper.beforeAssemble( (Serializable[]) cacheable.get( i ), returnTypes, session ); it receives Object[] from the cache and tries to cast to Serializable[] which obviously fails. It seems to be a generic place and should not work for anyone, but somehow it is working? I did check put operation and it actually puts Serializable[] into the cache. Any pointers on how to resolve the issue? Cache config used is very simple taken from examples. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite 2.4 & 2.5 together?
Hi, It is recommended to set baseline at first if grid was upgraded from <2.4 version. Then you need to deactivate the grid to force checkpoint, this can reduce time of next startup. Then stop grid and update jars to newer version and start the grid. Then you may be need to activate grid if it was deactivated on first step. On Mon, Jun 25, 2018 at 12:19 PM Michaelikus wrote: > ok, thanks > > So where i can find information about migration procedure? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Best regards, Andrey V. Mashenkov
Re: Ignite 2.4 & 2.5 together?
ok, thanks So where i can find information about migration procedure? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite 2.4 & 2.5 together?
Hi, It should not be possible by default and highly not recommended. As there can be differences in communication protocol between versions ans compatibility is not tested. However, persistence format should be backward compatible that means Ignite newer version can work with persistence created with older version. On Mon, Jun 25, 2018 at 11:30 AM Michaelikus wrote: > Hi ppl! > > I have cluser of 5 nodes ignite 2.4. > > Is it possible to add new nodes with 2.5 version to it? > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Best regards, Andrey V. Mashenkov
Re: Re-post: java.io.IOException: Too many open files
Hi,
Thank you for attaching the code sample and configration. Most likely this
information will be enough to reproduce the issue.
Meanwhile could you please try to add `defer client.Close()` into your code:
func main() {
client, err := redis.DialTimeout("tcp", "10.1.14.221:11211", 500 *
time.Microsecond)
if err != nil {
panic(err)
}
defer client.Close()
foo, err := client.Cmd("GET", "foo").Str()
if err != nil {
panic(err)
}
fmt.Println("foo: ", foo)
}
I think it should help to fix "Connection reset by peer" exceptions on
Ignite nodes.
Best Regards,
Roman
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite 2.4 & 2.5 together?
Hi ppl! I have cluser of 5 nodes ignite 2.4. Is it possible to add new nodes with 2.5 version to it? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re-post: java.io.IOException: Too many open files
Hi, Sorry I have no knowledge about maven. Here is my config file and sample code of the client. I reproduced "Connection reset by peer" by adjust the timeout setting, but ignite's file descriptor count didn't increase. Before ignite was halted by "Too many open files", there was a close wait spike in network metrics (see network.png). I'm still looking for the reason for that. Thanks. http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd;> radix.go Description: Binary data
Re: Re-post: java.io.IOException: Too many open files
Hi, I checked the logs and it saw there are a lot of "Connection reset by peer" exceptions there which can be a cause of "Too many open files". It seems that the clients connect to Ignite server node via REST API. Could you please share a small reproducer (maven project at github) and server node configuration as well? I will try to reproduce the issue in my environment. I think we need to analyze how a client app interacts with a server node. Thanks. Best Regards, Roman -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
