Re: problem of using object as key in cache configurations
I think ignite should add example codes or document to tell the beginners. I did search the forum but no results. but the error log message did give me hint on where the problem might be occurred (in my case, the object serialization part). -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/problem-of-using-object-as-key-in-cache-configurations-tp4116p4151.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: problem of using object as key in cache configurations
Hello,A few teams at our company have independently run into this issue and ended up each wasting time figuring out how to fix it. And this is not the first time I see people asking this very same question here. I wonder if it is possible to improve diagnostics in Ignite code and log a more helpful message?ThanksAndrey _ From: kevin.zhengSent: Wednesday, April 13, 2016 6:15 PM Subject: Re: problem of using object as key in cache configurations To: Finally, I found the solution: I need to use "cache.withKeepBinary()" to call the query method when the cache Key is an object type. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/problem-of-using-object-as-key-in-cache-configurations-tp4116p4146.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Schema Import Utility - Mismatch when loading data between Date and Timestamp
Hello. Issue with loading of Date data https://issues.apache.org/jira/browse/IGNITE-2936 is closed. It will be available at master branch or in next nightly build. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Schema-Import-Utility-Mismatch-when-loading-data-between-Date-and-Timestamp-tp3790p4149.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: re: querying, sql performance of apache ignite
Read-through works for key-based access (methods like get, remove, invoke, etc.), but not for SQL queries. That's because Ignite doesn't know what data it will need for the query prior to running it. But you can preload it manually. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/querying-sql-performance-of-apache-ignite-tp4135p4147.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: problem of using object as key in cache configurations
Finally, I found the solution: I need to use "cache.withKeepBinary()" to call the query method when the cache Key is an object type. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/problem-of-using-object-as-key-in-cache-configurations-tp4116p4146.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: multiple value key
Hi, If you're querying value by key, it's better to use get() method instead of query. The key can be either a primitive, or any Java object. In the latter case don't forget to properly implement equals() and hashCode() methods. With queries you can do any other more complicated requests. Ignite uses standard SQL, so all your examples should work. See [1] for more details. [1] https://apacheignite.readme.io/docs/sql-queries -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/multiple-value-key-tp4138p4144.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: querying, sql performance of apache ignite
Hi, 1. This is correct, current implementation of SQL queries only data that is already in memory. This is enough for majority of use cases, because usually in-memory systems deal with operational data sets (i.e. subsets of "hot", most frequently accessed data) and can safely ignore the rest of the data. For example, you can maintain the data for the last several days and discard historical data. If you have a different case, let us know. 2. IgniteCache is a key-value storage, but it's up to you how to map objects to relational tables. This is fully defined by CacheStore interface [1] and you can always provide your own implementation. Also take a look at [2] for information about bulk loading the data. There are more efficient ways then using get() method. [1] https://apacheignite.readme.io/docs/persistent-store [2] https://apacheignite.readme.io/docs/data-loading -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/querying-sql-performance-of-apache-ignite-tp4135p4143.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: executing jar application with ignite server
Hi, You can use Compute Grid [1] to execute custom code on the server nodes. Is that what you're looking for? [1] https://apacheignite.readme.io/docs/compute-grid -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/executing-jar-application-with-ignite-server-tp4129p4142.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite: ERROR: Java classpath is empty
This thread duplicates http://stackoverflow.com/questions/36594113/ignite-error-java-classpath-is-empty -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-ERROR-Java-classpath-is-empty-tp4118p4140.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Unexpected heap usage by GridDhtPartitionTopologyImpl
Hi, You can find the list of commands to build the project here: https://github.com/apache/ignite/blob/master/DEVNOTES.txt Let us know if something doesn't work for you. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unexpected-heap-usage-by-GridDhtPartitionTopologyImpl-tp3967p4141.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: What does mean caching database exactly ?
Data loading happens on the server (in case you're using CacheStore for this). But someone has to initiate the process by calling loadCache method. It can be done by any of the apps, I believe it's better to do it by one that starts first. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/What-does-mean-caching-database-exactly-tp4043p4139.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: how to check whether server and client connected?
Hello Ravi, Why are you not using Ignite client () node for access to cluster? Client node special created for using as gateway: doesn't sores data and doesn't compute job by default. On Wed, Apr 13, 2016 at 6:01 PM, Ravi kumar Puriwrote: > How do i connect to client with my server . Tell me something related to > port or host adrzs > On 13-Apr-2016 17:46, "Alexei Scherbakov" > wrote: > >> Hi, >> >> Every Ignite node works either in server or client mode. >> Server nodes actually contain data and perform grid tasks, while client >> nodes provide API for working with server nodes remotely. >> Refer to https://apacheignite.readme.io/docs/clients-vs-servers for >> details. >> >> Note: where is a notion of "near" cache, which may be useful on client >> nodes for speeding up access to frequently accessed data. >> Refer to https://apacheignite.readme.io/docs/near-caches for details. >> >> 2016-04-13 12:25 GMT+03:00 Ravi Puri : >> >>> where to configure the client and server to interact. >>> >>> Ignition.setClientMode(true); >>> >>> this has been set but how to access local and remote nodes and how to >>> start >>> caching the data remotely available? >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-ignite-users.70518.x6.nabble.com/how-to-check-whether-server-and-client-connected-tp4117.html >>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>> >> >> >> >> -- >> >> Best regards, >> Alexei Scherbakov >> >
Re: How to load 2 tables in a cache
OK, UPDATE: This method is working, but how do I add setArgs (I mean, I want to pass the argument for orgname as "Ignite" i.e. to only print person names in Ignite organization. How to do that??? It's giving error if I simply do as given on the ignite website: private static void sqlFieldsQueryWithJoin() { IgniteCache, Person> cache = Ignition.ignite().cache("PCache"); //String oname = "Barclays"; // Execute query to get names of all employees. String sql = "select concat(firstName, ' ', lastName), org.orgname " + "from Person, \"" + ORG_CACHE + "\".Organization as org " + "where Person.orgid = org.orgid"; QueryCursor > cursor = cache.query(new SqlFieldsQuery(sql)); // In this particular case each row will have one element with full name of an employees. List
> res = cursor.getAll(); // Print persons' names and organizations' names. print("Names of all employees and organizations they belong to:", res); } -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-load-2-tables-in-a-cache-tp4026p4136.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
querying, sql performance of apache ignite
Hi, I consider some issues about cache in apache-ignite. 1. Firstly, from what you said me, we can query apache-ignite using SQL. However, then results will be only from cache RAM. So, what about situation when table from database doesn't fit into RAM. Then SELECT * on ignite returns only the rows that are loaded into cache. Am I ok ? 2. We always must set key-value. Now, key should be the same as in my database ? Then, before doing select sql we can use get method to load into cache needed rows. Am I ok ? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/querying-sql-performance-of-apache-ignite-tp4135.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Vij, No, it doesn't. IGFS serves the very different purpose - it is Hadoop-compatible file system. It means that, for example, you can load data to IGFS and then query it using Hive. But native Ignite SQL is not applicable here. Vladimir. On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhatiwrote: > Thanks Vladimir, > > I have not gone through complete documentation but if you could let me > know does IGFS provide SQL support like Ignite cache does ? > > Regards, > Vij > > > On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov < > voze...@gridgain.com> wrote: > > > Hi Vijayendra, > > IGFS is designed to be a distributed file system which could cache data > from Hadoop file systems. It cannot be used as cache store by design. > Ignite doesn't have store implementation for HDFS, so you should implement > your own if needed. Particularly, you should implement > org.apache.ignite.cache.store.CacheStore interface. > > Vladimir. > > On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati > wrote: > > Hi, > > Can some body please provide me any pointers regarding how I can use > Ignite Data Grid/ In Memory caching with write through/write behind mode > and writing to HDFS ? > > I know Ignite provides IGFS but its different from what I am looking for. > > The other way could be I can use IGFS as my In Memory store but is it the > right approach ? > > Regards, > Vijayendra Bhati > > > > >
Re: How to load 2 tables in a cache
Could this error be because I haven't done setIndexedType?: cfg1.setIndexedTypes(OrganizationKey.class, Organization.class); OR because I have same column name "orgid" in both the tables?? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-load-2-tables-in-a-cache-tp4026p4133.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: how to check whether server and client connected?
How do i connect to client with my server . Tell me something related to port or host adrzs On 13-Apr-2016 17:46, "Alexei Scherbakov"wrote: > Hi, > > Every Ignite node works either in server or client mode. > Server nodes actually contain data and perform grid tasks, while client > nodes provide API for working with server nodes remotely. > Refer to https://apacheignite.readme.io/docs/clients-vs-servers for > details. > > Note: where is a notion of "near" cache, which may be useful on client > nodes for speeding up access to frequently accessed data. > Refer to https://apacheignite.readme.io/docs/near-caches for details. > > 2016-04-13 12:25 GMT+03:00 Ravi Puri : > >> where to configure the client and server to interact. >> >> Ignition.setClientMode(true); >> >> this has been set but how to access local and remote nodes and how to >> start >> caching the data remotely available? >> >> >> >> -- >> View this message in context: >> http://apache-ignite-users.70518.x6.nabble.com/how-to-check-whether-server-and-client-connected-tp4117.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > > > > -- > > Best regards, > Alexei Scherbakov >
Re: link cache to object class?
So i need to define jst d cache name in configuratn n nothng else as their attributes .how can i wil be able to relate to class .. even i dnr need any bean or class path/package in bean form to get name in ignite.xml configuration On 13-Apr-2016 17:36, "Denis Magda"wrote: > Give a cache a name that is related to an objects simple class name. > > As an example, you have a class “Person”. Create the cache with the same > name and get a reference to it using ignite.cache(“Person”) > > — > Denis > > > On Apr 13, 2016, at 1:35 PM, Ravi Puri > wrote: > > > > how can i link the cache created in configuration to the object class so > that > > i can decide which cache is for which class? > > > > > > > > -- > > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/link-cache-to-object-class-tp4119.html > > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > >
executing jar application with ignite server
Hello, Maybe, I don't understand something, but how to execute program on server node ? I mean something like spark-submit. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/executing-jar-application-with-ignite-server-tp4129.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Thanks Vladimir, I have not gone through complete documentation but if you could let me know does IGFS provide SQL support like Ignite cache does ? Regards,Vij On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerovwrote: Hi Vijayendra, IGFS is designed to be a distributed file system which could cache data from Hadoop file systems. It cannot be used as cache store by design.Ignite doesn't have store implementation for HDFS, so you should implement your own if needed. Particularly, you should implement org.apache.ignite.cache.store.CacheStore interface. Vladimir. On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati wrote: Hi, Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In Memory caching with write through/write behind mode and writing to HDFS ? I know Ignite provides IGFS but its different from what I am looking for. The other way could be I can use IGFS as my In Memory store but is it the right approach ? Regards,Vijayendra Bhati
Re: how to check whether server and client connected?
Hi, Every Ignite node works either in server or client mode. Server nodes actually contain data and perform grid tasks, while client nodes provide API for working with server nodes remotely. Refer to https://apacheignite.readme.io/docs/clients-vs-servers for details. Note: where is a notion of "near" cache, which may be useful on client nodes for speeding up access to frequently accessed data. Refer to https://apacheignite.readme.io/docs/near-caches for details. 2016-04-13 12:25 GMT+03:00 Ravi Puri: > where to configure the client and server to interact. > > Ignition.setClientMode(true); > > this has been set but how to access local and remote nodes and how to start > caching the data remotely available? > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/how-to-check-whether-server-and-client-connected-tp4117.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Alexei Scherbakov
Re: Unexpected heap usage by GridDhtPartitionTopologyImpl
Hi, Currently we are using 1.5.0.final version, since we have an urgency with our timings , we decided to change the source of 1.5.0.final. But i cant find a developer documentation, can you please share the link. A short brief also can be helpful. Thanks. On Fri, Apr 8, 2016 at 2:56 PM, Tolga Kavukcuwrote: > Thanks for the response, we are following the issue. After its done we can > keep in touch. > > Regards. > > On Fri, Apr 8, 2016 at 12:56 PM, Yakov Zhdanov > wrote: > >> Guys, I like the idea of separate pool. But, please note that >> write-behind pool may be slower and may not be able to flush all cache >> updates to DB. We will have to force system threads to help with this. >> >> Tolga, I know that Nick is currently working on async cache callbacks and >> he will be introducing a new pool to system. I think you will be able to >> use it. Ticket for tracking is >> https://issues.apache.org/jira/browse/IGNITE-2004 >> >> For now you can start reviewing >> - org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore >> I think we will need to refactor it in the way we move flusher threads >> logic to execute in separate pool instead of dedicated threads we have now. >> >> --Yakov >> >> 2016-04-08 11:11 GMT+03:00 Tolga Kavukcu : >> >>> Hi Denis, >>> >>> Yes we don't need to have expiration policy, so setting >>> CacheConfiguraiton.setEagerTtl to false solved this problem. >>> >>> So we are testing the whole system, we also found out that using a cache >>> with writeBehind enabled causes a new thread creation for each one. So >>> if we think about future plans and possibilities, it's not a best practice >>> to have increasing number of threads within jvm. >>> >>> We wonder that if there is a option to use a thread pool for writeBehind >>> jobs. If there is not, we could implement it for the community. So if you >>> can guide us where to start, i would be glad :) >>> >>> Thanks. >>> >>> On Fri, Apr 8, 2016 at 1:54 AM, Denis Magda wrote: >>> Tolga, The cleanup threads "ttl-cleanup-workers" are used to eagerly remove expired cache entries. Expiration policy can be set either a cache wide in CacheConfiguration or can be used later with cache.withExpirePolicy(...) calls. I failed to reproduce your case. What I've done is started 30 caches and destroyed all of them later. Visual VM showed that all "ttl-cleanup-workers" were stopped successfully. What Ignite version do you use? In any case if you are not planing to use expiration policy you can set CacheConfiguraiton.setEagerTtl to false and the ttl workers Threads won't be created at all. Regards, Denis On 4/7/2016 3:43 PM, Tolga Kavukcu wrote: Hi Denis, IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter seems like decreased heap usage. I will run longer tests to check heap behaviour. Also i need another help with thread's created by ignite. I found out that ignite creates a cleanup thread named "ttl-cleanup-worker" for each cache. But when cache is destroyed, clean up thread does not deleted. Instead it waits sleeping state at all. My first question is that , is it possible to decrease thread count with a configuration, like "thread pool with x threads" for all caches. Secondly, is "unremoved threads" are expected behaviour. Thanks. On Thu, Apr 7, 2016 at 2:40 PM, Denis Magda wrote: > Hi Tolga, > > GridDhtPartitionTopologyImpl is created per cache. If you destroy a > cache this object should be GCed. However you should use cache.destroy() > for that. > > Please also make sure that you make "live set" heap dumps only. Try to > perform GC explicitly before making the dump because a collector may clean > dead objects much later depending on its heuristics. > > -- > Denis > > On 4/7/2016 8:27 AM, Tolga Kavukcu wrote: > > Hi Denis, > > Thanks for the response. I will try IGNITE_ > ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter. The screnshots taken from > eclipse memory analyser which opens and analyses heap dump. I understand > heap requirement for wrapping and indexing off-heap entry positions. But > also found out that instances of > *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl > *is constantly increasing within jvm. > > > I also create and destroy so many small caches during the lifecycle, > do you think that it is possible to destroyed caches leaves a footprint in > heap. > > The previous scrreenshots was dominator tree view of memory analyser. > I attached again with headers. > > You can see that each of GridDhtPartitionTopologyImpl uses 20mb~
Re: spark and ignite - possibilities
Hi Tomk, IGFS is essentially a caching layer which could be used on top of any Hadoop file system. If ThriftServer performs heave data processing, then it is definitely make sense to try IGFS. BTW, please properly subscribe to the user list (this way we will not have to manually approve your emails). All you need to do is send an email to “ user-subscr...@ignite.apache.org” and follow simple instructions in the reply. Vladimir. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/spark-and-ignite-possibilities-tp4055p4120.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
link cache to object class?
how can i link the cache created in configuration to the object class so that i can decide which cache is for which class? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/link-cache-to-object-class-tp4119.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
how to check whether server and client connected?
where to configure the client and server to interact. Ignition.setClientMode(true); this has been set but how to access local and remote nodes and how to start caching the data remotely available? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-check-whether-server-and-client-connected-tp4117.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
problem of using object as key in cache configurations
Dear all,I have a table which has no primary index (need to use two columns as composite index).at the beginning, I use only one column as key to load all the data into the cache. However, lots of records were ignored ( because they have the same key).Then I create an java object to store two column data as a key in cache configuration object. now I can load all the data into the cache. However, when I test the query function. I got exceptions info as follows?Exception in thread "main" class org.apache.ignite.binary.BinaryObjectException: Failed resolve class for ID: 524756802 at org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:499) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1443) at org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537) at org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117) at org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280) at org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaries(CacheObjectContext.java:228) at org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinariesIfNeeded(CacheObjectContext.java:165) at org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinariesIfNeeded(CacheObjectContext.java:154) at org.apache.ignite.internal.processors.cache.GridCacheContext.unwrapBinariesIfNeeded(GridCacheContext.java:1737) at org.apache.ignite.internal.processors.query.GridQueryCacheObjectsIterator.next(GridQueryCacheObjectsIterator.java:64) at org.apache.ignite.internal.processors.query.GridQueryCacheObjectsIterator.next(GridQueryCacheObjectsIterator.java:29) at org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:73) at org.apache.ignite.examples.datagrid.store.auto.CacheAutoStoreExample_new.simpleQuery(CacheAutoStoreExample_new.java:544) at org.apache.ignite.examples.datagrid.store.auto.CacheAutoStoreExample_new.composedIndexDemo(CacheAutoStoreExample_new.java:246) at org.apache.ignite.examples.datagrid.store.auto.CacheAutoStoreExample_new.main(CacheAutoStoreExample_new.java:213)Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read class name from file [id=524756802, file=D:\ignite-master\work\marshaller\524756802.classname] at org.apache.ignite.internal.MarshallerContextImpl.className(MarshallerContextImpl.java:159) at org.apache.ignite.internal.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:174) at org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:483) ... 14 moreCaused by: java.io.FileNotFoundException: D:\ignite-master\work\marshaller\524756802.classnameany idea what was wrong?best regards,Kevin -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/problem-of-using-object-as-key-in-cache-configurations-tp4116.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: What does mean caching database exactly ?
Yes, firstly I am going to loadCache contain of database. Then, I am going to write newly data (into cache and db - write throught). Loader app will be only read data from cache (not from db). Thanks to read-through it will see new data. Tell me please, what exactly I should say in .xml configure file ? only name for cache ? Writer and reader are reader, server is running by ./ignite.sh -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/What-does-mean-caching-database-exactly-tp4043p4115.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: visor keep trigger alert when condition no longer satisfied?
thanks Vasiliy!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/visor-keep-trigger-alert-when-condition-no-longer-satisfied-tp3772p4114.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.