Re: sql query in case of cluster group
Hi Neeraj, Looks like queryParallelizm has buggy in ignite-2.0 that already fixed in master. Fix will be available in 2.1 version that it coming. Seems, it should be released in few days. On Mon, Jul 17, 2017 at 3:32 PM, neerajbhatt wrote: > Hi Andrey, > > we are not using query timeout, because we want to get partial results, in > case 1 node is slow, but other nodes are faster. > So, we are using cluster groups. > > Yes, just checked. If we don't use queryParallelism, then the query works > well.. :) > > Kindly create a jira if you have same observation > > Thanks > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p15000.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
Re: sql query in case of cluster group
Hi Andrey, we are not using query timeout, because we want to get partial results, in case 1 node is slow, but other nodes are faster. So, we are using cluster groups. Yes, just checked. If we don't use queryParallelism, then the query works well.. :) Kindly create a jira if you have same observation Thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p15000.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
You can use SqlFieldsQuery.setTimeout() for this. Would you please check if it is workable without queryParalellism? Looks weird, I need a time to make a test and check if local queries with queryParalellism>1 is broken. On Mon, Jul 17, 2017 at 1:44 PM, neerajbhatt wrote: > We are using cluster groups to run query on each nodes of the cluster so > that > we can set the timeout on each query. > We are running local queries and in TestIgniteCallable > we have given List> res = > cache.query(qry.setArgs().setLocal(true)).getAll(); > > What are we missing ? > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14998.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
Re: sql query in case of cluster group
We are using cluster groups to run query on each nodes of the cluster so that we can set the timeout on each query. We are running local queries and in TestIgniteCallable we have given List> res = cache.query(qry.setArgs().setLocal(true)).getAll(); What are we missing ? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14998.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
1. LIMIT make no sense without ORDER for non-local queries and when sqlQueryParallelizm > 1. As in that case, data from different index segments will be merged out of order and you can get different results with same query from run to run . 2. You use ClusterGroup to make your jobs to be run on certain nodes, but it doesn't affect operations inside a job. So, non-local sql query will be run on all data nodes. On Mon, Jul 17, 2017 at 12:44 PM, neerajbhatt wrote: > Hi Andrew > > It seems to be an order by issue. We have created a test project in github > https://github.com/neerajbhatt/testIgnite > > If we use PutItems to put some records in a partitioned cache(TESTCACHE) > args[0]=number of records, and then Get the items by > GetItemsWithClusterGroup args[0] = query type (0 or 1), args[1]= number of > queries and args [3] as last digit of ip. > > for first query ("SELECT T.id FROM " + "TESTCache.TestItem as T LIMIT 3") > different ids are given for different cluster groups( each cluster group > having one node) > > for second query ("SELECT T.id FROM " + "TESTCache.TestItem as T order by > field1 desc LIMIT 3") we are getting same ids for each cluster group. It > seems in case of order by instead of cluster group data is being fetched > from whole cluster. > > Please suggest > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14994.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
Re: sql query in case of cluster group
Hi Andrew It seems to be an order by issue. We have created a test project in github https://github.com/neerajbhatt/testIgnite If we use PutItems to put some records in a partitioned cache(TESTCACHE) args[0]=number of records, and then Get the items by GetItemsWithClusterGroup args[0] = query type (0 or 1), args[1]= number of queries and args [3] as last digit of ip. for first query ("SELECT T.id FROM " + "TESTCache.TestItem as T LIMIT 3") different ids are given for different cluster groups( each cluster group having one node) for second query ("SELECT T.id FROM " + "TESTCache.TestItem as T order by field1 desc LIMIT 3") we are getting same ids for each cluster group. It seems in case of order by instead of cluster group data is being fetched from whole cluster. Please suggest -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14994.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
I don't think it is ORDER BY issue. Are you sure you run queries locally and on different nodes? On Fri, Jul 14, 2017 at 3:52 PM, neerajbhatt wrote: > Hi Andrew > > Thanks for your inputs now we are not getting duplicate records as we are > now using different cache and ignite objects for each compute. > > But for queries which have order by (sort by) clause we are getting exactly > same result from all cluster groups(nodes). It looks in case of order by > the > query is getting executed in all nodes > > Thanks > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14906.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
Re: sql query in case of cluster group
Hi Andrew Thanks for your inputs now we are not getting duplicate records as we are now using different cache and ignite objects for each compute. But for queries which have order by (sort by) clause we are getting exactly same result from all cluster groups(nodes). It looks in case of order by the query is getting executed in all nodes Thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14906.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Here you use same cache object from all computes. >> cache.query(qry.setArgs(args.toArray()).setLocal(true)).getAll(); So, actually, you run query on same cache node from threads belongs to other nodes. This will not work in multi JVM env. Just keep in mind that your task\job object can be transfered to another node (JVM or machine). You should get cache instance inside task [1]. [1] https://apacheignite.readme.io/docs/resource-injection On Fri, Jul 14, 2017 at 2:10 PM, neerajbhatt wrote: > Any update ?, Also we have data as back up as one. In case of back ups can > we > get duplicate result as we might be getting result from both master and > slave > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14903.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
Re: sql query in case of cluster group
Any update ?, Also we have data as back up as one. In case of back ups can we get duplicate result as we might be getting result from both master and slave -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14903.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
--code is as below --all caches are partitioned --args[0] is type of query, args[1] is number of queries, args[2] is ip of cluster group (each node is a separate cluster group. group name is 'i-' --we are trying with different args[2] (cluster groups) and we are getting duplicate result. Web console shows that a particular id is in only in one node (hence data partition and collocation is working fine( import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; import java.util.List; import java.util.concurrent.TimeUnit; import org.apache.ignite.Ignite; import org.apache.ignite.IgniteCache; import org.apache.ignite.IgniteCluster; import org.apache.ignite.IgniteCompute; import org.apache.ignite.Ignition; import org.apache.ignite.cache.query.SqlFieldsQuery; import org.apache.ignite.cluster.ClusterGroup; import org.apache.ignite.configuration.IgniteConfiguration; import org.apache.ignite.lang.IgniteCallable; import org.apache.ignite.lang.IgniteFuture; import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; public class GetItemsWithClusterGroup { public static void main(String args[]) { //args[0] is type of query (0,1,2), args[1] is number of queries, args[2] is ip of cluster group //we are trying with different args[2] and we are getting duplicate result int queryType = Integer.parseInt(args[0]); int maxNumberOfQueries = Integer.parseInt(args[1]); Ignition.setClientMode(true); IgniteConfiguration conf = new IgniteConfiguration(); conf.setPeerClassLoadingEnabled(true); TcpDiscoverySpi discovery = new TcpDiscoverySpi(); TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(); String ipaddress = args[2]; ipFinder.setAddresses(Arrays.asList(ipaddress)); discovery.setIpFinder(ipFinder); conf.setDiscoverySpi(discovery); Ignite ignite = Ignition.start(conf); System.out.println("IP=> " + ipaddress); IgniteCluster cluster = ignite.cluster(); List groups = Arrays.asList("i-"+ipaddress); List computes = new ArrayList<>(); IgniteCache itemCache = cluster.ignite().getOrCreateCache("ITEMCACHE"); for (String group : groups) { ClusterGroup cg = cluster.forAttribute("ROLE", group); System.out.println(group + " " + cg.nodes()); computes.add(ignite.compute(cg).withAsync().withTimeout(5000)); } int numberOfQueries = 0; while (numberOfQueries < maxNumberOfQueries) { numberOfQueries++; process(ignite, computes, itemCache, queryType); } } private static void process(Ignite ignite, List computes, IgniteCache cache, int queryType) { List>> futures = new ArrayList<>(); String geoId = "144"; StringBuilder buff = new StringBuilder(); final List args = new ArrayList<>(); if (queryType == 0) { buff = new StringBuilder("SELECT T.uniqueSkuId FROM " + "ITEMCACHE.Item as T LIMIT 2"); } if (queryType == 1) { buff = new StringBuilder( "SELECT r.id FROM IGPCACHE.Rank as r " + " WHERE r.geoId=? " + " order by r.rank desc limit 3"); args.add(geoId); } if (queryType == 2) { buff = new StringBuilder("SELECT r.id FROM IGPCACHE.Rank as r " + " WHERE r.geoId=? " + " limit 3"); args.add(geoId); } if (queryType == 3) { buff = new StringBuilder("SELECT r.id FROM IGPCACHE.Rank as r " + " limit 3"); } SqlFieldsQuery qry = new SqlFieldsQuery(buff.toString()); qry.setEnforceJoinOrder(true); for (IgniteCompute async : computes) { async.call(new IgniteCallable>() { @Override public List call() throws Exception { try { List> res = cache.query(qry.setArgs(args.toArray()).setLocal(true)).getAll(); List items = convert(res); return items; } catch (Exception e) { System.out.println("excep
Re: sql query in case of cluster group
Hi, Sound weird. Query qry.setLocal(true) should return local query result. Would you please share a code? On Fri, Jul 14, 2017 at 11:04 AM, neerajbhatt wrote: > Hello All > > We have a requirement to set time out for each node so that we can get a > partial result , in case of a time out from a particular node. To solve > this > we have created a cluster group for each node and run compute > > ClusterGroup cg = cluster.forAttribute("ROLE", group); > computes.add(ignite.compute(cg).withAsync().withTimeout(5000)); > > with qry.setLocal(true); > > Since the caches are partitioned on a id we should get different result > from > each cluster group but we are getting same records from different cluster > groups > > So does cluster group works in case of sql query like it works in job > execution, service deployment, messaging, events (as given in > documentation) > ? > > Thanks > > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov