Re: how to organize groups in cluster?

[mamaco]

Hi Andrew,
Thank you for the response.
1. yes, I will try this way to build sub caches
2. yes, make sense.
3. I tested EvictionPolicy, it seemed to be not available in ignite 2.0+, 
please refer to my earlier discuss (with denis)

 
, is there sth else that I need to know?

I'm fighting against none-blocking real-time processes, I hope I could have
better understanding of ignite documents


Andrew Mashenkov wrote
> Hi,
> 
> 1. NodeFilter can be used [1] to bound cache to certain data nodes.
> 2. You can subscribe to cache updates via ContinuousQuery [2].
> But keep in mind, ContinuousQuery listener is called from sensitive code
> and it is usually bad idea to make blocking operations in it.
> 3. To keep only TOP N entries in cache, a EvictionPolicy [3] can be used.
> EvictionPolicy doesn't support persistence.
> 
> Hope, this helps.
> 
> [1]
> https://stackoverflow.com/questions/39612201/apache-ignite-how-to-deploy-cache-to-some-certain-cluster-group
> [2] https://apacheignite.readme.io/docs/continuous-queries
> [3]
> https://apacheignite.readme.io/docs/evictions#section-first-in-first-out-fifo-
> 
> On Thu, Mar 29, 2018 at 2:42 AM, vkulichenko <

> valentin.kulichenko@

> >
> wrote:
> 
>> Sorry, I'm still failing to understand what you're trying to achieve.
>> What
>> is
>> the reason to manually maintain a tree structure which is basically an
>> index? Why not use Ignite indexes that are provided out of the box?
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
> 
> 
> 
> -- 
> Best regards,
> Andrey V. Mashenkov





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to organize groups in cluster?

[mamaco]

Hi Val,
Here's the background:
I'm thinking about creating message pipelines right in ignite cluster. 
it's a nightmare to save data in a centralized cache and run everything on
it, 
so my whole idea is to guarantee *none-blocking process* and scalable
consumers.

1. (producer) put real-time stream into internal message pipeline
2. (consumer) top N algorithm process
3. (consumer) cross-cache replication


 

//Topic based message receiver
IgniteMessaging msg = ignite.message(ignite.cluster().forRemotes());
msg.localListen("Item", (nodeId, message) -> {
this.topitems.Process((AdobeItem) message);
return true; 
});


//algorithm class
public class AdobeTopViewedItems {
private static Ignite ignite; 
private static IgniteCache cacheItems;

private Map ListTopN = new LinkedHashMap();
private AdobeItem smallest=null;
private Integer TopN=Integer.parseInt(System.getProperty("TopN"));
private static WebsocketUtil socket;
public AdobeTopViewedItems(String accountid) {
ignite=Ignition.ignite();
cacheItems = ignite.cache("Items");
Initialize(accountid);
socket = new WebsocketUtil("s7biapp10",9001,"ItemChannel");
}

public Map Get(){
return ListTopN;
}

public void Initialize(String accountid){
this.ListTopN=load(accountid);
}


public Map load(String accountid){
Map output =new LinkedHashMap();
String script="from AdobeItem \n"
+ "where \n"
+ 
"datekey>='"+DateUtil.DateTime2DateString(DateUtil.DateTime2Date(new
Date()))+"' \n"
+ "and accountid='"+accountid+"' \n"
+ "order by visits desc \n"
+ "limit "+Integer.toString(TopN)+" \n";
SqlQuery sql=new SqlQuery(AdobeItem.class, script);
QueryCursor>
cursor=cacheItems.query(sql);
for (javax.cache.Cache.Entry row : cursor){
AdobeItem entry=row.getValue();
output.put(entry.key(), entry);
}
return output;
}

public void Process(AdobeItem item){
if(this.smallest==null){
this.smallest=item;
this.ListTopN.put(item.key(), item);
this.ListTopN=sort(ListTopN,false);

}else{
if(item.visits>this.smallest.visits){
this.smallest=item;
this.ListTopN.put(item.key(), item);
this.ListTopN=sort(ListTopN,false);
*send(item);*
}
}
 }

 private void send(AdobeItem item){
 String message=new Gson().toJson(item);
 socket.send(message, MessageRoute.server2client.toString());
 }
   
 private Map sort(Map map,Boolean
descending) {
final Boolean is_descending=descending;
List> list = new
LinkedList>(map.entrySet());
Collections.sort(list, new Comparator>() {
   public int compare(Entry o1, Entry o2) {
   if (is_descending) {
return o2.getValue().visits.compareTo(o1.getValue().visits);
   } else {
   return
o2.getValue().visits.compareTo(o1.getValue().visits);
   }
   }
});
Map sortedMap = new LinkedHashMap();
Integer counter=0;
for (Entry entry : list) {
sortedMap.put(entry.getKey(), entry.getValue());
counter+=1;
smallest=entry.getValue();
if(counter==TopN) break;
}
list.clear();
list=null;
return sortedMap;
 }
}




vkulichenko wrote
> Sorry, I'm still failing to understand what you're trying to achieve. What
> is
> the reason to manually maintain a tree structure which is basically an
> index? Why not use Ignite indexes that are provided out of the box?
> 
> -Val
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to organize groups in cluster?

[Andrey Mashenkov]

Hi,

1. NodeFilter can be used [1] to bound cache to certain data nodes.
2. You can subscribe to cache updates via ContinuousQuery [2].
But keep in mind, ContinuousQuery listener is called from sensitive code
and it is usually bad idea to make blocking operations in it.
3. To keep only TOP N entries in cache, a EvictionPolicy [3] can be used.
EvictionPolicy doesn't support persistence.

Hope, this helps.

[1]
https://stackoverflow.com/questions/39612201/apache-ignite-how-to-deploy-cache-to-some-certain-cluster-group
[2] https://apacheignite.readme.io/docs/continuous-queries
[3]
https://apacheignite.readme.io/docs/evictions#section-first-in-first-out-fifo-

On Thu, Mar 29, 2018 at 2:42 AM, vkulichenko 
wrote:

> Sorry, I'm still failing to understand what you're trying to achieve. What
> is
> the reason to manually maintain a tree structure which is basically an
> index? Why not use Ignite indexes that are provided out of the box?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Andrey V. Mashenkov

Re: how to organize groups in cluster?

[vkulichenko]

Sorry, I'm still failing to understand what you're trying to achieve. What is
the reason to manually maintain a tree structure which is basically an
index? Why not use Ignite indexes that are provided out of the box?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to organize groups in cluster?

[mamaco]

Hi Val,
1M per day in memory sales data, 100 parallel financial aggregations in each
batch plus additional  PUB/SUB requests for both instant query and server
push. It has been confirmed to play group count distinct is way too slow, so
we decided to play some of TOP N query in separate domain, for example:

Capture a cache event and export to messaging-->client node subscribe to the
TOPIC-->a IN-MEMORY tree set with particular comparator for filtering and
sorting -->publish result once TOP N map is changed.-->update user
visualization

I believe this way won't block cache.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to organize groups in cluster?

[vkulichenko]

Can you clarify what you mean by "real-time query" in this case? Why not just
start node C as a client and run a query from it?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

how to organize groups in cluster?

[mamaco]

assume we a ignite cluster of nodes A,B,C, is it possible to create partion
cache only on node A and node B, and use topic message to replicate data to
node C for real-time query? if not, is it possible to create 2 independant
clusters,  cluster1 for node A+node B, cluster2 for node C, and let them
communicate through topic messaging? it looks like a solution for separated
data groups, sometimes I want to isolated write and read processes.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/