On Wed, May 4, 2016 at 9:12 AM, Alexey Kuznetsov <akuznet...@gridgain.com> wrote:
> > May be, but sending closures is seems as workaround for me. > This will require from user to write quite a lot of code. > Just call of loadCache(...some descriptors...) will be much more user > friendly. > I think documenting this properly will take us a long way and will help our users. After this is documented, we can then come up with a better design and include it into one of future releases. > Also I'm not sure what will happen with key loaded from DB if it will be > not affinity key for node where localLoadCache will be executed. > Will it be ignored or put into cache? > They will be ignored. > > Semen, Alexey Goncharuk - could you tell what will happen with such keys > inside of localLoadCache ? > > On Tue, May 3, 2016 at 11:10 PM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > > > Alexey, > > > > What you are saying should already be possible by sending compute > closures > > to nodes and calling localLoadCache(“myCustomSqlStatement”). > > > > Isn’t it just a case of providing proper example and documentation? > > > > D. > > > > On Tue, May 3, 2016 at 7:22 AM, Alexey Kuznetsov < > akuznet...@gridgain.com> > > wrote: > > > > > I totally agree with Vladimir. > > > > > > From JdbcPojo store side we could introduce support of some kind load > > > descriptor > > > that will contains SQL to execute and node filter. > > > > > > On each node store will check node filter and execute SQL if node match > > the > > > filter. > > > > > > This will solve first problem - "do not load full database on each > node" > > . > > > > > > As for second problem - "not ignore non-primary non-backup entries" - I > > > think this should be solved on a cache level, > > > because store does not know anything about primary / backup. > > > > > > Thoughts? > > > > > > > > > On Wed, Apr 27, 2016 at 9:27 PM, Vladimir Ozerov <voze...@gridgain.com > > > > > wrote: > > > > > > > Igniters, > > > > > > > > We receive more and more questions about the same problem: "I have > big > > a > > > > database. How should I load it to Ignite?" > > > > > > > > Obviously, users try to use POJO store as a most convenient approach, > > but > > > > it cannot handle this case properly. > > > > 1) If user invoke *IgniteCache.loadCache()*, then the same request - > > and > > > > usually this is full table scan - will be invoked on every node > leading > > > to > > > > very poor performance. For instance, we have a report of a load of > 47M > > > > entries to cache on 16 nodes which took ... 8 hours!!! > > > > 2) If user invoke IgniteCache.localLoadCache(), then our internal > cache > > > > logic will filter out non-primary and non-backup entries. So this > > > approach > > > > doesn't work either. > > > > 3) User could try using *IgniteDataStreamer*, but in this case he had > > to > > > > deal with all JDBC-related stuff on his own - not convenient. > > > > 4) Another approach I heard several times - "user should have an > > > attribute > > > > for affinity in the table ...". And the idea that this way user will > be > > > > able to divide the whole data set into several disjoint sets with > > > specific > > > > affinity. Doesn't work. Consider the user with some legacy database - > > the > > > > most common use case. How is he going to work with affinity? > > > > > > > > Bottom line: Ignite has *no convenient way *to load millions of > entries > > > > from a database. > > > > > > > > We need to start thinking of possible solutions. Several ideas from > my > > > > side: > > > > > > > > 1) POJO store must be much more flexible. We should be able to pass > > > > different queries to different nodes when calling "loadCache". > > > > > > > > 2) Cache store could have additional mode when it will not ignore > > > > non-primary non-backup entries, but rather *distribute *it to other > > > nodes. > > > > E.g. with help of data streamer. > > > > > > > > Thoughts? > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > -- > > > Alexey Kuznetsov > > > GridGain Systems > > > www.gridgain.com > > > > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com >
