Sergey, I think we should start more caches, like 1000 in one time. But we have to have enough memory on our TC agents. As I know, empty cache is require about 50 mb (without indexing), am I right?
You are right, I keep in mind that *backups* and *REPLICATED* mode make no sense together, but we still have to test it in one node and multi node casesю Any other *no sense* combinations? I forgot about custom BinaryConfiguration at IgniteConfiguration for BinaryMarshaller. So, at least 6 IgniteConfigurations. -- Artem -- On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <skoz...@gridgain.com> wrote: > Hi Artem > > It's good idea to create 20-30 cache configurations at once and then to > iterate tests over those caches in parallel (but make sure that cache names > are unique). > Another point that some combinations make no sense like *backups *and > *REPLICATED > *cache > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <ashu...@gridgain.com> wrote: > > > Hi all, > > > > I have an update. > > > > I've started from *CacheConfiguration* permutations. I wrote out a list > > with all CacheConfiguration setters and filtered it with Alexey G. > > > > Finally we have: > > > > 1. CacheMode - 3 variants > > 2. CacheAtomicityMode - 2 variants > > 3. CacheMemoryMode - 3 variants > > 4. setLoadPreviousValue - 2 variants > > 5. setReadFromBackup - 2 variants > > 6. setStoreKeepBinary - 2 variants > > 7. setRebalanceMode - SYNC and ASYNC (2 variants) > > 8. setSwapEnabled - 2 variants > > 9. setCopyOnRead - 2 variants > > 10. NearConfiguration disabled / default NearConfiguration / custom > > NearConfiguration - 3 variants > > 11. With and without a complex parameter. The complex parameter > defines > > not-default Eviction policy and filter, cache store configuration > > (storeFactory and storeSessionListenerFactory), rebalancing > > configuration, > > affinity function, offHeapMaxMemory, interceptor, topology validator > and > > CacheEntryListener. > > > > I've run 123 Cache Full Api test cases for all permutations of parameters > > 1-9 and got 256896 test cases (1152 configuration variants * 123 test > > cases). All these tests take 4 hours 40 minutes. Not all tests pass, so > MAY > > BE when all tests will pass it will take less time (3,5 hours for > example). > > > > As we can see the tests take a lot of time. > > > > The following permutation should be supported too: > > > > 1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1 > > backups, 4 nodes and 2 backups - 3 variants > > 2. Client and Server nodes - 2 variants > > 3. Indexing enabled and disabled for cache - 2 variants > > 4. IgniteConfiguration permutations - how many variants? I see at > least > > 4 (2 Marshallers, P2P). > > > > Plus we need add new test cases to test different key and value types and > > etc. > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will split > all > > tests on 250 suites and run on all 30 TC agents it will take about 30-40 > > hours. Ok, we can do it during weekends. > > > > I think it will take too much time. > > > > As an option we can start a cache for each configuration and run tests > > concurrently. But we need to implement this opportunity in our test > > framework. > > > > Any other thoughts how we can decrease time for tests? > > > > Thanks, > > -- Artem -- > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <sboi...@gridgain.com> > > wrote: > > > > > Artem, > > > > > > One more thing for new tests: I think test should start both server and > > > client nodes and use Ignite API from all nodes. > > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <ashu...@gridgain.com> > > wrote: > > > > > > > Dmitriy, > > > > > > > > Actually, I don't have a list with all the permutations. > > > > > > > > At first, we need to split in our discussion test cases and Ignite > > > > configuration which should be covered. > > > > > > > > For example, new Full Api test cases for cache are based on old Full > > Api > > > > test cases. So, it need to think what the test cases was not covered > > > > before. > > > > > > > > About Ignite configurations, I'm going to add permutation for each > > > > IgniteConfiguration and CacheConfiguration property. > > > > > > > > By the way, the jira contains the following list of permutation (feel > > > free > > > > to add something): > > > > > > > > The following tests should be added (for functional blocks): > > > > > > > > 1. Interceptor > > > > 2. Queries: continuous, scan, SQL, fields and text queries. > > > > 3. cache events > > > > 4. We should also test with Serializable, Externalizable, and > plain > > > > Pojos for keys and values. > > > > 5. The Pojo in the above test should contain an enum value > > > > 6. We should also test Enums as keys and Enums as values > > > > 7. All operations should have single-key and multi-key operations > > > > > > > > New tests should cover all combinations for following properties: > > > > > > > > 1. cache modes > > > > 2. operation from client nodes and server nodes > > > > 3. store enabled/disabled > > > > 4. evicts sycn/non-sync > > > > 5. eviction policies > > > > 6. near on/off > > > > 7. marshallers (+ Binary marshaller with different mappers) > > > > 8. keys and values - externalizable, serializable, binaryzable, > > "none > > > of > > > > previous" > > > > 9. classes available on servers: true/false > > > > 10. Peer loading on/off > > > > 11. Affinity functions > > > > 12. expiry policies > > > > > > > > > > > > Thanks, > > > > -- Artem -- > > > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan < > > dsetrak...@apache.org > > > > > > > > wrote: > > > > > > > > > Artem, I think it is best to specify all the permutations here, so > > > others > > > > > can make additional suggestions. Otherwise, we cannot get a full > > > picture. > > > > > > > > > > Thanks, > > > > > D. > > > > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <ashu...@gridgain.com > > > > > > wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I thought a little bit more and think we need to add a support > for > > > the > > > > > > following permutations too (I've added these to the jira > > > description): > > > > > > - We should also test with Serializable, Externalizable, and > plain > > > > Pojos > > > > > > for keys and values. > > > > > > - The Pojo in the above test should contain an enum value > > > > > > - We should also test Enums as keys and Enums as values > > > > > > - All operations should have single-key and multi-key operations > > > > > > > > > > > > Maybe someone see any other permutation to be supported? > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak < > > ashu...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > There is a branch at my fork and a pull request at Ignite. See > > > > comment > > > > > > > about pull request at the ticket (PR-446). > > > > > > > > > > > > > > But I have to notice that the branch under hard development and > > you > > > > it > > > > > > can > > > > > > > not work (have compilation or test errors) at some moments. > > > > > > > > > > > > > > Good luck! > > > > > > > > > > > > > > -- Artem -- > > > > > > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan < > > > > > dsetrak...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> Artem, > > > > > > >> > > > > > > >> This is great. I have noticed from the ticket that you have > > > created > > > > > some > > > > > > >> initial suite already. Is there a branch I can look at it? > > > > > > >> > > > > > > >> D. > > > > > > >> > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak < > > > ashu...@gridgain.com > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Igniters, > > > > > > >> > > > > > > > >> > I'm working on an enhancement of Full API coverage [1] [2]. > > > > > > >> > > > > > > > >> > Ignite already has Full API test, but currently it's hard to > > > test > > > > > all > > > > > > >> > configuration combinations. > > > > > > >> > > > > > > > >> > Feel free to add comments in the jira if you have any > thought. > > > > > > >> > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521 > > > > > > >> > [2] > > > > > > >> > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > -- Artem -- > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
