I have some blurriness about what a shard is, that I still didn’t figure out. I have some guesses:
1) Every yang tree == one shard. 2) Shard can be a collection of a number of yang trees. 3) None of the above? From: Tom Pantelis [mailto:tompante...@gmail.com] Sent: Wednesday, January 11, 2017 3:46 PM To: Kochba, Alon <alo...@hpe.com> Cc: Sela, Guy <guy.s...@hpe.com>; Williams, Marcus <marcus.willi...@intel.com>; Daniel Farrell <dfarr...@redhat.com>; odl netvirt dev <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org; Robert Varga <n...@hq.sk>; integration-...@lists.opendaylight.org Subject: Re: [netvirt-dev] [controller-dev] [mdsal-dev] Netvirt Scale tests: OutOfMemory from datastore It's not the while data store - snapshots occur per shard. But since everything probably goes into one shard, it essentially is the whole DS. No behavior change - this limit hadn't been hit until now. On Wed, Jan 11, 2017 at 8:34 AM, Kochba, Alon <alo...@hpe.com<mailto:alo...@hpe.com>> wrote: + Daniel and Marcus Jumping in here with some questions.. Is the bottom line here that the whole config DS size cannot exceed 2GB or did I get something wrong? Is it possible we never encountered such a scenario in the S3P performance tests done in Beryllium/Boron [1], or did something change in the behavior? It seems that this limitation is extremely problematic for scale, and should be easily recreated by doing NetVirt scale tests (loading ~200 computes and creating a VM on each, or configuring a TZ for all of them manually). I see 1800 OVSDB nodes were installed in [1], but not sure if any flows were configured on the OpenFlow side which might cause the DS to scale.. We also need to monitor what part of the DS is taking so much space - we might have redundant data being stored catching up space, but in any case projects would hit 2GB at some point. [1] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf --alon From: netvirt-dev-boun...@lists.opendaylight.org<mailto:netvirt-dev-boun...@lists.opendaylight.org> [mailto:netvirt-dev-boun...@lists.opendaylight.org<mailto:netvirt-dev-boun...@lists.opendaylight.org>] On Behalf Of Sela, Guy Sent: Wednesday, 11 January 2017 11:23 To: Tom Pantelis <tompante...@gmail.com<mailto:tompante...@gmail.com>> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org>>; controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org>; Robert Varga <n...@hq.sk<mailto:n...@hq.sk>> Subject: Re: [netvirt-dev] [controller-dev] [mdsal-dev] Netvirt Scale tests: OutOfMemory from datastore Opened a bug: https://bugs.opendaylight.org/show_bug.cgi?id=7521 From: Tom Pantelis [mailto:tompante...@gmail.com] Sent: Wednesday, January 11, 2017 1:06 AM To: Sela, Guy <guy.s...@hpe.com<mailto:guy.s...@hpe.com>> Cc: Robert Varga <n...@hq.sk<mailto:n...@hq.sk>>; controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev <netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org>> Subject: Re: [controller-dev] [mdsal-dev] Netvirt Scale tests: OutOfMemory from datastore operational won't snapshot if not persisted or replicated. Can you open a bug for tracking? On Tue, Jan 10, 2017 at 4:12 PM, Sela, Guy <guy.s...@hpe.com<mailto:guy.s...@hpe.com>> wrote: Do the snapshot happen only to the configuration data store or to the operational as well? -----Original Message----- From: Robert Varga [mailto:n...@hq.sk<mailto:n...@hq.sk>] Sent: Tuesday, January 10, 2017 10:21 PM To: Tom Pantelis <tompante...@gmail.com<mailto:tompante...@gmail.com>>; Sela, Guy <guy.s...@hpe.com<mailto:guy.s...@hpe.com>> Cc: controller-dev@lists.opendaylight.org<mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev <netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org>> Subject: Re: [controller-dev] [mdsal-dev] Netvirt Scale tests: OutOfMemory from datastore On 01/10/2017 08:16 PM, Tom Pantelis wrote: > Since the length field of an array is an int, it's constrained to > Integer.MAX_VALUE (~2G). To handle snapshots larger than 2G we'll have > to chuck it into multiple byte[]. That could get funky on the reassembly side of things. If we go that route, we may as well switch to something more friendly, like java.nio.ByteBuffer(), too. Guy, just out of curiosity: how much duplication is in your data set? Regards, Robert
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
