Hi Pieter, Your startup times are definitely too long - probably at least an order of magnitude. My first guess is that this is network related. This may either be a DNS lookup issue or, if the the cluster is isolated from the internet, it may be some problem with XSD validation needing internet access (even though we do bundle the XSD files with Geode - should be the same for Spring too). I will see if I can find any potential XSD issue.
--Jens On Wed, Oct 17, 2018 at 3:22 AM Pieter van Zyl <[email protected]> wrote: > Good day. > > We are currently running a 3 node Geode cluster. > > We are running the locator from gfsh and then staring up 3 servers with > Spring that connects to the central locator. > > We are using persistence on all the regions and have basically one data > and pdx store per node. > > The problem we are experiencing is that with no data aka clean cluster it > take 75minutes to start up. > > Once data has been imported into the cluster and we shutdown all > nodes/server and startup again it takes 128 to 160 minutes > This is very slow. > > Question is is there anyway to improve the startup speed? Is this normal > and expected speed? > > We have a 100gig database distributed across the 3 nodes. > Server 1: 100 gig memory and 90 gig assigned heap and db size of 49gig and > 32 cores. > Server 2: 64 gig memory and 60 gig assigned heap and db size of 34gig and > 16 cores > Server 3: 64 gig memory and 60 gig assigned heap and db size of 34gig and > 16 cores > > Should we have more data stores? Maybe separate stores for the partition > vs replicated regions? > > <gfe:disk-store id="pdx-disk-store" allow-force-compaction="true" > auto-compact="true" max-oplog-size="1024"> > * <gfe:disk-dir location="geode/pdx"/>* > </gfe:disk-store> > > <gfe:disk-store id="tauDiskStore" allow-force-compaction="true" > auto-compact="true" max-oplog-size="5120" > compaction-threshold="90"> > * <gfe:disk-dir location="geode/tauDiskStore"/>* > </gfe:disk-store> > > We have a mix of regions: > > Example partitioned region: > > <gfe:replicated-region id="net.lautus.gls.domain.life.accounting.Account" > disk-store-ref="tauDiskStore" > statistics="true" > persistent="true"><!--<gfe:cache-listener ref="cacheListener"/>--> > <gfe:eviction type="HEAP_PERCENTAGE" action="OVERFLOW_TO_DISK"/> > </gfe:replicated-region> > > Example replicated region: > <gfe:replicated-region id="org.rdb.internal.session.rootmap.RootMapHolder" > disk-store-ref="tauDiskStore" > statistics="true" persistent="true" > > > <!--<gfe:cache-listener ref="cacheListener"/>--> > <gfe:eviction type="ENTRY_COUNT" action="OVERFLOW_TO_DISK" > threshold="100"> > <gfe:object-sizer ref="objectSizer"/> > </gfe:eviction> > </gfe:replicated-region> > > > Any advice would be appreciated > > Kindly > Pieter >
