Re: Capacity planning for production deployment on kubernetes
Shiva, What version of Ignite do you use? Before version 2.7 Ignite used a different mechanism to limit the size of the WAL history. It used DataStorageConfiguration#walHistorySize property, which is currently deprecated. This is what’s explained on the internal documentation page. In Ignite versions starting from 2.7 DataStorageConfiguration#maxWalArchiveSize property is used. It specifies the maximum size of WAL history in bytes. As you said, it’s 4 times the checkpoint buffer size. Since you didn’t change the size of the checkpoint buffer, by default it’s 1 GB in your case (rules for its calculation can be found here: https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size) So, maximum WAL history size is 4 GB. Plus you need to add WAL size itself which is (64 MB per segment) x (10 segments) = 640 megabytes by default. If you use Ignite 2.7 or newer, then 10 GB should be enough for WAL with archive. Failures should be investigated in this case. But Ignite 2.6 and older use a different approach, so there is no strict limitation on the WAL archive size. Denis On 22 Aug 2019, 22:35 +0300, Denis Magda , wrote: > Please share the whole log file. It might be the case that something goes > wrong with volumes you attached to Ignite pods. > > - > Denis > > > > On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar > > wrote: > > > Hi Denis, > > > > > > Thanks for your response, > > > yes in our test also we have seen OOM errors and pod crash. > > > so we will follow the recommendation for RAM requirements and also I was > > > checking to ignite documentation on disk space required for WAL + WAL > > > archive. > > > here in this link > > > https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive > > > > > > it says: archive size is defined as 4 times the size of the checkpointing > > > buffer and checkpointing buffer is a function of the data region > > > (https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size) > > > > > > but in this link > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration > > > > > > under Estimating disk space section it explains something to estimate > > > disk space required for WAL but it is not clear, can you please help me > > > the correct recommendation for calculating the disk space required for > > > WAL+WAL archive. > > > > > > In one of my testing, I configured 4GB for data region and 10GB for > > > WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive > > > runs out of space. > > > > > > [ignite@ignite-cluster-ignite-node-2 ignite]$ df -h > > > Filesystem Size Used Avail Use% Mounted on > > > overlay 158G 39G 112G 26% / > > > tmpfs 63G 0 63G 0% /dev > > > tmpfs 63G 0 63G 0% /sys/fs/cgroup > > > /dev/vda1 158G 39G 112G 26% /etc/hosts > > > shm 64M 0 64M 0% /dev/shm > > > /dev/vdq 9.8G 9.7G 44M 100% /opt/ignite/wal > > > /dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence > > > tmpfs 63G 12K 63G 1% > > > /run/secrets/kubernetes.io/serviceaccount > > > tmpfs 63G 0 63G 0% /proc/acpi > > > tmpfs 63G 0 63G 0% /proc/scsi > > > tmpfs 63G 0 63G 0% /sys/firmware > > > > > > > > > and this is the error message in ignite node: > > > > > > "ERROR","JVM will be halted immediately due to the failure: > > > [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class > > > o.a.i.IgniteCheckedException: Failed to archive WAL segment > > > [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0006.wal, > > > > > > dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0236.wal.tmp]]]" > > > > > > > On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov > > > > wrote: > > > > > Shivakumar, > > > > > > > > > > Such allocation doesn’t allow full memory utilization, so it’s > > > > > possible, that nodes will crash because of out of memory errors. > > > > > So, it’s better to follow the given recommendation. > > > > > > > > > > If you want us to investigate reasons of the failures, please provide > > > > > logs and configuration of the failed nodes. > > > > > > > > > > Denis > > > > > On 21 Aug 2019, 16:17 +0300, Shiva Kumar , > > > > > wrote: > > > > > > Hi all, > > > > > > we are testing field use case before deploying in the field and we > > > > > > want to know whether below resource limits are suitable in > > > > > > production. > > > > > > There are 3 nodes (3 pods on kubernetes) running. Each having below > > > > > > configuration > > > > > > > > > > > > DefaultDataRegion: 60GB > > > > > > JVM: 32GB > > > > > > Resource allocated for each contain
Re: Capacity planning for production deployment on kubernetes
Please share the whole log file. It might be the case that something goes wrong with volumes you attached to Ignite pods. - Denis On Thu, Aug 22, 2019 at 8:07 AM Shiva Kumar wrote: > Hi Denis, > > Thanks for your response, > yes in our test also we have seen OOM errors and pod crash. > so we will follow the recommendation for RAM requirements and also I was > checking to ignite documentation on disk space required for WAL + WAL > archive. > here in this link > https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive > > it says: archive size is defined as 4 times the size of the checkpointing > buffer and checkpointing buffer is a function of the data region ( > https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size > ) > > but in this link > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration > > under *Estimating disk space* section it explains something to estimate > disk space required for WAL but it is not clear, can you please help me the > correct recommendation for calculating the disk space required for WAL+WAL > archive. > > In one of my testing, I configured 4GB for data region and 10GB for > WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive > runs out of space. > > [ignite@ignite-cluster-ignite-node-2 ignite]$* df -h* > Filesystem Size Used Avail Use% Mounted on > overlay 158G 39G 112G 26% / > tmpfs63G 0 63G 0% /dev > tmpfs63G 0 63G 0% /sys/fs/cgroup > /dev/vda1 158G 39G 112G 26% /etc/hosts > shm 64M 0 64M 0% /dev/shm > */dev/vdq9.8G 9.7G 44M 100% /opt/ignite/wal* > /dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence > tmpfs63G 12K 63G 1% /run/secrets/ > kubernetes.io/serviceaccount > tmpfs63G 0 63G 0% /proc/acpi > tmpfs63G 0 63G 0% /proc/scsi > tmpfs63G 0 63G 0% /sys/firmware > > > and this is the error message in ignite node: > > "ERROR","JVM will be halted immediately due to the failure: > [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class > o.a.i.IgniteCheckedException: Failed to archive WAL segment > [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0006.wal, > dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0236.wal.tmp]]]" > > > On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov > wrote: > >> Shivakumar, >> >> Such allocation doesn’t allow full memory utilization, so it’s possible, >> that nodes will crash because of out of memory errors. >> So, it’s better to follow the given recommendation. >> >> If you want us to investigate reasons of the failures, please provide >> logs and configuration of the failed nodes. >> >> Denis >> On 21 Aug 2019, 16:17 +0300, Shiva Kumar , >> wrote: >> >> Hi all, >> we are testing field use case before deploying in the field and we want >> to know whether below resource limits are suitable in production. >> There are 3 nodes (3 pods on kubernetes) running. Each having below >> configuration >> >>DefaultDataRegion: 60GB >> JVM: 32GB >> Resource allocated for each container: 64GB >> >> And ignite documents says (JVM+ All DataRegion) should not exceed 70% of >> total RAM allocated to each node(container). >> but we started testing with the above configuration and up to 9 days >> ignite cluster was running successfully and there was some data ingestion >> but suddenly pods crashed and they were unable to recover from the crash. >> does the above resource configuration not good for node recovery?? >> >>
Re: Capacity planning for production deployment on kubernetes
Hi Denis, Thanks for your response, yes in our test also we have seen OOM errors and pod crash. so we will follow the recommendation for RAM requirements and also I was checking to ignite documentation on disk space required for WAL + WAL archive. here in this link https://apacheignite.readme.io/docs/write-ahead-log#section-wal-archive it says: archive size is defined as 4 times the size of the checkpointing buffer and checkpointing buffer is a function of the data region ( https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size ) but in this link https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-SubfoldersGeneration under *Estimating disk space* section it explains something to estimate disk space required for WAL but it is not clear, can you please help me the correct recommendation for calculating the disk space required for WAL+WAL archive. In one of my testing, I configured 4GB for data region and 10GB for WAL+WAL archive but our pods crashing as disk mounted for WAL+WAL archive runs out of space. [ignite@ignite-cluster-ignite-node-2 ignite]$* df -h* Filesystem Size Used Avail Use% Mounted on overlay 158G 39G 112G 26% / tmpfs63G 0 63G 0% /dev tmpfs63G 0 63G 0% /sys/fs/cgroup /dev/vda1 158G 39G 112G 26% /etc/hosts shm 64M 0 64M 0% /dev/shm */dev/vdq9.8G 9.7G 44M 100% /opt/ignite/wal* /dev/vdr 50G 1.4G 48G 3% /opt/ignite/persistence tmpfs63G 12K 63G 1% /run/secrets/ kubernetes.io/serviceaccount tmpfs63G 0 63G 0% /proc/acpi tmpfs63G 0 63G 0% /proc/scsi tmpfs63G 0 63G 0% /sys/firmware and this is the error message in ignite node: "ERROR","JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed to archive WAL segment [srcFile=/opt/ignite/wal/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0006.wal, dstFile=/opt/ignite/wal/archive/node00-37ea8ba6-3198-46a1-9e9e-38aff27ed9c9/0236.wal.tmp]]]" On Thu, Aug 22, 2019 at 8:04 PM Denis Mekhanikov wrote: > Shivakumar, > > Such allocation doesn’t allow full memory utilization, so it’s possible, > that nodes will crash because of out of memory errors. > So, it’s better to follow the given recommendation. > > If you want us to investigate reasons of the failures, please provide logs > and configuration of the failed nodes. > > Denis > On 21 Aug 2019, 16:17 +0300, Shiva Kumar , > wrote: > > Hi all, > we are testing field use case before deploying in the field and we want to > know whether below resource limits are suitable in production. > There are 3 nodes (3 pods on kubernetes) running. Each having below > configuration > >DefaultDataRegion: 60GB > JVM: 32GB > Resource allocated for each container: 64GB > > And ignite documents says (JVM+ All DataRegion) should not exceed 70% of > total RAM allocated to each node(container). > but we started testing with the above configuration and up to 9 days > ignite cluster was running successfully and there was some data ingestion > but suddenly pods crashed and they were unable to recover from the crash. > does the above resource configuration not good for node recovery?? > >
Re: Capacity planning for production deployment on kubernetes
Shivakumar, Such allocation doesn’t allow full memory utilization, so it’s possible, that nodes will crash because of out of memory errors. So, it’s better to follow the given recommendation. If you want us to investigate reasons of the failures, please provide logs and configuration of the failed nodes. Denis On 21 Aug 2019, 16:17 +0300, Shiva Kumar , wrote: > Hi all, > we are testing field use case before deploying in the field and we want to > know whether below resource limits are suitable in production. > There are 3 nodes (3 pods on kubernetes) running. Each having below > configuration > > DefaultDataRegion: 60GB > JVM: 32GB > Resource allocated for each container: 64GB > > And ignite documents says (JVM+ All DataRegion) should not exceed 70% of > total RAM allocated to each node(container). > but we started testing with the above configuration and up to 9 days ignite > cluster was running successfully and there was some data ingestion but > suddenly pods crashed and they were unable to recover from the crash. > does the above resource configuration not good for node recovery??
