space left for compaction
Dear All, As for STCS,datastax suggest us to keep half of the free space for compaction,this is not strict,could anyone advise how many space should we left for one node? Thanks, Peng Xiao
RE: space left for compaction
Hi, half of free space does not make sense. Imagine your SSTables need 100G space and you have 20G free disk. Compaction won't be able to do its job with 10G. Half free of total disk makes more sense and is what you need for a major compaction worst case. Thomas From: Peng Xiao [mailto:2535...@qq.com] Sent: Samstag, 30. September 2017 10:21 To: user Subject: space left for compaction Dear All, As for STCS,datastax suggest us to keep half of the free space for compaction,this is not strict,could anyone advise how many space should we left for one node? Thanks, Peng Xiao The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313
RE: space left for compaction
Hi Thomas , So IIUC in this case you should leave at least 50G for compaction (half of the sstables size). Is that makes sense? Cheers Avi On Oct 1, 2017 11:39 AM, "Steinmaurer, Thomas" < thomas.steinmau...@dynatrace.com> wrote: Hi, half of free space does not make sense. Imagine your SSTables need 100G space and you have 20G free disk. Compaction won’t be able to do its job with 10G. Half free of total disk makes more sense and is what you need for a major compaction worst case. Thomas *From:* Peng Xiao [mailto:2535...@qq.com] *Sent:* Samstag, 30. September 2017 10:21 *To:* user *Subject:* space left for compaction Dear All, As for STCS,datastax suggest us to keep half of the free space for compaction,this is not strict,could anyone advise how many space should we left for one node? Thanks, Peng Xiao The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> ädterstra <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> ße 313 <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
Re: space left for compaction
Hi Avi, Actually, in Thomas' example you would need an additional 100G of free disk space to complete the compaction, in the worst-case situation (the worst-case would be that neither input SSTable contains any overlapping data or tombstones, therefore the output SSTable would also be roughly 100G). STCS progressively compacts SSTables of similar size together, with the output being a single SSTable containing the data of the input SSTables. Eventually you may end up with some very large SSTables that combined will take up 50% of your total disk space. In order to compact those SSTables together, STCS requires an equal amount of free disk space, which would be the other (unused) 50% of your total disk space. Cheers, Justin On Mon, 2 Oct 2017 at 12:42 Avi Levi wrote: > Hi Thomas , > So IIUC in this case you should leave at least 50G for compaction (half > of the sstables size). Is that makes sense? > Cheers > Avi > > > On Oct 1, 2017 11:39 AM, "Steinmaurer, Thomas" < > thomas.steinmau...@dynatrace.com> wrote: > > Hi, > > > > half of free space does not make sense. Imagine your SSTables need 100G > space and you have 20G free disk. Compaction won’t be able to do its job > with 10G. > > > > Half free of total disk makes more sense and is what you need for a major > compaction worst case. > > > > Thomas > > > > *From:* Peng Xiao [mailto:2535...@qq.com] > *Sent:* Samstag, 30. September 2017 10:21 > *To:* user > *Subject:* space left for compaction > > > > Dear All, > > > > As for STCS,datastax suggest us to keep half of the free space for > compaction,this is not strict,could anyone advise how many space should we > left for one node? > > > > Thanks, > > Peng Xiao > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freist > <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> > ädterstra > <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> > ße 313 > <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> > > > -- *Justin Cameron*Senior Software Engineer <https://www.instaclustr.com/> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
Re: space left for compaction
Got it. Thanks On Mon, Oct 2, 2017 at 4:54 AM, Justin Cameron wrote: > Hi Avi, > > Actually, in Thomas' example you would need an additional 100G of free > disk space to complete the compaction, in the worst-case situation (the > worst-case would be that neither input SSTable contains any overlapping > data or tombstones, therefore the output SSTable would also be roughly > 100G). > > STCS progressively compacts SSTables of similar size together, with the > output being a single SSTable containing the data of the input SSTables. > > Eventually you may end up with some very large SSTables that combined will > take up 50% of your total disk space. In order to compact those SSTables > together, STCS requires an equal amount of free disk space, which would be > the other (unused) 50% of your total disk space. > > Cheers, > Justin > > On Mon, 2 Oct 2017 at 12:42 Avi Levi wrote: > >> Hi Thomas , >> So IIUC in this case you should leave at least 50G for compaction (half >> of the sstables size). Is that makes sense? >> Cheers >> Avi >> >> >> On Oct 1, 2017 11:39 AM, "Steinmaurer, Thomas" < >> thomas.steinmau...@dynatrace.com> wrote: >> >> Hi, >> >> >> >> half of free space does not make sense. Imagine your SSTables need 100G >> space and you have 20G free disk. Compaction won’t be able to do its job >> with 10G. >> >> >> >> Half free of total disk makes more sense and is what you need for a major >> compaction worst case. >> >> >> >> Thomas >> >> >> >> *From:* Peng Xiao [mailto:2535...@qq.com] >> *Sent:* Samstag, 30. September 2017 10:21 >> *To:* user >> *Subject:* space left for compaction >> >> >> >> Dear All, >> >> >> >> As for STCS,datastax suggest us to keep half of the free space for >> compaction,this is not strict,could anyone advise how many space should we >> left for one node? >> >> >> >> Thanks, >> >> Peng Xiao >> The contents of this e-mail are intended for the named addressee only. It >> contains information that may be confidential. Unless you are the named >> addressee or an authorized designee, you may not copy or use it, or >> disclose it to anyone else. If you received it in error please notify us >> immediately and then destroy it. Dynatrace Austria GmbH (registration >> number FN 91482h) is a company registered in Linz whose registered office >> is at 4040 Linz, Austria, Freist >> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> >> ädterstra >> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> >> ße 313 >> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> >> >> >> -- > > > *Justin Cameron*Senior Software Engineer > > > <https://www.instaclustr.com/> > > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. >