Re: Big Data Question

[Jeff Jirsa]

(Yes, just somewhat less likely to be the same order of speed-up in STCS
where sstables are more likely to cross token boundaries, modulo some stuff
around sstable splitting at token ranges a la 6696)

On Mon, Aug 21, 2023 at 11:35 AM Dinesh Joshi  wrote:

> Minor correction, zero copy streaming aka faster streaming also works for
> STCS.
>
> Dinesh
>
> On Aug 21, 2023, at 8:01 AM, Jeff Jirsa  wrote:
>
> 
> There's a lot of questionable advice scattered in this thread. Set aside
> most of the guidance like 2TB/node, it's old and super nuanced.
>
> If you're bare metal, do what your organization is good at. If you have
> millions of dollars in SAN equipment and you know how SANs work and fail
> and get backed up, run on a SAN if your organization knows how to properly
> operate a SAN. Just make sure you understand it's a single point of failure.
>
> If you're in the cloud, EBS is basically the same concept. You can lose
> EBS in an AZ, just like you can lose SAN in a DC. Persist outside of that.
> Have backups. Know how to restore them.
>
> The reason the "2TB/node" limit was a thing was around time to recover
> from failure more than anything else. I described this in detail here, in
> 2015, before faster-streaming in 4.0 was a thing :
> https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279
> . With faster streaming, IF you use LCS (so faster streaming works), you
> can probably go at least 4-5x more dense than before, if you understand how
> likely your disks are to fail and you can ensure you dont have correlated
> failures when they age out (that means if you're on bare metal, measuring
> flash life, and ideally mixing vendors to avoid firmware bugs).
>
> You'll still see risks of huge clusters, largely in gossip and schema
> propagation. Upcoming CEPs address those. 4.0 is better there (with schema,
> especially) than 3.0 was, but for "max nodes in a cluster", what you're
> really comparing is "how many gossip speakers and tokens are in the
> cluster" (which means your vnode settings matter, for things like pending
> range calculators).
>
> Looking at the roadmap, your real question comes down to :
> - If you expect to use the transactional features in Accord/5.0 to
> transact across rows/keys, you probably want to keep one cluster
> - If you dont ever expect to use multi-key transactions, just de-risk by
> sharding your cluster into many smaller clusters now, with consistent
> hashing to map keys to clusters, and have 4 clusters of the same smaller
> size, with whatever node density you think you can do based on your
> compaction strategy and streaming rate (and disk type).
>
> If you have time and budget, create a 3 node cluster with whatever disks
> you have, fill them, start working on them - expand to 4, treat one as
> failed and replace it - simulate the operations you'll do at that size.
> It's expensive to mimic a 500 host cluster, but if you've got budget, try
> it in AWS and see what happens when you apply your real schema, and then do
> a schema change.
>
>
>
>
>
> On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
>> For our scenario, the goal is to minimize down-time for a single (at
>> least initially) data center system.  Data-loss is basically unacceptable.
>> I wouldn't say we have a "rusty slow data center" - we can certainly use
>> SSDs and have servers connected via 10G copper to a fast back-plane.  For
>> our specific use case with Cassandra (lots of writes, small number of
>> reads), the network load is usually pretty low.  I suspect that would
>> change if we used Kubernetes + central persistent storage.
>> Good discussion.
>>
>> -Joe
>> On 8/17/2023 7:37 PM, daemeon reiydelle wrote:
>>
>> I started to respond, then realized I and the other OP posters are not
>> thinking the same: What is the business case for availability, data
>> los/reload/recoverability? You all argue for higher availability and damn
>> the cost. But noone asked "can you lose access, for 20 minutes, to a
>> portion of the data, 10 times a year, on a 250 node cluster in AWS, if it
>> is not lost"? Can you lose access 1-2 times a year for the cost of a 500
>> node cluster holding the same data?
>>
>> Then we can discuss 32/64g JVM and SSD's.
>> *.*
>> *Arthur C. Clarke famously said that "technology sufficiently advanced is
>> indistinguishable from magic." Magic is coming, and it's coming for all of
>> us*
>>
>> *Daemeon Reiydelle*
>> *email: daeme...@gmail.com *
>> *LI: https://www.linkedin.com/in/daemeonreiydelle/
>> *
>> *San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*
>>
>>
>> On Thu, Aug 17, 2023 at 1:53 PM Joe Obernberger <
>> joseph.obernber...@gmail.com> wrote:
>>
>>> Was assuming reaper did incremental?  That was probably a bad assumption.
>>>
>>> nodetool repair -pr
>>> I know it well now!
>>>
>>> :)
>>>
>>> -Joe
>>>
>>> On 8/

Re: Big Data Question

[Dinesh Joshi]

Minor correction, zero copy streaming aka faster streaming also works for STCS.DineshOn Aug 21, 2023, at 8:01 AM, Jeff Jirsa  wrote:There's a lot of questionable advice scattered in this thread. Set aside most of the guidance like 2TB/node, it's old and super nuanced.If you're bare metal, do what your organization is good at. If you have millions of dollars in SAN equipment and you know how SANs work and fail and get backed up, run on a SAN if your organization knows how to properly operate a SAN. Just make sure you understand it's a single point of failure.If you're in the cloud, EBS is basically the same concept. You can lose EBS in an AZ, just like you can lose SAN in a DC. Persist outside of that. Have backups. Know how to restore them. The reason the "2TB/node" limit was a thing was around time to recover from failure more than anything else. I described this in detail here, in 2015, before faster-streaming in 4.0 was a thing : https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279 . With faster streaming, IF you use LCS (so faster streaming works), you can probably go at least 4-5x more dense than before, if you understand how likely your disks are to fail and you can ensure you dont have correlated failures when they age out (that means if you're on bare metal, measuring flash life, and ideally mixing vendors to avoid firmware bugs). You'll still see risks of huge clusters, largely in gossip and schema propagation. Upcoming CEPs address those. 4.0 is better there (with schema, especially) than 3.0 was, but for "max nodes in a cluster", what you're really comparing is "how many gossip speakers and tokens are in the cluster" (which means your vnode settings matter, for things like pending range calculators). Looking at the roadmap, your real question comes down to : - If you expect to use the transactional features in Accord/5.0 to transact across rows/keys, you probably want to keep one cluster- If you dont ever expect to use multi-key transactions, just de-risk by sharding your cluster into many smaller clusters now, with consistent hashing to map keys to clusters, and have 4 clusters of the same smaller size, with whatever node density you think you can do based on your compaction strategy and streaming rate (and disk type). If you have time and budget, create a 3 node cluster with whatever disks you have, fill them, start working on them - expand to 4, treat one as failed and replace it - simulate the operations you'll do at that size. It's expensive to mimic a 500 host cluster, but if you've got budget, try it in AWS and see what happens when you apply your real schema, and then do a schema change.On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger  wrote:

  

  
  
For our scenario, the goal is to minimize down-time for a single
  (at least initially) data center system.  Data-loss is basically
  unacceptable.  I wouldn't say we have a "rusty slow data center" -
  we can certainly use SSDs and have servers connected via 10G
  copper to a fast back-plane.  For our specific use case with
  Cassandra (lots of writes, small number of reads), the network
  load is usually pretty low.  I suspect that would change if we
  used Kubernetes + central persistent storage.  
  Good discussion.
-Joe

On 8/17/2023 7:37 PM, daemeon reiydelle
  wrote:


  
  
I
  started to respond, then realized I and the other OP posters
  are not thinking the same: What is the business case for
  availability, data los/reload/recoverability? You all argue
  for higher availability and damn the cost. But noone asked
  "can you lose access, for 20 minutes, to a portion of the
  data, 10 times a year, on a 250 node cluster in AWS, if it is
  not lost"? Can you lose access 1-2 times a year for the cost
  of a 500 node cluster holding the same data?


Then
  we can discuss 32/64g JVM and SSD's.

  

  .
  Arthur C. Clarke famously said that "technology
sufficiently advanced is indistinguishable from magic."
Magic is coming, and it's coming for all of us
  

  

  
Daemeon
Reiydelle

  email: daeme...@gmail.com
LI: https://www.linkedin.com/in/daemeonreiydelle/

San Francisco
1.415.501.0198/Skype daemeon.c.m.reiydelle
  

  

  


  
  
  
On Thu, Aug 17, 2023 at
  1:53 PM Joe Obernberger 
 

Re: Big Data Question

[daemeon reiydelle]

- k8s

   1. depending on the version and networking, number of containers per
   node, nodepooling, etc. you can expect to see 1-2% additional storage IO
   latency (depends on whether all are on the same network vs. separate
   storage IO TCP network)
   2. System overhead may be 3-15% depending on what security mitigations
   are in place (if you own the systems and workload is dedicated, turn them
   off!)
   3. c* pod loss recovery is the big win here. pod failure and recovery
   (e.g. to another node) will bring up the SAME c* node as of the node
   failure (so only a few updates). Perhaps 2x replication, or none if the
   storage itself is replicated.

I wonder if you folks have already set out OLA's for "minimum outage" with
no data loss? Write amplification is mostly only a problem when networks
are heavily used. May not even be an issue in your case.
*.*
*Arthur C. Clarke famously said that "technology sufficiently advanced is
indistinguishable from magic." Magic is coming, and it's coming for all of
us*

*Daemeon Reiydelle*
*email: daeme...@gmail.com *
*LI: https://www.linkedin.com/in/daemeonreiydelle/
*
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*


On Mon, Aug 21, 2023 at 8:49 AM Patrick McFadin  wrote:

> ...and a shameless plug for the Cassandra Summit in December. We have a
> talk from somebody that is doing 70TB per node and will be digging into all
> the aspects that make that work for them. I hope everyone in this thread is
> at that talk! I can't wait to hear all the questions.
>
> Patrick
>
> On Mon, Aug 21, 2023 at 8:01 AM Jeff Jirsa  wrote:
>
>> There's a lot of questionable advice scattered in this thread. Set aside
>> most of the guidance like 2TB/node, it's old and super nuanced.
>>
>> If you're bare metal, do what your organization is good at. If you have
>> millions of dollars in SAN equipment and you know how SANs work and fail
>> and get backed up, run on a SAN if your organization knows how to properly
>> operate a SAN. Just make sure you understand it's a single point of failure.
>>
>> If you're in the cloud, EBS is basically the same concept. You can lose
>> EBS in an AZ, just like you can lose SAN in a DC. Persist outside of that.
>> Have backups. Know how to restore them.
>>
>> The reason the "2TB/node" limit was a thing was around time to recover
>> from failure more than anything else. I described this in detail here, in
>> 2015, before faster-streaming in 4.0 was a thing :
>> https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279
>> . With faster streaming, IF you use LCS (so faster streaming works), you
>> can probably go at least 4-5x more dense than before, if you understand how
>> likely your disks are to fail and you can ensure you dont have correlated
>> failures when they age out (that means if you're on bare metal, measuring
>> flash life, and ideally mixing vendors to avoid firmware bugs).
>>
>> You'll still see risks of huge clusters, largely in gossip and schema
>> propagation. Upcoming CEPs address those. 4.0 is better there (with schema,
>> especially) than 3.0 was, but for "max nodes in a cluster", what you're
>> really comparing is "how many gossip speakers and tokens are in the
>> cluster" (which means your vnode settings matter, for things like pending
>> range calculators).
>>
>> Looking at the roadmap, your real question comes down to :
>> - If you expect to use the transactional features in Accord/5.0 to
>> transact across rows/keys, you probably want to keep one cluster
>> - If you dont ever expect to use multi-key transactions, just de-risk by
>> sharding your cluster into many smaller clusters now, with consistent
>> hashing to map keys to clusters, and have 4 clusters of the same smaller
>> size, with whatever node density you think you can do based on your
>> compaction strategy and streaming rate (and disk type).
>>
>> If you have time and budget, create a 3 node cluster with whatever disks
>> you have, fill them, start working on them - expand to 4, treat one as
>> failed and replace it - simulate the operations you'll do at that size.
>> It's expensive to mimic a 500 host cluster, but if you've got budget, try
>> it in AWS and see what happens when you apply your real schema, and then do
>> a schema change.
>>
>>
>>
>>
>>
>> On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger <
>> joseph.obernber...@gmail.com> wrote:
>>
>>> For our scenario, the goal is to minimize down-time for a single (at
>>> least initially) data center system.  Data-loss is basically unacceptable.
>>> I wouldn't say we have a "rusty slow data center" - we can certainly use
>>> SSDs and have servers connected via 10G copper to a fast back-plane.  For
>>> our specific use case with Cassandra (lots of writes, small number of
>>> reads), the network load is usually pretty low.  I suspect that would
>>> change if we used Kubernetes + central 

Re: Big Data Question

[Patrick McFadin]

...and a shameless plug for the Cassandra Summit in December. We have a
talk from somebody that is doing 70TB per node and will be digging into all
the aspects that make that work for them. I hope everyone in this thread is
at that talk! I can't wait to hear all the questions.

Patrick

On Mon, Aug 21, 2023 at 8:01 AM Jeff Jirsa  wrote:

> There's a lot of questionable advice scattered in this thread. Set aside
> most of the guidance like 2TB/node, it's old and super nuanced.
>
> If you're bare metal, do what your organization is good at. If you have
> millions of dollars in SAN equipment and you know how SANs work and fail
> and get backed up, run on a SAN if your organization knows how to properly
> operate a SAN. Just make sure you understand it's a single point of failure.
>
> If you're in the cloud, EBS is basically the same concept. You can lose
> EBS in an AZ, just like you can lose SAN in a DC. Persist outside of that.
> Have backups. Know how to restore them.
>
> The reason the "2TB/node" limit was a thing was around time to recover
> from failure more than anything else. I described this in detail here, in
> 2015, before faster-streaming in 4.0 was a thing :
> https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279
> . With faster streaming, IF you use LCS (so faster streaming works), you
> can probably go at least 4-5x more dense than before, if you understand how
> likely your disks are to fail and you can ensure you dont have correlated
> failures when they age out (that means if you're on bare metal, measuring
> flash life, and ideally mixing vendors to avoid firmware bugs).
>
> You'll still see risks of huge clusters, largely in gossip and schema
> propagation. Upcoming CEPs address those. 4.0 is better there (with schema,
> especially) than 3.0 was, but for "max nodes in a cluster", what you're
> really comparing is "how many gossip speakers and tokens are in the
> cluster" (which means your vnode settings matter, for things like pending
> range calculators).
>
> Looking at the roadmap, your real question comes down to :
> - If you expect to use the transactional features in Accord/5.0 to
> transact across rows/keys, you probably want to keep one cluster
> - If you dont ever expect to use multi-key transactions, just de-risk by
> sharding your cluster into many smaller clusters now, with consistent
> hashing to map keys to clusters, and have 4 clusters of the same smaller
> size, with whatever node density you think you can do based on your
> compaction strategy and streaming rate (and disk type).
>
> If you have time and budget, create a 3 node cluster with whatever disks
> you have, fill them, start working on them - expand to 4, treat one as
> failed and replace it - simulate the operations you'll do at that size.
> It's expensive to mimic a 500 host cluster, but if you've got budget, try
> it in AWS and see what happens when you apply your real schema, and then do
> a schema change.
>
>
>
>
>
> On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
>> For our scenario, the goal is to minimize down-time for a single (at
>> least initially) data center system.  Data-loss is basically unacceptable.
>> I wouldn't say we have a "rusty slow data center" - we can certainly use
>> SSDs and have servers connected via 10G copper to a fast back-plane.  For
>> our specific use case with Cassandra (lots of writes, small number of
>> reads), the network load is usually pretty low.  I suspect that would
>> change if we used Kubernetes + central persistent storage.
>> Good discussion.
>>
>> -Joe
>> On 8/17/2023 7:37 PM, daemeon reiydelle wrote:
>>
>> I started to respond, then realized I and the other OP posters are not
>> thinking the same: What is the business case for availability, data
>> los/reload/recoverability? You all argue for higher availability and damn
>> the cost. But noone asked "can you lose access, for 20 minutes, to a
>> portion of the data, 10 times a year, on a 250 node cluster in AWS, if it
>> is not lost"? Can you lose access 1-2 times a year for the cost of a 500
>> node cluster holding the same data?
>>
>> Then we can discuss 32/64g JVM and SSD's.
>> *.*
>> *Arthur C. Clarke famously said that "technology sufficiently advanced is
>> indistinguishable from magic." Magic is coming, and it's coming for all of
>> us*
>>
>> *Daemeon Reiydelle*
>> *email: daeme...@gmail.com *
>> *LI: https://www.linkedin.com/in/daemeonreiydelle/
>> *
>> *San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*
>>
>>
>> On Thu, Aug 17, 2023 at 1:53 PM Joe Obernberger <
>> joseph.obernber...@gmail.com> wrote:
>>
>>> Was assuming reaper did incremental?  That was probably a bad assumption.
>>>
>>> nodetool repair -pr
>>> I know it well now!
>>>
>>> :)
>>>
>>> -Joe
>>>
>>> On 8/17/2023 4:47 PM, Bowen Song via user wrote:
>>> > I don't have experie

Re: Big Data Question

[Jeff Jirsa]

There's a lot of questionable advice scattered in this thread. Set aside
most of the guidance like 2TB/node, it's old and super nuanced.

If you're bare metal, do what your organization is good at. If you have
millions of dollars in SAN equipment and you know how SANs work and fail
and get backed up, run on a SAN if your organization knows how to properly
operate a SAN. Just make sure you understand it's a single point of failure.

If you're in the cloud, EBS is basically the same concept. You can lose EBS
in an AZ, just like you can lose SAN in a DC. Persist outside of that. Have
backups. Know how to restore them.

The reason the "2TB/node" limit was a thing was around time to recover from
failure more than anything else. I described this in detail here, in 2015,
before faster-streaming in 4.0 was a thing :
https://stackoverflow.com/questions/31563447/cassandra-cluster-data-density-data-size-per-node-looking-for-feedback-and/31690279
. With faster streaming, IF you use LCS (so faster streaming works), you
can probably go at least 4-5x more dense than before, if you understand how
likely your disks are to fail and you can ensure you dont have correlated
failures when they age out (that means if you're on bare metal, measuring
flash life, and ideally mixing vendors to avoid firmware bugs).

You'll still see risks of huge clusters, largely in gossip and schema
propagation. Upcoming CEPs address those. 4.0 is better there (with schema,
especially) than 3.0 was, but for "max nodes in a cluster", what you're
really comparing is "how many gossip speakers and tokens are in the
cluster" (which means your vnode settings matter, for things like pending
range calculators).

Looking at the roadmap, your real question comes down to :
- If you expect to use the transactional features in Accord/5.0 to transact
across rows/keys, you probably want to keep one cluster
- If you dont ever expect to use multi-key transactions, just de-risk by
sharding your cluster into many smaller clusters now, with consistent
hashing to map keys to clusters, and have 4 clusters of the same smaller
size, with whatever node density you think you can do based on your
compaction strategy and streaming rate (and disk type).

If you have time and budget, create a 3 node cluster with whatever disks
you have, fill them, start working on them - expand to 4, treat one as
failed and replace it - simulate the operations you'll do at that size.
It's expensive to mimic a 500 host cluster, but if you've got budget, try
it in AWS and see what happens when you apply your real schema, and then do
a schema change.





On Mon, Aug 21, 2023 at 7:31 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:

> For our scenario, the goal is to minimize down-time for a single (at least
> initially) data center system.  Data-loss is basically unacceptable.  I
> wouldn't say we have a "rusty slow data center" - we can certainly use SSDs
> and have servers connected via 10G copper to a fast back-plane.  For our
> specific use case with Cassandra (lots of writes, small number of reads),
> the network load is usually pretty low.  I suspect that would change if we
> used Kubernetes + central persistent storage.
> Good discussion.
>
> -Joe
> On 8/17/2023 7:37 PM, daemeon reiydelle wrote:
>
> I started to respond, then realized I and the other OP posters are not
> thinking the same: What is the business case for availability, data
> los/reload/recoverability? You all argue for higher availability and damn
> the cost. But noone asked "can you lose access, for 20 minutes, to a
> portion of the data, 10 times a year, on a 250 node cluster in AWS, if it
> is not lost"? Can you lose access 1-2 times a year for the cost of a 500
> node cluster holding the same data?
>
> Then we can discuss 32/64g JVM and SSD's.
> *.*
> *Arthur C. Clarke famously said that "technology sufficiently advanced is
> indistinguishable from magic." Magic is coming, and it's coming for all of
> us*
>
> *Daemeon Reiydelle*
> *email: daeme...@gmail.com *
> *LI: https://www.linkedin.com/in/daemeonreiydelle/
> *
> *San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*
>
>
> On Thu, Aug 17, 2023 at 1:53 PM Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
>> Was assuming reaper did incremental?  That was probably a bad assumption.
>>
>> nodetool repair -pr
>> I know it well now!
>>
>> :)
>>
>> -Joe
>>
>> On 8/17/2023 4:47 PM, Bowen Song via user wrote:
>> > I don't have experience with Cassandra on Kubernetes, so I can't
>> > comment on that.
>> >
>> > For repairs, may I interest you with incremental repairs? It will make
>> > repairs hell of a lot faster. Of course, occasional full repair is
>> > still needed, but that's another story.
>> >
>> >
>> > On 17/08/2023 21:36, Joe Obernberger wrote:
>> >> Thank you.  Enjoying this conversation.
>> >> Agree on blade servers, where each blade has a small number of SSDs.
>> >> Yeh/Nah to a kubernetes app

Re: Big Data Question

[Joe Obernberger]

For our scenario, the goal is to minimize down-time for a single (at 
least initially) data center system.  Data-loss is basically 
unacceptable.  I wouldn't say we have a "rusty slow data center" - we 
can certainly use SSDs and have servers connected via 10G copper to a 
fast back-plane.  For our specific use case with Cassandra (lots of 
writes, small number of reads), the network load is usually pretty low.  
I suspect that would change if we used Kubernetes + central persistent 
storage.

Good discussion.

-Joe

On 8/17/2023 7:37 PM, daemeon reiydelle wrote:
I started to respond, then realized I and the other OP posters are not 
thinking the same: What is the business case for availability, data 
los/reload/recoverability? You all argue for higher availability and 
damn the cost. But noone asked "can you lose access, for 20 minutes, 
to a portion of the data, 10 times a year, on a 250 node cluster in 
AWS, if it is not lost"? Can you lose access 1-2 times a year for the 
cost of a 500 node cluster holding the same data?


Then we can discuss 32/64g JVM and SSD's.
/./
/Arthur C. Clarke famously said that "technology sufficiently advanced 
is indistinguishable from magic." Magic is coming, and it's coming for 
all of us/

/
/
*Daemeon Reiydelle*
*email: daeme...@gmail.com*
*LI: https://www.linkedin.com/in/daemeonreiydelle/*
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*


On Thu, Aug 17, 2023 at 1:53 PM Joe Obernberger 
 wrote:


Was assuming reaper did incremental?  That was probably a bad
assumption.

nodetool repair -pr
I know it well now!

:)

-Joe

On 8/17/2023 4:47 PM, Bowen Song via user wrote:
> I don't have experience with Cassandra on Kubernetes, so I can't
> comment on that.
>
> For repairs, may I interest you with incremental repairs? It
will make
> repairs hell of a lot faster. Of course, occasional full repair is
> still needed, but that's another story.
>
>
> On 17/08/2023 21:36, Joe Obernberger wrote:
>> Thank you.  Enjoying this conversation.
>> Agree on blade servers, where each blade has a small number of
SSDs.
>> Yeh/Nah to a kubernetes approach assuming fast persistent
storage?  I
>> think that might be easier to manage.
>>
>> In my current benchmarks, the performance is excellent, but the
>> repairs are painful.  I come from the Hadoop world where it was
all
>> about large servers with lots of disk.
>> Relatively small number of tables, but some have a high number of
>> rows, 10bil + - we use spark to run across all the data.
>>
>> -Joe
>>
>> On 8/17/2023 12:13 PM, Bowen Song via user wrote:
>>> The optimal node size largely depends on the table schema and
>>> read/write pattern. In some cases 500 GB per node is too
large, but
>>> in some other cases 10TB per node works totally fine. It's
hard to
>>> estimate that without benchmarking.
>>>
>>> Again, just pointing out the obvious, you did not count the
off-heap
>>> memory and page cache. 1TB of RAM for 24GB heap * 40 instances is
>>> definitely not enough. You'll most likely need between 1.5 and
2 TB
>>> memory for 40x 24GB heap nodes. You may be better off with blade
>>> servers than single server with gigantic memory and disk sizes.
>>>
>>>
>>> On 17/08/2023 15:46, Joe Obernberger wrote:
 Thanks for this - yeah - duh - forgot about replication in my
example!
 So - is 2TBytes per Cassandra instance advisable?  Better to use
 more/less?  Modern 2u servers can be had with 24 3.8TBtyte
SSDs; so
 assume 80Tbytes per server, you could do:
 (1024*3)/80 = 39 servers, but you'd have to run 40 instances of
 Cassandra on each server; maybe 24G of heap per instance, so a
 server with 1TByte of RAM would work.
 Is this what folks would do?

 -Joe

 On 8/17/2023 9:13 AM, Bowen Song via user wrote:
> Just pointing out the obvious, for 1PB of data on nodes with
2TB
> disk each, you will need far more than 500 nodes.
>
> 1, it is unwise to run Cassandra with replication factor 1. It
> usually makes sense to use RF=3, so 1PB data will cost 3PB of
> storage space, minimal of 1500 such nodes.
>
> 2, depending on the compaction strategy you use and the write
> access pattern, there's a disk space amplification to consider.
> For example, with STCS, the disk usage can be many times of the
> actual live data size.
>
> 3, you will need some extra free disk space as temporary
space for
> running compactions.
>
> 4, the data is rarely going to be perfectly evenly distributed
> among all nodes, and you need to take that into
consideration and
> size the nodes based on the node with the