2020 ASF Community Survey: Users

[Nate McCall]

Hello everyone,

If you have an apache.org email, you should have received an email with an
invitation to take the 2020 ASF Community Survey. Please take 15 minutes to
complete it.

If you do not have an apache.org email address or you didn’t receive a
link, please follow this link to the survey:
https://communitysurvey.limequery.org/454363

This survey is important because it will provide us with scientific
information about our community, and shed some light on how we can
collaborate better and become more diverse. Our last survey of this kind
was implemented in 2016, which means that our existing data about Apache
communities is outdated. The deadline to complete the survey is January
4th, 2020. You can find information about privacy on the survey’s
Confluence page [1].

Your participation is paramount to the success of this project! Please
consider filling out the survey, and share this news with your fellow
Apache contributors. As individuals form the Apache community, your opinion
matters: we want to hear your voice.

If you have any questions about the survey or otherwise, please reach out
to us!

Kindly,
ASF Diversity & Inclusion
https://diversity.apache.org/


[1]
https://cwiki.apache.org/confluence/display/EDI/Launch+Plan+-+The+2020+ASF+Community+Survey

Re: AWS ephemeral instances + backup

[daemeon reiydelle]

If you can handle the slower IO of S3 this can work, but you will have a
window of out of date images. YOu don't have a concept of persistent
snapshots.

<==>
Life lived is not about the size of the dog in the fight:
It is about the size of the fight in the dog.

*Daemeon Reiydelle*

*email: daeme...@gmail.com *
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*



On Thu, Dec 5, 2019 at 2:06 PM Jon Haddad  wrote:

> You can easily do this with bcache or LVM
> http://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/.
>
> Medusa might be a good route to go down if you want to do backups instead:
> https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html
>
>
>
> On Thu, Dec 5, 2019 at 12:21 PM Carl Mueller
>  wrote:
>
>> Does anyone have experience tooling written to support this strategy:
>>
>> Use case: run cassandra on i3 instances on ephemerals but synchronize the
>> sstables and commitlog files to the cheapest EBS volume type (those have
>> bad IOPS but decent enough throughput)
>>
>> On node replace, the startup script for the node, back-copies the
>> sstables and commitlog state from the EBS to the ephemeral.
>>
>> As can be seen:
>> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
>>
>> the (presumably) spinning rust tops out at 2375 MB/sec (using
>> multiple EBS volumes presumably) that would incur about a ten minute delay
>> for node replacement for a 1TB node, but I imagine this would only be used
>> on higher IOPS r/w nodes with smaller densities, so 100GB would be about a
>> minute of delay only, already within the timeframes of an AWS node
>> replacement/instance restart.
>>
>>
>>

Re: AWS ephemeral instances + backup

[Jon Haddad]

You can easily do this with bcache or LVM
http://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/.

Medusa might be a good route to go down if you want to do backups instead:
https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html



On Thu, Dec 5, 2019 at 12:21 PM Carl Mueller
 wrote:

> Does anyone have experience tooling written to support this strategy:
>
> Use case: run cassandra on i3 instances on ephemerals but synchronize the
> sstables and commitlog files to the cheapest EBS volume type (those have
> bad IOPS but decent enough throughput)
>
> On node replace, the startup script for the node, back-copies the sstables
> and commitlog state from the EBS to the ephemeral.
>
> As can be seen:
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
>
> the (presumably) spinning rust tops out at 2375 MB/sec (using multiple EBS
> volumes presumably) that would incur about a ten minute delay for node
> replacement for a 1TB node, but I imagine this would only be used on higher
> IOPS r/w nodes with smaller densities, so 100GB would be about a minute of
> delay only, already within the timeframes of an AWS node
> replacement/instance restart.
>
>
>

Re: AWS ephemeral instances + backup

[Ben Slater]

We have some tooling that does that kind of thing using S3 rather  than
attached EBS but a similar principle. There is a bit of an overview here:
https://www.instaclustr.com/advanced-node-replace/

It's become a pretty core part of our ops toolbox since we introduced it.

Cheers
Ben

---


*Ben Slater**Chief Product Officer*



   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.


On Fri, 6 Dec 2019 at 08:32, Jeff Jirsa  wrote:

>
> No experience doing it that way personally, but I'm curious: Are you
> backing up in case of ephemeral instance dying, or backing up in case of
> data problems / errors / etc?
>
> On instance dying, you're probably fine with just straight normal
> replacements, not restoring from backup. For the rest, is it cheaper to use
> something like tablesnap and go straight to s3?
>
> On Thu, Dec 5, 2019 at 12:21 PM Carl Mueller
>  wrote:
>
>> Does anyone have experience tooling written to support this strategy:
>>
>> Use case: run cassandra on i3 instances on ephemerals but synchronize the
>> sstables and commitlog files to the cheapest EBS volume type (those have
>> bad IOPS but decent enough throughput)
>>
>> On node replace, the startup script for the node, back-copies the
>> sstables and commitlog state from the EBS to the ephemeral.
>>
>> As can be seen:
>> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
>>
>> the (presumably) spinning rust tops out at 2375 MB/sec (using
>> multiple EBS volumes presumably) that would incur about a ten minute delay
>> for node replacement for a 1TB node, but I imagine this would only be used
>> on higher IOPS r/w nodes with smaller densities, so 100GB would be about a
>> minute of delay only, already within the timeframes of an AWS node
>> replacement/instance restart.
>>
>>
>>

Re: AWS ephemeral instances + backup

[Jeff Jirsa]

No experience doing it that way personally, but I'm curious: Are you
backing up in case of ephemeral instance dying, or backing up in case of
data problems / errors / etc?

On instance dying, you're probably fine with just straight normal
replacements, not restoring from backup. For the rest, is it cheaper to use
something like tablesnap and go straight to s3?

On Thu, Dec 5, 2019 at 12:21 PM Carl Mueller
 wrote:

> Does anyone have experience tooling written to support this strategy:
>
> Use case: run cassandra on i3 instances on ephemerals but synchronize the
> sstables and commitlog files to the cheapest EBS volume type (those have
> bad IOPS but decent enough throughput)
>
> On node replace, the startup script for the node, back-copies the sstables
> and commitlog state from the EBS to the ephemeral.
>
> As can be seen:
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
>
> the (presumably) spinning rust tops out at 2375 MB/sec (using multiple EBS
> volumes presumably) that would incur about a ten minute delay for node
> replacement for a 1TB node, but I imagine this would only be used on higher
> IOPS r/w nodes with smaller densities, so 100GB would be about a minute of
> delay only, already within the timeframes of an AWS node
> replacement/instance restart.
>
>
>

AWS ephemeral instances + backup

[Carl Mueller]

Does anyone have experience tooling written to support this strategy:

Use case: run cassandra on i3 instances on ephemerals but synchronize the
sstables and commitlog files to the cheapest EBS volume type (those have
bad IOPS but decent enough throughput)

On node replace, the startup script for the node, back-copies the sstables
and commitlog state from the EBS to the ephemeral.

As can be seen:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

the (presumably) spinning rust tops out at 2375 MB/sec (using multiple EBS
volumes presumably) that would incur about a ten minute delay for node
replacement for a 1TB node, but I imagine this would only be used on higher
IOPS r/w nodes with smaller densities, so 100GB would be about a minute of
delay only, already within the timeframes of an AWS node
replacement/instance restart.

Re: No progress in compactionstats

[Hossein Ghiyasi Mehr]

Hi,
Check nodetool tpstats output.
*---*
*VafaTech  : A Total Solution for Data Gathering &
Analysis*
*---*


On Mon, Dec 2, 2019 at 10:56 AM Dipan Shah  wrote:

> Hello,
>
> I am running a 5 node Cassandra cluster on V 3.7 and did not understand
> why the following thing was happening. I had altered the compaction
> strategy of a table from Size to Leveled and while running "nodetool
> compactionstats" found that the SSTables were stuck and not getting
> compacted. This was happening on majority of the nodes while the remaining
> were still showing some progress at compacting the SSTables.
>
>
>
> There were no errors in system.log and a service restart also did not
> help. I reverted the compaction strategy to Size to see what happens and
> that sent the value of pending tasks back to 0.
>
> I have done this earlier for a similar tables and it has worked perfectly
> fine for me. What could have gone wrong over here?
>
> Thanks,
>
> Dipan Shah
>

Repair Strategies

[Adarsh Kumar]

Hi,

We are in the process of designing a new solution around cassandra. As
repairs are very critical tasks for cassandra clusters want some pointers
on the following:

   1. What are the recommended strategies for repairs for production
   cluster (>100 nodes)
   2. Is there any special consideration(s) do multi DC repairs
   3. Steps to recover a DC through repairs
   4. How much incremental repairs are useful and community recommendations
   about it(i see TLP recommendation cassandra-reaper, that say to use inc
   repair >4.0)

Any link, blogs or SOP will be helpful. Thanks in advance.

Thanks and regards,
Adarsh Kumar