Re: Can't select count(*)

[Stefania Alborghetti]

Regarding select count(*), the timeout is probably client side. Try
changing the default connect timeout in cqlsh via --request-timeout. By
default it is 10 seconds. Refer to "cqlsh --help" for more details but
basically "cqlsh --request-timeout=30" should work.

Regarding COPY TO/FROM, these commands were recently enhanced and should be
available in 2.2.5 (not yet released). More details in this blog post:
http://www.datastax.com/dev/blog/new-features-in-cqlsh-copy. The problem
with COPY FROM prior to this enhancement is that it only contacts one
replica, so it is subject to coordinator timeouts regardless of the size of
the cluster, and it does not retry on timeouts. It just aborts.

Release 2.2.5 should be available soon but if you cannot wait till then you
can try downloading the source code for 2.2 HEAD and running cqlsh from
there. It should be compatible.

EC2 t2.small is probably too weak but I don't know enough about AWS
instances to comment further about this.

On Mon, Feb 1, 2016 at 3:56 PM, Ivan Zelensky  wrote:

> Hi all! I have a table with simple primary key (one field on primary key
> only), and ~1 million records. Table stored on single-node C* 2.2.4.
> Problem: when I'm trying to execute "SELECT count(*) FROM my_table;",
> operation is timed out.
> As I understand, 1 mln rows is not so big dataset to use MapRed to count
> it, so I thing it is smth wrong with configuration.
> Also I can't do "COPY TO/FROM" when dataset > 30 000 rows.
> Also maybe it is too weak hadrware (AWS EC2 t2.small), but even on
> t2.large I had timeout on COPY, just on 300 000 rows.
>
> My configuration is default config from deb package. Maybe somebody know
> what I should to tweak there?
>
> Thank you.
>



-- 


[image: datastax_logo.png] 

Stefania Alborghetti

Apache Cassandra Software Engineer

|+852 6114 9265| stefania.alborghe...@datastax.com

cassandra-stress tool - InvalidQueryException: Batch too large

[Ralf Steppacher]

I am using Cassandra 2.2.4 and I am struggling to get the cassandra-stress tool 
to work for my test scenario. I have followed the example on 
http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema
 

 to create a yaml file describing my test.

I am collecting events per user id (text, partition key). Events have a session 
type (text), event type (text), and creation time (timestamp) (clustering keys, 
in that order). Plus some more attributes required for rendering the events in 
a UI. For testing purposes I ended up with the following column spec and insert 
distribution:

columnspec:
  - name: created_at
cluster: uniform(10..1)
  - name: event_type
size: uniform(5..10)
population: uniform(1..30)
cluster: uniform(1..30)
  - name: session_type
size: fixed(5)
population: uniform(1..4)
cluster: uniform(1..4)
  - name: user_id
size: fixed(15)
population: uniform(1..100)
  - name: message
size: uniform(10..100)
population: uniform(1..100B)

insert:
  partitions: fixed(1)
  batchtype: UNLOGGED
  select: fixed(1)/120


Running stress tool for just the insert prints 

Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
total rows in the partitions)

and then immediately starts flooding me with 
"com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large”. 

Why I should be exceeding the "batch_size_fail_threshold_in_kb: 50” in the 
cassandra.yaml I do not understand. My understanding is that the stress tool 
should generate one row per batch. The size of a single row should not exceed 
8+10*3+5*3+15*3+100*3 = 398 bytes. Assuming a worst case of all text characters 
being 3 byte unicode characters. 

How come I end up with batches that exceed the 50kb threshold? Am I missing the 
point about the “select” attribute?


Thanks!
Ralf

Asynchronous, one-way data replication to stand-alone nodes?

[Pelle Jansson]

Hello.

I am not sure this have been asked before. I've tried to search and either I am 
terrible at searching, or the question haven't been asked.

We are trying to figure out how to solve the following problem;

We have systems in various regions in AWS, and we are trying to find a way to 
sync configuration data for various applications we have running there. This 
configuration data is located in an existing 5-node Cassandra cluster today. It 
is a single, small keyspace, just a few MB's worth of data (for now, anyway..).

It might look something like this (I hope formatting survives):

AB
\  /
  C* Centralized, existing, 5-node cluster
/ \
D   E

Where A, B, D, E = single nodes (or small clusters) in for example AWS.

So A, B, D and E would get data pushed to them from C*, but they should not 
sync any data back to C*.

One thought is to run a stand-alone, single Cassandra node (or a small cluster) 
in each region and sync the data over. One way to do this is to use 
sstableloader, but is there a more convenient, more "out-of-the-box"-way to get 
the data to these nodes?
Would it be possible to somehow use the Multi DC feature in this setup, and 
would it be possible to prevent the stand alone nodes to sync their data back 
to the "main" DC in that case? Or is there features of Cassandra that can do 
this that we've missed?

I hope I've made myself understandable ;)

Thanks in advance for any input on this.

Br,
Pelle Jansson

Re: EC2 storage options for C*

[Jack Krupansky]

I'm not a fan of guy - this appears to be the slideshare corresponding to
the video:
http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second

My apologies if my questions are actually answered on the video or slides,
I just did a quick scan of the slide text.

I'm curious where the EBS physical devices actually reside - are they in
the same rack, the same data center, same availability zone? I mean, people
try to minimize network latency between nodes, so how exactly is EBS able
to avoid network latency?

Did your test use Amazon EBS–Optimized Instances?

SSD or magnetic or does it make any difference?

What info is available on EBS performance at peak times, when multiple AWS
customers have spikes of demand?

Is RAID much of a factor or help at all using EBS?

How exactly is EBS provisioned in terms of its own HA - I mean, with a
properly configured Cassandra cluster RF provides HA, so what is the
equivalent for EBS? If I have RF=3, what assurance is there that those
three EBS volumes aren't all in the same physical rack?

For multi-data center operation, what configuration options assure that the
EBS volumes for each DC are truly physically separated?

In terms of syncing data for the commit log, if the OS call to sync an EBS
volume returns, is the commit log data absolutely 100% synced at the
hardware level on the EBS end, such that a power failure of the systems on
which the EBS volumes reside will still guarantee availability of the
fsynced data. As well, is return from fsync an absolute guarantee of
sstable durability when Cassandra is about to delete the commit log,
including when the two are on different volumes? In practice, we would like
some significant degree of pipelining of data, such as during the full
processing of flushing memtables, but for the fsync at the end a solid
guarantee is needed.


-- Jack Krupansky

On Mon, Feb 1, 2016 at 12:56 AM, Eric Plowe  wrote:

> Jeff,
>
> If EBS goes down, then EBS Gp2 will go down as well, no? I'm not
> discounting EBS, but prior outages are worrisome.
>
>
> On Sunday, January 31, 2016, Jeff Jirsa 
> wrote:
>
>> Free to choose what you'd like, but EBS outages were also addressed in
>> that video (second half, discussion by Dennis Opacki). 2016 EBS isn't the
>> same as 2011 EBS.
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Jan 31, 2016, at 8:27 PM, Eric Plowe  wrote:
>>
>> Thank you all for the suggestions. I'm torn between GP2 vs Ephemeral. GP2
>> after testing is a viable contender for our workload. The only worry I have
>> is EBS outages, which have happened.
>>
>> On Sunday, January 31, 2016, Jeff Jirsa 
>> wrote:
>>
>>> Also in that video - it's long but worth watching
>>>
>>> We tested up to 1M reads/second as well, blowing out page cache to
>>> ensure we weren't "just" reading from memory
>>>
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Jan 31, 2016, at 9:52 AM, Jack Krupansky 
>>> wrote:
>>>
>>> How about reads? Any differences between read-intensive and
>>> write-intensive workloads?
>>>
>>> -- Jack Krupansky
>>>
>>> On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa 
>>> wrote:
>>>
 Hi John,

 We run using 4T GP2 volumes, which guarantee 10k iops. Even at 1M
 writes per second on 60 nodes, we didn’t come close to hitting even 50%
 utilization (10k is more than enough for most workloads). PIOPS is not
 necessary.



 From: John Wong
 Reply-To: "user@cassandra.apache.org"
 Date: Saturday, January 30, 2016 at 3:07 PM
 To: "user@cassandra.apache.org"
 Subject: Re: EC2 storage options for C*

 For production I'd stick with ephemeral disks (aka instance storage) if
 you have running a lot of transaction.
 However, for regular small testing/qa cluster, or something you know
 you want to reload often, EBS is definitely good enough and we haven't had
 issues 99%. The 1% is kind of anomaly where we have flush blocked.

 But Jeff, kudo that you are able to use EBS. I didn't go through the
 video, do you actually use PIOPS or just standard GP2 in your production
 cluster?

 On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng 
 wrote:

> Yep, that motivated my question "Do you have any idea what kind of
> disk performance you need?". If you need the performance, its hard to beat
> ephemeral SSD in RAID 0 on EC2, and its a solid, battle tested
> configuration. If you don't, though, EBS GP2 will save a _lot_ of 
> headache.
>
> Personally, on small clusters like ours (12 nodes), we've found our
> choice of instance dictated much more by the balance of price, CPU, and
> memory. We're using GP2 SSD and we find that for our patterns the disk is
> rarely the bottleneck. YMMV, of course.
>
> On Fri, Jan 29, 2016 at 7:32 PM, Jeff Jirsa <
> jeff.ji...@crowdstrike.com> wrote:
>
>> If you have to ask that question, I strongly recommend m4 or c4
>> instances with GP2 EBS.  Wh

Re: EC2 storage options for C*

[Jack Krupansky]

Oops... that was supposed to be "not a fan of video"! I have no problem
with the guys in the video!

-- Jack Krupansky

On Mon, Feb 1, 2016 at 8:51 AM, Jack Krupansky 
wrote:

> I'm not a fan of guy - this appears to be the slideshare corresponding to
> the video:
>
> http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second
>
> My apologies if my questions are actually answered on the video or slides,
> I just did a quick scan of the slide text.
>
> I'm curious where the EBS physical devices actually reside - are they in
> the same rack, the same data center, same availability zone? I mean, people
> try to minimize network latency between nodes, so how exactly is EBS able
> to avoid network latency?
>
> Did your test use Amazon EBS–Optimized Instances?
>
> SSD or magnetic or does it make any difference?
>
> What info is available on EBS performance at peak times, when multiple AWS
> customers have spikes of demand?
>
> Is RAID much of a factor or help at all using EBS?
>
> How exactly is EBS provisioned in terms of its own HA - I mean, with a
> properly configured Cassandra cluster RF provides HA, so what is the
> equivalent for EBS? If I have RF=3, what assurance is there that those
> three EBS volumes aren't all in the same physical rack?
>
> For multi-data center operation, what configuration options assure that
> the EBS volumes for each DC are truly physically separated?
>
> In terms of syncing data for the commit log, if the OS call to sync an EBS
> volume returns, is the commit log data absolutely 100% synced at the
> hardware level on the EBS end, such that a power failure of the systems on
> which the EBS volumes reside will still guarantee availability of the
> fsynced data. As well, is return from fsync an absolute guarantee of
> sstable durability when Cassandra is about to delete the commit log,
> including when the two are on different volumes? In practice, we would like
> some significant degree of pipelining of data, such as during the full
> processing of flushing memtables, but for the fsync at the end a solid
> guarantee is needed.
>
>
> -- Jack Krupansky
>
> On Mon, Feb 1, 2016 at 12:56 AM, Eric Plowe  wrote:
>
>> Jeff,
>>
>> If EBS goes down, then EBS Gp2 will go down as well, no? I'm not
>> discounting EBS, but prior outages are worrisome.
>>
>>
>> On Sunday, January 31, 2016, Jeff Jirsa 
>> wrote:
>>
>>> Free to choose what you'd like, but EBS outages were also addressed in
>>> that video (second half, discussion by Dennis Opacki). 2016 EBS isn't the
>>> same as 2011 EBS.
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Jan 31, 2016, at 8:27 PM, Eric Plowe  wrote:
>>>
>>> Thank you all for the suggestions. I'm torn between GP2 vs Ephemeral.
>>> GP2 after testing is a viable contender for our workload. The only worry I
>>> have is EBS outages, which have happened.
>>>
>>> On Sunday, January 31, 2016, Jeff Jirsa 
>>> wrote:
>>>
 Also in that video - it's long but worth watching

 We tested up to 1M reads/second as well, blowing out page cache to
 ensure we weren't "just" reading from memory



 --
 Jeff Jirsa


 On Jan 31, 2016, at 9:52 AM, Jack Krupansky 
 wrote:

 How about reads? Any differences between read-intensive and
 write-intensive workloads?

 -- Jack Krupansky

 On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa >>> > wrote:

> Hi John,
>
> We run using 4T GP2 volumes, which guarantee 10k iops. Even at 1M
> writes per second on 60 nodes, we didn’t come close to hitting even 50%
> utilization (10k is more than enough for most workloads). PIOPS is not
> necessary.
>
>
>
> From: John Wong
> Reply-To: "user@cassandra.apache.org"
> Date: Saturday, January 30, 2016 at 3:07 PM
> To: "user@cassandra.apache.org"
> Subject: Re: EC2 storage options for C*
>
> For production I'd stick with ephemeral disks (aka instance storage)
> if you have running a lot of transaction.
> However, for regular small testing/qa cluster, or something you know
> you want to reload often, EBS is definitely good enough and we haven't had
> issues 99%. The 1% is kind of anomaly where we have flush blocked.
>
> But Jeff, kudo that you are able to use EBS. I didn't go through the
> video, do you actually use PIOPS or just standard GP2 in your production
> cluster?
>
> On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng 
> wrote:
>
>> Yep, that motivated my question "Do you have any idea what kind of
>> disk performance you need?". If you need the performance, its hard to 
>> beat
>> ephemeral SSD in RAID 0 on EC2, and its a solid, battle tested
>> configuration. If you don't, though, EBS GP2 will save a _lot_ of 
>> headache.
>>
>> Personally, on small clusters like ours (12 nodes), we've found our
>> choice of instance dictated much more by the balance of price

Re: cassandra-stress tool - InvalidQueryException: Batch too large

[Jake Luciani]

Yeah that looks like a bug.  Can you open a JIRA and attach the full .yaml?

Thanks!


On Mon, Feb 1, 2016 at 5:09 AM, Ralf Steppacher 
wrote:

> I am using Cassandra 2.2.4 and I am struggling to get the cassandra-stress
> tool to work for my test scenario. I have followed the example on
> http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema
>  to
> create a yaml file describing my test.
>
> I am collecting events per user id (text, partition key). Events have a
> session type (text), event type (text), and creation time (timestamp)
> (clustering keys, in that order). Plus some more attributes required for
> rendering the events in a UI. For testing purposes I ended up with the
> following column spec and insert distribution:
>
> columnspec:
>   - name: created_at
> cluster: uniform(10..1)
>   - name: event_type
> size: uniform(5..10)
> population: uniform(1..30)
> cluster: uniform(1..30)
>   - name: session_type
> size: fixed(5)
> population: uniform(1..4)
> cluster: uniform(1..4)
>   - name: user_id
> size: fixed(15)
> population: uniform(1..100)
>   - name: message
> size: uniform(10..100)
> population: uniform(1..100B)
>
> insert:
>   partitions: fixed(1)
>   batchtype: UNLOGGED
>   select: fixed(1)/120
>
>
> Running stress tool for just the insert prints
>
> Generating batches with [1..1] partitions and [0..1] rows (of
> [10..120] total rows in the partitions)
>
> and then immediately starts flooding me with
> "com.datastax.driver.core.exceptions.InvalidQueryException: Batch too
> large”.
>
> Why I should be exceeding the "batch_size_fail_threshold_in_kb: 50” in the
> cassandra.yaml I do not understand. My understanding is that the stress
> tool should generate one row per batch. The size of a single row should not
> exceed 8+10*3+5*3+15*3+100*3 = 398 bytes. Assuming a worst case of all text
> characters being 3 byte unicode characters.
>
> How come I end up with batches that exceed the 50kb threshold? Am I
> missing the point about the “select” attribute?
>
>
> Thanks!
> Ralf
>



-- 
http://twitter.com/tjake

Re: missing rows while importing data using sstable loader

[Romain Hardouin]

Did you run "nodetool flush" on the source node? If not, the missing rows could 
be in memtables.

Re: Cassandra's log is full of mesages reset by peers even without traffic

[Jean Carlo]

Hello Annuj,,

I checked my settings and this what I got.
root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep
net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_probes = 9
root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep
net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_keepalive_intvl = 75
root@node001[SPH][BENCH][PnS3]:~$ sysctl -A | grep net.ipv4 | grep
net.ipv4.tcp_keepalive_time
net.ipv4.tcp_keepalive_time = 300The tcp_keepalive_time is quite high in
comparation to that written on the doc
https://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootIdleFirewall.html
Do you think that is ok?
Best regards

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Fri, Jan 29, 2016 at 11:02 AM, Anuj Wadehra 
wrote:

> Hi Jean,
>
> Please make sure that your Firewall is not dropping TCP connections which
> are in use. Tcp keep alive on all nodes must be less than the firewall
> setting. Please refer to
>
> https://docs.datastax.com/en/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html
>  for
> details on TCP settings.
>
>
> Thanks
> Anuj
>
> Sent from Yahoo Mail on Android
> 
>
> On Fri, 29 Jan, 2016 at 3:21 pm, Jean Carlo
>  wrote:
> Hello guys,
>
> I have a cluster cassandra 2.1.12 with 6 nodes. All the logs of my nodes
> are having this messages marked as INFO
>
> INFO  [SharedPool-Worker-1] 2016-01-29 10:40:57,745 Message.java:532 -
> Unexpected exception during request; channel = [id: 0xff15eb8c, /
> 172.16.162.4:9042]
> java.io.IOException: Error while read(...): Connection reset by peer
> at io.netty.channel.epoll.Native.readAddress(Native Method)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
> This happens either the cluster is stressed or not. Btw it is not
> production. The ip marked there (172.16.162.4) belongs to a node of the
> cluster, this is not the only node that appears, acctually we are having
> all the node's ip having that reset by peer problem.
>
> Our cluster is having more reads than writes. like 50 reads per second.
>
> Any one got the same problem?
>
>
> Best regards
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
>

Re: Session timeout

[oleg yusim]

Jeff,

You mentioned that both Authentication and Authorization are pluggable. In
relation to that, is logging pluggable as well? I.e. if I'm not happy with
what logback has to provide and want to replace it with some alternative
logging module, can I do it?

Thanks,

Oleg

On Fri, Jan 29, 2016 at 11:42 AM, Jeff Jirsa 
wrote:

>
> > For instance, way AAA (authentication, authorization, audit) is done,
> doesn't allow for centralized account and access control management, which
> in reality translates into shared accounts and no hierarchy.
>
> Authentication and Authorization are both pluggable. Any organization can
> write their own, and tie it to any AAA system they currently have. If they
> were feeling generous, they could open source it for the community, and
> perhaps bring it upstream. There’s nothing fundamentally preventing your
> organization from writing an Authenticator (
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/IAuthenticator.java
>  )
> or Authorizor (
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/auth/IAuthorizer.java
>  )
> if they were so inclined.
>
> Audit is something that’s being actively discussed (
> https://issues.apache.org/jira/browse/CASSANDRA-8844 ).
>
> It’s an open source project with a very small number of commercial
> vendors. In general, that means there are 3 options:
>
>1. Wait for someone else to write it to fit their need, and hopefully
>they open source it.
>2. Write it yourself
>3. Pay a vendor (such as Datastax), and let them know in advance it’s
>a requirement to get it on their roadmap. This is really #2 with some
>polish to make it easier to get through your legal/AP systems.
>
> > So far it doesn't work quite well, and from what you are saying, it
> wouldn't, because of lack of knowledge and lack of motivation to get it.
> What would be your suggestion? Who is capable of answering my questions? Is
> there another community, I should turn to?
>
> The cassandra-user and cassandra-dev mailing lists are the primary sources
> of knowledge outside of support contracts. For paid support, companies like
> Datastax and The Last Pickle tend to be well respected options. Both of
> those companies will probably answer some of your questions for free if you
> post on these mailing lists. They’ll likely answer even more if you pay
> them.
>
>
>
> From: oleg yusim
> Reply-To: "user@cassandra.apache.org"
> Date: Friday, January 29, 2016 at 9:16 AM
> To: "user@cassandra.apache.org"
> Subject: Re: Session timeout
>
> Jon,
>
> I suspected something like that. I did a bit of learning on Cassandra
> before starting my assessment, and I understand that you are right, and it
> is generally not used like that.
>
> However (taking off my developer hat and putting on my security architect
> hat), from the security point of view the way Cassandra is used now is not
> very secure. For instance, way AAA (authentication, authorization, audit)
> is done, doesn't allow for centralized account and access control
> management, which in reality translates into shared accounts and no
> hierarchy. That in turn translates into situation when one person
> compromising credentials means complete disaster - administrative access to
> DB was just given up, with all the consequences. To top it all logging
> currently implemented in horrible manner too. It doesn't even allow to log
> username - basic requirement for any product, which would allow DBA or ISSO
> to figure out who did what on DB and recover in case of attack or crash. In
> general, logs the way they are today are targeted toward developer, making
> changes in DB, not toward the DBA, using it, and doesn't make much sense in
> my opinion.
>
> Now if you are interested in that subject, that document:
> http://iasecontent.disa.mil/stigs/zip/Jan2016/U_Database_V2R3_SRG.zip
> covers security concerns which should be taken in the account, when we are
> designing database. It also explains why each of them is important and what
> exactly would happen if it would be neglected.
>
> Jon, I would also appreciate suggestion. What I do right now is called
> "writing a STIG".That is when somebody takes concepts from SRG (the
> document I gave you link to above) and figures out how those are applied to
> that particular product. What is met (and what configuration on product
> leads to it, exactly), what is not met, but can be with little enhancement
> (and again - what those would be exactly), and what is not met and can't be
> met at current design. All that is combined into one document, called STIG
> and published by government (DISA) on
> http://iase.disa.mil/stigs/Pages/a-z.aspx page. Those STIGs mean a great
> deal from the security point of view because they:
>
>- Allow to save a lot of time on re-assessment of the product every
>single time
>- Allow to know what are the products limitations are from the
>security point of view before ha

Re: EC2 storage options for C*

[Jeff Jirsa]

> My apologies if my questions are actually answered on the video or slides, I 
> just did a quick scan of the slide text.

Virtually all of them are covered.

> I'm curious where the EBS physical devices actually reside - are they in the 
> same rack, the same data center, same availability zone? I mean, people try 
> to minimize network latency between nodes, so how exactly is EBS able to 
> avoid network latency?

Not published,and probably not a straight forward answer (probably have 
redundancy cross-az, if it matches some of their other published behaviors). 
The promise they give you is ‘iops’, with a certain block size. Some instance 
types are optimized for dedicated, ebs-only network interfaces. Like most 
things in cassandra / cloud, the only way to know for sure is to test it 
yourself and see if observed latency is acceptable (or trust our testing, if 
you assume we’re sufficiently smart and honest). 

> Did your test use Amazon EBS–Optimized Instances?

We tested dozens of instance type/size combinations (literally). The best 
performance was clearly with ebs-optimized instances that also have enhanced 
networking (c4, m4, etc) - slide 43

> SSD or magnetic or does it make any difference?

SSD, GP2 (slide 64)

> What info is available on EBS performance at peak times, when multiple AWS 
> customers have spikes of demand?

Not published, but experiments show that we can hit 10k iops all day every day 
with only trivial noisy neighbor problems, not enough to impact a real cluster 
(slide 58)

> Is RAID much of a factor or help at all using EBS?

You can use RAID to get higher IOPS than you’d normally get by default (GP2 
IOPS cap is 10k, which you get with a 3.333T volume – if you need more than 
10k, you can stripe volumes together up to the ebs network link max) (hinted at 
in slide 64)

> How exactly is EBS provisioned in terms of its own HA - I mean, with a 
> properly configured Cassandra cluster RF provides HA, so what is the 
> equivalent for EBS? If I have RF=3, what assurance is there that those three 
> EBS volumes aren't all in the same physical rack?

There is HA, I’m not sure that AWS publishes specifics. Occasionally specific 
volumes will have issues (hypervisor’s dedicated ethernet link to EBS network 
fails, for example). Occasionally instances will have issues. The 
volume-specific issues seem to be less common than the instance-store “instance 
retired” or “instance is running on degraded hardware” events. Stop/Start and 
you’ve recovered (possible with EBS, not possible with instance store). The 
assurances are in AWS’ SLA – if the SLA is insufficient (and it probably is 
insufficient), use more than one AZ and/or AWS region or cloud vendor.

> For multi-data center operation, what configuration options assure that the 
> EBS volumes for each DC are truly physically separated?

It used to be true that EBS control plane for a given region spanned AZs. 
That’s no longer true. AWS asserts that failure modes for each AZ are isolated 
(data may replicate between AZs, but a full outage in us-east-1a shouldn’t 
affect running ebs volumes in us-east-1b or us-east-1c). Slide 65

> In terms of syncing data for the commit log, if the OS call to sync an EBS 
> volume returns, is the commit log data absolutely 100% synced at the hardware 
> level on the EBS end, such that a power failure of the systems on which the 
> EBS volumes reside will still guarantee availability of the fsynced data. As 
> well, is return from fsync an absolute guarantee of sstable durability when 
> Cassandra is about to delete the commit log, including when the two are on 
> different volumes? In practice, we would like some significant degree of 
> pipelining of data, such as during the full processing of flushing memtables, 
> but for the fsync at the end a solid guarantee is needed.

Most of the answers in this block are “probably not 100%, you should be writing 
to more than one host/AZ/DC/vendor to protect your organization from failures”. 
AWS targets something like 0.1% annual failure rate per volume and 99.999% 
availability (slide 66). We believe they’re exceeding those goals (at least 
based with the petabytes of data we have on gp2 volumes).  



From:  Jack Krupansky
Reply-To:  "user@cassandra.apache.org"
Date:  Monday, February 1, 2016 at 5:51 AM
To:  "user@cassandra.apache.org"
Subject:  Re: EC2 storage options for C*

I'm not a fan of guy - this appears to be the slideshare corresponding to the 
video: 
http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second

My apologies if my questions are actually answered on the video or slides, I 
just did a quick scan of the slide text.

I'm curious where the EBS physical devices actually reside - are they in the 
same rack, the same data center, same availability zone? I mean, people try to 
minimize network latency between nodes, so how exactly is EBS able to avoid 
network latency? 

Did your test use Amazon EBS

Moving Away from Compact Storage

[Anuj Wadehra]

Hi
Whats the fastest and reliable way to migrate data from a Compact Storage table 
to Non-Compact storage table?
I was not able to find any command for dropping the compact storage 
directive..so I think migrating data is the only way...any suggestions?
ThanksAnuj

Re: Moving Away from Compact Storage

[DuyHai Doan]

Use Apache Spark to parallelize the data migration. Look at this piece of
code
https://github.com/doanduyhai/Cassandra-Spark-Demo/blob/master/src/main/scala/usecases/MigrateAlbumsData.scala#L58-L60

If your source and target tables have the SAME structure (except for the
COMPACT STORAGE clause), migration with Spark is a 2 lines of code

On Mon, Feb 1, 2016 at 8:14 PM, Anuj Wadehra  wrote:

> Hi
>
> Whats the fastest and reliable way to migrate data from a Compact Storage
> table to Non-Compact storage table?
>
> I was not able to find any command for dropping the compact storage
> directive..so I think migrating data is the only way...any suggestions?
>
> Thanks
> Anuj
>
>

Re: EC2 storage options for C*

[Jack Krupansky]

Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
question, it seems like magnetic (HDD) is no longer a recommended storage
option for databases on AWS. In particular, only the C2 Dense Storage
instances have local magnetic storage - all the other instance types are
SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
Access."

For the record, that AWS doc has Cassandra listed as a use case for i2
instance types.

Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
only for the "small to medium databases" use case.

Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is the
doc simply for any newly started instances?

See:
https://aws.amazon.com/ec2/instance-types/
http://aws.amazon.com/ebs/details/


-- Jack Krupansky

On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
wrote:

> > My apologies if my questions are actually answered on the video or
> slides, I just did a quick scan of the slide text.
>
> Virtually all of them are covered.
>
> > I'm curious where the EBS physical devices actually reside - are they in
> the same rack, the same data center, same availability zone? I mean, people
> try to minimize network latency between nodes, so how exactly is EBS able
> to avoid network latency?
>
> Not published,and probably not a straight forward answer (probably have
> redundancy cross-az, if it matches some of their other published
> behaviors). The promise they give you is ‘iops’, with a certain block size.
> Some instance types are optimized for dedicated, ebs-only network
> interfaces. Like most things in cassandra / cloud, the only way to know for
> sure is to test it yourself and see if observed latency is acceptable (or
> trust our testing, if you assume we’re sufficiently smart and honest).
>
> > Did your test use Amazon EBS–Optimized Instances?
>
> We tested dozens of instance type/size combinations (literally). The best
> performance was clearly with ebs-optimized instances that also have
> enhanced networking (c4, m4, etc) - slide 43
>
> > SSD or magnetic or does it make any difference?
>
> SSD, GP2 (slide 64)
>
> > What info is available on EBS performance at peak times, when multiple
> AWS customers have spikes of demand?
>
> Not published, but experiments show that we can hit 10k iops all day every
> day with only trivial noisy neighbor problems, not enough to impact a real
> cluster (slide 58)
>
> > Is RAID much of a factor or help at all using EBS?
>
> You can use RAID to get higher IOPS than you’d normally get by default
> (GP2 IOPS cap is 10k, which you get with a 3.333T volume – if you need more
> than 10k, you can stripe volumes together up to the ebs network link max)
> (hinted at in slide 64)
>
> > How exactly is EBS provisioned in terms of its own HA - I mean, with a
> properly configured Cassandra cluster RF provides HA, so what is the
> equivalent for EBS? If I have RF=3, what assurance is there that those
> three EBS volumes aren't all in the same physical rack?
>
> There is HA, I’m not sure that AWS publishes specifics. Occasionally
> specific volumes will have issues (hypervisor’s dedicated ethernet link to
> EBS network fails, for example). Occasionally instances will have issues.
> The volume-specific issues seem to be less common than the instance-store
> “instance retired” or “instance is running on degraded hardware” events.
> Stop/Start and you’ve recovered (possible with EBS, not possible with
> instance store). The assurances are in AWS’ SLA – if the SLA is
> insufficient (and it probably is insufficient), use more than one AZ and/or
> AWS region or cloud vendor.
>
> > For multi-data center operation, what configuration options assure that
> the EBS volumes for each DC are truly physically separated?
>
> It used to be true that EBS control plane for a given region spanned AZs.
> That’s no longer true. AWS asserts that failure modes for each AZ are
> isolated (data may replicate between AZs, but a full outage in us-east-1a
> shouldn’t affect running ebs volumes in us-east-1b or us-east-1c). Slide 65
>
> > In terms of syncing data for the commit log, if the OS call to sync an
> EBS volume returns, is the commit log data absolutely 100% synced at the
> hardware level on the EBS end, such that a power failure of the systems on
> which the EBS volumes reside will still guarantee availability of the
> fsynced data. As well, is return from fsync an absolute guarantee of
> sstable durability when Cassandra is about to delete the commit log,
> including when the two are on different volumes? In practice, we would like
> some significant degree of pipelining of data, such as during the full
> processing of flushing memtables, but for the fsync at the end a solid
> guarantee is needed.
>
> Most of the answers in this block are “probably not 100%, you should be
> writing to more than one host/AZ/DC/vendor to protect your organization
> from failures”. AWS targets something like 0.1% annual failure rate per
> vo

Re: EC2 storage options for C*

[Steve Robenalt]

Hi Jack,

At the bottom of the instance-types page, there is a link to the previous
generations, which includes the older series (m1, m2, etc), many of which
have HDD options.

There are also the d2 (Dense Storage) instances in the current generation
that include various combos of local HDDs.

The i2 series has good sized SSDs available, and has the advanced
networking option, which is also useful for Cassandra. The enhanced
networking is available with other instance types as well, as you'll see on
the feature list under each type.

Steve



On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky 
wrote:

> Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
> question, it seems like magnetic (HDD) is no longer a recommended storage
> option for databases on AWS. In particular, only the C2 Dense Storage
> instances have local magnetic storage - all the other instance types are
> SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
> Access."
>
> For the record, that AWS doc has Cassandra listed as a use case for i2
> instance types.
>
> Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
> only for the "small to medium databases" use case.
>
> Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is
> the doc simply for any newly started instances?
>
> See:
> https://aws.amazon.com/ec2/instance-types/
> http://aws.amazon.com/ebs/details/
>
>
> -- Jack Krupansky
>
> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
> wrote:
>
>> > My apologies if my questions are actually answered on the video or
>> slides, I just did a quick scan of the slide text.
>>
>> Virtually all of them are covered.
>>
>> > I'm curious where the EBS physical devices actually reside - are they
>> in the same rack, the same data center, same availability zone? I mean,
>> people try to minimize network latency between nodes, so how exactly is EBS
>> able to avoid network latency?
>>
>> Not published,and probably not a straight forward answer (probably have
>> redundancy cross-az, if it matches some of their other published
>> behaviors). The promise they give you is ‘iops’, with a certain block size.
>> Some instance types are optimized for dedicated, ebs-only network
>> interfaces. Like most things in cassandra / cloud, the only way to know for
>> sure is to test it yourself and see if observed latency is acceptable (or
>> trust our testing, if you assume we’re sufficiently smart and honest).
>>
>> > Did your test use Amazon EBS–Optimized Instances?
>>
>> We tested dozens of instance type/size combinations (literally). The best
>> performance was clearly with ebs-optimized instances that also have
>> enhanced networking (c4, m4, etc) - slide 43
>>
>> > SSD or magnetic or does it make any difference?
>>
>> SSD, GP2 (slide 64)
>>
>> > What info is available on EBS performance at peak times, when multiple
>> AWS customers have spikes of demand?
>>
>> Not published, but experiments show that we can hit 10k iops all day
>> every day with only trivial noisy neighbor problems, not enough to impact a
>> real cluster (slide 58)
>>
>> > Is RAID much of a factor or help at all using EBS?
>>
>> You can use RAID to get higher IOPS than you’d normally get by default
>> (GP2 IOPS cap is 10k, which you get with a 3.333T volume – if you need more
>> than 10k, you can stripe volumes together up to the ebs network link max)
>> (hinted at in slide 64)
>>
>> > How exactly is EBS provisioned in terms of its own HA - I mean, with a
>> properly configured Cassandra cluster RF provides HA, so what is the
>> equivalent for EBS? If I have RF=3, what assurance is there that those
>> three EBS volumes aren't all in the same physical rack?
>>
>> There is HA, I’m not sure that AWS publishes specifics. Occasionally
>> specific volumes will have issues (hypervisor’s dedicated ethernet link to
>> EBS network fails, for example). Occasionally instances will have issues.
>> The volume-specific issues seem to be less common than the instance-store
>> “instance retired” or “instance is running on degraded hardware” events.
>> Stop/Start and you’ve recovered (possible with EBS, not possible with
>> instance store). The assurances are in AWS’ SLA – if the SLA is
>> insufficient (and it probably is insufficient), use more than one AZ and/or
>> AWS region or cloud vendor.
>>
>> > For multi-data center operation, what configuration options assure that
>> the EBS volumes for each DC are truly physically separated?
>>
>> It used to be true that EBS control plane for a given region spanned AZs.
>> That’s no longer true. AWS asserts that failure modes for each AZ are
>> isolated (data may replicate between AZs, but a full outage in us-east-1a
>> shouldn’t affect running ebs volumes in us-east-1b or us-east-1c). Slide 65
>>
>> > In terms of syncing data for the commit log, if the OS call to sync an
>> EBS volume returns, is the commit log data absolutely 100% synced at the
>> hardware level on the EBS end, such that a powe

Re: Questions about the replicas selection and remote coordinator

[Jun Wu]

Hi Steve,

Thank you so much for your kind reply and now it makes more sense. But for 
the remote coordinator issue, it’s definitely a interesting topic. If you have 
any other conclusion  on this. I’d be pretty happy to learn from you. 

Thanks again!

Jun
> On Jan 29, 2016, at 13:09, Steve Robenalt  wrote:
> 
> Hi Jun,
> 
> The replicas are chosen according to factors that are generally more easily 
> selected internally, as is the case with coordinators. Even if the replicas 
> were selected in a completely round-robin fashion initially, they could end 
> up being re-distributed as a result of node failures, additions/removals 
> to/from the cluster, etc, particularly when vnodes are used. As such, the 
> diagrams and the nodes they refer to are hypothetical, but accurate in the 
> sense that they are non-contiguous, and that different sets of replicas are 
> distributed to various parts of the cluster.
> 
> As far as the remote coordinator is concerned, I'm not sure what motivated 
> the change from 1.2 to 2.1 and would be interested in understanding that 
> change myself. I do know that improved performance was a big part of the 2.1 
> release, but I'm not sure if the change in coordinators was part of that 
> effort or not.
> 
> Steve
> 
> 
> On Fri, Jan 29, 2016 at 10:13 AM, Jun Wu  > wrote:
> Hi Steve,
> 
>Thank you so much for your reply. 
> 
>Yes, you're right, I'm using the version of 2.1. So based on this, I think 
> I'm outdated. 
> 
> However, this comes to another interesting question: why we change this 
> part from version 1 to version 2. As we can see that in version 1, there's 
> connections from node 10 in DC 1 with node 10 in DC 2, then node 10 in DC 2 
> send 3 copies to 3 nodes in DC 2, which should be more time-saving than 
> version 2.1, which send data from node 10 in DC 1 to 3 nodes in DC 2 directly.
> 
>  Also, is there any information on how to choose the replicas. Like here 
> https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/architectureClientRequestsMultiDCWrites_c.html
>  
> 
> Why we choose node 1, 3, 6 as replicas and 4, 8, 11 as another 3 replicas?
> 
> Also, is node 11 working as remote coordinator here? Or is the concept of 
> remote coordinator really existed, as the figure shows, we even don't need 
> the remote coordinator. 
> 
> Thanks!
> 
> Jun
> 
> 
> 
> 
> Date: Fri, 29 Jan 2016 09:55:58 -0800
> Subject: Re: Questions about the replicas selection and remote coordinator
> From: sroben...@highwire.org 
> To: user@cassandra.apache.org 
> 
> 
> Hi Jun,
> 
> The 2 diagrams you are comparing come from versions of Cassandra that are 
> significantly different - 1.2 in the first case and 2.1 in the second case, 
> so it's not surprising that there are differences. since you haven't 
> qualified your question with the Cassandra version you are asking about, I 
> would assume that the 2.1 example is more representative of what you would be 
> likely to see. In any case, it's best to use a consistent version for your 
> documentation because Cassandra changes quite rapidly with many of the 
> releases.
> 
> As far as choosing the coordinator node, I don't think there's a way to force 
> it, nor would it be a good idea to do so. In order to make a reasonable 
> selection of coordinators, you would need a lot of internal knowledge about 
> load on the nodes in the cluster and you'd need to also handle certain 
> classes of failures and retries, so you would end up duplicating what is 
> already being done for you internally.
> 
> Steve
> 
> 
> On Fri, Jan 29, 2016 at 9:11 AM, Jun Wu  > wrote:
> Hi there,
> 
> I have some questions about the replicas selection. 
> 
> Let's say that we have 2 data centers: DC1 and DC2, the figure also be 
> got from link here: 
> https://docs.datastax.com/en/cassandra/1.2/cassandra/images/write_access_multidc_12.png
>  
> .
>  There're 10 nodes in each data center. We set the replication factor to be 3 
> and 3 in each data center, which means there'll be 3 and 3 replicas in each 
> data center.
> 
> (1) My first question is how to choose which 3 nodes to write data to, in 
> the link above, the 3 replicas are node 1, 2, 7. But, is there any mechanism 
> to select these 3?
> 
> (2) Another question is about the remote coordinator, the previous figure 
> shows that node 10 in DC1 will write data to node 10  in DC 2, then node 10 
> in DC2 will write 3 copies to 3 nodes in DC2.
> 
> But, another figure from datastax shows different method, the figure can 
> be found here, 
> https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/architectureClientRequestsMu

Re: EC2 storage options for C*

[Jack Krupansky]

Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2 Dense
Storage".

The remaining question is whether any of the "Previous Generation
Instances" should be publicly recommended going forward.

And whether non-SSD instances should be recommended going forward as well.
sure, technically, someone could use the legacy instances, but the question
is what we should be recommending as best practice going forward.

Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.

-- Jack Krupansky

On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt 
wrote:

> Hi Jack,
>
> At the bottom of the instance-types page, there is a link to the previous
> generations, which includes the older series (m1, m2, etc), many of which
> have HDD options.
>
> There are also the d2 (Dense Storage) instances in the current generation
> that include various combos of local HDDs.
>
> The i2 series has good sized SSDs available, and has the advanced
> networking option, which is also useful for Cassandra. The enhanced
> networking is available with other instance types as well, as you'll see on
> the feature list under each type.
>
> Steve
>
>
>
> On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky 
> wrote:
>
>> Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
>> question, it seems like magnetic (HDD) is no longer a recommended storage
>> option for databases on AWS. In particular, only the C2 Dense Storage
>> instances have local magnetic storage - all the other instance types are
>> SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
>> Access."
>>
>> For the record, that AWS doc has Cassandra listed as a use case for i2
>> instance types.
>>
>> Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
>> only for the "small to medium databases" use case.
>>
>> Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is
>> the doc simply for any newly started instances?
>>
>> See:
>> https://aws.amazon.com/ec2/instance-types/
>> http://aws.amazon.com/ebs/details/
>>
>>
>> -- Jack Krupansky
>>
>> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
>> wrote:
>>
>>> > My apologies if my questions are actually answered on the video or
>>> slides, I just did a quick scan of the slide text.
>>>
>>> Virtually all of them are covered.
>>>
>>> > I'm curious where the EBS physical devices actually reside - are they
>>> in the same rack, the same data center, same availability zone? I mean,
>>> people try to minimize network latency between nodes, so how exactly is EBS
>>> able to avoid network latency?
>>>
>>> Not published,and probably not a straight forward answer (probably have
>>> redundancy cross-az, if it matches some of their other published
>>> behaviors). The promise they give you is ‘iops’, with a certain block size.
>>> Some instance types are optimized for dedicated, ebs-only network
>>> interfaces. Like most things in cassandra / cloud, the only way to know for
>>> sure is to test it yourself and see if observed latency is acceptable (or
>>> trust our testing, if you assume we’re sufficiently smart and honest).
>>>
>>> > Did your test use Amazon EBS–Optimized Instances?
>>>
>>> We tested dozens of instance type/size combinations (literally). The
>>> best performance was clearly with ebs-optimized instances that also have
>>> enhanced networking (c4, m4, etc) - slide 43
>>>
>>> > SSD or magnetic or does it make any difference?
>>>
>>> SSD, GP2 (slide 64)
>>>
>>> > What info is available on EBS performance at peak times, when multiple
>>> AWS customers have spikes of demand?
>>>
>>> Not published, but experiments show that we can hit 10k iops all day
>>> every day with only trivial noisy neighbor problems, not enough to impact a
>>> real cluster (slide 58)
>>>
>>> > Is RAID much of a factor or help at all using EBS?
>>>
>>> You can use RAID to get higher IOPS than you’d normally get by default
>>> (GP2 IOPS cap is 10k, which you get with a 3.333T volume – if you need more
>>> than 10k, you can stripe volumes together up to the ebs network link max)
>>> (hinted at in slide 64)
>>>
>>> > How exactly is EBS provisioned in terms of its own HA - I mean, with a
>>> properly configured Cassandra cluster RF provides HA, so what is the
>>> equivalent for EBS? If I have RF=3, what assurance is there that those
>>> three EBS volumes aren't all in the same physical rack?
>>>
>>> There is HA, I’m not sure that AWS publishes specifics. Occasionally
>>> specific volumes will have issues (hypervisor’s dedicated ethernet link to
>>> EBS network fails, for example). Occasionally instances will have issues.
>>> The volume-specific issues seem to be less common than the instance-store
>>> “instance retired” or “instance is running on degraded hardware” events.
>>> Stop/Start and you’ve recovered (possible with EBS, not possible with
>>> instance store). The assurances are in AWS’ SLA – if the SLA is
>>> insufficient (and it probably is insufficient), use more than one AZ a

Re: EC2 storage options for C*

[Steve Robenalt]

Yeah, I would only start a previous-generation instance for non-production
purposes at this point and I suspect that many of them will be retired
sooner or later.

Given a choice, I'd use i2 instances for most purposes, and use the d2
instances where storage volume and access patterns demanded it.

I have seen reports of other instances running SSD-backed EBS. Not sure if
the network lag is a significant issue in that case or not, but it would
open up other options for Cassandra clusters.

On Mon, Feb 1, 2016 at 1:55 PM, Jack Krupansky 
wrote:

> Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2
> Dense Storage".
>
> The remaining question is whether any of the "Previous Generation
> Instances" should be publicly recommended going forward.
>
> And whether non-SSD instances should be recommended going forward as well.
> sure, technically, someone could use the legacy instances, but the question
> is what we should be recommending as best practice going forward.
>
> Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.
>
> -- Jack Krupansky
>
> On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt 
> wrote:
>
>> Hi Jack,
>>
>> At the bottom of the instance-types page, there is a link to the previous
>> generations, which includes the older series (m1, m2, etc), many of which
>> have HDD options.
>>
>> There are also the d2 (Dense Storage) instances in the current generation
>> that include various combos of local HDDs.
>>
>> The i2 series has good sized SSDs available, and has the advanced
>> networking option, which is also useful for Cassandra. The enhanced
>> networking is available with other instance types as well, as you'll see on
>> the feature list under each type.
>>
>> Steve
>>
>>
>>
>> On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky 
>> wrote:
>>
>>> Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
>>> question, it seems like magnetic (HDD) is no longer a recommended storage
>>> option for databases on AWS. In particular, only the C2 Dense Storage
>>> instances have local magnetic storage - all the other instance types are
>>> SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
>>> Access."
>>>
>>> For the record, that AWS doc has Cassandra listed as a use case for i2
>>> instance types.
>>>
>>> Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
>>> only for the "small to medium databases" use case.
>>>
>>> Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is
>>> the doc simply for any newly started instances?
>>>
>>> See:
>>> https://aws.amazon.com/ec2/instance-types/
>>> http://aws.amazon.com/ebs/details/
>>>
>>>
>>> -- Jack Krupansky
>>>
>>> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
>>> wrote:
>>>
 > My apologies if my questions are actually answered on the video or
 slides, I just did a quick scan of the slide text.

 Virtually all of them are covered.

 > I'm curious where the EBS physical devices actually reside - are they
 in the same rack, the same data center, same availability zone? I mean,
 people try to minimize network latency between nodes, so how exactly is EBS
 able to avoid network latency?

 Not published,and probably not a straight forward answer (probably have
 redundancy cross-az, if it matches some of their other published
 behaviors). The promise they give you is ‘iops’, with a certain block size.
 Some instance types are optimized for dedicated, ebs-only network
 interfaces. Like most things in cassandra / cloud, the only way to know for
 sure is to test it yourself and see if observed latency is acceptable (or
 trust our testing, if you assume we’re sufficiently smart and honest).

 > Did your test use Amazon EBS–Optimized Instances?

 We tested dozens of instance type/size combinations (literally). The
 best performance was clearly with ebs-optimized instances that also have
 enhanced networking (c4, m4, etc) - slide 43

 > SSD or magnetic or does it make any difference?

 SSD, GP2 (slide 64)

 > What info is available on EBS performance at peak times, when
 multiple AWS customers have spikes of demand?

 Not published, but experiments show that we can hit 10k iops all day
 every day with only trivial noisy neighbor problems, not enough to impact a
 real cluster (slide 58)

 > Is RAID much of a factor or help at all using EBS?

 You can use RAID to get higher IOPS than you’d normally get by default
 (GP2 IOPS cap is 10k, which you get with a 3.333T volume – if you need more
 than 10k, you can stripe volumes together up to the ebs network link max)
 (hinted at in slide 64)

 > How exactly is EBS provisioned in terms of its own HA - I mean, with
 a properly configured Cassandra cluster RF provides HA, so what is the
 equivalent for EBS? If I have RF=3, what assurance is there that 

Re: EC2 storage options for C*

[Jeff Jirsa]

A lot of people use the old gen instances (m1 in particular) because they came 
with a ton of effectively free ephemeral storage (up to 1.6TB). Whether or not 
they’re viable is a decision for each user to make. They’re very, very commonly 
used for C*, though. At a time when EBS was not sufficiently robust or 
reliable, a cluster of m1 instances was the de facto standard. 

The canonical “best practice” in 2015 was i2. We believe we’ve made a 
compelling argument to use m4 or c4 instead of i2. There exists a company we 
know currently testing d2 at scale, though I’m not sure they have much in terms 
of concrete results at this time. 

- Jeff

From:  Jack Krupansky
Reply-To:  "user@cassandra.apache.org"
Date:  Monday, February 1, 2016 at 1:55 PM
To:  "user@cassandra.apache.org"
Subject:  Re: EC2 storage options for C*

Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2 Dense 
Storage". 

The remaining question is whether any of the "Previous Generation Instances" 
should be publicly recommended going forward.

And whether non-SSD instances should be recommended going forward as well. 
sure, technically, someone could use the legacy instances, but the question is 
what we should be recommending as best practice going forward.

Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.

-- Jack Krupansky

On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt  wrote:
Hi Jack, 

At the bottom of the instance-types page, there is a link to the previous 
generations, which includes the older series (m1, m2, etc), many of which have 
HDD options. 

There are also the d2 (Dense Storage) instances in the current generation that 
include various combos of local HDDs.

The i2 series has good sized SSDs available, and has the advanced networking 
option, which is also useful for Cassandra. The enhanced networking is 
available with other instance types as well, as you'll see on the feature list 
under each type. 

Steve



On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky  wrote:
Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic question, 
it seems like magnetic (HDD) is no longer a recommended storage option for 
databases on AWS. In particular, only the C2 Dense Storage instances have local 
magnetic storage - all the other instance types are SSD or EBS-only - and EBS 
Magnetic is only recommended for "Infrequent Data Access." 

For the record, that AWS doc has Cassandra listed as a use case for i2 instance 
types.

Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2 only 
for the "small to medium databases" use case.

Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is the doc 
simply for any newly started instances?

See:
https://aws.amazon.com/ec2/instance-types/
http://aws.amazon.com/ebs/details/


-- Jack Krupansky

On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa  wrote:
> My apologies if my questions are actually answered on the video or slides, I 
> just did a quick scan of the slide text.

Virtually all of them are covered.

> I'm curious where the EBS physical devices actually reside - are they in the 
> same rack, the same data center, same availability zone? I mean, people try 
> to minimize network latency between nodes, so how exactly is EBS able to 
> avoid network latency?

Not published,and probably not a straight forward answer (probably have 
redundancy cross-az, if it matches some of their other published behaviors). 
The promise they give you is ‘iops’, with a certain block size. Some instance 
types are optimized for dedicated, ebs-only network interfaces. Like most 
things in cassandra / cloud, the only way to know for sure is to test it 
yourself and see if observed latency is acceptable (or trust our testing, if 
you assume we’re sufficiently smart and honest). 

> Did your test use Amazon EBS–Optimized Instances?

We tested dozens of instance type/size combinations (literally). The best 
performance was clearly with ebs-optimized instances that also have enhanced 
networking (c4, m4, etc) - slide 43

> SSD or magnetic or does it make any difference?

SSD, GP2 (slide 64)

> What info is available on EBS performance at peak times, when multiple AWS 
> customers have spikes of demand?

Not published, but experiments show that we can hit 10k iops all day every day 
with only trivial noisy neighbor problems, not enough to impact a real cluster 
(slide 58)

> Is RAID much of a factor or help at all using EBS?

You can use RAID to get higher IOPS than you’d normally get by default (GP2 
IOPS cap is 10k, which you get with a 3.333T volume – if you need more than 
10k, you can stripe volumes together up to the ebs network link max) (hinted at 
in slide 64)

> How exactly is EBS provisioned in terms of its own HA - I mean, with a 
> properly configured Cassandra cluster RF provides HA, so what is the 
> equivalent for EBS? If I have RF=3, what assurance is there that those three 
> EBS volumes aren't all in the same physical

Re: EC2 storage options for C*

[Steve Robenalt]

Hi Jeff,

I'm going to go back and review your presentation. I missed it at Cassandra
Summit and didn't make it to re:Invent last year. The opinion I voiced was
from my own direct experience. Didn't mean to imply that there weren't
other good options available.

Thanks,
Steve

On Mon, Feb 1, 2016 at 2:12 PM, Jeff Jirsa 
wrote:

> A lot of people use the old gen instances (m1 in particular) because they
> came with a ton of effectively free ephemeral storage (up to 1.6TB).
> Whether or not they’re viable is a decision for each user to make. They’re
> very, very commonly used for C*, though. At a time when EBS was not
> sufficiently robust or reliable, a cluster of m1 instances was the de facto
> standard.
>
> The canonical “best practice” in 2015 was i2. We believe we’ve made a
> compelling argument to use m4 or c4 instead of i2. There exists a company
> we know currently testing d2 at scale, though I’m not sure they have much
> in terms of concrete results at this time.
>
> - Jeff
>
> From: Jack Krupansky
> Reply-To: "user@cassandra.apache.org"
> Date: Monday, February 1, 2016 at 1:55 PM
>
> To: "user@cassandra.apache.org"
> Subject: Re: EC2 storage options for C*
>
> Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2
> Dense Storage".
>
> The remaining question is whether any of the "Previous Generation
> Instances" should be publicly recommended going forward.
>
> And whether non-SSD instances should be recommended going forward as well.
> sure, technically, someone could use the legacy instances, but the question
> is what we should be recommending as best practice going forward.
>
> Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.
>
> -- Jack Krupansky
>
> On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt 
> wrote:
>
>> Hi Jack,
>>
>> At the bottom of the instance-types page, there is a link to the previous
>> generations, which includes the older series (m1, m2, etc), many of which
>> have HDD options.
>>
>> There are also the d2 (Dense Storage) instances in the current generation
>> that include various combos of local HDDs.
>>
>> The i2 series has good sized SSDs available, and has the advanced
>> networking option, which is also useful for Cassandra. The enhanced
>> networking is available with other instance types as well, as you'll see on
>> the feature list under each type.
>>
>> Steve
>>
>>
>>
>> On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky 
>> wrote:
>>
>>> Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
>>> question, it seems like magnetic (HDD) is no longer a recommended storage
>>> option for databases on AWS. In particular, only the C2 Dense Storage
>>> instances have local magnetic storage - all the other instance types are
>>> SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
>>> Access."
>>>
>>> For the record, that AWS doc has Cassandra listed as a use case for i2
>>> instance types.
>>>
>>> Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
>>> only for the "small to medium databases" use case.
>>>
>>> Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is
>>> the doc simply for any newly started instances?
>>>
>>> See:
>>> https://aws.amazon.com/ec2/instance-types/
>>> http://aws.amazon.com/ebs/details/
>>>
>>>
>>> -- Jack Krupansky
>>>
>>> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
>>> wrote:
>>>
 > My apologies if my questions are actually answered on the video or
 slides, I just did a quick scan of the slide text.

 Virtually all of them are covered.

 > I'm curious where the EBS physical devices actually reside - are they
 in the same rack, the same data center, same availability zone? I mean,
 people try to minimize network latency between nodes, so how exactly is EBS
 able to avoid network latency?

 Not published,and probably not a straight forward answer (probably have
 redundancy cross-az, if it matches some of their other published
 behaviors). The promise they give you is ‘iops’, with a certain block size.
 Some instance types are optimized for dedicated, ebs-only network
 interfaces. Like most things in cassandra / cloud, the only way to know for
 sure is to test it yourself and see if observed latency is acceptable (or
 trust our testing, if you assume we’re sufficiently smart and honest).

 > Did your test use Amazon EBS–Optimized Instances?

 We tested dozens of instance type/size combinations (literally). The
 best performance was clearly with ebs-optimized instances that also have
 enhanced networking (c4, m4, etc) - slide 43

 > SSD or magnetic or does it make any difference?

 SSD, GP2 (slide 64)

 > What info is available on EBS performance at peak times, when
 multiple AWS customers have spikes of demand?

 Not published, but experiments show that we can hit 10k iops all day
 every day with only trivial noi

Re: EC2 storage options for C*

[Jack Krupansky]

Ah, yes, the good old days of m1.large.

-- Jack Krupansky

On Mon, Feb 1, 2016 at 5:12 PM, Jeff Jirsa 
wrote:

> A lot of people use the old gen instances (m1 in particular) because they
> came with a ton of effectively free ephemeral storage (up to 1.6TB).
> Whether or not they’re viable is a decision for each user to make. They’re
> very, very commonly used for C*, though. At a time when EBS was not
> sufficiently robust or reliable, a cluster of m1 instances was the de facto
> standard.
>
> The canonical “best practice” in 2015 was i2. We believe we’ve made a
> compelling argument to use m4 or c4 instead of i2. There exists a company
> we know currently testing d2 at scale, though I’m not sure they have much
> in terms of concrete results at this time.
>
> - Jeff
>
> From: Jack Krupansky
> Reply-To: "user@cassandra.apache.org"
> Date: Monday, February 1, 2016 at 1:55 PM
>
> To: "user@cassandra.apache.org"
> Subject: Re: EC2 storage options for C*
>
> Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2
> Dense Storage".
>
> The remaining question is whether any of the "Previous Generation
> Instances" should be publicly recommended going forward.
>
> And whether non-SSD instances should be recommended going forward as well.
> sure, technically, someone could use the legacy instances, but the question
> is what we should be recommending as best practice going forward.
>
> Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.
>
> -- Jack Krupansky
>
> On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt 
> wrote:
>
>> Hi Jack,
>>
>> At the bottom of the instance-types page, there is a link to the previous
>> generations, which includes the older series (m1, m2, etc), many of which
>> have HDD options.
>>
>> There are also the d2 (Dense Storage) instances in the current generation
>> that include various combos of local HDDs.
>>
>> The i2 series has good sized SSDs available, and has the advanced
>> networking option, which is also useful for Cassandra. The enhanced
>> networking is available with other instance types as well, as you'll see on
>> the feature list under each type.
>>
>> Steve
>>
>>
>>
>> On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky 
>> wrote:
>>
>>> Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic
>>> question, it seems like magnetic (HDD) is no longer a recommended storage
>>> option for databases on AWS. In particular, only the C2 Dense Storage
>>> instances have local magnetic storage - all the other instance types are
>>> SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data
>>> Access."
>>>
>>> For the record, that AWS doc has Cassandra listed as a use case for i2
>>> instance types.
>>>
>>> Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2
>>> only for the "small to medium databases" use case.
>>>
>>> Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is
>>> the doc simply for any newly started instances?
>>>
>>> See:
>>> https://aws.amazon.com/ec2/instance-types/
>>> http://aws.amazon.com/ebs/details/
>>>
>>>
>>> -- Jack Krupansky
>>>
>>> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa 
>>> wrote:
>>>
 > My apologies if my questions are actually answered on the video or
 slides, I just did a quick scan of the slide text.

 Virtually all of them are covered.

 > I'm curious where the EBS physical devices actually reside - are they
 in the same rack, the same data center, same availability zone? I mean,
 people try to minimize network latency between nodes, so how exactly is EBS
 able to avoid network latency?

 Not published,and probably not a straight forward answer (probably have
 redundancy cross-az, if it matches some of their other published
 behaviors). The promise they give you is ‘iops’, with a certain block size.
 Some instance types are optimized for dedicated, ebs-only network
 interfaces. Like most things in cassandra / cloud, the only way to know for
 sure is to test it yourself and see if observed latency is acceptable (or
 trust our testing, if you assume we’re sufficiently smart and honest).

 > Did your test use Amazon EBS–Optimized Instances?

 We tested dozens of instance type/size combinations (literally). The
 best performance was clearly with ebs-optimized instances that also have
 enhanced networking (c4, m4, etc) - slide 43

 > SSD or magnetic or does it make any difference?

 SSD, GP2 (slide 64)

 > What info is available on EBS performance at peak times, when
 multiple AWS customers have spikes of demand?

 Not published, but experiments show that we can hit 10k iops all day
 every day with only trivial noisy neighbor problems, not enough to impact a
 real cluster (slide 58)

 > Is RAID much of a factor or help at all using EBS?

 You can use RAID to get higher IOPS than you’d normally get by default

Extensions

[oleg yusim]

Greetings,

Is it a way to find out (list or otherwise) if any extensions were
installed with Cassandra base package?

Thanks,

Oleg

Re: missing rows while importing data using sstable loader

[Arindam Choudhury]

What is the best practise to create sstables?

On 1 February 2016 at 15:21, Romain Hardouin  wrote:

> Did you run "nodetool flush" on the source node? If not, the missing rows
> could be in memtables.
>