Doesn't paging help with this ? Also if we select a range via the cluster key we're never really selecting the full partition. Or is that wrong ?
On Fri, Oct 28, 2016, at 05:00 PM, Edward Capriolo wrote: > Big partitions are an anti-pattern here is why: > > First Cassandra is not an analytic datastore. Sure it has some UDFs > and aggregate UDFs, but the true purpose of the data store is to > satisfy point reads. Operations have strict timeouts: > > # How long the coordinator should wait for read operations to complete > read_request_timeout_in_ms: 5000 > > # How long the coordinator should wait for seq or index scans to > # complete > range_request_timeout_in_ms: 10000 > > This means you need to be able to satisfy the operation in 5 seconds. > Which is not only the "think time" for 1 server, but if you are doing > a quorum the operation has to complete and compare on 2 or more > servers. Beyond these cutoffs are thread pools which fill up and start > dropping requests once full. > > Something has to give, either functionality or physics. Particularly > the physics of aggregating an ever-growing data set across N replicas > in less than 5 seconds. How many 2ms point reads will be blocked by > 50 ms queries etc. > > I do not see the technical limitations of big partitions on disk is > the only hurdle to climb here. > > > On Fri, Oct 28, 2016 at 10:39 AM, Alexander Dejanovski > <a...@thelastpickle.com> wrote: >> Hi Eric, >> >> that would be >> https://issues.apache.org/jira/browse/CASSANDRA-9754 by >> Michael Kjellman and >> https://issues.apache.org/jira/browse/CASSANDRA-11206 by >> Robert Stupp. >> If you haven't seen it yet, Robert's summit talk on big partitions is >> totally worth it : >> Video : https://www.youtube.com/watch?v=N3mGxgnUiRY >> Slides : >> http://www.slideshare.net/DataStax/myths-of-big-partitions-robert-stupp-datastax-cassandra-summit-2016 >> >> Cheers, >> >> >> On Fri, Oct 28, 2016 at 4:09 PM Eric Evans >> <john.eric.ev...@gmail.com> wrote: >>> On Thu, Oct 27, 2016 at 4:13 PM, Alexander Dejanovski >>> <a...@thelastpickle.com> wrote: >>> > A few patches are pushing the limits of partition sizes so we may >>> > soon be >>> > more comfortable with big partitions. >>> >>> You don't happen to have Jira links to these handy, do you? >>> >>> >>> -- >>> Eric Evans john.eric.ev...@gmail.com >> >> -- >> ----------------- >> Alexander Dejanovski >> France >> @alexanderdeja >> >> Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com[1] >> Links: 1. http://www.thelastpickle.com/
