[ https://issues.apache.org/jira/browse/CASSANDRA-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491820#comment-13491820 ]
Sylvain Lebresne commented on CASSANDRA-4915: --------------------------------------------- bq. and relying on the limit being X vs 10X or 0.1X is silly Why I agree on the silliness, I don't fully share your optimism that people won't start relying on it. I also would prefer being able to clearly specify that "without limit we return as much result as there is with the technical limitation that it's Integer.MAX_VALUE", rather than having to settle for "without limit we return results with a limit that depends on the weather and the exact value of which you shouldn't rely on". I also think that having an arbitrary default limit is a very bad OOM protection (I think it's still fairly easy to OOM even with the 10,000 limit unless you are mindful of your query). But I'd rather discuss that in CASSANDRA-4918 for the sake of not mixing unrelated issues. Because I do think there is an issue here that has nothing to do whatsoever with the limit and preventing OOMing. That issue is that we allow some queries that do not scale with the number of records in the database. And to be clear, 'not scale with the number of records in the database' means that even for a *constant* query output it doesn't scale. Those queries are: # the one in the description of this ticket # as Jonathan said (and I don't disagree with it's statement), secondary index queries with additional restrictions. Now I agree that we can't completely protect people against those short of refusing the queries. But I do think we have some discrepancies in what we support and don't support: we refuse 'SELECT * FROM t WHERE partition_key = .. AND clustering_key_part2 = ...' based on the argument than because clustering_key_part1 is not provided, we would have to do a full scan of the internal row and the inefficiency of that would be too surprising for the user. But we do allow the query in the description of this ticket even though honestly it's the same kind of query (I.e, it's a query where we don't have *any* index to really start with). And I don't like discrepancies. Or in other words, we've claimed that an advantage of Cassandra is that that query performance is predictable, but queries that for the same output (even a very small one) have an execution time that is proportional to the number of record in the database is imho the exact definition of query performance being non predictable (or at least non-scalable). So I think it would be of interest to clarify what it is exactly that we guarantee in term of query performance being predictable. And for that I see a number of options: # We leave thing as they are, but then the rule of when a query will have a predicable performance (which for me means that the performance will be almost only dependant on the query output) are fairly opaque and not very coherent. And in particular in that case it feels random to refuse queries that would require a full internal row scan when we happily do the ones that require an entire ring scan. # We get strict about allowing only queries that we can guarantee have predictable performance (with the definition above that I think is reasonable). That does mean refusing the query in the description, but also indeed queries on 2ndary indexes that have more than one restriction, which probably make that solution too restrictive to be desirable. # We try to hit some middle ground, where while we allow some guarantee we can't guarantee the predictability, we at least make it so that the rule for when the predictability is guaranteed easy to understand/follow. My proposition for "ALLOW FULL SCAN" above was a tentative of that. If we allow that, and unless I forget something which is possible, I think we can say that: a query will have predictable performance unless it either use 2ndary index or it uses 'allow full scan'. And for 2ndary index we can refine that a bit and say 'it still will have guaranteed predictable performance if you only use one restriction in the query'. But at least, we'd have clear guarantee without 2ndary index, and I do thing that 1) it's very useful and 2) it's not crazy to say that 2ndary index involves more complex processing and offer thus less guarantee in term of predictability. In favor of my third point, I want to mention that this is exactly the guarantee that thrift provides today, because today a non-2ndary query in thrift always give you predictable performance in the sense that the query performance will be proportional to the query ouptut (that you can control with the limit), because a get_range_slice in thrift (without IndexExpression) with a count of 1 will only ever scan one row (and if that one row doesn't have anything for the filter, the result will be an empty row), but that is *not* how CQL3 works today. > CQL should force limit when query samples data. > ----------------------------------------------- > > Key: CASSANDRA-4915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4915 > Project: Cassandra > Issue Type: Improvement > Affects Versions: 1.2.0 beta 1 > Reporter: Edward Capriolo > Priority: Minor > > When issuing a query like: > {noformat} > CREATE TABLE videos ( > videoid uuid, > videoname varchar, > username varchar, > description varchar, > tags varchar, > upload_date timestamp, > PRIMARY KEY (videoid,videoname) > ); > SELECT * FROM videos WHERE videoname = 'My funny cat'; > {noformat} > Cassandra samples some data using get_range_slice and then applies the query. > This is very confusing to me, because as an end user am not sure if the query > is fast because Cassandra is performing an optimized query (over an index, or > using a slicePredicate) or if cassandra is simple sampling some random rows > and returning me some results. > My suggestions: > 1) force people to supply a LIMIT clause on any query that is going to > page over get_range_slice > 2) having some type of explain support so I can establish if this > query will work in the > I will champion suggestion 1) because CQL has put itself in a rather unique > un-sql like position by applying an automatic limit clause without the user > asking for them. I also do not believe the CQL language should let the user > issue queries that will not work as intended with "larger-then-auto-limit" > size data sets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira