Re: implementing a 'sorted set' on top of cassandra

2017-01-17 Thread Benjamin Roth
rmance hit there? My thought was to make a >> tradeoff by using more cassandra disk space (ie pre calculating all sets), >> in exchange for faster reads when requests actually come in that need this >> data. >> >> From: Benjamin Roth >> Reply-To: "user@

Re: implementing a 'sorted set' on top of cassandra

2017-01-17 Thread Jonathan Haddad
in that need this > data. > > From: Benjamin Roth > Reply-To: "user@cassandra.apache.org" > Date: Saturday, January 14, 2017 at 1:25 PM > To: "user@cassandra.apache.org" > Subject: Re: implementing a 'sorted set' on top of cassandra > >

Re: implementing a 'sorted set' on top of cassandra

2017-01-17 Thread Edward Capriolo
gt; data. > > From: Benjamin Roth > Reply-To: "user@cassandra.apache.org" > Date: Saturday, January 14, 2017 at 1:25 PM > To: "user@cassandra.apache.org" > Subject: Re: implementing a 'sorted set' on top of cassandra > > Mike mentioned "inc

Re: implementing a 'sorted set' on top of cassandra

2017-01-17 Thread Mike Torra
g>" mailto:user@cassandra.apache.org>> Date: Saturday, January 14, 2017 at 1:25 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: implementing a 'sorted set' on top of cassandra Mik

Re: implementing a 'sorted set' on top of cassandra

2017-01-14 Thread Benjamin Roth
Mike mentioned "increment" in his initial post. That let me think of a case with increments and fetching a top list by a counter like https://redis.io/commands/zincrby https://redis.io/commands/zrangebyscore 1. Cassandra is absolutely not made to sort by a counter (or a non-counter numeric increme

Re: implementing a 'sorted set' on top of cassandra

2017-01-14 Thread Jonathan Haddad
Sorted sets don't have a requirement of incrementing / decrementing. They're commonly used for thing like leaderboards where the values are arbitrary. In Redis they are implemented with 2 data structures for efficient lookups of either key or value. No getting around that as far as I know. In Cas

Re: implementing a 'sorted set' on top of cassandra

2017-01-14 Thread DuyHai Doan
Sorting on an "incremented" numeric value has always been a nightmare to be done properly in C* Either use Counter type but then no sorting is possible since counter cannot be used as type for clustering column (which allows sort) Or use simple numeric type on clustering column but then to increm

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Benjamin Roth
If your proposed solution is crazy depends on your needs :) It sounds like you can live with not-realtime data. So it is ok to cache it. Why preproduce the results if you only need 5% of them? Why not use redis as a cache with expiring sorted sets that are filled on demand from cassandra partitions

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Benjamin Roth
Not if you want to sort by score (a counter) Am 14.01.2017 08:33 schrieb "DuyHai Doan" : > Clustering column can be seen as sorted set > > Table abstraction == Map> > > > On Sat, Jan 14, 2017 at 2:28 AM, Edward Capriolo > wrote: > >> >> >> On Fri, Jan 13, 2017 at 8:14 PM, Jonathan Haddad >> wro

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread DuyHai Doan
Clustering column can be seen as sorted set Table abstraction == Map> On Sat, Jan 14, 2017 at 2:28 AM, Edward Capriolo wrote: > > > On Fri, Jan 13, 2017 at 8:14 PM, Jonathan Haddad > wrote: > >> I've thought about this for years and have never arrived on a >> particularly great implementation

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Edward Capriolo
On Fri, Jan 13, 2017 at 8:14 PM, Jonathan Haddad wrote: > I've thought about this for years and have never arrived on a particularly > great implementation. Your idea will be maybe OK if the sets are very > small and if the values don't change very often. But in a system where the > values of t

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Jonathan Haddad
I've thought about this for years and have never arrived on a particularly great implementation. Your idea will be maybe OK if the sets are very small and if the values don't change very often. But in a system where the values of the keys in the set change frequently (lots of tombstones) or the s

Re: implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Edward Capriolo
On Fri, Jan 13, 2017 at 5:14 PM, Mike Torra wrote: > We currently use redis to store sorted sets that we increment many, many > times more than we read. For example, only about 5% of these sets are ever > read. We are getting to the point where redis is becoming difficult to > scale (currently at

implementing a 'sorted set' on top of cassandra

2017-01-13 Thread Mike Torra
We currently use redis to store sorted sets that we increment many, many times more than we read. For example, only about 5% of these sets are ever read. We are getting to the point where redis is becoming difficult to scale (currently at >20 nodes). We've started using cassandra for other thin