Geohash nearby query implementation in Cassandra.
Hello everyone,
I'm working on a application that uses Cassandra and has a geolocation
component.
I was wondering beside the slides and video at
http://www.readwriteweb.com/cloud/2011/02/video-simplegeo-cassandra.php that
simplegeo published regarding their strategy if anyone has implemented geohash
storage and search in cassandra.
The basic usage is to allow a user to find things close to a geo location based
on distance radius.
I though about a couple of approaches.
1. Have the geohashes be the keys using the Ordered partitioner and get a group
of rows between keys then store the items as columns in what it would end up
looking like wide rows since each column would point to another row in a
different column family representing the item nearby.
2. Simply store the geohash prefixes as columns and use secondary indexes to do
queries such as >= and <=.
The problem I'm facing in both cases is ordering by distance and searching
neighbors.
The neighbors problem is clearly explained here:
https://github.com/davetroy/geohash-js
Once the neighbors are calculated an item can be fetched with SQL similar to
this.
SELECT * FROM table WHERE LEFT(geohash,6) IN ('dqcjqc',
'dqcjqf','dqcjqb','dqcjr1','dqcjq9','dqcjqd','dqcjr4','dqcjr0','dqcjq8')
Since Cassandra does not currently support OR or a IN statement with elements
that are not keys I'm not sure what the best way to implement geohashes may be.
Thanks in advance for any tips.
Cassandra - row range and column slice
Hallo,
assuming Ordered Partitioner I would like to have possibility to find
records by row key range and columns by slice - for example:
Give me all rows between 2001 and 2003 and all columns between A and C.
For such data:
{
2001: {A:"v1", Z:"v2"},
2002: {R:"v2", Z:"v3"},
2003: {C:"v4", Z:"v5"},
2004: {A:"v1",B:"v33", Z:"v2"}
}
Result would be:
2001: {A:"v1""},
2003: {C:"v4"}
Is such multi-slice query possible with Cassandra? Are there any
performance Issues (bedsides unbalanced cluster)?
Thanks,
Maciej
Re: Cassandra - row range and column slice
In this case, you have one query predicate that operates on a much
lower range (years) you could use it as the row key and issue a
multigetslicequery where you set all row keys and specify the slice
you're interested in (here: 2001 2002 2003, then >= "A", < "D")
On Fri, Feb 17, 2012 at 11:46 AM, Maciej Miklas
wrote:
> Hallo,
>
> assuming Ordered Partitioner I would like to have possibility to find
> records by row key range and columns by slice - for example:
>
> Give me all rows between 2001 and 2003 and all columns between A and C.
>
> For such data:
> {
> 2001: {A:"v1", Z:"v2"},
> 2002: {R:"v2", Z:"v3"},
> 2003: {C:"v4", Z:"v5"},
> 2004: {A:"v1",B:"v33", Z:"v2"}
> }
> Result would be:
> 2001: {A:"v1""},
> 2003: {C:"v4"}
>
> Is such multi-slice query possible with Cassandra? Are there any
> performance Issues (bedsides unbalanced cluster)?
>
> Thanks,
> Maciej
Streaming sessions from BulkOutputFormat job being listed long after they were killed
Hi! If I run a hadoop job that uses BulkOutputFormat to write data to Cassandra, and that hadoop job is aborted, i.e. streaming sessions are not completed, it seems like the streaming sessions hang around for a very long time, I've observed at least 12-15h, in output from 'nodetool netstats'. To me it seems like they go away only after a restart of Cassandra. Is this a known behaviour? Does it cause any problems, f. ex. consuming memory, or should I just ignore it? Regards, \EF
General questions about Cassandra
Hi, we have developed a software that store logs from mail servers in MySQL, but for huge enviroments we are developing a version that store this data in HBase. Raw logs are, once a day, first normalized, so the output is like this: username,date of login, IP Address, protocol username,date of login, IP Address, protocol username,date of login, IP Address, protocol [...] and after inserted into the database. As I was saying, for huge installation (from 1 to 10 million of logins per day, keep for 12 months) we are working with HBase, but I would also consider Cassandra. The advantage of HBase is MapReduce which makes searching the logs very fast by splitting the "query" concurrently on multiple hosts. Query will be launched from a web interface (will be few requests per day) and the search keys are user and time range. But Cassandra seems less complex to manage and simply to run, so I want to evaluate it instead of HBase. My question is, can also Cassandra split a "query" over the cluster like MapReduce? Reading on-line Cassandra seems fast in insert data but slower than HBase to "query". Is it really so? We want not install Hadoop over Cassandra. Any suggestion is welcome :-) -- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it @ LOLUG -> Socio http://www.lolug.net
cassandra on ec2 lock-ups
Hi, I've experienced several node lock-ups on EC2 instances. I'm running with the following set-up: heap-new: 800M max-heap: 8G instance type: m2.xlarge java is java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) running on ubuntu 11.04 and the apache cassandra packages + jna installed All my initial tokens are computed and set in cassandra.yaml according to the number of nodes present, the seed list is the ip of the first node. I am seeing a very strange behavior where on startup a node will lock-up the machine, ssh is still possible but ps hangs on the cassandra process which seems to be doing nothing.
Re: cassandra on ec2 lock-ups
sorry for not doing my homework properly: http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs On Fri, Feb 17, 2012 at 2:00 PM, Pierre-Yves Ritschard wrote: > Hi, > > I've experienced several node lock-ups on EC2 instances. I'm running > with the following set-up: > > heap-new: 800M > max-heap: 8G > instance type: m2.xlarge > > java is > java version "1.6.0_26" > Java(TM) SE Runtime Environment (build 1.6.0_26-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) > > running on ubuntu 11.04 and the apache cassandra packages + jna installed > > All my initial tokens are computed and set in cassandra.yaml according > to the number of nodes present, the seed list is the ip of the first > node. > > I am seeing a very strange behavior where on startup a node will > lock-up the machine, ssh is still possible but ps hangs on the > cassandra process which seems to be doing nothing.
Re: cassandra on ec2 lock-ups
http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs On Fri, Feb 17, 2012 at 7:00 AM, Pierre-Yves Ritschard wrote: > Hi, > > I've experienced several node lock-ups on EC2 instances. I'm running > with the following set-up: > > heap-new: 800M > max-heap: 8G > instance type: m2.xlarge > > java is > java version "1.6.0_26" > Java(TM) SE Runtime Environment (build 1.6.0_26-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) > > running on ubuntu 11.04 and the apache cassandra packages + jna installed > > All my initial tokens are computed and set in cassandra.yaml according > to the number of nodes present, the seed list is the ip of the first > node. > > I am seeing a very strange behavior where on startup a node will > lock-up the machine, ssh is still possible but ps hangs on the > cassandra process which seems to be doing nothing.
Re: deleting rows and tombstones
Deleting the entire row at once only creates a row-level tombstone. This is almost free. "Tombstone buildup" happens when performing column-level deletes, then inserting more (different) columns. The classic example is modeling a queue in a row. On Tue, Feb 14, 2012 at 1:54 PM, Todd Burruss wrote: > my design calls for deleting a row (by key, not individual columns) and > re-inserting it a lot and I'm concerned about tombstone build up slowing > down reads. I know if I delete a lot of individual columns the tombstones > will build up and slow down reads until they are cleaned up, but not sure if > the same holds for deleting the whole role. > > thoughts? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Replication factor per column family
Ok, that's clear, thank you for your time! 2012/2/16 aaron morton > yes. > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 16/02/2012, at 10:15 PM, R. Verlangen wrote: > > Hmm ok. This means if I want to have a CF with RF = 3 and another CF with > RF = 1 (e.g. some debug logging) I will have to create 2 keyspaces? > > 2012/2/16 aaron morton > >> Multiple CF mutations for a row are treated atomically in the commit log, >> and they are sent together to the replicas. Replication occurs at the row >> level, not the row+cf level. >> >> If each CF had it's own RF, odd things may happen. Like sending a batch >> mutation for one row and two CF's that fails because there is not enough >> nodes for one of the CF's. >> >> Would be other reasons as well. In short it's baked in. >> >> Cheers >> >> - >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 16/02/2012, at 9:54 PM, R. Verlangen wrote: >> >> Hi there, >> >> As the subject states: "Is it possible to set a replication factor per >> column family?" >> >> Could not find anything of recent releases. I'm running Cassandra 1.0.7 >> and I think it should be possible on a per CF basis instead of the whole >> keyspace. >> >> With kind regards, >> Robin >> >> >> > >
RE: General questions about Cassandra
Are there plans to build-in some sort of map-reduce framework into Cassandra and CQL? It seems that users should be able to apply a Java method to selected rows in parallel on the distributed Cassandra JVMs. I believe Solandra uses such an integration. Don From: Alessio Cecchi [ales...@skye.it] Sent: Friday, February 17, 2012 4:42 AM To: user@cassandra.apache.org Subject: General questions about Cassandra Hi, we have developed a software that store logs from mail servers in MySQL, but for huge enviroments we are developing a version that store this data in HBase. Raw logs are, once a day, first normalized, so the output is like this: username,date of login, IP Address, protocol username,date of login, IP Address, protocol username,date of login, IP Address, protocol [...] and after inserted into the database. As I was saying, for huge installation (from 1 to 10 million of logins per day, keep for 12 months) we are working with HBase, but I would also consider Cassandra. The advantage of HBase is MapReduce which makes searching the logs very fast by splitting the "query" concurrently on multiple hosts. Query will be launched from a web interface (will be few requests per day) and the search keys are user and time range. But Cassandra seems less complex to manage and simply to run, so I want to evaluate it instead of HBase. My question is, can also Cassandra split a "query" over the cluster like MapReduce? Reading on-line Cassandra seems fast in insert data but slower than HBase to "query". Is it really so? We want not install Hadoop over Cassandra. Any suggestion is welcome :-) -- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it @ LOLUG -> Socio http://www.lolug.net
Re: General questions about Cassandra
Don, That's a good idea, but you have to be careful not to preclude the use of dynamic column families (e.g. CF's with time series-like schemas) which is what Cassandra's best at. The right approach is to build your own "ORM"/persistence layer (or generate one with some tools) that can hide the API differences between static and dynamic CF's. Once you're there, hadoop and Pig both come very close to what you're asking for. In other words, you should be asking for a means to apply a Java method to selected objects (not rows) that are persisted in a Cassandra column family. thx - Chris Chris Gerken chrisger...@mindspring.com 512.587.5261 http://www.linkedin.com/in/chgerken On Feb 17, 2012, at 9:35 AM, Don Smith wrote: > Are there plans to build-in some sort of map-reduce framework into Cassandra > and CQL? It seems that users should be able to apply a Java method to > selected rows in parallel on the distributed Cassandra JVMs. I believe > Solandra uses such an integration. > > Don > > From: Alessio Cecchi [ales...@skye.it] > Sent: Friday, February 17, 2012 4:42 AM > To: user@cassandra.apache.org > Subject: General questions about Cassandra > > Hi, > > we have developed a software that store logs from mail servers in MySQL, > but for huge enviroments we are developing a version that store this > data in HBase. Raw logs are, once a day, first normalized, so the output > is like this: > > username,date of login, IP Address, protocol > username,date of login, IP Address, protocol > username,date of login, IP Address, protocol > [...] > > and after inserted into the database. > > As I was saying, for huge installation (from 1 to 10 million of logins > per day, keep for 12 months) we are working with HBase, but I would also > consider Cassandra. > > The advantage of HBase is MapReduce which makes searching the logs very > fast by splitting the "query" concurrently on multiple hosts. > > Query will be launched from a web interface (will be few requests per > day) and the search keys are user and time range. > > But Cassandra seems less complex to manage and simply to run, so I want > to evaluate it instead of HBase. > > My question is, can also Cassandra split a "query" over the cluster like > MapReduce? Reading on-line Cassandra seems fast in insert data but > slower than HBase to "query". Is it really so? > > We want not install Hadoop over Cassandra. > > Any suggestion is welcome :-) > > -- > Alessio Cecchi is: > @ ILS -> http://www.linux.it/~alessice/ > on LinkedIn -> http://www.linkedin.com/in/alessice > Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ > @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it > @ LOLUG -> Socio http://www.lolug.net >
Re: General questions about Cassandra
MapReduce and Hadoop generally are pluggable so you can do queries over HDFS, over HBase, or over Cassandra. Cassandra has good Hadoop support as outlined here: http://wiki.apache.org/cassandra/HadoopSupport. If you're looking for a simpler solution, there is DataStax's enterprise product which makes configuration and operations much easier for doing both realtime and analytic queries - http://www.datastax.com/products/enterprise/compare We currently use CDH3 with Cassandra 0.8 and it works fine. In bocca al lupo, Jeremy On Feb 17, 2012, at 6:42 AM, Alessio Cecchi wrote: > Hi, > > we have developed a software that store logs from mail servers in MySQL, but > for huge enviroments we are developing a version that store this data in > HBase. Raw logs are, once a day, first normalized, so the output is like this: > > username,date of login, IP Address, protocol > username,date of login, IP Address, protocol > username,date of login, IP Address, protocol > [...] > > and after inserted into the database. > > As I was saying, for huge installation (from 1 to 10 million of logins per > day, keep for 12 months) we are working with HBase, but I would also consider > Cassandra. > > The advantage of HBase is MapReduce which makes searching the logs very fast > by splitting the "query" concurrently on multiple hosts. > > Query will be launched from a web interface (will be few requests per day) > and the search keys are user and time range. > > But Cassandra seems less complex to manage and simply to run, so I want to > evaluate it instead of HBase. > > My question is, can also Cassandra split a "query" over the cluster like > MapReduce? Reading on-line Cassandra seems fast in insert data but slower > than HBase to "query". Is it really so? > > We want not install Hadoop over Cassandra. > > Any suggestion is welcome :-) > > -- > Alessio Cecchi is: > @ ILS -> http://www.linux.it/~alessice/ > on LinkedIn -> http://www.linkedin.com/in/alessice > Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ > @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it > @ LOLUG -> Socio http://www.lolug.net >
Re: General questions about Cassandra
In response to an offline question… There are two usage patterns for Cassandra column families, static and dynamic. With both approaches you store objects of a given type into a column family. With static usage the object type you're persisting has a single key and each row in the column family maps to a single object. The value of an object's key is stored in the row key and each of the object's properties is stored in a column whose name is the name of the property and whose value is the property value. There are the same number of columns in a row as there are non-null property values. This usage is very much like traditional relational database usage. With dynamic usage the object type to be persisted has two keys (I'll get to composite keys in a bit). With this approach the value of an object's primary key is stored as a row key and the entire object is stored in a single column whose name is the value of the object's secondary key and whose value is the entire object (serialized into a ByteBuffer). This results in persisting potentially many objects in a single row. All of those objects have the same primary key and there are as many columns as there are objects with the same primary key. An example of this approach is a time series column family in which each row holds weather readings for a different city and each column in a row holds all of the weather observations for that city at a certain time. The timestamp is used as a column name and an object holding all the observations is serialized and stored in the corresponding column value. Cassandra is a really powerful database, but it excels performance-wise with reading and writing time series data stored using a dynamic column family. There are variations of the above patterns. You can use composite types to define a row key or column name that are made up of values of multiple keys, for example. I gave a presentation on the topic of Cassandra patterns recently to the Austin Cassandra Meetup. You can find my charts there in the archives or posted to my box at the linkedin site below…. or contact me offline. To bring this back to the original question. Asking for the ability to apply a Java method to selected rows makes sense for static column families, but I think the more general need is to be able to apply a Java method to selected persisted objects in a column family regardless of static or dynamic usage. While I'm on my soapbox, I think this requirement applies to Pig support as well. thx Chris Gerken chrisger...@mindspring.com 512.587.5261 http://www.linkedin.com/in/chgerken On Feb 17, 2012, at 10:07 AM, Chris Gerken wrote: > Don, > > That's a good idea, but you have to be careful not to preclude the use of > dynamic column families (e.g. CF's with time series-like schemas) which is > what Cassandra's best at. The right approach is to build your own > "ORM"/persistence layer (or generate one with some tools) that can hide the > API differences between static and dynamic CF's. Once you're there, hadoop > and Pig both come very close to what you're asking for. > > In other words, you should be asking for a means to apply a Java method to > selected objects (not rows) that are persisted in a Cassandra column family. > > thx > > - Chris > > Chris Gerken > > chrisger...@mindspring.com > 512.587.5261 > http://www.linkedin.com/in/chgerken > > > > On Feb 17, 2012, at 9:35 AM, Don Smith wrote: > >> Are there plans to build-in some sort of map-reduce framework into Cassandra >> and CQL? It seems that users should be able to apply a Java method to >> selected rows in parallel on the distributed Cassandra JVMs. I believe >> Solandra uses such an integration. >> >> Don >> >> From: Alessio Cecchi [ales...@skye.it] >> Sent: Friday, February 17, 2012 4:42 AM >> To: user@cassandra.apache.org >> Subject: General questions about Cassandra >> >> Hi, >> >> we have developed a software that store logs from mail servers in MySQL, >> but for huge enviroments we are developing a version that store this >> data in HBase. Raw logs are, once a day, first normalized, so the output >> is like this: >> >> username,date of login, IP Address, protocol >> username,date of login, IP Address, protocol >> username,date of login, IP Address, protocol >> [...] >> >> and after inserted into the database. >> >> As I was saying, for huge installation (from 1 to 10 million of logins >> per day, keep for 12 months) we are working with HBase, but I would also >> consider Cassandra. >> >> The advantage of HBase is MapReduce which makes searching the logs very >> fast by splitting the "query" concurrently on multiple hosts. >> >> Query will be launched from a web interface (will be few requests per >> day) and the search keys are user and time range. >> >> But Cassandra seems less complex to manage and simply to run, so I want >> to evaluate it instead of HBas
Re: Streaming sessions from BulkOutputFormat job being listed long after they were killed
Erik, Currently, streaming failure handling is poorly functioning. There are several discussions and bug reports regarding streaming failure on jira. Hanged streaming session will be left in memory unless you restart C*, but it does not cause problem I believe. -- Yuki Morishita On Friday, February 17, 2012 at 6:18 AM, Erik Forsberg wrote: > Hi! > > If I run a hadoop job that uses BulkOutputFormat to write data to > Cassandra, and that hadoop job is aborted, i.e. streaming sessions are > not completed, it seems like the streaming sessions hang around for a > very long time, I've observed at least 12-15h, in output from 'nodetool > netstats'. > > To me it seems like they go away only after a restart of Cassandra. > > Is this a known behaviour? Does it cause any problems, f. ex. consuming > memory, or should I just ignore it? > > Regards, > \EF > >
Re: Key cache hit rate issue
ah, I missed the part about "key" cache .. I read "row" cache. thx On 2/16/12 6:14 PM, "Jonathan Ellis" wrote: >Look for this code in SSTableReader.getPosition: > >Pair unifiedKey = new >Pair(descriptor, decoratedKey); >Long cachedPosition = getCachedPosition(unifiedKey, true); > >On Thu, Feb 16, 2012 at 4:23 PM, Todd Burruss >wrote: >> jonathan, you said the key to the cache is key + sstable? looking at >>the >> code it looks like a DecoratedKey is the "row key". how does sstable >>come >> into play? >> >> On 2/16/12 1:20 PM, "Jonathan Ellis" wrote: >> >>>So, you have roughly 1/6 of your (physical) row keys cached and about >>>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember, >>>each logical key may be spread across multiple physical sstables -- >>>each (key, sstable) pair is one entry in the key cache. >>> >>>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana >>> wrote: Hi Aaron, Here it is. Keyspace: Read Count: 1123637972 Read Latency: 5.757938114343114 ms. Write Count: 128201833 Write Latency: 0.0682576607387509 ms. Pending Tasks: 0 Column Family: YY SSTable count: 18 Space used (live): 103318720685 Space used (total): 103318720685 Number of Keys (estimate): 92404992 Memtable Columns Count: 1425580 Memtable Data Size: 359655747 Memtable Switch Count: 2522 Read Count: 1123637972 Read Latency: 14.731 ms. Write Count: 128201833 Write Latency: NaN ms. Pending Tasks: 0 Bloom Filter False Postives: 1488 Bloom Filter False Ratio: 0.0 Bloom Filter Space Used: 331522920 Key cache capacity: 16637958 Key cache size: 16637958 Key cache hit rate: 0.2708 Row cache: disabled Compacted row minimum size: 51 Compacted row maximum size: 6866 Compacted row mean size: 2560 Thanks, Eran Chinthaka Withana On Thu, Feb 16, 2012 at 12:30 AM, aaron morton wrote: > > Its in the order of 261 to 8000 and the ratio is 0.00. But i guess >8000 is > bit high. Is there a way to fix/improve it? > > Sorry I don't understand what you mean. But if the ratio is 0.0 all >is > good. > > Could you include the full output from cfstats for the CF you are >looking > at ? > > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote: > > Its in the order of 261 to 8000 and the ratio is 0.00. But i guess >8000 is > bit high. Is there a way to fix/improve it? > > Thanks, > Eran Chinthaka Withana > > > On Tue, Feb 14, 2012 at 3:42 PM, aaron morton > > wrote: >> >> Out of interest what does cfstats say about the bloom filter stats >>? A >> high false positive could lead to a low key cache hit rate. >> >> Also, is there a way to warm start the key cache, meaning pre-load >>the >> amount of keys I set as keys_cached? >> >> See key_cache_save_period when creating the CF. >> >> Cheers >> >> >> - >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote: >> >> Hi, >> >> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% >> (using the numerical values). This is visible in cfstats too. But >>I'm >> getting less than 20% (or sometimes even 0%) key cache hit rate. >>Well, the >> data access pattern is not the issue here as I know they are >>retrieving the >> same row multiple times. I'm using hector client with dynamic load >>balancing >> policy with consistency ONE for both reads and writes. Any ideas on >>how to >> find the issue and fix this? >> >> Here is what I see on cfstats. >> >> Key cache capacity: 16637958 >> Key cache size: 16637958 >> Key cache hit rate: 0.045454545454545456 >> >> Also, is there a way to warm start the key cache, meaning pre-load >>the >> amount of keys I set as keys_cached? >> >> Thanks, >> Eran >> >> > > >>> >>> >>> >>>-- >>>Jonathan Ellis >>>Project Chair, Apache Cassandra >>>co-founder of DataStax, the source for professional Cassandra support >>>http://www.datastax.com >> > > > >-- >Jonathan Ellis >Project Chair, Apache Cassandra >co-founder of DataStax, the source for professional Cassandra support >http://www.datastax.com
Re: Geohash nearby query implementation in Cassandra.
2012/2/17 Raúl Raja Martínez
> Hello everyone,
>
> I'm working on a application that uses Cassandra and has a geolocation
> component.
> I was wondering beside the slides and video at
> http://www.readwriteweb.com/cloud/2011/02/video-simplegeo-cassandra.php that
> simplegeo published regarding their strategy if anyone has implemented
> geohash storage and search in cassandra.
> The basic usage is to allow a user to find things close to a geo location
> based on distance radius.
>
> I though about a couple of approaches.
>
> 1. Have the geohashes be the keys using the Ordered partitioner and get a
> group of rows between keys then store the items as columns in what it would
> end up looking like wide rows since each column would point to another row
> in a different column family representing the item nearby.
>
That's what we did early on at SimpleGeo.
> 2. Simply store the geohash prefixes as columns and use secondary indexes
> to do queries such as >= and <=.
>
This seems like a reasonable approach now that secondary indexes are
available. It might even address some of the hotspot problems we had with
the order preserving partitioner since the indices are distributed across
all hosts. Of course there are tradeoffs there too. Seems like a viable
option for sure.
> The problem I'm facing in both cases is ordering by distance and searching
> neighbors.
>
This will always be a problem with dimensionality reduction techniques like
geohashes. A brief bit of pedantry: it is mathematically impossible to do
dimensionality reduction without losing information. You can't embed a 2
dimensional space in a 1 dimensional space and preserve the 2D
topology. This manifests itself all sorts of ways, but when it comes to
doing kNN queries it's particularly obvious. Things that are near in 2D
space can be far apart in 1D space and vice versa. Doing a 1D embedding
like this will always result in suboptimal performance for at least some
queries. You'll have to over-fetch and post-process to get the correct
results.
That said, a 1D embedding is certainly easier to code since
multidimensional indexes are not available in Cassandra. And there are
plenty of data sets that don't hit any degenerate cases. Moreover, if
you're mostly doing bounding-radius queries the geohash approach isn't
nearly as bad (the only trouble comes when you want to limit the results,
in which case you often want things ordered by distance from centroid and
the query is no longer a bounding radius query - rather, it's a kNN with a
radius constraint). In any case, geohash is a reasonable starting point, at
least.
The neighbors problem is clearly explained here:
> https://github.com/davetroy/geohash-js
>
> Once the neighbors are calculated an item can be fetched with SQL similar
> to this.
>
> SELECT * FROM table WHERE LEFT(geohash,6) IN ('dqcjqc',
> 'dqcjqf','dqcjqb','dqcjr1','dqcjq9','dqcjqd','dqcjr4','dqcjr0','dqcjq8')
>
> Since Cassandra does not currently support OR or a IN statement with
> elements that are not keys I'm not sure what the best way to implement
> geohashes may be.
>
Can't you use the thrift interface and use multiget_slice? If I recall
correctly, we implemented a special version of multiget_slice that stopped
when we got a certain number of columns across all rows. I don't have that
code handy but we did that work early in our Cassandra careers and,
starting from the thrift interface and following control flow for the
multiget_slice command, it wasn't terribly difficult to add.
Mike
Re: Key cache hit rate issue
Only thing I can think of is that if you've set the cache size
manually over JMX it will preserve that size if you change it via a
schema update.
On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
wrote:
> Hi Jonathan,
>>
>>
>> > For some reason 16637958 (the keys cached) has become a golden number
>> > and I
>> > don't see key cache increasing beyond that.
>>
>> 16637958 is your configured cache capacity according to the cfstats you
>> pasted.
>
>
> this is another weird part. If you look at the schema[1] (pasted here,
> extracted through cli), the keys_cached is set to 40million. And cfstats
> shows that cassandra is only using 16 million.
>
> Please note that I enabled row cache since I posted the cfstats. And pasting
> the latest cfstats[2] herewith.
>
> [1]
> create column family YY
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'LongType'
> and rows_cached = 1.0
> and row_cache_save_period = 0
> and row_cache_keys_to_save = 2147483647
> and keys_cached = 4.0E7
> and key_cache_save_period = 14400
> and read_repair_chance = 0.1
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and row_cache_provider = 'SerializingCacheProvider'
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and compression_options = {'chunk_length_kb' : '64', 'sstable_compression'
> : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> [2]
> Keyspace: X
> Read Count: 590667
> Read Latency: 11.644107261790484 ms.
> Write Count: 12720068
> Write Latency: 0.06049896832312532 ms.
> Pending Tasks: 0
> Column Family: YY
> SSTable count: 12
> Space used (live): 66300297338
> Space used (total): 66300297338
> Number of Keys (estimate): 61152896
> Memtable Columns Count: 881972
> Memtable Data Size: 211749842
> Memtable Switch Count: 267
> Read Count: 590667
> Read Latency: 16.859 ms.
> Write Count: 12720068
> Write Latency: 0.060 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 507
> Bloom Filter False Ratio: 0.00099
> Bloom Filter Space Used: 168118968
> Key cache capacity: 4000
> Key cache size: 3838755
> Key cache hit rate: 0.2072072072072072
> Row cache capacity: 1
> Row cache size: 1
> Row cache hit rate: 0.03
> Compacted row minimum size: 51
> Compacted row maximum size: 6866
> Compacted row mean size: 2577
>
> Thanks,
> Eran
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
I never used JMX for any changes and use JMX only for monitoring. All my
updates goes through schema updates.
To give you little bit more context (not sure whether this will help but
anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
key cache hit rate. But after that point, I stopped all the reads and kept
only writes into the system. When I enabled reads this week, then only I
saw this read latency degradation. I didn't run nodetool repair for
sometime (gc grace is 10 days), since I stopped readers. Can this be
connected to the read degradation? I started running repairs last night but
didn't see any read latency improvements, yet.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis wrote:
> Only thing I can think of is that if you've set the cache size
> manually over JMX it will preserve that size if you change it via a
> schema update.
>
> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> wrote:
> > Hi Jonathan,
> >>
> >>
> >> > For some reason 16637958 (the keys cached) has become a golden number
> >> > and I
> >> > don't see key cache increasing beyond that.
> >>
> >> 16637958 is your configured cache capacity according to the cfstats you
> >> pasted.
> >
> >
> > this is another weird part. If you look at the schema[1] (pasted here,
> > extracted through cli), the keys_cached is set to 40million. And cfstats
> > shows that cassandra is only using 16 million.
> >
> > Please note that I enabled row cache since I posted the cfstats. And
> pasting
> > the latest cfstats[2] herewith.
> >
> > [1]
> > create column family YY
> > with column_type = 'Standard'
> > and comparator = 'UTF8Type'
> > and default_validation_class = 'UTF8Type'
> > and key_validation_class = 'LongType'
> > and rows_cached = 1.0
> > and row_cache_save_period = 0
> > and row_cache_keys_to_save = 2147483647
> > and keys_cached = 4.0E7
> > and key_cache_save_period = 14400
> > and read_repair_chance = 0.1
> > and gc_grace = 864000
> > and min_compaction_threshold = 4
> > and max_compaction_threshold = 32
> > and replicate_on_write = true
> > and row_cache_provider = 'SerializingCacheProvider'
> > and compaction_strategy =
> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> > and compression_options = {'chunk_length_kb' : '64',
> 'sstable_compression'
> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >
> > [2]
> > Keyspace: X
> > Read Count: 590667
> > Read Latency: 11.644107261790484 ms.
> > Write Count: 12720068
> > Write Latency: 0.06049896832312532 ms.
> > Pending Tasks: 0
> > Column Family: YY
> > SSTable count: 12
> > Space used (live): 66300297338
> > Space used (total): 66300297338
> > Number of Keys (estimate): 61152896
> > Memtable Columns Count: 881972
> > Memtable Data Size: 211749842
> > Memtable Switch Count: 267
> > Read Count: 590667
> > Read Latency: 16.859 ms.
> > Write Count: 12720068
> > Write Latency: 0.060 ms.
> > Pending Tasks: 0
> > Bloom Filter False Postives: 507
> > Bloom Filter False Ratio: 0.00099
> > Bloom Filter Space Used: 168118968
> > Key cache capacity: 4000
> > Key cache size: 3838755
> > Key cache hit rate: 0.2072072072072072
> > Row cache capacity: 1
> > Row cache size: 1
> > Row cache hit rate: 0.03
> > Compacted row minimum size: 51
> > Compacted row maximum size: 6866
> > Compacted row mean size: 2577
> >
> > Thanks,
> > Eran
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Key cache hit rate issue
I suspect the main difference is that 2-3 weeks ago almost none of
your reads had to hit disk.
On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
wrote:
> I never used JMX for any changes and use JMX only for monitoring. All my
> updates goes through schema updates.
>
> To give you little bit more context (not sure whether this will help but
> anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
> key cache hit rate. But after that point, I stopped all the reads and kept
> only writes into the system. When I enabled reads this week, then only I saw
> this read latency degradation. I didn't run nodetool repair for sometime (gc
> grace is 10 days), since I stopped readers. Can this be connected to the
> read degradation? I started running repairs last night but didn't see any
> read latency improvements, yet.
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis wrote:
>>
>> Only thing I can think of is that if you've set the cache size
>> manually over JMX it will preserve that size if you change it via a
>> schema update.
>>
>> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
>> wrote:
>> > Hi Jonathan,
>> >>
>> >>
>> >> > For some reason 16637958 (the keys cached) has become a golden number
>> >> > and I
>> >> > don't see key cache increasing beyond that.
>> >>
>> >> 16637958 is your configured cache capacity according to the cfstats you
>> >> pasted.
>> >
>> >
>> > this is another weird part. If you look at the schema[1] (pasted here,
>> > extracted through cli), the keys_cached is set to 40million. And cfstats
>> > shows that cassandra is only using 16 million.
>> >
>> > Please note that I enabled row cache since I posted the cfstats. And
>> > pasting
>> > the latest cfstats[2] herewith.
>> >
>> > [1]
>> > create column family YY
>> > with column_type = 'Standard'
>> > and comparator = 'UTF8Type'
>> > and default_validation_class = 'UTF8Type'
>> > and key_validation_class = 'LongType'
>> > and rows_cached = 1.0
>> > and row_cache_save_period = 0
>> > and row_cache_keys_to_save = 2147483647
>> > and keys_cached = 4.0E7
>> > and key_cache_save_period = 14400
>> > and read_repair_chance = 0.1
>> > and gc_grace = 864000
>> > and min_compaction_threshold = 4
>> > and max_compaction_threshold = 32
>> > and replicate_on_write = true
>> > and row_cache_provider = 'SerializingCacheProvider'
>> > and compaction_strategy =
>> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
>> > and compression_options = {'chunk_length_kb' : '64',
>> > 'sstable_compression'
>> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>> >
>> > [2]
>> > Keyspace: X
>> > Read Count: 590667
>> > Read Latency: 11.644107261790484 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.06049896832312532 ms.
>> > Pending Tasks: 0
>> > Column Family: YY
>> > SSTable count: 12
>> > Space used (live): 66300297338
>> > Space used (total): 66300297338
>> > Number of Keys (estimate): 61152896
>> > Memtable Columns Count: 881972
>> > Memtable Data Size: 211749842
>> > Memtable Switch Count: 267
>> > Read Count: 590667
>> > Read Latency: 16.859 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.060 ms.
>> > Pending Tasks: 0
>> > Bloom Filter False Postives: 507
>> > Bloom Filter False Ratio: 0.00099
>> > Bloom Filter Space Used: 168118968
>> > Key cache capacity: 4000
>> > Key cache size: 3838755
>> > Key cache hit rate: 0.2072072072072072
>> > Row cache capacity: 1
>> > Row cache size: 1
>> > Row cache hit rate: 0.03
>> > Compacted row minimum size: 51
>> > Compacted row maximum size: 6866
>> > Compacted row mean size: 2577
>> >
>> > Thanks,
>> > Eran
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
True, the high hit rate has translated to low read latency. But the
question is how can I debug the reason for low hit rate now assuming read
patterns haven't changed.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 3:07 PM, Jonathan Ellis wrote:
> I suspect the main difference is that 2-3 weeks ago almost none of
> your reads had to hit disk.
>
> On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
> wrote:
> > I never used JMX for any changes and use JMX only for monitoring. All my
> > updates goes through schema updates.
> >
> > To give you little bit more context (not sure whether this will help but
> > anyway), about 2-3 weeks back the read latency was 4-8ms with about
> 90-95%
> > key cache hit rate. But after that point, I stopped all the reads and
> kept
> > only writes into the system. When I enabled reads this week, then only I
> saw
> > this read latency degradation. I didn't run nodetool repair for sometime
> (gc
> > grace is 10 days), since I stopped readers. Can this be connected to the
> > read degradation? I started running repairs last night but didn't see any
> > read latency improvements, yet.
> >
> > Thanks,
> > Eran Chinthaka Withana
> >
> >
> >
> > On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis
> wrote:
> >>
> >> Only thing I can think of is that if you've set the cache size
> >> manually over JMX it will preserve that size if you change it via a
> >> schema update.
> >>
> >> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> >> wrote:
> >> > Hi Jonathan,
> >> >>
> >> >>
> >> >> > For some reason 16637958 (the keys cached) has become a golden
> number
> >> >> > and I
> >> >> > don't see key cache increasing beyond that.
> >> >>
> >> >> 16637958 is your configured cache capacity according to the cfstats
> you
> >> >> pasted.
> >> >
> >> >
> >> > this is another weird part. If you look at the schema[1] (pasted here,
> >> > extracted through cli), the keys_cached is set to 40million. And
> cfstats
> >> > shows that cassandra is only using 16 million.
> >> >
> >> > Please note that I enabled row cache since I posted the cfstats. And
> >> > pasting
> >> > the latest cfstats[2] herewith.
> >> >
> >> > [1]
> >> > create column family YY
> >> > with column_type = 'Standard'
> >> > and comparator = 'UTF8Type'
> >> > and default_validation_class = 'UTF8Type'
> >> > and key_validation_class = 'LongType'
> >> > and rows_cached = 1.0
> >> > and row_cache_save_period = 0
> >> > and row_cache_keys_to_save = 2147483647
> >> > and keys_cached = 4.0E7
> >> > and key_cache_save_period = 14400
> >> > and read_repair_chance = 0.1
> >> > and gc_grace = 864000
> >> > and min_compaction_threshold = 4
> >> > and max_compaction_threshold = 32
> >> > and replicate_on_write = true
> >> > and row_cache_provider = 'SerializingCacheProvider'
> >> > and compaction_strategy =
> >> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> >> > and compression_options = {'chunk_length_kb' : '64',
> >> > 'sstable_compression'
> >> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >> >
> >> > [2]
> >> > Keyspace: X
> >> > Read Count: 590667
> >> > Read Latency: 11.644107261790484 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.06049896832312532 ms.
> >> > Pending Tasks: 0
> >> > Column Family: YY
> >> > SSTable count: 12
> >> > Space used (live): 66300297338
> >> > Space used (total): 66300297338
> >> > Number of Keys (estimate): 61152896
> >> > Memtable Columns Count: 881972
> >> > Memtable Data Size: 211749842
> >> > Memtable Switch Count: 267
> >> > Read Count: 590667
> >> > Read Latency: 16.859 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.060 ms.
> >> > Pending Tasks: 0
> >> > Bloom Filter False Postives: 507
> >> > Bloom Filter False Ratio: 0.00099
> >> > Bloom Filter Space Used: 168118968
> >> > Key cache capacity: 4000
> >> > Key cache size: 3838755
> >> > Key cache hit rate: 0.2072072072072072
> >> > Row cache capacity: 1
> >> > Row cache size: 1
> >> > Row cache hit rate: 0.03
> >> > Compacted row minimum size: 51
> >> > Compacted row maximum size: 6866
> >> > Compacted row mean size: 2577
> >> >
> >> > Thanks,
> >> > Eran
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
