Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Josh Gibbs
On 12/11/2011 5:02 p.m., Simon Slavin wrote: On 12 Nov 2011, at 3:43am, Josh Gibbs wrote: We are struggling to find a way to rework this query in a way that performs efficiently for large data sets. It's all about the indexes. The problem with this query seems to be related to the

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Igor Tandetnik
Josh Gibbs jgi...@imailds.com wrote: The timestamps are already integers. We stumbled across that CAST operation optimization purely by accident. I don't remember what led to it, but we found that it gave a measurable performance boost casting the integer as an integer. This works by

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Igor Tandetnik
Josh Gibbs jgi...@imailds.com wrote: To reiterate from my original question, if we don't add the order by then the results come back at a very acceptable speed. But of course we'd then have to iterate through the results ourselves to siphon off the top item set that we're after. I'd really

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Simon Slavin
On 12 Nov 2011, at 2:37pm, Igor Tandetnik wrote: ANALYZE Yeah. Do an ANALYZE. Then test timing with and without the CAST. See if it helps. I don't understand why your query should be faster when you suppress a useful index. Oh wait. Your entire WHERE clause is about the joined table.

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Josh Gibbs
On 13/11/2011 3:37 a.m., Igor Tandetnik wrote: Josh Gibbsjgi...@imailds.com wrote: The timestamps are already integers. We stumbled across that CAST operation optimization purely by accident. I don't remember what led to it, but we found that it gave a measurable performance boost casting

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-12 Thread Josh Gibbs
On 13/11/2011 3:45 a.m., Igor Tandetnik wrote: Josh Gibbsjgi...@imailds.com wrote: To reiterate from my original question, if we don't add the order by then the results come back at a very acceptable speed. But of course we'd then have to iterate through the results ourselves to siphon off

[sqlite] Any thoughts on optimizations for this query?

2011-11-11 Thread Josh Gibbs
Hi all, We are struggling to find a way to rework this query in a way that performs efficiently for large data sets. The goal is to find the top x most active senders of e-mails within a date range. The killer of this query is the ORDER BY clause. Without it the results are quick and snappy.

Re: [sqlite] Any thoughts on optimizations for this query?

2011-11-11 Thread Simon Slavin
On 12 Nov 2011, at 3:43am, Josh Gibbs wrote: We are struggling to find a way to rework this query in a way that performs efficiently for large data sets. It's all about the indexes. The goal is to find the top x most active senders of e-mails within a date range. Do you frequently do