[tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-09-05 Thread Kostas Jakeliunas
Hello, This status update is less extensive/dramatic compared to the last one, but I'm still happy to report to be slowly moving ahead towards a stable searchable archive system. In short, I've been working on what I said I should work on in the last report, more or less: I should now move on wit

[tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-23 Thread Kostas Jakeliunas
Hello! Updating on my Searchable Tor metrics archive project. (As is very evident) I'm very open for naming suggestions. :) To the best of my understanding and current satisfaction, I solved the database bottlenecks, or at least I am, as of now, satisfied with the current output from my benchmark

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-19 Thread Kostas Jakeliunas
Hi Karsten, On Mon, Aug 19, 2013 at 3:49 PM, Karsten Loesing wrote: > I wonder, is there a document describing the new API somewhere? If not, > do you mind creating one? > I have an outdated draft that covers more than the API. Looks like it's high time I updated and published it (at least the

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-19 Thread Karsten Loesing
Hi Kostas, On 8/15/13 9:50 PM, Kostas Jakeliunas wrote: > On Wed, Aug 14, 2013 at 1:33 PM, Karsten Loesing > wrote: > >> >> Looks like pg_trgm is contained in postgresql-contrib-9.1, so it's more >> likely that we can run something requiring this extension on a >> torproject.org machine. Still,

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-15 Thread Kostas Jakeliunas
On Wed, Aug 14, 2013 at 1:33 PM, Karsten Loesing wrote: > > Looks like pg_trgm is contained in postgresql-contrib-9.1, so it's more > likely that we can run something requiring this extension on a > torproject.org machine. Still, requiring extensions should be the last > resort if no other soluti

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-14 Thread Karsten Loesing
On 8/13/13 3:17 PM, Kostas Jakeliunas wrote: > On Tue, Aug 13, 2013 at 2:15 PM, Karsten Loesing > wrote: >> >> I suggest putting pg_prewarm on the future work list. I sense there's a >> lot of unused potential in stock PostgreSQL. Tweaking the database at >> this point has the word "premature op

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-13 Thread Kostas Jakeliunas
On Tue, Aug 13, 2013 at 2:15 PM, Karsten Loesing wrote: > > I suggest putting pg_prewarm on the future work list. I sense there's a > lot of unused potential in stock PostgreSQL. Tweaking the database at > this point has the word "premature optimization" written on it in big > letters for me. >

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-13 Thread Karsten Loesing
On 8/12/13 2:58 PM, Kostas Jakeliunas wrote: > Karsten, > > this won't be a very short email, but I honestly swear I did revise it a > couple of times. :) Okay. I'm putting in feedback below where I think it makes most sense. > This is not urgent by any measure, so whenever you find > time to r

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-12 Thread Kostas Jakeliunas
Karsten, this won't be a very short email, but I honestly swear I did revise it a couple of times. :) This is not urgent by any measure, so whenever you find time to reply will be fine. ctrl+f to "observe:" for some precise data / support for my plan re: using the pg_prewarm extension. On Mon, Au

Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-12 Thread Karsten Loesing
On 8/10/13 9:28 PM, Kostas Jakeliunas wrote: > * I don't think we can avoid using certain postgresql extensions (if only > one) - which means that deploying will always take more than apt-get && pip > install, but I believe it is needed; Can you give an example of a query that won't be executed

[tor-dev] [GSoC 2013] Status report - Searchable metrics archive

2013-08-10 Thread Kostas Jakeliunas
Hello, another busy benchmarking + profiling period for database querying, but this time more rigorous and awesome. * wrote a generic query analyzer which logs query statements, EXPLAIN, ANALYZE, spots and informs of particular queries that yield inefficient query plans; * wrote a very simple