Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread knizhnik
On 12/12/2013 11:42 AM, Pavel Stehule wrote: it is interesting idea. For me, a significant information from comparation, so we do some significantly wrong. Memory engine should be faster naturally, but I don't tkink it can be 1000x. Sorry, but I didn't fabricate this results: Below is

Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread Merlin Moncure
On Thu, Dec 12, 2013 at 4:02 AM, knizhnik knizh...@garret.ru wrote: On 12/12/2013 11:42 AM, Pavel Stehule wrote: it is interesting idea. For me, a significant information from comparation, so we do some significantly wrong. Memory engine should be faster naturally, but I don't tkink it can be

Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread knizhnik
On 12/12/2013 07:03 PM, Merlin Moncure wrote: On Thu, Dec 12, 2013 at 4:02 AM, knizhnik knizh...@garret.ru wrote: Yeah. It's not fair to compare vs an implementation that is constrained to use only 1mb. For analytics work huge work mem is pretty typical setting. 10x improvement is believable

Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread Merlin Moncure
On Thu, Dec 12, 2013 at 12:18 PM, knizhnik knizh...@garret.ru wrote: IMHO it is strange to see such small default values in postgresql configuration. This (low default work mem) is because of three things: 1) Most queries do not really need a lot of work mem 2) Work mem stacks with each query

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread Merlin Moncure
On Mon, Dec 9, 2013 at 1:40 PM, knizhnik knizh...@garret.ru wrote: Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread k...@rice.edu
On Mon, Dec 09, 2013 at 11:40:41PM +0400, knizhnik wrote: Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any feedbacks,

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
Hello! Implementation of IMCS itself took me about two months (with testing and writing documentation). But huge part of the code was previously written by me for other projects, so I have reused them. Most of the time I have spent in integration of this code with PostgreSQL (I was not so

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
Hi, I depends on what you mean by transparently substitute. I f you want to be able to execute standard SQL queries using columnar store, then it seems to be impossible without rewriting of executor. I provided another approach based on calling standard functions which perform manipulations

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread Merlin Moncure
On Wed, Dec 11, 2013 at 10:08 AM, knizhnik knizh...@garret.ru wrote: 1. Calls in PL/pgSQL are very slow - about 1-2 micsroseconds at my computer. Just defining insertion per-row trigger with empty procedure increase time of insertion of 6 million records twice - from 7 till 15 seconds. If

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread Kevin Grittner
k...@rice.edu k...@rice.edu wrote: The question I have, which applies to the matview support as well, is How can we transparently substitute usage of the in-memory columnar store/matview in a SQL query?. My take on that regarding matviews is: (1)  It makes no sense to start work on this

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread desmodemone
2013/12/9 knizhnik knizh...@garret.ru Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any feedbacks, bug reports and

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
Thank you very much for reporting the problem. And sorry for this bug and lack of negative tests. Attempt to access unexisted value cause autoloading of data from the table to columnar store (because autoload property is enabled by default) and as far as this entry is not present in the table,

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread Pavel Stehule
it is interesting idea. For me, a significant information from comparation, so we do some significantly wrong. Memory engine should be faster naturally, but I don't tkink it can be 1000x. Yesterday we did a some tests, that shows so for large tables (5G)a our hashing is not effective. Disabling

[HACKERS] In-Memory Columnar Store

2013-12-09 Thread knizhnik
Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any feedbacks, bug reports and suggestions are welcome. Vertical representation