Re: [HACKERS] Hash tables in dynamic shared memory

2017-04-06 Thread Thomas Munro
On Mon, Dec 19, 2016 at 12:33 PM, Thomas Munro wrote: > Since Dilip Kumar's Parallel Bitmap Heap Scan project is no longer > using this, I think I should park it here unless/until another > potential use case pops up. Some interesting candidates have been > mentioned already, and I'm fairly sure

Re: [HACKERS] Hash tables in dynamic shared memory

2016-12-18 Thread Thomas Munro
On Sun, Nov 20, 2016 at 11:54 AM, John Gorman wrote: > I reviewed the dht-v2.patch and found that it is in excellent shape. Thanks for reviewing! And sorry for the late reply. > Benchmarking shows that this performs somewhat faster than > dynahash which is surprising because it is doing DSA add

Re: [HACKERS] Hash tables in dynamic shared memory

2016-12-04 Thread Haribabu Kommi
On Sun, Nov 20, 2016 at 9:54 AM, John Gorman wrote: > I reviewed the dht-v2.patch and found that it is in excellent shape. > > The overall performance will be faster due to not having to examine > more than one hash bucket array most of the time. > > Reviewer didn't find any problems in the appro

Re: [HACKERS] Hash tables in dynamic shared memory

2016-11-19 Thread John Gorman
I reviewed the dht-v2.patch and found that it is in excellent shape. Benchmarking shows that this performs somewhat faster than dynahash which is surprising because it is doing DSA address translations on the fly. One area where this could run faster is to reduce the amount of time when the hash

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Andres Freund
On 2016-10-05 08:02:42 +0200, Magnus Hagander wrote: > On Oct 5, 2016 1:23 AM, "Thomas Munro" > wrote: > > Another thought: it could be used to make things like > > pg_stat_statements not have to be in shared_preload_libraries. > That would indeed be a great improvement. And possibly also allow t

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Thomas Munro
On Wed, Oct 5, 2016 at 7:02 PM, Magnus Hagander wrote: > On Oct 5, 2016 1:23 AM, "Thomas Munro" > wrote: >> >> On Wed, Oct 5, 2016 at 12:11 PM, Thomas Munro >> wrote: >> > On Wed, Oct 5, 2016 at 11:22 AM, Andres Freund >> > wrote: >> >>> Potential use cases for DHT include caches, in-memory dat

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Thomas Munro
On Thu, Oct 6, 2016 at 12:02 AM, Dilip Kumar wrote: > While reviewing , I found that in dht_iterate_begin function, we are > not initializing > iterator->last_item_pointer to InvalidDsaPointer; Fixed, thanks. -- Thomas Munro http://www.enterprisedb.com dht-v2.patch Description: Binary data -

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-05 Thread Dilip Kumar
On Wed, Oct 5, 2016 at 3:10 AM, Thomas Munro wrote: > Here is an experimental hash table implementation that uses DSA memory > so that hash tables can be shared by multiple backends and yet grow > dynamically. Development name: "DHT". +dht_iterate_begin(dht_hash_table *hash_table, + dht_iterat

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-04 Thread Magnus Hagander
On Oct 5, 2016 1:23 AM, "Thomas Munro" wrote: > > On Wed, Oct 5, 2016 at 12:11 PM, Thomas Munro > wrote: > > On Wed, Oct 5, 2016 at 11:22 AM, Andres Freund wrote: > >>> Potential use cases for DHT include caches, in-memory database objects > >>> and working state for parallel execution. > >> > >

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-04 Thread Thomas Munro
On Wed, Oct 5, 2016 at 12:11 PM, Thomas Munro wrote: > On Wed, Oct 5, 2016 at 11:22 AM, Andres Freund wrote: >>> Potential use cases for DHT include caches, in-memory database objects >>> and working state for parallel execution. >> >> Is there a more concrete example, i.e. a user we'd convert to

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-04 Thread Thomas Munro
On Wed, Oct 5, 2016 at 11:22 AM, Andres Freund wrote: >> Potential use cases for DHT include caches, in-memory database objects >> and working state for parallel execution. > > Is there a more concrete example, i.e. a user we'd convert to this at > the same time as introducing this hashtable? A c

Re: [HACKERS] Hash tables in dynamic shared memory

2016-10-04 Thread Andres Freund
Hi, > It's impossible to write a general purpose hash table that will be > suitable for every use case, considering all the combinations of > design trade offs and subsets of functionality someone might want. Very much agreed. > There is other work being done in this space: I'm aware of Andres

[HACKERS] Hash tables in dynamic shared memory

2016-10-04 Thread Thomas Munro
Hi hackers, Here is an experimental hash table implementation that uses DSA memory so that hash tables can be shared by multiple backends and yet grow dynamically. Development name: "DHT". It's impossible to write a general purpose hash table that will be suitable for every use case, considerin