Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2014-11-10 Thread Jeff Janes
t, issue spotted by the isolation tests integrated in the patch. > - Isolation tests for reindex concurrently are re-enabled by default. > Regards, > It looks like this needs another rebase, I get failures on index.c, toasting.c, indexcmds.c, and index.h Thanks, Jeff

Re: [HACKERS] pgcrypto: PGP signatures

2014-11-11 Thread Jeff Janes
ms to be related but different. Cheers, Jeff pgcrypto3.pl Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-11 Thread Jeff Davis
tly with an extra field in the lock tag, and perhaps some catalog knowledge? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-12 Thread Jeff Davis
, it would see: (A1 A2) -> B -> (A1 A2) which is a cycle, and can be detected regardless of the synchronization method used between A1 and A2. There are some details to work out to avoid false positives, of course. Is that about right? Regards, Jeff Davis -- Sent via pgsql-hackers m

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Jeff Davis
Exclusion Constraints, I went to a lot of effort to make deadlocks impossible, and was quite proud. When Tom saw it, he told me not to bother, and to do it the simple way instead, because deadlocks can happen even with UNIQUE constraints (which I didn't even know). We should use the sam

Re: [HACKERS] controlling psql's use of the pager a bit more

2014-11-13 Thread Jeff Janes
en other things get screwy once you set that to a non-actual value. I don't recall of the top of my head which things those are, though. Cheers, Jeff

Re: [HACKERS] controlling psql's use of the pager a bit more

2014-11-13 Thread Jeff Janes
27;d be useful to force the > pager if the result is wider than the screen. > For which formats do you want that? It is already done that way under \pset format aligned. Cheers, Jeff

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Jeff Davis
On Thu, Nov 13, 2014 at 11:26 AM, Robert Haas wrote: > > On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis wrote: > > If two backends both have an exclusive lock on the relation for a join > > operation, that implies that they need to do their own synchronization, > > be

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-11-16 Thread Jeff Davis
which would be one less context to visit. And if those don't work, perhaps I could resort to a sampling method of some kind, as you allude to above. Regards, Jeff Davis [1] I'm fairly sure I tested something very similar on Robert's POWER machine a while ago, and it was

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-11-17 Thread Jeff Davis
ve isReset there, and keep mem_allocated as a uint64. That way, if I find later that I want to track the aggregated value for the child contexts as well, I can split it into two uint32s. I'll hold off any any such optimizations until I see some numbers from HashAgg though. Attached new versio

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-18 Thread Jeff Davis
d I think we'll make a better decision about the exact form it takes. In other words: lock groups is important, but I don't see the rush for lock sharing specifically. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] pg_test_fsync file descriptor leak

2014-11-18 Thread Jeff Janes
unlink from working, as the file is still open. Trivial fix attached. Thanks, Jeff pg_test_fsync_leak.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Jeff Davis
argue that we know we're headed for this problem, and therefore we should solve it now. I disagree. You are assuming that sharing exclusive heavyweight locks among a group will be a fundamental part of everything postgres does with parallelism; but not every design requires it. Regards,

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Jeff Davis
cessShare lock, and A2 tries to acquire an exclusive lock. B is waiting on A2. That's still a deadlock, right? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] 9.2 recovery/startup problems

2014-11-25 Thread Jeff Janes
ot. I can't distribute a tarball of this particular database. How would I go about debugging this? Should I track down the source of the FATAL and convert it to a PANIC so I can get a core dump to look at? A second attempt to start up the server completes successfully. Cheers, Jeff

Re: [HACKERS] 9.2 recovery/startup problems

2014-11-26 Thread Jeff Janes
On Wed, Nov 26, 2014 at 5:06 AM, Alvaro Herrera wrote: > Jeff Janes wrote: > > > This is what I get in the log from the attempted restart: > > > > PST LOG: database system was interrupted; last known up at 2014-11-25 > > 15:40:33 PST > > PST LOG: databa

Re: [HACKERS] 9.2 recovery/startup problems

2014-11-26 Thread Jeff Janes
On Tue, Nov 25, 2014 at 9:30 PM, Jeff Janes wrote: > Using both 9.2.9 and 9_2_STABLE 9b468bcec15f1ea7433d4, I get a fairly > reproducible startup failure. > > What I was doing is restore a database from a base backup and roll it > forward with a recovery.conf until it complete

Re: [HACKERS] Parallel Seq Scan

2015-11-16 Thread Jeff Janes
On Sat, Nov 14, 2015 at 10:12 PM, Amit Kapila wrote: > On Fri, Nov 13, 2015 at 11:05 PM, Jeff Janes wrote: >> >> On Wed, Nov 11, 2015 at 6:53 AM, Robert Haas >> wrote: >> > >> > I've committed most of this, except for some planner bits that I &g

Re: [HACKERS] Erroneous cost estimation for nested loop join

2015-11-16 Thread Jeff Janes
worry about "double-counting the cache effects" comes from. The effective_cache_size and io read-ahead can both apply and both give benefits simultaneously and cumulatively. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] check for interrupts in set_rtable_names

2015-11-16 Thread Jeff Janes
might as > well back-patch it in toto. Comments? As-committed it has solved the problem, as best as I can tell. Thanks, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Tab completion for ALTER COLUMN SET STATISTICS

2015-11-17 Thread Jeff Janes
On Mon, Sep 28, 2015 at 8:48 AM, Robert Haas wrote: > On Sat, Sep 26, 2015 at 7:24 AM, Michael Paquier > wrote: >> On Sat, Sep 26, 2015 at 7:18 AM, Jeff Janes wrote: >>> If I have "alter table foo alter COLUMN bar SET STATISTICS" in the line >>> buffer,

Re: [HACKERS] Using quicksort for every external sort run

2015-11-18 Thread Jeff Janes
eed up quite a bit less, only 22%. But 22% is still very worthwhile, and who wants to spend their time fine-tuning the memory use for every index build? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-18 Thread Jeff Janes
r elsewhere) against the impossible state of being all frozen but not all visible? I get an error when running pg_upgrade from 9.4 to 9.6-this error while copying relation "mediawiki.archive" ("/tmp/data/base/16414/21043_vm" to "/tmp/data_fm/base/16400/21043_vm"): No

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-18 Thread Jeff Janes
On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: > > I get an error when running pg_upgrade from 9.4 to 9.6-this > > error while copying relation "mediawiki.archive" > ("/tmp/data/base/16414/21043_vm" to > "/tmp/data_fm/base/16400/21043_vm"): No

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-20 Thread Jeff Janes
On Thu, Nov 19, 2015 at 6:44 AM, Masahiko Sawada wrote: > On Thu, Nov 19, 2015 at 5:54 AM, Jeff Janes wrote: >> On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: >>> >>> I get an error when running pg_upgrade from 9.4 to 9.6-this >>> >>> er

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-22 Thread Jeff Janes
recomputed on both new pages. It looks like there is no precedence for doing that in pg_upgrade so this will be breaking new ground. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN pending list clean up exposure to SQL

2015-11-25 Thread Jeff Janes
On Thu, Nov 19, 2015 at 8:37 AM, Jaime Casanova wrote: > On 12 August 2015 at 20:19, Jeff Janes wrote: >> >> But where does this belong? Core? Its own separate extension? >> > > I will say core. Gin indexes are in core and i don't see why this > function sh

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2015-11-28 Thread Jeff Janes
s, so it is not possible to use such a proc in group status > updation as each group member needs to wait on semaphore. It is not tad > difficult to add the support for that case if we are okay with creating > additional > semaphore for each such dummy proc which I was not sure, so I

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-28 Thread Jeff Janes
issue on Windows. I've fixed (I think) the conflict in order to do some testing, but I'd like to get an updated patch from the author in case I did it wrong. I don't want to find bugs that I just introduced myself. Thanks, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2015-11-28 Thread Jeff Janes
On Wed, Nov 18, 2015 at 3:29 PM, Peter Geoghegan wrote: > On Wed, Nov 18, 2015 at 10:31 AM, Jeff Janes wrote: > >> I agree we don't want to optimize for low memory, but I don't think we >> should throw it under the bus, either. Right now we are effectively >>

Re: [HACKERS] Using quicksort for every external sort run

2015-11-28 Thread Jeff Janes
e data. If we only go slightly above the limits, it is much more graceful. It will happily do a 3 way merge followed by a 1023 way final merge (or something like that) so only 0.3 percent of the data needs a second pass, not all of it. If course by the time you get a factor of 2 over the limit

Re: [HACKERS] Using quicksort for every external sort run

2015-11-30 Thread Jeff Janes
On Sat, Nov 28, 2015 at 4:05 PM, Peter Geoghegan wrote: > On Sat, Nov 28, 2015 at 2:04 PM, Jeff Janes wrote: ... >> >> The final merging is intermixed with whatever other work goes on to >> build the actual index files out of the sorted data, so I don't know >>

Re: [HACKERS] Using quicksort for every external sort run

2015-11-30 Thread Jeff Janes
application_name and IP address instead of ROLE. I have no idea how I would implement that, it is just how I would like to use it as the end user. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Jeff Janes
On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada wrote: > On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >> On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada >> wrote: >>> >>> Yeah, we need to consider to compute checksum if enabled. >>> I'

Re: [HACKERS] 9.5Beta1 psql wrapped format expanded output

2015-11-30 Thread Jeff Janes
On Mon, Nov 30, 2015 at 2:59 PM, Tom Lane wrote: > Jeff Janes writes: >> On Fri, Oct 23, 2015 at 5:11 PM, Jeff Janes wrote: >>> Why swidth for border 2 is three greater than it is with border 1, I >>> don't really know. > >> Now I see why. Border 2 do

Fwd: [HACKERS] Another little thing about psql wrapped expanded output

2015-12-02 Thread Jeff Janes
no way of knowing that the dashes are separators and not actual data, the pager can't reasonably do a good job of that. For those who can't follow my hand-waving, try this: psql -c 'select * from pg_stats' -x ​ Cheers, Jeff

Re: Fwd: [HACKERS] Another little thing about psql wrapped expanded output

2015-12-03 Thread Jeff Janes
at times. Now, if I could just find a way to tell 'less' after-the-fact "pretend I started you up with the -X flag". Once I find the part I want, I want to quit the pager without clearing that stuff off of the screen. Sometimes. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-04 Thread Jeff Janes
On Tue, Dec 1, 2015 at 10:40 AM, Masahiko Sawada wrote: > On Tue, Dec 1, 2015 at 3:04 AM, Jeff Janes wrote: >> On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada >> wrote: >>> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >>>> On Tue, Nov 24, 2015 at 3

Re: [HACKERS] Using quicksort for every external sort run

2015-12-06 Thread Jeff Janes
set, each distinct value shows up a couple hundred times on average. I'm thinking of converting it to a btree_gin index once I've tested them a bit more, as the compression benefits are substantial. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2015-12-07 Thread Jeff Janes
land on a pre-existing infelicity, how big of a deal is that? It wouldn't be creating a regression, just shoving the region that experiences the problem around in such a way that it affects a different group of use cases. And perhaps more importantly, can anyone else reproduce th

Re: [HACKERS] Using quicksort for every external sort run

2015-12-08 Thread Jeff Janes
On Mon, Dec 7, 2015 at 9:01 AM, Jeff Janes wrote: > > The patched one ends with a 2-way, two sequential 233-way merges, and > a final 233-way merge: > > LOG: finished 2-way merge step: CPU 62.08s/435.70u sec elapsed 587.52 sec > LOG: finished 233-way merge step: CPU 77.94s/6

Re: [HACKERS] On columnar storage (2)

2015-12-08 Thread Jeff Janes
Could we get this rebased past the merge of the parallel execution commits? Thanks, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Speedup twophase transactions

2015-12-09 Thread Jeff Janes
; PREPARE TRANSACTION ':client_id.:scale'; > COMMIT PREPARED ':client_id.:scale'; > Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Cluster "stuck" in "not accepting commands to avoid wraparound data loss"

2015-12-11 Thread Jeff Janes
says "Nope, not me". Since with default settings each database/table gets frozen 10 times per real wrap-around, this is usually not going to be a problem as having 10 consecutive well timed crashes is very unlikely. But if you increase autovacuum_freeze_max_age a lot, or if the freeze s

Re: [HACKERS] Using quicksort for every external sort run

2015-12-12 Thread Jeff Janes
mat. This inflates the size quite a bit, at least for single-datum tuples. If we instead just read the disk format directly into the pool, and converted them into the in-memory format when each tuple came due for the merge heap, would that destroy the locality of reference you are seeking to gain? C

Re: [HACKERS] Using quicksort for every external sort run

2015-12-12 Thread Jeff Janes
On Sat, Dec 12, 2015 at 2:28 PM, Peter Geoghegan wrote: > On Sat, Dec 12, 2015 at 12:10 AM, Jeff Janes wrote: >> I have a question about the terminology used in this patch. What is a >> tuple proper? What is it in contradistinction to? I would think that >> a tuple which

Re: [HACKERS] Should TIDs be typbyval = FLOAT8PASSBYVAL to speed up CREATE INDEX CONCURRENTLY?

2015-12-12 Thread Jeff Janes
most of the access is just > reads, though those reads may have been part of a larger transaction that > did updates elsewhere. That sounds like the CIC is just blocked waiting for long-lived transactions to go away. There isn't much that changes to the sorting algorithm can do about tha

Re: [HACKERS] Using quicksort for every external sort run

2015-12-13 Thread Jeff Janes
On Tue, Dec 8, 2015 at 6:39 PM, Greg Stark wrote: > On Wed, Dec 9, 2015 at 12:02 AM, Jeff Janes wrote: >> >> >> Then in the next (final) merge, it is has to read in this huge >> fragmented tape run emulation, generating a lot of random IO to read >> it. > >

Re: [HACKERS] Using quicksort for every external sort run

2015-12-13 Thread Jeff Janes
On Sun, Dec 13, 2015 at 3:40 PM, Peter Geoghegan wrote: > On Sat, Dec 12, 2015 at 4:41 PM, Jeff Janes wrote: > >>>> Also, if I am reading this correctly, when we refill a pool from a >>>> logical tape we still transform each tuple as it is read from the disk &g

Re: [HACKERS] Disabling an index temporarily

2015-12-13 Thread Jeff Janes
n, but I think what Corey really wants here is to stop maintaining the index at retail while preserving the existing definition and existing index data, and then to do a wholesale fix-up, like what is done in the 2nd half of a create index concurrently, upon re-enabling it. Cheers, Jeff -- Sent

Re: [HACKERS] Disabling an index temporarily

2015-12-15 Thread Jeff Janes
entire table or an entire schema or an entire database, I certainly wouldn't object. I am not a masochist. But also, while loading 1.5 million records into a table with 250 million records is horribly, rebuilding all the indexes on a 251.5 million record table from scratch is even more horr

Fwd: [HACKERS] Cluster "stuck" in "not accepting commands to avoid wraparound data loss"

2015-12-17 Thread Jeff Janes
Sorry, accidentally failed to include the list originally, here it is for the list: On Dec 16, 2015 9:52 AM, "Robert Haas" wrote: > > On Fri, Dec 11, 2015 at 1:08 PM, Jeff Janes wrote: > > Since changes to datfrozenxid are WAL logged at the time they occur, > > bu

Re: [HACKERS] Cluster "stuck" in "not accepting commands to avoid wraparound data loss"

2015-12-17 Thread Jeff Janes
template1 get frozen with xid which were 5 past the xidStopLimit? Is that part of the mystery here, or is that normal? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Tab completion for ALTER COLUMN SET STATISTICS

2015-12-20 Thread Jeff Janes
ected. Thanks, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-21 Thread Jeff Janes
On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra wrote: > Hi, > > On 11/06/2015 02:09 AM, Tomas Vondra wrote: >> >> Hi, >> >> On 11/06/2015 01:05 AM, Jeff Janes wrote: >>> >>> On Thu, Nov 5, 2015 at 3:50 PM, Tomas Vondra >>> wrote: &g

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-23 Thread Jeff Janes
On Mon, Dec 21, 2015 at 11:51 AM, Tomas Vondra wrote: > > > On 12/21/2015 07:41 PM, Jeff Janes wrote: >> >> On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra >> wrote: > > > ... > >>> So both patches seem to do the trick, but (2) is faster. Not sure &

Re: [HACKERS] Using quicksort for every external sort run

2015-12-23 Thread Jeff Janes
nchmark the patch! > > While I think it's perfectly fair that you didn't apply the final > on-the-fly merge "memory pool" patch, I also think that it's quite > possible that the regression you see at the very low end would be > significantly ameliorated or even

Re: [HACKERS] COPY FREEZE and PD_ALL_VISIBLE

2015-12-23 Thread Jeff Janes
building the map as I go, in the latest version of this patch I'm >>> working on. >>> >> >> As, you are working on this patch, I have marked it as "Waiting on Author". > > There has not been an update for this patch in a while, switched to

Re: [HACKERS] Spurious standby query cancellations

2015-12-23 Thread Jeff Janes
On Wed, Sep 23, 2015 at 11:33 PM, Jeff Janes wrote: > > On further thought, neither do I. The attached patch inverts > ResolveRecoveryConflictWithLock to be called back from the lmgr code so that > is it like ResolveRecoveryConflictWithBufferPin code. It does not try to > cancel

Re: [HACKERS] Spurious standby query cancellations

2015-12-24 Thread Jeff Janes
On Wed, Dec 23, 2015 at 9:40 PM, Jeff Janes wrote: > On Wed, Sep 23, 2015 at 11:33 PM, Jeff Janes wrote: >> >> On further thought, neither do I. The attached patch inverts >> ResolveRecoveryConflictWithLock to be called back from the lmgr code

[HACKERS] 9.5rc1 brin_summarize_new_values

2015-12-25 Thread Jeff Janes
brin_summarize_new_values() does not check that it is called on the correct type of index, nor does it check permissions. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.5rc1 brin_summarize_new_values

2015-12-26 Thread Jeff Janes
On Sat, Dec 26, 2015 at 8:27 AM, Tom Lane wrote: > Michael Paquier writes: >> On Sat, Dec 26, 2015 at 7:10 AM, Jeff Janes wrote: >>> brin_summarize_new_values() does not check that it is called on the >>> correct type of index, nor does it check permissions. >

Re: [HACKERS]WIP: Covering + unique indexes.

2015-12-26 Thread Jeff Janes
shotData SnapshotDirty; ^ And the dblink contrib module fails its make check. I'm trying to find a good test case for it. Unfortunately in most of my natural use cases, the inclusion of the extra column causes the updates to become non-HOT, which causes more problems than it solves. Cheers,

Re: [HACKERS]WIP: Covering + unique indexes.

2015-12-26 Thread Jeff Janes
On Sat, Dec 26, 2015 at 5:58 PM, Jeff Janes wrote: > > And the dblink contrib module fails its make check. Ignore the dblink complaint. It seems to have been some wonky build issue that is not reproducible. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make c

Re: [HACKERS] Fuzzy substring searching with the pg_trgm extension

2015-12-26 Thread Jeff Janes
der the thread "Patch: pg_trgm: gin index scan performance for similarity search"? They conflict with your changes. Thanks, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Check for interrupts in bf and xdes crypt()

2015-12-27 Thread Jeff Janes
t from this. Looks good to me. Applies, builds, passes make check, does what it says and says what it does. No need for docs, no noticeable performance impact. I've marked it ready for committer. Also recommend for back-patching. Cheers, Jeff -- Sent via pgsql-hackers mailin

[HACKERS] Avoid endless futile table locks in vacuuming.

2015-12-27 Thread Jeff Janes
to scan the last page only if truncation is eminent. But those seem like needless complications to try to avoid sometimes scanning one block. Cheers, Jeff diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c new file mode 100644 index 2429889..abc2e28 *** a/src

Re: [HACKERS] GIN pending list clean up exposure to SQL

2015-12-28 Thread Jeff Janes
On Wed, Nov 25, 2015 at 9:29 AM, Jeff Janes wrote: > > I'll prepare a patch for core for the January commitfest, and see if > it flies. If not, there is always the extension to fall back to. Here is an updated patch. I've added type and permission checks, docs, and so

Re: [HACKERS] Avoid endless futile table locks in vacuuming.

2015-12-29 Thread Jeff Janes
On Mon, Dec 28, 2015 at 2:12 PM, Tom Lane wrote: > I wrote: >> Jeff Janes writes: >>> If a partially-active table develops a slug of stable all-visible, >>> non-empty pages at the end of it, then every autovacuum of that table >>> will skip the end pages on t

Fwd: [HACKERS] Avoid endless futile table locks in vacuuming.

2015-12-30 Thread Jeff Janes
Forgetting to CC the list is starting to become a bad habit, sorry. forwarding to list: -- Forwarded message -- From: Jeff Janes Date: Wed, Dec 30, 2015 at 10:03 AM Subject: Re: [HACKERS] Avoid endless futile table locks in vacuuming. To: Tom Lane On Dec 29, 2015 4:47 PM, &quo

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-06 Thread Jeff Janes
1 row) > > This index, as we've defined it should not be able to satisfy the query's > order by, although it does give correct results, that's because the index > seems to be built wrongly in cases where the rows are added after the index > exists. I think this just caus

Re: [HACKERS] No Issue Tracker - Say it Ain't So!

2016-01-07 Thread Jeff Janes
ginner can do them instead. It would need to be separate from a "we can't decide if we want this or can't decide how to do it or don't have the time to do it" list. There is no reason we have to have an issue tracker in order to create that separation, but it could help.

Re: [HACKERS] Do we need so many hint bits?

2012-11-15 Thread Jeff Davis
ontention for access to the visibility map pages would be a > problem. Can you think of a reasonable (or worst-case) test scenario that would illustrate this problem? Would I need a many-core box to actually test this? Also, what locks do you expect to be contended? The buffer header lock while pin

Re: [HACKERS] [PATCH] binary heap implementation

2012-11-15 Thread Jeff Davis
pretty ugly. Why not pass the allocator in as a function pointer? It could either be an argument to the initialization, or we could make a new global variable that's present inside the server and outside. (Slight complication: palloc is a macro) Regards, Jeff Davis -- Sent via pgsql

Re: [HACKERS] Do we need so many hint bits?

2012-11-16 Thread Jeff Davis
done in the visibility check. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Do we need so many hint bits?

2012-11-16 Thread Jeff Davis
XMAX_INVALID is set, then it > > should just be changed to InvalidTransactionId. > > However, that solution only works for invalid-xmin. For > committed-xmax, there could actually be quite a long time before the > page can be pruned, because there can be some other backend holding an

Re: [HACKERS] Do we need so many hint bits?

2012-11-16 Thread Jeff Janes
On Thu, Nov 15, 2012 at 4:42 PM, Jeff Davis wrote: > > Also, I am wondering about PD_ALL_VISIBLE. It was originally introduced > in the visibility map patch, apparently as a way to know when to clear > the VM bit when doing an update. It was then also used for scans, which > showe

Re: [HACKERS] Do we need so many hint bits?

2012-11-16 Thread Jeff Davis
ge patch, to do away with PD_ALL_VISIBLE without a lot of performance risk. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Do we need so many hint bits?

2012-11-16 Thread Jeff Davis
On Fri, 2012-11-16 at 17:04 -0800, Jeff Janes wrote: > On Thu, Nov 15, 2012 at 4:42 PM, Jeff Davis wrote: > > > > Also, I am wondering about PD_ALL_VISIBLE. It was originally introduced > > in the visibility map patch, apparently as a way to know when to clear > > the

Re: [HACKERS] Do we need so many hint bits?

2012-11-17 Thread Jeff Davis
he VM, hopefully using a buffer that we already have a pin on. I haven't really dug into this yet, but I don't see any obvious problem. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.pos

Re: [HACKERS] Do we need so many hint bits?

2012-11-17 Thread Jeff Davis
On Sat, 2012-11-17 at 16:53 -0500, Tom Lane wrote: > Jeff Davis writes: > > What's the problem with that? If you already have the VM buffer pinned > > (which should be possible if we keep the VM buffer in a longer-lived > > structure), then doing the test is a

Re: [HACKERS] Do we need so many hint bits?

2012-11-17 Thread Jeff Davis
setting hints > on a SeqScan then we can skip individual tuple hints also. If the > whole page becomes visible then we can set the VM. Hmm, that's an idea. Maybe we shouldn't bother setting the hints if it's already all-visible in the VM? Something else to think about

Re: [HACKERS] Do we need so many hint bits?

2012-11-17 Thread Jeff Janes
On Fri, Nov 16, 2012 at 5:35 PM, Jeff Davis wrote: > On Fri, 2012-11-16 at 17:04 -0800, Jeff Janes wrote: > >> Your question prompts me to post something I had been wondering. >> Might it be worthwhile to break the PD_ALL_VISIBLE / vm equivalence? >> Should the vm bit get

Re: [HACKERS] Do we need so many hint bits?

2012-11-18 Thread Jeff Davis
On Sun, 2012-11-18 at 15:19 +0100, Andres Freund wrote: > On Sunday, November 18, 2012 03:07:01 AM Jeff Davis wrote: > > Process A (process that clears a VM bit for a data page): > > 1. Acquires exclusive lock on data buffer > > 2. Acquires exclusive lock on VM buffer &

Re: [HACKERS] Enabling Checksums

2012-11-18 Thread Jeff Davis
On Wed, 2012-11-14 at 17:40 -0800, Jeff Davis wrote: > I'll do another pass to make sure I update all of the comments, and try > to self review it. Updated patches attached (the TLI patch wasn't changed though, only the main checksums patch). Changes: * A lot of cleanup

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
On Sun, Oct 21, 2012 at 12:59 AM, Amit kapila wrote: > On Saturday, October 20, 2012 11:03 PM Jeff Janes wrote: > >>Run the modes in reciprocating order? > Sorry, I didn't understood this, What do you mean by modes in reciprocating > order? Sorry for the long delay. In

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-11-18 Thread Jeff Janes
early, it could be expected to increase the amount of IO). So this also argues that the increase in performance is caused by the decrease in IO, but the patch isn't causing that decrease, it merely benefits from it due to an accident of timing. Cheers, Jeff -- Sent via pgsql

Re: [HACKERS] Enabling Checksums

2012-11-19 Thread Jeff Davis
ans that even an online background checking utility would be quite hard to do properly. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2012-11-19 Thread Jeff Davis
? But of course, that's a problem because a corrupt block might have the wrong LSN (in fact, it's likely, because garbage is more likely to make the LSN too high than too low). Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Do we need so many hint bits?

2012-11-19 Thread Jeff Davis
void taking new locks (or mutexes) during a read of the VM bit, because there is concern that it could cause contention. If we lock the entire VM page, that represents many, many data pages, so it's worrisome. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] Do we need so many hint bits?

2012-11-19 Thread Jeff Davis
> great idea after all... As I said elsewhere in the thread, I'm not planning to introduce any additional locking. There is already precedent in IndexOnlyNext. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Pg_upgrade speed for many tables

2012-11-19 Thread Jeff Janes
On Wed, Nov 14, 2012 at 3:55 PM, Bruce Momjian wrote: > On Mon, Nov 12, 2012 at 10:29:39AM -0800, Jeff Janes wrote: >> >> Is turning off synchronous_commit enough? What about turning off fsync? > > I did some testing with the attached patch on a magnetic disk with no >

Re: [HACKERS] Enabling Checksums

2012-11-19 Thread Jeff Davis
On Mon, 2012-11-19 at 10:35 -0800, Jeff Davis wrote: > Yes, the blocks written *after* the checkpoint might have a bad checksum > that will be fixed during recovery. But the blocks written *before* the > checkpoint should have a valid checksum, but if they don't, then > recovery d

Re: [HACKERS] StrategyGetBuffer questions

2012-11-20 Thread Jeff Janes
think it is all that big of a deal. I've implemented this patch to do that. It still applies to head. http://archives.postgresql.org/pgsql-hackers/2011-08/msg00305.php It was very effective at removing BufFreelistLock contention on the system I had at the time. Cheers, Jeff --

Re: [HACKERS] PANIC: could not write to log file

2012-11-21 Thread Jeff Janes
. How was the upgrade done? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] add -Wlogical-op to standard compiler options?

2012-11-21 Thread Jeff Janes
time. \ */ \ MEMSET_LOOP_LIMIT != 0) \ Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Do we need so many hint bits?

2012-11-21 Thread Jeff Davis
On Mon, 2012-11-19 at 14:46 -0500, Tom Lane wrote: > Jeff Davis writes: > > As I said elsewhere in the thread, I'm not planning to introduce any > > additional locking. There is already precedent in IndexOnlyNext. > > Of course, that just begs the question of whether t

Re: [HACKERS] tuplesort memory usage: grow_memtuples

2012-11-21 Thread Jeff Janes
On Thu, Nov 15, 2012 at 11:16 AM, Robert Haas wrote: > > So what's next here? Do you want to work on these issue some more? > Or does Jeff? This has been rewritten enough that I no longer feel much ownership of it. I'd prefer to leave it to Peter or Greg S., if they a

Re: [HACKERS] [PATCH 03/14] Add simple xlogdump tool

2012-11-21 Thread Jeff Janes
On Thu, Nov 15, 2012 at 9:13 AM, Andres Freund wrote: > On 2012-11-15 09:06:23 -0800, Jeff Janes wrote: >> On Wed, Nov 14, 2012 at 5:17 PM, Andres Freund >> wrote: >> > --- >> > src/bin/Makefile| 2 +- >> > src/bin/xlogdump/Makefile |

[HACKERS] Removing PD_ALL_VISIBLE

2012-11-21 Thread Jeff Davis
past 11/15. Regards, Jeff Davis rm-pd-all-visible-20121121.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

<    5   6   7   8   9   10   11   12   13   14   >