Re: d25ea01275 and partitionwise join

2019-10-13 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 03:02:17PM -0500, Justin Pryzby wrote: > On Thu, Sep 19, 2019 at 05:15:37PM +0900, Amit Langote wrote: > > Please find attached updated patches. > > Tom pointed me to this thread, since we hit it in 12.0 > https://www.postgresql.org/message-id/fla

Re: v12.0: segfault in reindex CONCURRENTLY

2019-10-14 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 06:06:43PM +0900, Michael Paquier wrote: > On Fri, Oct 11, 2019 at 07:44:46PM -0500, Justin Pryzby wrote: > > Unfortunately, there was no core file, and I'm still trying to reproduce it. > > Forgot to set ulimit -c? Having a backtrace would surel

Re: d25ea01275 and partitionwise join

2019-10-14 Thread Justin Pryzby
ut then it crashes in check-world (possibly due to misapplied hunks). -- Justin Pryzby System Administrator Telsasoft +1-952-707-8581

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-14 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 01:30:29PM -0500, Justin Pryzby wrote: > BTW it probably should've been documented as an "Open Item" for v12. https://commitfest.postgresql.org/25/2278/ I realized possibly people were thinking of that as a "feature" and not a bugfix for backp

Re: v12.0: segfault in reindex CONCURRENTLY

2019-10-14 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 03:10:21PM -0300, Alvaro Herrera wrote: > On 2019-Oct-13, Justin Pryzby wrote: > > > Looks like it's a race condition and dereferencing *holder=NULL. The first > > crash was probably the same bug, due to report query running during "reindex >

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-14 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 02:06:02PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Fri, Oct 11, 2019 at 10:48:37AM -0400, Tom Lane wrote: > >> Could you provide a self-contained test case please? > > > [ test case ] > > Oh, this is the same is

v12.0: interrupt reindex CONCURRENTLY: ccold: ERROR: could not find tuple for parent of relation ...

2019-10-15 Thread Justin Pryzby
On a badly-overloaded VM, we hit the previously-reported segfault in progress reporting. This left around some *ccold indices. I tried to drop them but: sentinel=# DROP INDEX child.alarms_null_alarm_id_idx1_ccold; -- child.alarms_null_alarm_time_idx_ccold; -- alarms_null_alarm_id_idx_ccold; ERR

Re: v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-15 Thread Justin Pryzby
On Tue, Oct 15, 2019 at 01:50:09PM -0700, Andres Freund wrote: > On 2019-10-13 07:51:06 -0700, Andres Freund wrote: > > On 2019-10-11 16:03:20 -0500, Justin Pryzby wrote: > > > ts=# CLUSTER huawei_m2000_config_enodebcell_enodeb USING > > > huawei_m2000_config_eno

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Justin Pryzby
This reminds me of an issue I reported several years ago where Btree index scans were chosen over seq scan of a large, INSERT-only table due to very high correlation, but performed poorly. I concluded that use of the the high "large scale" correlation on a large 50+GB table caused the planner to f

Re: v12 and pg_restore -f-

2019-10-16 Thread Justin Pryzby
On Sun, Oct 06, 2019 at 04:43:13PM -0400, Tom Lane wrote: > Nobody is going to wish that to mean "write to a file named '-'", so Probably right, but it occurs to me that someone could make a named pipe called that, possibly to make "dump to stdout" work with scripts that don't support dumping to s

Re: v12 and pg_restore -f-

2019-10-16 Thread Justin Pryzby
On Wed, Oct 16, 2019 at 03:04:52PM -0400, Stephen Frost wrote: > > On Wed, Oct 16, 2019 at 01:21:48PM -0400, Stephen Frost wrote: > > > [...] if they actually need to work with both concurrently (which strikes > > > me > > > as already at least relatively uncommon...). > > > > I doubt it's uncom

Re: v12.0: segfault in reindex CONCURRENTLY

2019-10-16 Thread Justin Pryzby
On Sun, Oct 13, 2019 at 04:18:34PM -0300, Alvaro Herrera wrote: > (FWIW I expect the crash is possible not just in reindex but also in > CREATE INDEX CONCURRENTLY.) FWIW, for sake of list archives, and for anyone running v12 hoping to avoid crashing, I believe we hit this for DROP INDEX CONCURRENT

Re: v12 and pg_restore -f-

2019-10-17 Thread Justin Pryzby
ossible. Also, I'm taking the opportunity to correct myself, before someone else does: On Wed, Oct 16, 2019 at 02:28:40PM -0500, Justin Pryzby wrote: > And vendors (something like pgadmin) will end up "having to" write to a file > to be portable, or else check the full version,

Re: v12.0: reindex CONCURRENTLY: lock ShareUpdateExclusiveLock on object 14185/39327/0 is already held

2019-10-18 Thread Justin Pryzby
Checking if anybody is working on either of these https://www.postgresql.org/message-id/20191013025145.GC4475%40telsasoft.com https://www.postgresql.org/message-id/20191015164047.GA22729%40telsasoft.com On Sat, Oct 12, 2019 at 09:51:45PM -0500, Justin Pryzby wrote: > I ran into this while try

Re: stress test for parallel workers

2019-10-22 Thread Justin Pryzby
8:01.526 EDT [5962] LOG: database system is ready to accept connections On Tue, Jul 23, 2019 at 11:27:03AM -0500, Justin Pryzby wrote: > Does anyone have a stress test for parallel workers ? > > On a customer's new VM, I got this several times while (trying to) migrate > their

Re: v12.0: reindex CONCURRENTLY: lock ShareUpdateExclusiveLock on object 14185/39327/0 is already held

2019-10-23 Thread Justin Pryzby
On Thu, Oct 24, 2019 at 11:42:04AM +0900, Michael Paquier wrote: > Please see the attached. Justin, does it fix your problems regarding > the locks? Confirmed. Thanks, Justin

Re: query logging of prepared statements

2019-03-04 Thread Justin Pryzby
On Mon, Mar 04, 2019 at 06:53:31PM +0300, Arthur Zakirov wrote: > Hello Justin, > > On 27.02.2019 21:06, Justin Pryzby wrote: > >I'm attaching a v2 patch which avoids repeated logging of PREPARE, rather > >than > >making such logs conditional on lo

Re: Should we add GUCs to allow partition pruning to be disabled?

2019-03-10 Thread Justin Pryzby
On Sun, Mar 10, 2019 at 10:53:02PM +1300, David Rowley wrote: > On Fri, 11 May 2018 at 17:37, Amit Langote > wrote: > > 5. The last sentence in caveats, that is, > > > > "Partitioning using these techniques will work well with up to perhaps a > > hundred partitions; don't try to use many thousand

Re: Should we add GUCs to allow partition pruning to be disabled?

2019-03-10 Thread Justin Pryzby
On Mon, Mar 11, 2019 at 01:06:08PM +0900, Amit Langote wrote: > On 2019/03/11 11:13, David Rowley wrote: > > On Mon, 11 Mar 2019 at 15:00, David Rowley > > wrote: > >> On Mon, 11 Mar 2019 at 14:33, Amit Langote > >> wrote: > >>> PG 11 moved the needle a bit for SELECT queries: > >>> > >>> Exclu

Re: using index or check in ALTER TABLE SET NOT NULL

2019-03-13 Thread Justin Pryzby
Hi, On Wed, Mar 13, 2019 at 01:25:11PM -0400, Robert Haas wrote: > On Wed, Mar 13, 2019 at 1:19 PM Tom Lane wrote: > > Oh, and yes, I think QueuePartitionConstraintValidation's usage > > is an unacceptable abuse of INFO level. I'm surprised we haven't > > gotten complaints about it yet. > > Per

Re: Change ereport level for QueuePartitionConstraintValidation

2019-03-16 Thread Justin Pryzby
On Fri, Mar 15, 2019 at 12:55:36PM +0300, Sergei Kornilov wrote: > We have INFO ereport messages in alter table attach partition like this: > > partition constraint for table \"%s\" is implied by existing constraints > > So now I am +1 to idea of change error level for this messages. I attach > p

Re: Re: query logging of prepared statements

2019-03-20 Thread Justin Pryzby
Hi, On Wed, Mar 20, 2019 at 02:46:00PM +0400, David Steele wrote: > >I perfectly understand your use case. I agree, it is duplicated. But I > >think some people may want to see it at every EXECUTE, if they don't want > >to grep for the prepared statement body which was logged earlier. > > > >I thi

clean up docs for log_statement_sample_rate

2019-03-28 Thread Justin Pryzby
>From 15d42c5a8f2f811a7add3e4179edcc1f7cd291f7 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 28 Mar 2019 08:53:26 -0500 Subject: [PATCH v1] Clean up docs for log_statement_sample_rate.. ..which was added at commit 88bdbd3f746049834ae3cc972e6e650586ec3c9d --- doc/src/sgml/config.sgml

clean up pg_checksums.sgml

2019-03-29 Thread Justin Pryzby
f340e5384d59ab4fc3f3d0b4891a5b1c0 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 28 Mar 2019 19:20:52 -0500 Subject: [PATCH] Clean up pg_checksums.sgml --- doc/src/sgml/ref/pg_checksums.sgml | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/src

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Justin Pryzby
On Fri, Mar 29, 2019 at 03:53:05PM +, Bossart, Nathan wrote: > I noticed a very small typo in the documentation for this feature. I submit a bunch more changes for consideration, attached. >From dafdb15fb3e7c69de82a2206c9bf07588b5665ce Mon Sep 17 00:00:00 2001 From: Justin Pryzby Dat

Re: [HACKERS] generated columns

2019-03-30 Thread Justin Pryzby
On Sat, Mar 30, 2019 at 09:03:03AM +0100, Peter Eisentraut wrote: > On 2019-03-26 20:50, Pavel Stehule wrote: > > It is great feature and I'll mark this feature as ready for commit > > Committed, thanks. create_table.sgml now has this: https://www.postgresql.org/docs/devel/sql-createtable.html#i

clean up docs for v12

2019-03-30 Thread Justin Pryzby
ady mailed this one separately: |Clean up docs for log_statement_sample_rate.. https://www.postgresql.org/message-id/flat/20190328135918.GA27808%40telsasoft.com Justin >From fb712dfe3cb3d64ac7d297ca5217193327f8b547 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 28 Mar 2019 08:53:26

stress test for parallel workers

2019-07-23 Thread Justin Pryzby
Does anyone have a stress test for parallel workers ? On a customer's new VM, I got this several times while (trying to) migrate their DB: < 2019-07-23 10:33:51.552 CDT postgres >FATAL: postmaster exited during a parallel transaction < 2019-07-23 10:33:51.552 CDT postgres >STATEMENT: CREATE U

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Tue, Jul 23, 2019 at 01:28:47PM -0400, Tom Lane wrote: > ... you'd think an OOM kill would show up in the kernel log. > (Not necessarily in dmesg, though. Did you try syslog?) Nothing in /var/log/messages (nor dmesg ring). I enabled abrtd while trying to reproduce it last week. Since you ask

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 10:03:25AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby wrote: > > #2 0x0085ddff in errfinish (dummy=) at > > elog.c:555 > > edata = > > If you have that core, it might be interesting to go to fr

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 10:46:42AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 10:42 AM Justin Pryzby wrote: > > On Wed, Jul 24, 2019 at 10:03:25AM +1200, Thomas Munro wrote: > > > On Wed, Jul 24, 2019 at 5:42 AM Justin Pryzby > > > wrote: > > > &

Re: stress test for parallel workers

2019-07-23 Thread Justin Pryzby
On Wed, Jul 24, 2019 at 11:32:30AM +1200, Thomas Munro wrote: > On Wed, Jul 24, 2019 at 11:04 AM Justin Pryzby wrote: > > I ought to have remembered that it *was* in fact out of space this AM when > > this > > core was dumped (due to having not touched it since scheduling t

crash 11.5~

2019-08-07 Thread Justin Pryzby
A daily report crashed repeatedly this morning running pg11.4. I compiled 11.5 and reproduced it there, too, so I'm including backtrace with -O0. I'm trying to dig further into it, but it seems to be crashing under load, but not when I try to narrow down to a single report, which seem to run to co

Re: crash 11.5~ (and 11.4)

2019-08-07 Thread Justin Pryzby
I checked this still happens with max_parallel_workers_per_gather=0. Now, I just reproduced using SELECT * FROM that table: (gdb) p thisatt->attrelid $4 = 2015128626 ts=# SELECT 2015128626::regclass; regclass | child.huawei_umts_ucell_201908 (gdb) p thisatt->attnum $1 = 2 (gdb) p attnum # For e

Re: crash 11.5~ (and 11.4)

2019-08-07 Thread Justin Pryzby
Just found this, although I'm not sure what to do about it. If it's corrupt table data, I can restore from backup. ts=# VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908; INFO: 0: aggressively vacuuming "child.huawei_umts_ucell_201908" LOCATION: lazy_scan_heap, vacuumlazy.c:502 ERROR: X

Re: crash 11.5~ (and 11.4)

2019-08-07 Thread Justin Pryzby
On Wed, Aug 07, 2019 at 04:51:54PM -0700, Andres Freund wrote: > Hi, > > On 2019-08-07 14:29:28 -0500, Justin Pryzby wrote: > > Just found this, although I'm not sure what to do about it. If it's corrupt > > table data, I can restore from backup. In the meantim

Re: Zedstore - compressed in-core columnar storage

2019-08-18 Thread Justin Pryzby
On Thu, Aug 15, 2019 at 01:05:49PM +0300, Heikki Linnakangas wrote: > We've continued hacking on Zedstore, here's a new patch version against > current PostgreSQL master (commit f1bf619acdf). If you want to follow the > development in real-time, we're working on this branch: > https://github.com/gr

Re: Zedstore - compressed in-core columnar storage

2019-08-19 Thread Justin Pryzby
On Mon, Aug 19, 2019 at 04:15:30PM -0700, Alexandra Wang wrote: > On Sun, Aug 18, 2019 at 12:35 PM Justin Pryzby wrote: > > > . I was missing a way to check for compression ratio; > > Here are the ways to check compression ratio for zedstore: > > Table level: >

pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-24 Thread Justin Pryzby
Core was generated by `postgres: telsasoft ts 10.100.2.162(33500) SELECT '. Program terminated with signal 6, Aborted. #0 0x0039ff6325e5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Missing separate debuginf

Re: pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-25 Thread Justin Pryzby
e this report. Unfortunately that tells us > very little. > > On Sun, Aug 25, 2019 at 2:25 PM Justin Pryzby wrote: > > #4 0x0039ff678dd0 in _int_free (av=0x39ff98e120, p=0x1d40b090, > > have_lock=0) at malloc.c:4846 > > #5 0x006269e5 in ExecHashJoinNewB

Re: pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-25 Thread Justin Pryzby
I'm not sure but maybe this is useful ? |(gdb) p VfdCache[2397] |$9 = {fd = -1, fdstate = 0, resowner = 0x24f93e0, nextFree = 2393, lruMoreRecently = 0, lruLessRecently = 2360, seekPos = 73016512, fileSize = 0, fileName = 0x0, fileFlags = 2, fileMode = 384} Knowing this report, very possibly th

Re: pg11.5: ExecHashJoinNewBatch: glibc detected...double free or corruption (!prev)

2019-08-26 Thread Justin Pryzby
On Mon, Aug 26, 2019 at 12:45:24PM -0400, Tom Lane wrote: > However ... there is some pretty interesting info at > https://bugzilla.redhat.com/show_bug.cgi?id=1338673 > suggesting that compiling with a late-model gcc against older RHEL6 > headers could result in bad code. I wonder whether the repo

Re: Doc tweak for huge_pages?

2018-01-21 Thread Justin Pryzby
On Mon, Jan 22, 2018 at 03:54:26PM +1300, Thomas Munro wrote: > On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro > wrote: > > On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob > > wrote: > > I don't know enough about this to make such a strong recommendation > > myself, which is why I was only trying t

Re: Doc tweak for huge_pages?

2018-01-21 Thread Justin Pryzby
On Mon, Jan 22, 2018 at 07:10:33PM +1300, Thomas Munro wrote: > On Mon, Jan 22, 2018 at 6:30 PM, Justin Pryzby wrote: > > On Mon, Jan 22, 2018 at 03:54:26PM +1300, Thomas Munro wrote: > >> On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro > >> wrote: > >> >

Re: Doc tweak for huge_pages?

2018-01-23 Thread Justin Pryzby
On Wed, Jan 24, 2018 at 07:46:41AM +0100, Catalin Iacob wrote: > I see Peter assigned himself as committer, some more information below > for him to decide on the strength of the anti THP message. Thanks for digging this up! > And it would be good if somebody could run benchmarks on pre 4.6 and >

update ALTER TABLE with ATTACH PARTITION lock mode

2019-10-27 Thread Justin Pryzby
n the future.. >From c820a81fba0a6c2388ec58fc0204ca833523e81e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 27 Oct 2019 18:54:24 -0500 Subject: [PATCH v1 1/2] Mention reduced locking strength of ATTACH PARTITION.. See commit 898e5e32 --- doc/src/sgml/ref/alter_table.sgml | 7 +++ 1 file changed

Re: update ALTER TABLE with ATTACH PARTITION lock mode (docs)

2019-10-28 Thread Justin Pryzby
On Mon, Oct 28, 2019 at 12:06:44PM -0300, Alvaro Herrera wrote: > On 2019-Oct-28, Michael Paquier wrote: > > > On Sun, Oct 27, 2019 at 07:12:07PM -0500, Justin Pryzby wrote: > > > commit #898e5e32 (Allow ATTACH PARTITION with only > > > ShareUpdateExclusiveLock)

Re: update ALTER TABLE with ATTACH PARTITION lock mode (docs)

2019-11-01 Thread Justin Pryzby
On Thu, Oct 31, 2019 at 06:07:34PM +0900, Michael Paquier wrote: > On Mon, Oct 28, 2019 at 10:56:33PM -0500, Justin Pryzby wrote: > > I suppose it should something other than partition(ed), since partitions > > can be > > partitioned, too... > > > > Attac

Re: update ALTER TABLE with ATTACH PARTITION lock mode (docs)

2019-11-01 Thread Justin Pryzby
On Fri, Nov 01, 2019 at 11:01:22PM +0900, Michael Paquier wrote: > On Fri, Nov 01, 2019 at 08:59:48AM -0500, Justin Pryzby wrote: > > I guess you mean because it's not a child until after the ALTER. Yes, that > > makes sense. > > Yes, perhaps you have another idea than

Re: bitmaps and correlation

2019-11-02 Thread Justin Pryzby
Attached is a fixed and rebasified patch for cfbot. Included inline for conceptual review. >From f3055a5696924427dda280da702c41d2d2796a24 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 1 Jan 2019 16:17:28 -0600 Subject: [PATCH v2] Use correlation statistic in costing bitmap sc

psql \d for wide tables / pattern for individual columns

2019-11-10 Thread Justin Pryzby
, btree (attrelid, attname) "pg_attribute_relid_attnum_index" UNIQUE, btree (attrelid, attnum) >From fdcde33f93af544eb1be0f327ffa49a133397da3 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 10 Nov 2019 15:02:00 -0600 Subject: [PATCH v1] psql: Allow filtering columns

checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-19 Thread Justin Pryzby
I (finally) noticed this morning on a server running PG12.1: < 2019-11-15 22:16:07.098 EST >PANIC: could not fsync file "base/16491/1731839470.2": No such file or directory < 2019-11-15 22:16:08.751 EST >LOG: checkpointer process (PID 27388) was terminated by signal 6: Aborted /dev/vdb on /

planner support functions: handle GROUP BY estimates ?

2019-11-19 Thread Justin Pryzby
Tom implemented "Planner support functions": https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a391ff3c3d418e404a2c6e4ff0865a107752827b https://www.postgresql.org/docs/12/xfunc-optimization.html I wondered whether there was any consideration to extend that to allow providing impro

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-19 Thread Justin Pryzby
On Wed, Nov 20, 2019 at 09:26:53AM +1300, Thomas Munro wrote: > Perhaps we should not panic if we failed to open (not fsync) the file, > but it's not the root problem here which is that somehow we thought we > should be fsyncing a file that had apparently been removed already > (due to CLUSTER, VAC

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-19 Thread Justin Pryzby
On Tue, Nov 19, 2019 at 04:49:10PM -0600, Justin Pryzby wrote: > On Wed, Nov 20, 2019 at 09:26:53AM +1300, Thomas Munro wrote: > > Perhaps we should not panic if we failed to open (not fsync) the file, > > but it's not the root problem here which is that somehow we thou

error context for vacuum to include block number

2019-11-20 Thread Justin Pryzby
19-11-20 14:52:49.521 CST [6319] STATEMENT: vacuum t; Justin >From 2aac5cdc16c222a053c02818ea2b3a6a5adfb89a Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 20 Nov 2019 14:53:20 -0600 Subject: [PATCH v1] vacuum errcontext to show block being processed As requested here. https://w

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-20 Thread Justin Pryzby
On Tue, Nov 19, 2019 at 07:22:26PM -0600, Justin Pryzby wrote: > I was trying to reproduce what was happening: > set -x; psql postgres -txc "DROP TABLE IF EXISTS t" -c "CREATE TABLE t(i int > unique); INSERT INTO t SELECT generate_series(1,99)"; echo "

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-25 Thread Justin Pryzby
I looked and found a new "hint". On Tue, Nov 19, 2019 at 05:57:59AM -0600, Justin Pryzby wrote: > < 2019-11-15 22:16:07.098 EST >PANIC: could not fsync file > "base/16491/1731839470.2": No such file or directory > < 2019-11-15 22:16:08.751 EST >L

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-26 Thread Justin Pryzby
This same crash occured on a 2nd server. Also qemu/KVM, but this time on a 2ndary ZFS tablespaces which (fails to) include the missing relfilenode. Linux database7 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux This is postgresql12-12.1-1PGDG.rhel7.

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2019-11-28 Thread Justin Pryzby
On Fri, Nov 29, 2019 at 10:50:36AM +1300, Thomas Munro wrote: > On Fri, Nov 29, 2019 at 3:13 AM Thomas Munro wrote: > > On Wed, Nov 27, 2019 at 7:53 PM Justin Pryzby wrote: > > > 2019-11-26 23:41:50.009-05 | could not fsync file > > > "pg_tblspc/16401/PG_12_2

Re: bitmaps and correlation

2019-12-01 Thread Justin Pryzby
On Sun, Dec 01, 2019 at 12:34:37PM +0900, Michael Paquier wrote: > On Sat, Nov 02, 2019 at 03:26:17PM -0500, Justin Pryzby wrote: > > Attached is a fixed and rebasified patch for cfbot. > > Included inline for conceptual review. > > Your patch still causes two regression

Re: error context for vacuum to include block number

2019-12-06 Thread Justin Pryzby
relation t 2019-11-27 20:04:53.640 CST [14244] STATEMENT: vacuum t; I tried to use BufferGetTag() to avoid using a 2ndary structure, but fails if the buffer is not pinned. >From 41e1d6d118346e84aac7cfe68424f7452c7dcb8d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 20 Nov 2019 14:53:20

verbose cost estimate

2019-12-07 Thread Justin Pryzby
Jeff said: https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bmsr6pOEoiwA7y5jrA%40mail.gmail.com |What would I find very useful is a verbosity option to get the cost |estimates expressed as a multiplier of each *_cost parameter, rather than |just as a scalar. I guess the g

Re: error context for vacuum to include block number

2019-12-11 Thread Justin Pryzby
On Wed, Dec 11, 2019 at 09:15:07PM +0900, Michael Paquier wrote: > On Fri, Dec 06, 2019 at 10:23:25AM -0600, Justin Pryzby wrote: > > Find attached updated patch: > > . Use structure to include relation name. > > . Split into a separate patch rename of "StringInfoData

shared tempfile was not removed on statement_timeout (unreproducible)

2019-12-12 Thread Justin Pryzby
I have a nagios check on ancient tempfiles, intended to catch debris left by crashed processes. But triggered on this file: $ sudo find /var/lib/pgsql/12/data/base/pgsql_tmp -ls 1429774 drwxr-x--- 3 postgres postgres 4096 Dec 12 11:32 /var/lib/pgsql/12/data/base/pgsql_tmp 1698684 d

Re: shared tempfile was not removed on statement_timeout (unreproducible)

2019-12-12 Thread Justin Pryzby
On Fri, Dec 13, 2019 at 03:03:47PM +1300, Thomas Munro wrote: > On Fri, Dec 13, 2019 at 7:05 AM Justin Pryzby wrote: > > I have a nagios check on ancient tempfiles, intended to catch debris left by > > crashed processes. But triggered on this file: > > > > $ sudo find

Re: error context for vacuum to include block number

2019-12-12 Thread Justin Pryzby
On Wed, Dec 11, 2019 at 12:33:53PM -0300, Alvaro Herrera wrote: > On 2019-Dec-11, Justin Pryzby wrote: > > + cbarg.blkno = 0; /* Not known yet */ > Shouldn't you use InvalidBlockNumber for this initialization? .. > > pgstat_progress_update_param(PROGRESS_V

pg_ls_tmpdir to show shared filesets

2019-12-12 Thread Justin Pryzby
On Thu, Dec 12, 2019, Justin Pryzby wrote in 20191212180506.gr2...@telsasoft.com: > Actually, I tried using pg_ls_tmpdir(), but it unconditionally masks > non-regular files and thus shared filesets. Maybe that's worth discussion on > a > new thread ? > > src/back

Re: error context for vacuum to include block number

2019-12-13 Thread Justin Pryzby
es, indstats); > vacrelstats->num_dead_tuples = 0; > - vacrelstats->num_index_scans++; > You are moving this comment within lazy_vacuum_heap_index, but it > applies to num_dead_tuples and not num_index_scans, no? You should > keep the incrementation of num_index_scans within the rou

Re: pg_ls_tmpdir to show shared filesets

2019-12-14 Thread Justin Pryzby
On Thu, Dec 12, 2019 at 11:39:31PM -0600, Justin Pryzby wrote: > I suggested that pg_ls_tmpdir should show shared filesets like > > 169347 5492 -rw-r- 1 postgres postgres 5619712 Dec 7 01:35 > > /var/lib/pgsql/12/data/base/pgsql_tmp/pgsql_tmp11025.0.sharedfileset/0.0 ..

Re: error context for vacuum to include block number

2019-12-15 Thread Justin Pryzby
On Sun, Dec 15, 2019 at 10:07:08PM +0900, Michael Paquier wrote: > On Fri, Dec 13, 2019 at 04:47:35PM -0600, Justin Pryzby wrote: > > It's related code which I cleaned up before adding new stuff. Not > > essential, > > thus separate (0002 should be backpatched). >

ERROR: could not resize shared memory segment...No space left on device

2019-12-16 Thread Justin Pryzby
A customer's report query hit this error. ERROR: could not resize shared memory segment "/PostgreSQL.2011322019" to 134217728 bytes: No space left on device I found: https://www.postgresql.org/message-id/flat/CAEepm%3D2D_JGb8X%3DLa-0PX9C8dBX9%3Dj9wY%2By1-zDWkcJu0%3DBQbA%40mail.gmail.com work_me

vacuum verbose detail logs are unclear (show debug lines at *start* of each stage?)

2019-12-20 Thread Justin Pryzby
This is a usability complaint. If one knows enough about vacuum and/or logging, I'm sure there's no issue. Right now vacuum shows: | 1 postgres=# VACUUM t; | 2 DEBUG: vacuuming "public.t" | 3 DEBUG: scanned index "t_i_key" to remove 999 row versions | 4 DETAIL: CPU: user: 0.00 s, sys

Re: planner support functions: handle GROUP BY estimates ?

2019-12-22 Thread Justin Pryzby
On Tue, Nov 19, 2019 at 01:34:21PM -0600, Justin Pryzby wrote: > Tom implemented "Planner support functions": > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a391ff3c3d418e404a2c6e4ff0865a107752827b > https://www.postgresql.org/docs/12/xfunc-optimization.h

Re: error context for vacuum to include block number

2019-12-23 Thread Justin Pryzby
On Mon, Dec 16, 2019 at 11:49:56AM +0900, Michael Paquier wrote: > On Sun, Dec 15, 2019 at 10:27:12AM -0600, Justin Pryzby wrote: > > I named it so because it calls both lazy_vacuum_index > > ("PROGRESS_VACUUM_PHASE_VACUUM_INDEX") and > > lazy_vacuum_heap(&qu

Re: error context for vacuum to include block number

2019-12-26 Thread Justin Pryzby
On Tue, Dec 24, 2019 at 01:19:09PM +0900, Michael Paquier wrote: > On Mon, Dec 23, 2019 at 07:24:28PM -0600, Justin Pryzby wrote: > > I renamed. > > Hmm. I have found what was partially itching me for patch 0002, and > that's actually the fact that we don't do

Re: planner support functions: handle GROUP BY estimates ?

2019-12-26 Thread Justin Pryzby
On Sun, Dec 22, 2019 at 06:16:48PM -0600, Justin Pryzby wrote: > On Tue, Nov 19, 2019 at 01:34:21PM -0600, Justin Pryzby wrote: > > Tom implemented "Planner support functions": > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a391ff3c3d418e404a2c6e4ff

doc: vacuum full, fillfactor, and "extra space"

2019-12-26 Thread Justin Pryzby
I started writing this patch to avoid the possibly-misleading phrase: "with no extra space" (since it's expected to typically take ~2x space, or 1x "extra" space). But the original phrase "with no extra space" seems to be wrong anyway, since it actually follows fillfactor, so say that. Possibly s

Re: ALTER INDEX fails on partitioned index

2019-12-26 Thread Justin Pryzby
On Mon, Jan 07, 2019 at 04:23:30PM -0300, Alvaro Herrera wrote: > On 2019-Jan-05, Justin Pryzby wrote: > > postgres=# CREATE TABLE t(i int)PARTITION BY RANGE(i); > > postgres=# CREATE INDEX ON t(i) WITH(fillfactor=11); > > postgres=# ALTER INDEX t_i_idx SET (fillfactor=1

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-27 Thread Justin Pryzby
Re-added -hackers. Thanks for reviewing. On Fri, Dec 27, 2019 at 05:22:47PM +0100, Fabien COELHO wrote: > The implementation simply extends an existing functions with a boolean to > allow for sub-directories. However, the function does not seem to show > subdir contents recursively. Should it be

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-27 Thread Justin Pryzby
On Fri, Dec 27, 2019 at 06:50:24PM +0100, Fabien COELHO wrote: > >On Fri, Dec 27, 2019 at 05:22:47PM +0100, Fabien COELHO wrote: > >>The implementation simply extends an existing functions with a boolean to > >>allow for sub-directories. However, the function does not seem to show > >>subdir conten

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-28 Thread Justin Pryzby
errmsg("could not stat directory \"%s\": %m", dir))); +errmsg("could not stat file \"%s\": %m", path))); >From fd88be5f1687354d9990fb1838adc0db36bc6dde Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 27

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-28 Thread Justin Pryzby
d be shown, too. And maybe the is_dir flag should be re-introduced (although someone could call pg_stat_file if needed). I'm interested to hear feedback on that, although this patch still isn't great. >From dd3b2779939fc1b396fed1fba2f7cefc9a6b1ad5 Mon Sep 17 00:00:00 2001 From: Justi

Re: error context for vacuum to include block number (atomic progress update)

2019-12-29 Thread Justin Pryzby
On Sat, Dec 28, 2019 at 07:21:31PM -0500, Robert Haas wrote: > On Thu, Dec 26, 2019 at 10:57 AM Justin Pryzby wrote: > > I agree that's better. > > I don't see any reason why the progress params need to be updated > > atomically. > > So rebasified against y

comment regarding double timestamps; and, infinite timestamps and NaN

2019-12-29 Thread Justin Pryzby
e: DROP TABLE t; CREATE TABLE t(t) AS SELECT generate_series(now(), now()+'1 day', '5 minutes'); INSERT INTO t VALUES('-infinity'); ALTER TABLE t ALTER t SET STATISTICS 1; ANALYZE t; explain SELECT * FROM t WHERE t>='2010-12-29'; >From b0151e24819499607eb2894dd92

Re: comment regarding double timestamps; and, infinite timestamps and NaN

2019-12-30 Thread Justin Pryzby
On Mon, Dec 30, 2019 at 09:05:24AM -0500, Tom Lane wrote: > Justin Pryzby writes: > > That seems to be only used for ineq_histogram_selectivity() interpolation of > > histogram bins. It looks to me that at least isn't working for "special > > values", and nee

infinite histogram bounds and nan (Re: comment regarding double timestamps; and, infinite timestamps and NaN)

2020-01-02 Thread Justin Pryzby
th=8) (actual time=0.014..0.020 rows=50 loops=1) I'm fine if the isnan() logic changes, but the comment indicates it's intended to be hit for an infinite histogram bound, but that doesn't work for timestamps (convert_to_scalar() should return (double)INFINITY and not (double)INT64_MIN/MAX)

Re: error context for vacuum to include block number

2020-01-02 Thread Justin Pryzby
On Thu, Dec 26, 2019 at 09:57:04AM -0600, Justin Pryzby wrote: > So rebasified against your patch. Rebased against your patch in master this time. >From dadb8dff6ea929d78f3695f606de9ade7674b7a1 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 27 Nov 2019 20:07:10 -0600 Subject: [PA

avoid some calls to memset with array initializer

2020-01-02 Thread Justin Pryzby
pfuncs.S1 >From 5117e66043b6c8c66c2f98fcd99fdaefec66f90e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 27 Dec 2019 17:30:36 -0600 Subject: [PATCH v1 1/2] Avoid some calls to memset.. ..in cases where that saves a couple lines of code. Note that gcc has builtin for memset, but inlined function is still not sa

explain HashAggregate to report bucket and memory stats

2020-01-03 Thread Justin Pryzby
emory and bucket stats; and if the Aggregate > node would report...anything. Find attached my WIP attempt to implement this. Jeff: can you suggest what details Aggregate should show ? Justin >From 5d0afe5d92649f575d9b09ae19b31d2bfd5bfd12 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: We

allow disabling indexscans without disabling bitmapscans

2020-01-04 Thread Justin Pryzby
ff. The high cost attributed to bitmap heapscan is topic for the other patch. Justin >From 6ad506879d8a754013b971197592fc9617850b7e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 4 Jan 2020 10:25:12 -0600 Subject: [PATCH v1] allow disabling indexscans but not bitmap scans --- src/b

Re: allow disabling indexscans without disabling bitmapscans

2020-01-04 Thread Justin Pryzby
On Sat, Jan 04, 2020 at 10:50:47AM -0600, Justin Pryzby wrote: > > Doesn't enable_indexscan=off accomplish this already? It is possible but > > not terribly likely to switch from index to seq, rather than from index to > > bitmap. (Unless the index scan was being used to o

doc: alter table references bogus table-specific planner parameters

2020-01-05 Thread Justin Pryzby
+ Justin >From 64699ee90ef6ebe9459e3b2b1f603f30ec2c49c8 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 5 Jan 2020 19:39:29 -0600 Subject: [PATCH v1] Fixes for commit 6f3a13ff Should backpatch to v10. --- doc/src/sgml/ref/alter_table.sgml | 8 +++- 1 file changed, 3 insertio

Re: doc: alter table references bogus table-specific planner parameters

2020-01-05 Thread Justin Pryzby
On Mon, Jan 06, 2020 at 03:48:52AM +, Simon Riggs wrote: > On Mon, 6 Jan 2020 at 02:56, Justin Pryzby wrote: > > > commit 6f3a13ff058f15d565a30c16c0c2cb14cc994e42 Enhance docs for ALTER > > TABLE lock levels of storage parms > > Author: Simon Riggs > > Date:

Re: bitmaps and correlation

2020-01-06 Thread Justin Pryzby
Find attached cleaned up patch. For now, I updated the regress/expected/, but I think the test maybe has to be updated to do what it was written to do. >From 36f547d69b8fee25869d6ef3ef26d327a8ba1205 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 1 Jan 2019 16:17:28 -0600 Subject: [PA

Re: bitmaps and correlation

2020-01-06 Thread Justin Pryzby
On Tue, Jan 07, 2020 at 09:21:03AM +0530, Dilip Kumar wrote: > On Tue, Jan 7, 2020 at 1:29 AM Justin Pryzby wrote: > > > > Find attached cleaned up patch. > > For now, I updated the regress/expected/, but I think the test maybe has to > > be > > updated to do

Re: vacuum verbose detail logs are unclear; log at *start* of each stage; show allvisible/frozen/hintbits

2020-01-12 Thread Justin Pryzby
On Sun, Dec 29, 2019 at 01:15:24PM -0500, Jeff Janes wrote: > On Fri, Dec 20, 2019 at 12:11 PM Justin Pryzby wrote: > > > This is a usability complaint. If one knows enough about vacuum and/or > > logging, I'm sure there's no issue. > > > | 11

Re: bitmaps and correlation

2020-01-12 Thread Justin Pryzby
On Mon, Jan 06, 2020 at 11:26:06PM -0600, Justin Pryzby wrote: > As Jeff has pointed out, high correlation has two effects in cost_index(): > 1) the number of pages read will be less; > 2) the pages will be read more sequentially; > > cost_index reuses the pages_fetched variabl

Re: doc: vacuum full, fillfactor, and "extra space"

2020-01-14 Thread Justin Pryzby
On Fri, Dec 27, 2019 at 11:58:18AM +0100, Fabien COELHO wrote: >> I started writing this patch to avoid the possibly-misleading phrase: "with >> no >> extra space" (since it's expected to typically take ~2x space, or 1x "extra" >> space). >> >> But the original phrase "with no extra space" seems

Re: [PATCH v1] pg_ls_tmpdir to show directories

2020-01-15 Thread Justin Pryzby
On Wed, Jan 15, 2020 at 11:21:36AM +0100, Fabien COELHO wrote: > I'm trying to think about how to get rid of the strange structure and hacks, > and the arbitrary looking size 2 array. > > Also the recursion is one step, but I'm not sure why, ISTM it could/should > go on always? Because tmpfiles o

<    3   4   5   6   7   8   9   10   11   12   >