Re: POC: Cleaning up orphaned files using undo logs

2019-05-01 Thread Dilip Kumar
On Tue, Apr 30, 2019 at 11:45 AM Dilip Kumar wrote: The attached patch will provide mechanism for masking the necessary bits in undo pages for supporting consistency checking for the undo pages. Ideally we can merge this patch with the main interface patch but currently I have kept it separate

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Amit Kapila
On Thu, May 2, 2019 at 7:36 AM John Naylor wrote: > > On Wed, May 1, 2019 at 11:24 PM Andres Freund wrote: > > > > Hi, > > > > On 2019-04-18 14:10:29 -0700, Andres Freund wrote: > > > My compromise suggestion would be to try to give John and Amit ~2 weeks > > > to come up with a cleanup

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Amit Kapila
On Thu, May 2, 2019 at 3:42 AM Alvaro Herrera wrote: > > On 2019-May-01, Amit Kapila wrote: > > > On Tue, Apr 30, 2019 at 7:52 PM Alvaro Herrera > > wrote: > > > > Hmm ... so, if vacuum runs and frees up any space from any of the pages, > > > then it should send out an invalidation -- it

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread David Rowley
On Thu, 2 May 2019 at 13:08, Tom Lane wrote: > > David Rowley writes: > > On Thu, 2 May 2019 at 06:18, Sergei Kornilov wrote: > >> PS: not think this is blocker for v12 > > > Probably not a blocker, but now does not seem like an unreasonable > > time to lower these INFO messages down to DEBUG.

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 22:01:53 -0400, Tom Lane wrote: > Andres Freund writes: > > Well, as I said before, I think hiding the to-be-rebuilt index from the > > list of indexes is dangerous too - if somebody added an actual > > CatalogUpdate/Insert (rather than inplace_update) anywhere along the > >

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread John Naylor
On Wed, May 1, 2019 at 11:24 PM Andres Freund wrote: > > Hi, > > On 2019-04-18 14:10:29 -0700, Andres Freund wrote: > > My compromise suggestion would be to try to give John and Amit ~2 weeks > > to come up with a cleanup proposal, and then decide whether to 1) revert > > 2) apply the new patch,

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Tom Lane
Andres Freund writes: > On 2019-05-01 19:41:24 -0400, Tom Lane wrote: >> OK, so per the other thread, it seems like the error recovery problem >> isn't going to affect this directly. However, I still don't like this >> proposal much; the reason being that it's a rather fundamental change >> in

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 19:41:24 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> On 2019-05-01 10:06:03 -0700, Andres Freund wrote: > >>> I'm not sure this is the right short-term answer. Why isn't it, for now, > >>> sufficient to do what I suggested with RelationSetNewRelfilenode()

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread Tom Lane
David Rowley writes: > On Thu, 2 May 2019 at 06:18, Sergei Kornilov wrote: >> PS: not think this is blocker for v12 > Probably not a blocker, but now does not seem like an unreasonable > time to lower these INFO messages down to DEBUG. Not a blocker perhaps, but it's better if we can get new

Re: PostgreSQL Asian language support for full text search using ICU (and also updating pg_trgm)

2019-05-01 Thread Tatsuo Ishii
[redirected to hackers list since I think this topic is related to adding new PostgreSQL feature.] I think there's no doubt that it would be nice if PostgreSQL natively supports Asian languages. For the first step, I briefly tested the ICU tokenizer (ubrk_open and other functions) with Japanese,

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread David Rowley
On Thu, 2 May 2019 at 06:18, Sergei Kornilov wrote: > > > I proposed that we didn't need those messages at all, which Robert pushed > > back on, but I'd be willing to compromise by reducing them to NOTICE or > > DEBUG level. > > I posted patch with such change in a separate topic: >

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Tom Lane
I wrote: > Andres Freund writes: >> On 2019-05-01 10:06:03 -0700, Andres Freund wrote: >>> I'm not sure this is the right short-term answer. Why isn't it, for now, >>> sufficient to do what I suggested with RelationSetNewRelfilenode() not >>> doing the CommandCounterIncrement(), and

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2019-05-01 Thread Tom Lane
Andres Freund writes: > On 2019-05-01 14:44:12 -0400, Tom Lane wrote: >> I'm busy looking into the REINDEX-on-pg_class mess, and one thing I found >> while testing HEAD is that with CLOBBER_CACHE_ALWAYS on, once you've >> gotten a failure, your session is hosed: >> >> regression=# select 1; >>

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Alvaro Herrera
On 2019-May-01, Amit Kapila wrote: > On Tue, Apr 30, 2019 at 7:52 PM Alvaro Herrera > wrote: > > Hmm ... so, if vacuum runs and frees up any space from any of the pages, > > then it should send out an invalidation -- it doesn't matter what the > > FSM had, just that there is more free space

Re: to_timestamp docs

2019-05-01 Thread Arthur Zakirov
On Thu, May 2, 2019 at 12:49 AM Alexander Korotkov wrote: > > It works globally (but only for remaining string if you don't put it > > at the beginning) > > and you can set it only once. For example: > > > > =# SELECT to_timestamp('JUL JUL JUL','FXMON_MON_MON'); > > ERROR: invalid value " J"

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
On Thu, May 2, 2019 at 12:49:23AM +0300, Alexander Korotkov wrote: > On Wed, May 1, 2019 at 11:20 PM Arthur Zakirov > wrote: > > Hello, > > Not sure if we need some additional checks here if FX is set. > > I'd like to add that this behavior is not new in 12. It was the same before. Agreed,

Re: to_timestamp docs

2019-05-01 Thread Alexander Korotkov
On Wed, May 1, 2019 at 11:20 PM Arthur Zakirov wrote: > Hello, > > On Wed, May 1, 2019 at 6:05 PM Bruce Momjian wrote: > > Thanks. I think I see the sentence you are thinking of: > > > >to_timestamp and to_date > >skip multiple blank spaces at the beginning of the input string >

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2019-05-01 Thread Tom Lane
Andres Freund writes: > On 2019-05-01 14:44:12 -0400, Tom Lane wrote: >> There might be an argument for treating rd_createSubid the same way, >> not sure. That field wouldn't ever be set on a system catalog, so >> it's less of a hazard, but there's something to be said for treating >> the two

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 11:20:05PM +0300, Arthur Zakirov wrote: > Hello, > > On Wed, May 1, 2019 at 6:05 PM Bruce Momjian wrote: > > Thanks. I think I see the sentence you are thinking of: > > > >to_timestamp and to_date > >skip multiple blank spaces at the beginning of the

Re: to_timestamp docs

2019-05-01 Thread Arthur Zakirov
Hello, On Wed, May 1, 2019 at 6:05 PM Bruce Momjian wrote: > Thanks. I think I see the sentence you are thinking of: > >to_timestamp and to_date >skip multiple blank spaces at the beginning of the input string >and around date and time values unless the FX >

pg_upgrade --clone error checking

2019-05-01 Thread Jeff Janes
With the new pg_upgrade --clone, if we are going to end up throwing the error "file cloning not supported on this platform" (which seems to depend only on ifdefs) I think we should throw it first thing, before any other checks are done and certainly before pg_dump gets run. This might result in

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 10:21:15 -0700, Andres Freund wrote: > FWIW, the dirty-hack version (attached) of the CommandCounterIncrement() > approach fixes the issue for a REINDEX pg_class_oid_index; in solation > even when using CCA. Started a whole CCA testrun with it, but the > results of that will

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2019-05-01 Thread Tom Lane
Andres Freund writes: > On 2019-05-01 14:44:12 -0400, Tom Lane wrote: >> This seems quite wrong, because it prevents us from rebuilding the >> entry with corrected values. In particular notice that the change >> causes us to skip the RelationInitPhysicalAddr call that would >> normally be

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Tom Lane
Andres Freund writes: > On 2019-05-01 10:06:03 -0700, Andres Freund wrote: >> I'm not sure this is the right short-term answer. Why isn't it, for now, >> sufficient to do what I suggested with RelationSetNewRelfilenode() not >> doing the CommandCounterIncrement(), and reindex_index() then doing

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 14:44:12 -0400, Tom Lane wrote: > I'm busy looking into the REINDEX-on-pg_class mess, and one thing I found > while testing HEAD is that with CLOBBER_CACHE_ALWAYS on, once you've > gotten a failure, your session is hosed: > > regression=# select 1; > ?column? > -- >

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2019-05-01 Thread Tom Lane
[ blast-from-the-past department ] Simon Riggs writes: > On 11 December 2012 03:01, Noah Misch wrote: >> Until these threads, I did not know that a relcache invalidation could trip >> up >> the WAL avoidance optimization, and now this. I poked at the relevant >> relcache.c code, and it

Re: Adding a test for speculative insert abort case

2019-05-01 Thread Andres Freund
Hi, On 2019-04-30 18:34:42 -0700, Melanie Plageman wrote: > On Tue, Apr 30, 2019 at 5:22 PM Andres Freund wrote: > > > > > Not easily so - that's why the ON CONFLICT patch didn't add code > > coverage for it :(. I wonder if you could whip something up by having > > another non-unique expression

Re: Adding a test for speculative insert abort case

2019-05-01 Thread Melanie Plageman
On Tue, Apr 30, 2019 at 7:14 PM Thomas Munro wrote: > I think it'd be nice to have a set of macros that can create wait > points in the C code that isolation tests can control, in a special > build. Perhaps there could be shm hash table of named wait points in > shared memory; if

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread Sergei Kornilov
Hi > I proposed that we didn't need those messages at all, which Robert pushed > back on, but I'd be willing to compromise by reducing them to NOTICE or > DEBUG level. I posted patch with such change in a separate topic: https://commitfest.postgresql.org/23/2076/ PS: not think this is blocker

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Andres Freund
On 2019-05-01 10:06:03 -0700, Andres Freund wrote: > I'm not sure this is the right short-term answer. Why isn't it, for now, > sufficient to do what I suggested with RelationSetNewRelfilenode() not > doing the CommandCounterIncrement(), and reindex_index() then doing the > SetReindexProcessing()

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Alexander Kukushkin
Hi, On Wed, 1 May 2019 at 17:02, Tomas Vondra wrote: > OK, so that seems like a separate issue, somewhat unrelated to the issue I > reported. And I'm not sure it's a walsender issue - it seems it might be a > psycopg2 issue in not reporting the flush properly, no? Agree, it is a different

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 12:50 PM Tom Lane wrote: > > Not strongly enough to argue about it very hard. The current behavior > > is a little weird, but it's a long way from being the weirdest thing > > we ship, and it appears that we have no tangible evidence that it > > causes a problem in

Re: hyrax vs. RelationBuildPartitionDesc

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 11:59 AM Andres Freund wrote: > The message I'm replying to is marked as an open item. Robert, what do > you think needs to be done here before release? Could you summarize, > so we then can see what others think? The original issue on this thread was that hyrax started

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 12:20:22 -0400, Tom Lane wrote: > Andres Freund writes: > > I see the bug. Turns out we need to figure out another way to solve the > > assertion triggered by doing catalog updates within > > RelationSetNewRelfilenode() - we can't just move the > > SetReindexProcessing() before

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Tom Lane
I wrote: > Did you figure out why this doesn't also happen in HEAD? ... actually, HEAD *is* broken with CCA, just differently. I'm on it. regards, tom lane

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Tom Lane
Robert Haas writes: > On Wed, May 1, 2019 at 12:15 PM Andres Freund wrote: >> My current inclination is to not do anything for v12. Robert, do you >> disagree? > Not strongly enough to argue about it very hard. The current behavior > is a little weird, but it's a long way from being the

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Tomas Vondra
On Wed, May 01, 2019 at 08:53:15AM -0700, Andres Freund wrote: Hi, On 2019-05-01 02:28:45 +0200, Tomas Vondra wrote: The reason for that is pretty simple - the walsenders are doing logical decoding, and during shutdown they're waiting for WalSndCaughtUp=true. But per XLogSendLogical() this

Re: New vacuum option to do only freezing

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 12:14 PM Andres Freund wrote: > On 2019-04-06 16:13:53 +0900, Michael Paquier wrote: > > On Sat, Apr 06, 2019 at 11:31:31AM +0900, Masahiko Sawada wrote: > > > Yes, but Fujii-san pointed out that this option doesn't support toast > > > tables and I think there is not

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread Tom Lane
Andres Freund writes: > There's still an open item "Debate INFO messages in ATTACH PARTITION and > SET NOT NULL" for this thread. Is there anything further to be done? Or > are we content with this for v12? IIRC, that's based on my complaint that these messages have no business being INFO

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 12:15 PM Andres Freund wrote: > On 2019-04-01 18:26:59 -0700, Andres Freund wrote: > > I'm not yet convinced it's necessary to create a new GUC, but also not > > strongly opposed. I've created an open items issue for it, so we don't > > forget. > > My current inclination

Re: REINDEX INDEX results in a crash for an index of pg_class since 9.6

2019-05-01 Thread Tom Lane
Andres Freund writes: > I see the bug. Turns out we need to figure out another way to solve the > assertion triggered by doing catalog updates within > RelationSetNewRelfilenode() - we can't just move the > SetReindexProcessing() before it. When CCA is enabled, the > CommandCounterIncrement()

Re: using index or check in ALTER TABLE SET NOT NULL

2019-05-01 Thread Andres Freund
On 2019-03-25 11:09:47 -0400, Robert Haas wrote: > On Mon, Mar 25, 2019 at 3:51 AM David Steele wrote: > > On 3/17/19 3:40 PM, David Rowley wrote: > > > On Thu, 14 Mar 2019 at 06:24, Robert Haas wrote: > > > > > > I think we can mark this patch as committed now as I don't think the > > > lack of

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 09:08:54AM -0700, Andres Freund wrote: > So sure, there's a few typo fixes, one bugfix, and one buildfarm test > stability issue. Doesn't seem crazy for a nontrivial improvement. OK, my ignorant opinion was just based on the length of discussion threads. -- Bruce

Re: compiler warning in pgcrypto imath.c

2019-05-01 Thread Andres Freund
Hi Noah, On 2019-03-23 00:02:36 -0700, Noah Misch wrote: > On Sat, Mar 23, 2019 at 10:20:16AM +0900, Michael Paquier wrote: > > On Fri, Mar 22, 2019 at 08:20:53PM -0400, Jeff Janes wrote: > > > PostgreSQL 12devel on aarch64-unknown-linux-gnu, compiled by gcc > > > (Ubuntu/Linaro

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Andres Freund
Hi, On 2019-04-01 18:26:59 -0700, Andres Freund wrote: > I'm not yet convinced it's necessary to create a new GUC, but also not > strongly opposed. I've created an open items issue for it, so we don't > forget. My current inclination is to not do anything for v12. Robert, do you disagree?

Re: New vacuum option to do only freezing

2019-05-01 Thread Andres Freund
Hi, On 2019-04-06 16:13:53 +0900, Michael Paquier wrote: > On Sat, Apr 06, 2019 at 11:31:31AM +0900, Masahiko Sawada wrote: > > Yes, but Fujii-san pointed out that this option doesn't support toast > > tables and I think there is not specific reason why not supporting > > them. So it might be

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-05-01 Thread Andres Freund
Hi, On 2019-04-08 19:22:04 +0900, Fujii Masao wrote: > On Mon, Apr 8, 2019 at 5:30 PM Masahiko Sawada wrote: > > "TRUNCATE" option for vacuum command should be added to the open items? > > Yes, I think. > Attached is the patch which adds TRUNCATE option into VACUUM. This has been an open item

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 11:28:11 -0400, Bruce Momjian wrote: > On Wed, May 1, 2019 at 08:24:25AM -0700, Andres Freund wrote: > > Hi, > > > > On 2019-04-18 14:10:29 -0700, Andres Freund wrote: > > > My compromise suggestion would be to try to give John and Amit ~2 weeks > > > to come up with a cleanup

Re: hyrax vs. RelationBuildPartitionDesc

2019-05-01 Thread Andres Freund
Hi The message I'm replying to is marked as an open item. Robert, what do you think needs to be done here before release? Could you summarize, so we then can see what others think? Greetings, Andres Freund

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Andres Freund
Hi, On 2019-05-01 02:28:45 +0200, Tomas Vondra wrote: > The reason for that is pretty simple - the walsenders are doing logical > decoding, and during shutdown they're waiting for WalSndCaughtUp=true. > But per XLogSendLogical() this only happens if > >if

Re: New vacuum option to do only freezing

2019-05-01 Thread Andres Freund
Hi, This thread is referenced an open item, and we ought to make some progress on it. On a more cosmetic note: On 2019-04-16 09:10:19 -0700, Andres Freund wrote: > On 2019-04-16 12:01:36 -0400, Tom Lane wrote: > > (BTW, I don't understand why that code will throw "found xmin %u from > > before

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 08:24:25AM -0700, Andres Freund wrote: > Hi, > > On 2019-04-18 14:10:29 -0700, Andres Freund wrote: > > My compromise suggestion would be to try to give John and Amit ~2 weeks > > to come up with a cleanup proposal, and then decide whether to 1) revert > > 2) apply the new

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Andres Freund
Hi, On 2019-04-18 14:10:29 -0700, Andres Freund wrote: > My compromise suggestion would be to try to give John and Amit ~2 weeks > to come up with a cleanup proposal, and then decide whether to 1) revert > 2) apply the new patch, 3) decide to live with the warts for 12, and > apply the patch in

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 10:01:50AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I don't think the changes made in PG 12 are documented accurately. > > That code is swapped out of my head at the moment, but it looks > to me like the para before the one you changed is where we discuss > the

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Tomas Vondra
On Wed, May 01, 2019 at 02:13:10PM +0200, Alexander Kukushkin wrote: Hi Tomas, On Wed, 1 May 2019 at 02:28, Tomas Vondra wrote: I see there's an ongoing discussion about race conditions in walreceiver blocking shutdown, so let me start a new thread about a race condition in walsender during

Re: to_timestamp docs

2019-05-01 Thread Tom Lane
Bruce Momjian writes: > I don't think the changes made in PG 12 are documented accurately. That code is swapped out of my head at the moment, but it looks to me like the para before the one you changed is where we discuss the behavior for whitespace. I'm not sure that this change is right, or

to_timestamp docs

2019-05-01 Thread Bruce Momjian
I don't think the changes made in PG 12 are documented accurately. It currently says: to_timestamp and to_date matches any single separator in the input string or is skipped However, I think it is more accurate to say _multiple_ whitespace can also be matched by a single

Re: walsender vs. XLogBackgroundFlush during shutdown

2019-05-01 Thread Alexander Kukushkin
Hi Tomas, On Wed, 1 May 2019 at 02:28, Tomas Vondra wrote: > I see there's an ongoing discussion about race conditions in walreceiver > blocking shutdown, so let me start a new thread about a race condition in > walsender during shutdown. > > The symptoms are rather simple - 'pg_ctl -m fast

Re: [PATCH v4] Add \warn to psql

2019-05-01 Thread Fabien COELHO
I guess that you have a verbose ~/.psqlrc. Can you try with adding -X to psql option when calling psql from the tap test? Ah, true. This patch works for me: diff --git a/src/bin/psql/t/001_psql.pl b/src/bin/psql/t/001_psql.pl index 32dd43279b..637baa94c9 100644 ---

Re: [PATCH v4] Add \warn to psql

2019-05-01 Thread David Fetter
On Wed, May 01, 2019 at 10:05:44AM +0300, Arthur Zakirov wrote: > (Unfortunately I accidentally sent my previous two messages using my personal > email address because of my email client configuration. This address is not > verified by PostgreSQL.org services and messages didn't reach hackers

Re: [PATCH v4] Add \warn to psql

2019-05-01 Thread Arthur Zakirov
(Unfortunately I accidentally sent my previous two messages using my personal email address because of my email client configuration. This address is not verified by PostgreSQL.org services and messages didn't reach hackers mailing lists, so I recent latest message). On Tue, Apr 30, 2019 at 4:46