Re: Partial aggregates pushdown

2024-06-11 Thread Jelte Fennema-Nio
On Tue, 11 Jun 2024 at 13:40, fujii.y...@df.mitsubishielectric.co.jp wrote: > > From: Bruce Momjian > > So, we need import/export text representation for the partial aggregate > > mode for these eight, and call the base data type > > text import/export functions for the zero ones when in this

Re: Addressing SECURITY DEFINER Function Vulnerabilities in PostgreSQL Extensions

2024-06-11 Thread Jelte Fennema-Nio
On Tue, 11 Jun 2024 at 11:54, Ashutosh Sharma wrote: > 1) Extends the CREATE EXTENSION command to support a new option, SET > SEARCH_PATH. I don't think it makes sense to add such an option to CREATE EXTENSION. I feel like such a thing should be part of the extension control file instead. That

Re: RFC: adding pytest as a supported test framework

2024-06-10 Thread Jelte Fennema-Nio
On Mon, 10 Jun 2024 at 22:47, Andrew Dunstan wrote: > As for what up and coming developers learn, they mostly don't learn C either, > and that's far more critical to what we do. I think many up and coming devs have at least touched C somewhere (e.g. in university). And because it's more

Re: RFC: adding pytest as a supported test framework

2024-06-10 Thread Jelte Fennema-Nio
On Mon, 10 Jun 2024 at 20:46, Jacob Champion wrote: > For the v18 cycle, I would like to try to get pytest [1] in as a > supported test driver, in addition to the current offerings. Huge +1 from me (but I'm definitely biased here) > Thoughts? Suggestions? I think the most important thing is

Re: Addressing SECURITY DEFINER Function Vulnerabilities in PostgreSQL Extensions

2024-06-06 Thread Jelte Fennema-Nio
On Thu, 6 Jun 2024 at 20:10, Isaac Morland wrote: > > On Thu, 6 Jun 2024 at 12:53, Jeff Davis wrote: > >> >> > I didn't get you completely here. w.r.t extensions how will this have >> > an impact if we set the search_path for definer functions. >> >> If we only set the search path for SECURITY

Re: ssl tests fail due to TCP port conflict

2024-06-06 Thread Jelte Fennema-Nio
On Wed, 5 Jun 2024 at 23:37, Tom Lane wrote: > > Andrew Dunstan writes: > > On 2024-06-05 We 16:00, Alexander Lakhin wrote: > >> That is, psql from the test instance 001_ssltests_34 opened a > >> connection to > >> the test server with the client port 50072 and it made using the port by > >> the

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-06-06 Thread Jelte Fennema-Nio
On Thu, 6 Jun 2024 at 18:01, Robert Haas wrote: > As I see it, the issue here is whether the default value would ever be > different from the latest value. If not, then using blank to mean the > latest seems fine, but if so, then I'd expect blank to mean the > default version and latest to mean

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-06-06 Thread Jelte Fennema-Nio
On Thu, 6 Jun 2024 at 03:03, Robert Haas wrote: > This makes some sense to me. I don't think that I believe > max_protocol_version=latest is future-proof: just because I want to > opt into this round of new features doesn't mean I feel the same way > about the next round. But it may still be a

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-06-05 Thread Jelte Fennema-Nio
On Wed, 5 Jun 2024 at 22:48, Robert Haas wrote: > I agree that we need such a mechanism, but if the driving feature is > cancel-key length, I expect that opt-in isn't going to work very well. > Opt-in seems like it would work well with compression or transparent > column encryption, but few users

Re: [multithreading] extension compatibility

2024-06-05 Thread Jelte Fennema-Nio
On Wed, 5 Jun 2024 at 22:05, Robert Haas wrote: > The attached patch is a sketch of one possible approach: PostgreSQL > signals whether it is multithreaded by defining or not defining > PG_MULTITHREADING in pg_config_manual.h, and an extension signals > thread-readiness by defining

Re: Extension security improvement: Add support for extensions with an owned schema

2024-06-05 Thread Jelte Fennema-Nio
On Wed, 5 Jun 2024 at 19:53, Jeff Davis wrote: > Is this orthogonal to relocatability? It's fairly orthogonal, but it has some impact on relocatability: You can only relocate to a schema name that does not exist yet (while currently you can only relocate to a schema that already exists). This is

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-06-05 Thread Jelte Fennema-Nio
On Wed, 5 Jun 2024 at 17:12, Robert Haas wrote: > Well, hang on. The discussion on that thread suggests that this is > only going to break connections to servers that don't have > NegotiateProtocolVersion. Yes, correct. > What > I really don't want is for v18 to break connections to v17

libpq: Trace StartupMessage/SSLRequest/GSSENCRequest correctly

2024-06-05 Thread Jelte Fennema-Nio
ace From 6fe11a1656e3bc85608e3d883848e42cd765e553 Mon Sep 17 00:00:00 2001 From: Jelte Fennema-Nio Date: Wed, 5 Jun 2024 12:07:09 +0200 Subject: [PATCH v1] libpq: Trace StartupMessage/SSLRequest/GSSENCRequest correctly These messages would previously be logged in the tracing output as: Unknown message: length %d With thi

Extension security improvement: Add support for extensions with an owned schema

2024-06-01 Thread Jelte Fennema-Nio
that function. From cca2fc5a820822a29d9bc3b810a811adb1d081e1 Mon Sep 17 00:00:00 2001 From: Jelte Fennema-Nio Date: Fri, 31 May 2024 02:04:31 -0700 Subject: [PATCH v1] Add support for extensions with an owned schema Writing the sql migration scripts that are run by CREATE EXTENSION and ALTER EXTENSION UPDATE are

Re: In-placre persistance change of a relation

2024-05-28 Thread Jelte Fennema-Nio
On Fri, 24 May 2024 at 00:09, Kyotaro Horiguchi wrote: > Along with rebasing, I changed the interface of XLogFsyncFile() to > return a boolean instead of an error message. Two notes after looking at this quickly during the advanced patch feedback session: 1. I would maybe split 0003 into two

Re: Fix calloc check if oom (PQcancelCreate)

2024-05-27 Thread Jelte Fennema-Nio
On Mon, 27 May 2024 at 18:16, Ranier Vilela wrote: > Is it mandatory to call PQcancelFinish in case PQcancelCreate fails? Yes, see the following line in the docs: Note that when PQcancelCreate returns a non-null pointer, you must call PQcancelFinish when you are finished with it, in order to

Re: Fix calloc check if oom (PQcancelCreate)

2024-05-27 Thread Jelte Fennema-Nio
On Mon, 27 May 2024 at 16:06, Ranier Vilela wrote: > Em seg., 27 de mai. de 2024 às 10:23, Daniel Gustafsson > escreveu: >> > On 27 May 2024, at 14:25, Ranier Vilela wrote: >> > I think that commit 61461a3, left some oversight. >> > The function *PQcancelCreate* fails in check, >> > return of

Re: DISCARD ALL does not force re-planning of plpgsql functions/procedures

2024-05-27 Thread Jelte Fennema-Nio
On Sun, 26 May 2024 at 19:39, Tom Lane wrote: > Hm, should it be? That's hard-won knowledge, and I'm not sure there > is a good reason to believe it's no longer applicable. Okay, so I looked into this a bit more and there's a similar case here that's imho very clearly doing something

Re: DISCARD ALL does not force re-planning of plpgsql functions/procedures

2024-05-26 Thread Jelte Fennema-Nio
On Sun, May 26, 2024, 12:26 Jelte Fennema-Nio wrote: > DISCARD PLANS should probably forget about it though indeed. > DISCARD PLANS should probably **not** forget about it > > Note that any change in behavior there would affect prepared > > statements in general, not only pl

Re: DISCARD ALL does not force re-planning of plpgsql functions/procedures

2024-05-26 Thread Jelte Fennema-Nio
On Sun, 26 May 2024 at 19:39, Tom Lane wrote: > Hm, should it be? That's hard-won knowledge, and I'm not sure there > is a good reason to believe it's no longer applicable. I think for DISCARD ALL it would probably make sense to forget this knowledge . Since that is advertised as "reset the

DISCARD ALL does not force re-planning of plpgsql functions/procedures

2024-05-26 Thread Jelte Fennema-Nio
I got a report on the PgBouncer repo[1] that running DISCARD ALL was not sufficient between connection handoffs to force replanning of stored procedures. Turns out that while DISCARD AL and DISCARD PLAN reset the plan cache they do not reset the num_custom_plans fields of the existing PlanSources.

Re: Volatile write caches on macOS and Windows, redux

2024-05-25 Thread Jelte Fennema-Nio
On Tue, 18 Jul 2023 at 05:29, Thomas Munro wrote: > It's possible that fcntl(F_FULLFSYNC) might fail with ENOSUPP or other > errors in obscure cases (eg unusual file systems). In that case, you > could manually lower fsync to just "on" and do your own research on > whether power loss can toast

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-25 Thread Jelte Fennema-Nio
(pressed send to early) On Sat, 25 May 2024 at 12:39, Jelte Fennema-Nio wrote: > But again if I'm alone in this, then I don't ... mind budging on this to move this decision along. Using _pq_.xxx parameters for all protocol changes would totally be acceptable to me.

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-25 Thread Jelte Fennema-Nio
On Thu, 23 May 2024 at 20:12, Tom Lane wrote: > > Jelte Fennema-Nio writes: > > On Fri, 17 May 2024 at 21:24, Robert Haas wrote: > >> Perhaps these are all minority positions, but I can't tell you what > >> everyone thinks, only what I think. > > > I'd

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-25 Thread Jelte Fennema-Nio
On Thu, 23 May 2024 at 20:40, Tom Lane wrote: > > Jacob Champion writes: > > Would it be good to expand on that idea of criticality? IIRC one of > > Jelte's complaints earlier was that middleware has to know all the > > extension types anyway, to be able to figure out whether it has to do > >

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-24 Thread Jelte Fennema-Nio
On Fri, 24 May 2024 at 15:28, Robert Haas wrote: > > On Thu, May 23, 2024 at 4:00 PM Tom Lane wrote: > > I don't recall exactly what I thought earlier, but now I think we'd > > be better off with separate infrastructure. guc.c is unduly complex > > already. Perhaps there are bits of it that

Re: libpq compression (part 3)

2024-05-21 Thread Jelte Fennema-Nio
On Mon, 20 May 2024 at 21:42, Jacob Champion wrote: > As Andrey points out, there was prior work done that started to take > this into account. I haven't reviewed it to see how good it is -- and > I think there are probably many use cases in which queries and tables > contain both private and

Re: Multiple startup messages over the same connection

2024-05-19 Thread Jelte Fennema-Nio
On Sat, 18 May 2024 at 23:10, Vladimir Churyukin wrote: > I guess the process of passing control from child processes to the parent > could be a bit tricky for that one, but doable? > Is there anything I'm missing that can be a no-go for this? One seriously difficult/possibly impossible thing

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 21:24, Robert Haas wrote: > I think the > fear that we're going to run into cases where such complex handshaking > is necessary is a major reason why I'm afraid of Jelte's ideas about > ParameterSet: it seems much more opinionated to me than he seems to > think it is. I

Re: libpq compression (part 3)

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 23:10, Jacob Champion wrote: > We're talking about a transport-level option, though -- I thought the > proposal enabled compression before authentication completed? Or has > that changed? I think it would make sense to only compress messages after authentication has

Re: libpq compression (part 3)

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 23:40, Robert Haas wrote: > To be clear, I am not arguing that it should be the receiver's choice. > I'm arguing it should be the client's choice, which means the client > decides what it sends and also tells the server what to send to it. > I'm open to counter-arguments,

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 17:59, Robert Haas wrote: > If they > get attention, it's much more likely to be because somebody saw their > email and wrote back than it is to be because somebody went through > the CommitFest and found their entry and was like "oh, I should review > this". I think this

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 14:19, Joe Conway wrote: > > On 5/16/24 22:26, Robert Haas wrote: > > For example, imagine that the CommitFest is FORCIBLY empty > > until a week before it starts. You can still register patches in the > > system generally, but that just means they get CI runs, not that > >

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 13:39, Robert Haas wrote: > > On Fri, May 17, 2024 at 7:11 AM Tomas Vondra > wrote: > > Yeah, I 100% agree with this. If a patch bitrots and no one cares enough > > to rebase it once in a while, then sure - it's probably fine to mark it > > RwF. But forcing all

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 11:02, Jelte Fennema-Nio wrote: > > On Fri, 17 May 2024 at 10:47, Daniel Gustafsson wrote: > > One way to ensure we capture detail could be if the system would send an > > automated email to the thread summarizing the entry when it's marked

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 10:47, Daniel Gustafsson wrote: > One way to ensure we capture detail could be if the system would send an > automated email to the thread summarizing the entry when it's marked as > "committed"? This sounds great! Especially if Going back from an archived thread to the

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-17 Thread Jelte Fennema-Nio
On Fri, 17 May 2024 at 06:58, Peter Eisentraut wrote: > Yeah, some fine-tuning might be required. But I would be wary of > over-designing too many new states at this point. I think the key idea > is that there ought to be a state that communicates "needs attention > from someone other than

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-16 Thread Jelte Fennema-Nio
On Thu, 16 May 2024 at 18:57, Robert Haas wrote: > Ugh, it's so hard to communicate clearly about this stuff. I didn't > really have any thought that we'd ever try to handle something as > complicated as compression using ParameterSet. Okay, then it's definitely very hard to communicate clearly

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-16 Thread Jelte Fennema-Nio
On Thu, 16 May 2024 at 17:29, Robert Haas wrote: > You're probably not going to like this answer, but I feel like this is > another sign that you're trying to use the protocol extensibility > facilities in the wrong way. In my first reply to the thread, I > proposed having the client send

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-16 Thread Jelte Fennema-Nio
On Thu, 16 May 2024 at 03:45, Jonathan S. Katz wrote: > Attached is a copy of the PostgreSQL 17 Beta 1 release announcement > draft. I think we can quickly mention c4ab7da6061 in the COPY paragraph, in some benchmarks it improved perf by close to 2x. Something like this: "has improved

Re: First draft of PG 17 release notes

2024-05-16 Thread Jelte Fennema-Nio
On Thu, 16 May 2024 at 05:48, Andres Freund wrote: > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-16 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 18:30, Dave Cramer wrote: > On Fri, 5 Apr 2024 at 12:09, Jelte Fennema-Nio wrote: >> I'll take a look at redesigning the protocol parameter stuff. To work >> with dedicated functions instead. > > +1 It's been a while, but I now actually

Re: Postgres and --config-file option

2024-05-15 Thread Jelte Fennema-Nio
On Wed, 15 May 2024 at 11:49, Peter Eisentraut wrote: > Yeah, some of this is becoming quite unwieldy, if we document and > mention each spelling variant of each option everywhere. > > Maybe if the original problem is that the option --config-file is not > explicitly in the --help output, let's

Re: First draft of PG 17 release notes

2024-05-14 Thread Jelte Fennema-Nio
On Tue, 14 May 2024 at 02:56, Bruce Momjian wrote: > > On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > > The buffering change improved performance up to ~40% in some of the > > > benchmarks. The c

Re: JIT compilation per plan node

2024-05-14 Thread Jelte Fennema-Nio
On Tue, 14 May 2024 at 10:19, David Rowley wrote: > Here's a plan where the total cost of a subnode is greater than the > total cost of the top node: Ah I didn't realize it was possible for that to happen. **reads up on plan costs** This actually makes me think that using total_cost of the

Re: JIT compilation per plan node

2024-05-14 Thread Jelte Fennema-Nio
On Tue, 20 Feb 2024 at 06:38, David Rowley wrote: > > On Tue, 20 Feb 2024 at 18:31, Tom Lane wrote: > > FWIW, I seriously doubt that an extra walk of the plan tree is even > > measurable compared to the number of cycles JIT compilation will > > expend if it's called. So I don't buy your

Re: Direct SSL connection with ALPN and HBA rules

2024-05-13 Thread Jelte Fennema-Nio
On Mon, 13 May 2024 at 18:14, Robert Haas wrote: > I disagree with Jacob's assertion that sslmode=require has no security > benefits over sslmode=prefer. That seems like the kind of pessimism > that makes people hate security professionals. There have got to be > some attacks that are foreclosed

Re: Direct SSL connection with ALPN and HBA rules

2024-05-13 Thread Jelte Fennema-Nio
On Mon, 13 May 2024 at 13:07, Heikki Linnakangas wrote: > "channel_binding=require sslmode=require" also protects from MITM attacks. Cool, I didn't realize we had this connection option and it could be used like this. But I think there's a few security downsides of channel_binding=require over

Re: Direct SSL connection with ALPN and HBA rules

2024-05-13 Thread Jelte Fennema-Nio
On Mon, 13 May 2024 at 15:38, Heikki Linnakangas wrote: > Here's a patch to implement that. + if (conn->sslnegotiation[0] == 'd' && + conn->sslmode[0] != 'r' && conn->sslmode[0] != 'v') I think these checks should use strcmp instead of checking magic first characters. I see this

Re: Direct SSL connection with ALPN and HBA rules

2024-05-13 Thread Jelte Fennema-Nio
On Sun, 12 May 2024 at 23:39, Heikki Linnakangas wrote: > You might miss that by changing sslnegotiation to 'postgres', or by > removing it altogether, you not only made it compatible with older > server versions, but you also allowed falling back to a plaintext > connection. Maybe you're fine

Re: Direct SSL connection with ALPN and HBA rules

2024-05-11 Thread Jelte Fennema-Nio
On Fri, 10 May 2024 at 15:50, Heikki Linnakangas wrote: > New proposal: > > - Remove the "try both" mode completely, and rename "requiredirect" to > just "direct". So there would be just two modes: "postgres" and > "direct". On reflection, the automatic fallback mode doesn't seem very > useful.

Re: First draft of PG 17 release notes

2024-05-11 Thread Jelte Fennema-Nio
On Fri, 10 May 2024 at 23:31, Tom Lane wrote: > > Bruce Momjian writes: > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now

Re: First draft of PG 17 release notes

2024-05-10 Thread Jelte Fennema-Nio
On Thu, 9 May 2024 at 06:04, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html Great work! There are two commits that I think would benefit from being listed (but maybe

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-06 Thread Jelte Fennema-Nio
On Tue, 23 Apr 2024 at 19:39, Jacob Champion wrote: > > On Mon, Apr 22, 2024 at 2:20 PM Jelte Fennema-Nio wrote: > > 1. I strongly believe minor protocol version bumps after the initial > > 3.1 one can be made painless for clients/poolers (so the ones to > > 3.2/3.3/etc)

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-05-06 Thread Jelte Fennema-Nio
On Tue, 23 Apr 2024 at 17:03, Robert Haas wrote: > If a client doesn't know about encrypted columns and sets that > bit at random, that will break things, and formally I think that's a > risk, because I don't believe we document anywhere that you shouldn't > set unused bits in the format mask.

Re: Reducing the log spam

2024-05-02 Thread Jelte Fennema-Nio
On Thu, 2 May 2024 at 13:08, Jelte Fennema-Nio wrote: > 2. Change the newly added check in errcode() to only set > output_to_server to false when IsLogicalWorker() returns false. Actually a third, and probably even better solution would be to only apply this new GUC to non-background

Re: Reducing the log spam

2024-05-02 Thread Jelte Fennema-Nio
On Thu, 2 May 2024 at 12:47, Laurenz Albe wrote: > Yes. But I'd argue that that is a shortcoming of logical replication: > there should be a ways to get this information via SQL. Having to look into > the log file is not a very useful option. Definitely agreed that accessing the error details

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-22 Thread Jelte Fennema-Nio
able to change your mind. On Thu, 18 Apr 2024 at 21:34, Jelte Fennema-Nio wrote: > When the server supports a lower version than the client, the client > should disable certain features because it gets the > ProtocolVersionNegotiation message. This is also true if we don't bump > th

Re: Support a wildcard in backtrace_functions

2024-04-21 Thread Jelte Fennema-Nio
On Sat, 20 Apr 2024 at 01:19, Michael Paquier wrote: > Removing this GUC and making the backend react by default the same way > as when this GUC was enabled sounds like a sensible route here. This > stuff is useful. I definitely agree it's useful. But I feel like changing the default of the GUC

Re: Support a wildcard in backtrace_functions

2024-04-19 Thread Jelte Fennema-Nio
On Fri, 19 Apr 2024 at 10:58, Peter Eisentraut wrote: > This presupposes that there is consensus about how the future > functionality should look like. This topic has gone through half a > dozen designs over a few months, and I think it would be premature to > randomly freeze that discussion now

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 23:36, Jelte Fennema-Nio wrote: > To clarify: My proposed approach is to use a protocol extension > parameter for to enable the new messages that the server can send > (like Peter is doing now). And **in addition to that** gate the new > Bind format type behi

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 22:17, Robert Haas wrote: > If we go with Peter's approach, every driver that supports his feature > will work perfectly, and every driver that doesn't will work exactly > as it does today. The risk of breaking anything is as near to zero as > human developers can

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-18 Thread Jelte Fennema-Nio
en still bail out if some >> feature is not supported. > > I don't think a client should ever bail out. They may not support something > but IMO bailing out is not an option. On Thu, 18 Apr 2024 at 21:01, Robert Haas wrote: > On Thu, Apr 18, 2024 at 1:49 PM Jelte F

Re: Transparent column encryption

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 18:46, Robert Haas wrote: > With regard to the Bind message, I suggest that we regard the protocol > change as reserving a currently-unused bit in the message to indicate > whether the value is pre-encrypted, without reference to the protocol > extension. It could be legal

Re: Transparent column encryption

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 13:25, Peter Eisentraut wrote: > Hopefully, the reason for key rotation is mainly that policies require > key rotation, not that keys get compromised all the time. These key rotation policies are generally in place to reduce the impact of a key compromise by limiting the

Re: Support a wildcard in backtrace_functions

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 09:02, Michael Paquier wrote: > > On Fri, Apr 12, 2024 at 09:36:36AM +0900, Michael Paquier wrote: > > log_backtrace speaks a bit more to me as a name for this stuff because > > it logs a backtrace. Now, there is consistency on HEAD as well > > because these GUCs are all

Re: Support a wildcard in backtrace_functions

2024-04-18 Thread Jelte Fennema-Nio
On Thu, 18 Apr 2024 at 10:50, Peter Eisentraut wrote: > Why exactly is this an open item? Is there anything wrong with the > existing feature? The name of the GUC backtrace_on_internal_error is so specific that it's impossible to extend our backtrace behaviour in future releases without adding

Re: Speed up clean meson builds by ~25%

2024-04-17 Thread Jelte Fennema-Nio
On Wed, 17 Apr 2024 at 16:00, Tom Lane wrote: > Break gram.y (say, misspell some token in a production) and > see what happens. The behavior's likely to be timing sensitive > though. Thanks for clarifying. It took me a little while to break gram.y in such a way that I was able to consistently

Re: Speed up clean meson builds by ~25%

2024-04-17 Thread Jelte Fennema-Nio
On Wed, 17 Apr 2024 at 13:41, Peter Eisentraut wrote: > > On 10.04.24 17:33, Tom Lane wrote: > > The immediate question then is do we want to take Jelte's patch > > as a way to ameliorate the pain meanwhile. I'm kind of down on > > it, because AFAICS what would happen if you break the core > >

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-15 Thread Jelte Fennema-Nio
On Mon, 15 Apr 2024 at 19:43, Robert Haas wrote: > > On Sat, Apr 6, 2024 at 6:14 PM Jelte Fennema-Nio wrote: > > I think for clients/drivers, the work would generally be pretty > > minimal. For almost all proposed changes, clients can "support" the > > protocol

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-11 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 23:51, Tom Lane wrote: > Is it really necessary for Citus' filter to be a security qual rather > than a plain ol' filter condition? No, it's not. I think using security quals simply required the least amount of code (and it worked just fine if you didn't have lots of

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 22:11, Tom Lane wrote: > There may be an argument for psql to do what you suggest, > but so far it seems like duplicative complication. > > If there's a case you can demonstrate where "\d foo" doesn't optimize > into an indexscan, we should look into exactly why that's

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 20:21, Kirill Reshke wrote: > Do we need to force Collaction here like in other branches? > if (PQserverVersion(conn) >= 12) >appendPQExpBufferStr(buf, " COLLATE pg_catalog.default"); According to the commit and codecomment that introduced the COLLATE, it was

Re: psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 20:06, Tom Lane wrote: > Really? ISTM this argument is ignoring an optimization the backend > has understood for a long time. Interesting. I didn't know about that optimization. I can't check right now, but probably the COLLATE breaks that optimization.

psql: Greatly speed up "\d tablename" when not using regexes

2024-04-10 Thread Jelte Fennema-Nio
Running "\d tablename" from psql could take multiple seconds when running on a system with 100k+ tables. The reason for this was that a sequence scan on pg_class takes place, due to regex matching being used. Regex matching is obviously unnecessary when we're looking for an exact match. This

Re: Transparent column encryption

2024-04-10 Thread Jelte Fennema-Nio
On Wed, 10 Apr 2024 at 12:13, Peter Eisentraut wrote: > > To kick some things off for PG18, here is an updated version of the > patch for automatic client-side column-level encryption. I only read the docs and none of the code, but here is my feedback on the current design: > (The CEK can't be

Re: Support a wildcard in backtrace_functions

2024-04-10 Thread Jelte Fennema-Nio
On Fri, 22 Mar 2024 at 11:09, Jelte Fennema-Nio wrote: > On Thu, 21 Mar 2024 at 15:41, Jelte Fennema-Nio wrote: > > Attached is a patch that implements this. Since the more I think about > > it, the more I like this approach. > > I now added a 3rd patch to this p

Re: Add trim_trailing_whitespace to editorconfig file

2024-04-09 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 16:58, Jelte Fennema-Nio wrote: > > ISTM that with a small shell script, .editorconfig could be generated > > from .gitattributes? > > Honestly, I don't think building such automation is worth the effort. Okay, I spent the time to add a script to generat

Re: PostgreSQL 17 Release Management Team & Feature Freeze

2024-04-08 Thread Jelte Fennema-Nio
On Mon, 8 Apr 2024 at 20:15, Tomas Vondra wrote: > I 100% understand how frustrating the lack of progress can be, and I > agree we need to do better. I tried to move a number of stuck patches > this CF, and I hope (and plan) to do more of this in the future. > > But I don't quite see how would

Re: Flushing large data immediately in pqcomm

2024-04-08 Thread Jelte Fennema-Nio
On Sun, 7 Apr 2024 at 11:34, David Rowley wrote: > That seems to require modifying the following function signatures: > secure_write(), be_tls_write(), be_gssapi_write(). That's not an area > I'm familiar with, however. Attached is a new patchset where 0003 does exactly that. The only place

Re: Flushing large data immediately in pqcomm

2024-04-08 Thread Jelte Fennema-Nio
On Sun, 7 Apr 2024 at 14:41, David Rowley wrote: > Looking at the code in socket_putmessage_noblock(), I don't understand > why it's ok for PqSendBufferSize to be int but "required" must be > size_t. There's a line that does "PqSendBufferSize = required;". It > kinda looks like they both should

Re: Speed up clean meson builds by ~25%

2024-04-08 Thread Jelte Fennema-Nio
On Mon, 8 Apr 2024 at 10:02, Peter Eisentraut wrote: > I have tested this with various compilers, and I can confirm that this > shaves off about 5 seconds from the build wall-clock time, which > represents about 10%-20% of the total time. I think this is a good patch. Great to hear. >

Re: Speed up clean meson builds by ~25%

2024-04-08 Thread Jelte Fennema-Nio
On Mon, 8 Apr 2024 at 10:00, Alexander Lakhin wrote: > As I wrote in [1], I didn't observe the issue with clang-18, so maybe it > is fixed already. > Perhaps it's worth rechecking... Using the attached script I got these timings. Clang is significantly slower in all of them. But especially with

Re: Flushing large data immediately in pqcomm

2024-04-07 Thread Jelte Fennema-Nio
On Sun, 7 Apr 2024 at 03:39, Andres Freund wrote: > Changing the global vars to size_t seems mildly bogus to me. All it's > achieving is to use slightly more memory. It also just seems unrelated to the > change. I took a closer look at this. I agree that changing PqSendBufferSize to size_t is

Re: Flushing large data immediately in pqcomm

2024-04-06 Thread Jelte Fennema-Nio
On Sat, 6 Apr 2024 at 22:21, Andres Freund wrote: > The small regression for small results is still kinda visible, I haven't yet > tested the patch downthread. Thanks a lot for the faster test script, I'm also impatient. I still saw the small regression with David his patch. Here's a v6 where I

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-06 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 18:48, Robert Haas wrote: > Maybe we'd be better off adding a libpq connection > option that forces the use of a specific minor protocol version, but > then we'll need backward-compatibility code in libpq basically > forever. But maybe we need that anyway to avoid older and

Re: Flushing large data immediately in pqcomm

2024-04-06 Thread Jelte Fennema-Nio
On Sat, 6 Apr 2024 at 03:34, David Rowley wrote: > Does anyone else want to try the attached script on the v5 patch to > see if their numbers are better? On my machine (i9-10900X, in Ubuntu 22.04 on WSL on Windows) v5 consistently beats master by ~0.25 seconds: master: Run 100 100 500:

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 18:43, Tom Lane wrote: > I don't buy that argument, actually. libpq, and pretty much every > other client AFAIK, has provisions to let higher code levels insert > random options into the startup packet. So to make this work libpq > would have to filter or at least inspect

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 18:30, Dave Cramer wrote: >> > I really intended the _pq_ prefix as a way of taking something out of >> > the GUC namespace, not as a part of the GUC namespace that users would >> > see. And I'm reluctant to go back on that. If we want to make >> > pg_protocol.${NAME} mean a

Re: Speed up clean meson builds by ~25%

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 17:24, Andres Freund wrote: > I recommend opening a bug report for clang, best with an already preprocessed > input file. > We're going to need to do something about this from our side as well, I > suspect. The times aren't great with gcc either, even if not as bad as with

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 16:02, Robert Haas wrote: > Often? > > I kind of hope that the protocol starts to evolve a bit more than it > has, but I don't want a continuous stream of changes. That will be > very hard to test and verify correctness, and a hassle for drivers to > keep up with, and a mess

Re: Add new protocol message to change GUCs for usage with future protocol-only GUCs

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 16:04, Robert Haas wrote: > > On Thu, Apr 4, 2024 at 1:10 PM Jelte Fennema-Nio wrote: > > Attached is a rebased patchset > > We should keep talking about this, but I think we're far too close to > the wire at this point to think about commit

Re: Speed up clean meson builds by ~25%

2024-04-05 Thread Jelte Fennema-Nio
On Fri, 5 Apr 2024 at 00:45, Jelte Fennema-Nio wrote: > It improved clean build times on my machine (10 cores/20 threads) from ~40 > seconds to ~30 seconds. After discussing this off-list with Bilal, I realized that this gain is only happening for clang builds on my system. Because thos

Speed up clean meson builds by ~25%

2024-04-04 Thread Jelte Fennema-Nio
Building the generated ecpg preproc file can take a long time. You can check how long using: ninja -C build src/interfaces/ecpg/preproc/ecpg.p/meson-generated_.._preproc.c.o This moves that file much closer to the top of our build order, so building it can be pipelined much better with other

Re: WIP Incremental JSON Parser

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 23:52, Jelte Fennema-Nio wrote: > Fair enough, I guess I'll change my invocation to include the ninja > "test" target too: > ninja -C build test install-quiet && meson test -C build Actually that doesn't do what I want either because that

Re: WIP Incremental JSON Parser

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 23:34, Jacob Champion wrote: > > On Thu, Apr 4, 2024 at 1:42 PM Jelte Fennema-Nio wrote: > > Maybe that's something worth addressing too. I expected that > > install/install-quiet was a strict superset of a plain ninja > > invocation. > > Maybe

Re: Security lessons from liblzma

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 23:06, Daniel Gustafsson wrote: > > > On 4 Apr 2024, at 23:02, Jelte Fennema-Nio wrote: > > > How about we make it meson/make targets, so they are simply cached > > just like any of our other build artefacts are cached. Then only clean > >

Re: Security lessons from liblzma

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 22:56, Daniel Gustafsson wrote: > > > On 4 Apr 2024, at 22:47, Tom Lane wrote: > > > > Robert Haas writes: > >> On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: > >>> I don't disagree, like I said that very email: it's non-trivial and I > >>> wish we > >>> could

Re: WIP Incremental JSON Parser

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 20:48, Jacob Champion wrote: > > On Thu, Apr 4, 2024 at 11:12 AM Jacob Champion > wrote: > > What's in the `...`? I wouldn't expect to find the test binary in your > > tmp_install. > > Oh, I wonder if this is just a build dependency thing? I typically run > a bare `ninja`

Re: WIP Incremental JSON Parser

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 18:13, Andrew Dunstan wrote: > Argh. You get out of the habit when you're running with meson :-( I'm having some issues with meson too actually. "meson test -C build" is now failing with this for me: Command 'test_json_parser_incremental' not found in

  1   2   3   4   >