Re: Guiding principle for dropping LLVM versions?
On 1/25/24 13:41, Thomas Munro wrote: On Thu, Jan 25, 2024 at 4:44 PM Thomas Munro wrote: ... A few build farm animals will now fail in the configure step as discussed, and need some adjustment (ie disable LLVM or upgrade to LLVM 10+ for the master branch). Owners pinged. I think I fixed up the 4 or 6 under my name... Regards, Mark
Re: LLVM 16 (opaque pointers)
On Mon, Oct 23, 2023 at 10:47:24PM -0400, Tom Lane wrote: > Andres Freund writes: > > FC 29 is well out of support, so I don't think it makes sense to invest any > > further time in this. Personally, I don't think it's useful to have years > > old > > fedora in the buildfarm... > > +1. It's good to test old LTS distros, but Fedora releases have a > short shelf life by design. I'll start retiring those old Fedora ones I have. :) Regards, Mark
Re: LLVM 16 (opaque pointers)
On Tue, Oct 24, 2023 at 10:17:22AM +1300, Thomas Munro wrote: > On Tue, Oct 24, 2023 at 4:27 AM Mark Wong wrote: > > I haven't gotten around to disabling llvm on any of my animals with llvm > > < 7 yet. Do you still want to hold on that? > > Yes, please disable --with-llvm on s390x and POWER animals with LLVM < > 7 (see below). Also, you have a bunch of machines with LLVM 16 that > are failing to compile on REL_11_STABLE. That is expected, because we > agreed not to back-patch the LLVM 16 API changes into REL_11_STABLE: > > > kingsnake: 16.0.6 Fedora Linux 38 > > krait: CentOS 16.0.6 Stream 8 > > lancehead: CentOS 16.0.6 Stream 8 I should have updated these to not use --with-llvm for REL_11_STABLE. > These POWER machines fail as expected, and it's unfixable: > > > elasmobranch: 5.0.1 openSUSE Leap 15.0 > > demoiselle: 5.0.1 openSUSE Leap 15.0 > > cavefish: 6.0.0 Ubuntu 18.04.6 LTS These should now be updated to not use --with-llvm at all. > These s390x animals are failing, but don't show the layout complaint. > I can see that LLVM 6 also lacked a case for s390x in > llvm::orc::createLocalIndirectStubsManagerBuilder(), the thing that > was fixed in 7 with the addition of a default case. Therefore these > presumably fail just like old LLVM on POWER, and it's unfixable. So I > suggest turning off --with-llvm on these two: > > > cotinga: 6.0.0 Ubuntu 18.04.6 LTS > > perch: 6.0.0 Ubuntu 18.04.6 LTS Ok, I should have removed --with-llvm here too. > This s390x animal doesn't actually have --with-llvm enabled so it > passes, but surely it'd be just like lora: > > > mamushi: 15.0.7 Red Hat Enterprise Linux 9.2 Oops, I think I added it now. I think I made all the recommended changes, and trimmed out the lines where I didn't need to do anything. :) Andres pointed out to me that my animals aren't set up to collect core file so I'm also trying to update that too... Regards, Mark
Re: LLVM 16 (opaque pointers)
On Mon, Oct 23, 2023 at 01:15:04PM +1300, Thomas Munro wrote: > On Sun, Oct 22, 2023 at 2:44 PM Thomas Munro wrote: > > On Sat, Oct 21, 2023 at 2:45 PM Tom Lane wrote: > > > Thomas Munro writes: > > > > (It'd be nice if the > > > > build farm logged "$LLVM_CONFIG --version" somewhere.) > > > > > > It's not really the buildfarm script's responsibility to do that, > > > but feel free to make configure do so. > > > > Done, copying the example of how we do it for perl and various other things. > > Build farm measles update: > > With that we can see that nicator (LLVM 15 on POWER) is green. We can > see that cavefish (LLVM 6 on POWER) is red as expected. We can also > see that bonito (LLVM 7 on POWER) is red, so my earlier theory that > this might be due to the known bug we got fixed in LLVM 7 is not > enough. Maybe there are other things fixed on POWER somewhere between > those LLVM versions? I suspect it'll be hard to figure out without > debug builds and backtraces. I haven't gotten around to disabling llvm on any of my animals with llvm < 7 yet. Do you still want to hold on that? > One thing is definitely our fault, though. xenodermus shows failures > on REL_12_STABLE and REL_13_STABLE, using debug LLVM 6 on x86. I > couldn't reproduce this locally on (newer) debug LLVM, so I bugged > Andres for access to the host/libraries and chased it down. There is > some type punning for a function parameter REL_13_STABLE and earlier, > removed by Andres in REL_14_STABLE, and when I back-patched my > refactoring I effectively back-patched a few changes from his commit > df99ddc70b97 that removed the type punning, but I should have brought > one more line from that commit to remove another trace of it. See > attached. Here are my list of llvm-config versions and distros for s390x and POWER (didn't see any issues on aarch64 but I grabbed all the info at the same time.) s390x: branta: 10.0.0 Ubuntu 20.04.4 LTS cotinga: 6.0.0 Ubuntu 18.04.6 LTS perch: 6.0.0 Ubuntu 18.04.6 LTS sarus: 14.0.0 Ubuntu 22.04.1 LTS aracari: 15.0.7 Red Hat Enterprise Linux 8.6 pipit: 15.0.7 Red Hat Enterprise Linux 8.6 lora: 15.0.7 Red Hat Enterprise Linux 9.2 mamushi: 15.0.7 Red Hat Enterprise Linux 9.2 pike: 11.0.1 Debian GNU/Linux 11 rinkhals: 11.0.1 Debian GNU/Linux 11 POWER: bonito: 7.0.1 Fedora 29 cavefish: 6.0.0 Ubuntu 18.04.6 LTS demoiselle: 5.0.1 openSUSE Leap 15.0 elasmobranch: 5.0.1 openSUSE Leap 15.0 babbler: 15.0.7 AlmaLinux 8.8 pytilia: 15.0.7 AlmaLinux 8.8 nicator: 15.0.7 AlmaLinux 9.2 twinspot: 15.0.7 AlmaLinux 9.2 cascabel: 11.0.1 Debian GNU/Linux 11 habu: 16.0.6 Fedora Linux 38 kingsnake: 16.0.6 Fedora Linux 38 krait: CentOS 16.0.6 Stream 8 lancehead: CentOS 16.0.6 Stream 8 aarch64: boiga: 14.0.5 Fedora Linux 36 corzo: 14.0.5 Fedora Linux 36 desman: 16.0.6 Fedora Linux 38 motmot: 16.0.6 Fedora Linux 38 whinchat: 11.0.1 Debian GNU/Linux 11 jackdaw: 11.0.1 Debian GNU/Linux 11 blackneck: 7.0.1 Debian GNU/Linux 10 alimoche: 7.0.1 Debian GNU/Linux 10 bulbul: 15.0.7 AlmaLinux 8.8 broadbill: 15.0.7 AlmaLinux 8.8 oystercatcher: 15.0.7 AlmaLinux 9.2 potoo: 15.0.7 AlmaLinux 9.2 whiting: 6.0.0 Ubuntu 18.04.5 LTS vimba: 6.0.0 Ubuntu 18.04.5 LTS splitfin: 10.0.0 Ubuntu 20.04.6 LTS rudd: 10.0.0 Ubuntu 20.04.6 LTS turbot: 14.0.0 Ubuntu 22.04.3 LTS shiner: 14.0.0 Ubuntu 22.04.3 LTS ziege: 16.0.6 CentOS Stream 8 chevrotain: 11.0.1 Debian GNU/Linux 11 Regards, Mark
Re: LLVM 16 (opaque pointers)
On Sat, Oct 21, 2023 at 10:48:47AM +1300, Thomas Munro wrote: > On Thu, Oct 19, 2023 at 6:20 AM Thomas Munro wrote: > > Interestingly, a new problem just showed up on the the RHEL9 s390x > > machine "lora", where a previously reported problem [1] apparently > > re-appeared. It complains about incompatible layout, previously > > blamed on mismatch between clang and LLVM versions. I can see that > > its clang is v15 from clues in the conflig log, but I don't know which > > version of LLVM is being used. However, I see now that --with-llvm > > was literally just turned on, so there is no reason to think that this > > would have worked before or this work is relevant. Strange though -- > > we must be able to JIT further than that on s390x because we have > > crash reports in other threads (ie we made it past this and into other > > more advanced brokenness). > > I see that Mark has also just enabled --with-llvm on some POWER Linux > animals, and they have failed in various ways. The failures are > strangely lacking in detail. It seems we didn't have coverage before, > and I recall that there were definitely versions of LLVM that *didn't* > work for our usage in the past, which I'll need to dredge out of the > archives. I will try to get onto a cfarm POWER machine and see if I > can reproduce that, before and after these commits, and whose bug is > it etc. Yeah, I'm slowing enabling --with-llvm on POWER, s390x, and aarch64 (but none here yet as I write this)... > I doubt I can get anywhere near an s390x though, and we definitely had > pre-existing problems on that arch. If you want to send me your ssh key, I have access to these systems through OSUOSL and LinuxFoundation programs. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
Re: Buildfarm failures on urocryon
Hi, On Fri, Sep 01, 2023 at 11:27:47AM +0530, vignesh C wrote: > Hi, > > Recently urocryon has been failing with the following errors at [1]: > checking for icu-uc icu-i18n... no > configure: error: ICU library not found > If you have ICU already installed, see config.log for details on the > failure. It is possible the compiler isn't looking in the proper directory. > Use --without-icu to disable ICU support. > > configure:8341: checking whether to build with ICU support > configure:8371: result: yes > configure:8378: checking for icu-uc icu-i18n > configure:8440: result: no > configure:8442: error: ICU library not found > If you have ICU already installed, see config.log for details on the > failure. It is possible the compiler isn't looking in the proper directory. > Use --without-icu to disable ICU support. > > Urocryon has been failing for the last 17 days. > > I think ICU libraries need to be installed in urocryon to fix this issue. Oops, that's when I upgraded the build farm client (from v14 to v17). I think it's fixed now... Regards, Mark
Re: real/float example for testlibpq3
On Mon, Mar 20, 2023 at 01:37:57PM -0400, Gregory Stark (as CFM) wrote: > On Mon, 23 Jan 2023 at 11:54, Mark Wong wrote: > fficient way to communicate useful information. > > > > Yeah, I will try to cover all the data types we ship. :) I'll keep at > > it and drop the code examples. > > I assume this isn't going to happen for this commitfest? If you think > it is then shout otherwise I'll mark it Returned with Feedback and > move it on to the next release. Sounds good. I actually thought I did that already, thanks for catching that. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
Re: real/float example for testlibpq3
On Fri, Jan 20, 2023 at 01:12:07PM -0500, Robert Haas wrote: > On Fri, Jan 20, 2023 at 12:58 PM Tom Lane wrote: > > I don't mind if you write something like > > > > A float4 value is a 4-byte IEEE single-precision floating point > > number. It is transmitted in network byte order, so you must > > convert to local byte order. (C users can do this portably > > using the standard ntohl() function.) > > > > but I'm not sure an example is worth more than such a parenthetical > > comment. Perhaps others disagree, though. > > I don't disagree with that. > > I do think that when you suggested documenting this rather than just > adding some examples, you moved the goalposts a long way. If we're > going to add this to the documentation, it probably ought to cover > every data type we ship. Overall, I think that would be a better > result than just adding a few examples for the most common data types > to testlibpq*.c, but it's also substantially more work. I do agree > with you that if we're going to document this, as opposed to provide > examples, then a narrative style is more appropriate than a whole > bunch of small sample programs; maintaining working code in the > documentation seems like an annoying amount of maintenance and is > probably not the most efficient way to communicate useful information. Yeah, I will try to cover all the data types we ship. :) I'll keep at it and drop the code examples. Regards, Mark -- Mark Wong EDB: http://www.enterprisedb.com
Re: real/float example for testlibpq3
On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote: > Peter Eisentraut writes: > > On 01.11.22 09:15, Tom Lane wrote: > >> Agreed that the libpq manual is not the place for this, but I feel > >> like it will also be clutter in "Data Types". Perhaps we should > >> invent a new appendix or the like? Somewhere near the wire protocol > >> docs seems sensible. > > > Would that clutter the protocol docs? ;-) > > I said "near", not "in". At the time I was thinking "new appendix", > but I now recall that the wire protocol docs are not an appendix > but a chapter in the Internals division. So that doesn't seem like > quite the right place anyway. > > Perhaps a new chapter under "IV. Client Interfaces" is the right > place? > > If we wanted to get aggressive, we could move most of the nitpicky details > about datatype text formatting (e.g., the array quoting rules) there too. > I'm not set on that, but it'd make datatype.sgml smaller which could > hardly be a bad thing. > > > I suppose figuring out exactly where to put it and how to mark it up, > > etc., in a repeatable fashion is part of the job here. > > Yup. How does this look? I've simply moved things around into a new "Binary Format" section with the few parts that I've started for some quick feedback about whether this is looking like the right landing place. Regards, Mark diff --git a/doc/src/sgml/binary-format.sgml b/doc/src/sgml/binary-format.sgml index a297ece784..779b606ec9 100644 --- a/doc/src/sgml/binary-format.sgml +++ b/doc/src/sgml/binary-format.sgml @@ -6,9 +6,102 @@ pgsql binary format - This chapter describes the binary format used in the wire protocol. There - are a number of C examples for the data types used in PostgreSQL. We will - try to be as comprehensive as possible with the native data types. + This chapter describes the binary representation of the native PostgreSQL + data types and gives examples on how to handle each data type's binary format + by offering C code examples for each data types. + + We will try to cover all of the native data types... + + + + boolean + + + A boolean is transmitted as single byte that, when cast to an + int, will be 0 for + false and 1 for + true. + + + + + + + + real + + + A real is composed of 4 bytes and needs to be handled correctly + for byte order. + + + + + + + + + timestamp without time zone + + + A timestamp without time zone is a 64-bit data type + representing the number of microseconds since January 1, 2000. It can be + converted into a broken-down time representation by converting the time into + seconds and saving the microseconds elsewhere. + + + + Note that in C time is counted from January 1, 1970, so this difference + needs to be accounted for in addition to handling the network byte order. + + + + + + diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index 0d6be9a2fa..688f947107 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -51,6 +51,7 @@ + diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 2e271862fc..705b03f4aa 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -196,6 +196,7 @@ break is not needed in a wider output rendering. &lobj; &ecpg; &infoschema; + &binary-format;
Re: real/float example for testlibpq3
On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote: > Peter Eisentraut writes: > > On 01.11.22 09:15, Tom Lane wrote: > >> Agreed that the libpq manual is not the place for this, but I feel > >> like it will also be clutter in "Data Types". Perhaps we should > >> invent a new appendix or the like? Somewhere near the wire protocol > >> docs seems sensible. > > > Would that clutter the protocol docs? ;-) > > I said "near", not "in". At the time I was thinking "new appendix", > but I now recall that the wire protocol docs are not an appendix > but a chapter in the Internals division. So that doesn't seem like > quite the right place anyway. > > Perhaps a new chapter under "IV. Client Interfaces" is the right > place? > > If we wanted to get aggressive, we could move most of the nitpicky details > about datatype text formatting (e.g., the array quoting rules) there too. > I'm not set on that, but it'd make datatype.sgml smaller which could > hardly be a bad thing. > > > I suppose figuring out exactly where to put it and how to mark it up, > > etc., in a repeatable fashion is part of the job here. > > Yup. I'll take a stab at adding a new chapter and share how that looks. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
Re: s390x builds on buildfarm
Hi everyone, On Wed, Aug 10, 2022 at 6:56 AM Andrew Dunstan wrote: > > On 2022-08-10 We 09:04, Vivian Kong wrote: > > > > Hi, > > > > > > > > Are builds being paused on s390x as it looks like the s390x builds > > were last run 15 days ago. If so, wondering what is the reason for > > the pause and what is required to resume the builds? > > The OS the builds were running on seems to have reached end of life. > > Please let me know if we can help with getting them updated and resume > > the builds. > > > > > > > > > > Mark, I think you run most or all of these. Yeah, IBM moved me to new hardware and I haven't set them up yet. I will try to do that soon. Regards, Mark
Re: Asking for feedback on Pgperffarm
Hi Yedil, On Mon, Aug 08, 2022 at 02:50:17PM +0200, Yedil Serzhan wrote: > Dear hackers, > > I'm Yedil. I'm working on the project "Postgres Performance Farm" during > Gsoc. Pgperffarm is a project like Postgres build farm but focuses on the > performance of the database. Now it has 2 types of benchmarks, pgbench and > tpc-h. The website is online here <http://140.211.168.145/>, and the repo > is here <https://github.com/PGPerfFarm/pgperffarm_server>. > > I would like you to take a look at our website and, if possible, give some > feedback on, for example, what other data should be collected or what other > metrics could be used to compare performance. Nice work! We need to be careful with how results based on the TPC-H specification are presented. It needs to be changed, but maybe not dramatically. Something like "Fair use derivation of TPC-H". It needs to be clear that it's not an official TPC-H result. I think I've hinted at it in the #perffarm slack channel, that I think it would be better if you leveraged one of the already existing TPC-H derived kits. While I'm partial to dbt-3, because I'm trying to maintain it and because it sounded like you were starting to do something similar to that, I think you can save a good amount of effort from reimplementing another kit from scratch. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
Re: real/float example for testlibpq3
On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote: > Robert Haas writes: > > On Tue, Jun 14, 2022 at 1:40 PM Mark Wong wrote: > >> I've created a function for each data type with the idea that an example > >> for handling a specific data type can be more easily reviewed by looking > >> in a single place. > >> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN > >> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if > >> this is a good way to go. > > > I'm not sure that we want to let these test programs grow very large. > > In particular, I don't think we should let ourselves get sucked into > > adding an example for every data type under the sun -- if that's > > wanted, the solution is perhaps to add documentation for the binary > > formats, not hide impromptu documentation inside a test program. But > > doing this much seems OK to me. > > Yeah, "hiding impromptu documentation inside a test program" is what > this looks like, and I'm not sure that's a reasonable way to go. > > (1) Who's going to think to look in src/test/examples/testlibpq3.c for > documentation of binary formats? > > (2) The useful details are likely to get buried in notational and > portability concerns, as I think your build failure illustrates. > > (3) I bet few if any packagers install these files, so that the new > info would be unavailable to many people. > > I think some new appendix in the main SGML docs would be the appropriate > place if we want to provide real documentation. Just wanted to do another quick check in before I go too far. How do does this start look? Extending the libpq section with a code snippet and description per data type? Regards, Mark diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index 37ec3cb4e5..44c60223ee 100644 --- a/doc/src/sgml/libpq.sgml +++ b/doc/src/sgml/libpq.sgml @@ -8874,6 +8874,109 @@ int PQisthreadsafe(); + + Binary Format + +Various libpq functions support sending or +receiving data in binary format. Data in binary format requires the +application to be handle any differences between the system and the network +byte order. + + + + demonstrates how to write a complete +program that uses libpq functions that request +data in binary foramt. This example shows how to handle only a few data +types. This section will detail the handling of as many of the data types +as possible. See for descriptions of each of the +built-in data types. + + + + boolean + + A boolean is transmitted as single byte that, when cast to + an int, will be 0 for + false and 1 for + true. + + + + + + + + real + + A real is composed of 4 bytes and needs to be handled + correctly for byte order. + + + + + + + + timestamp without time zone + + A timestamp without time zone is a 64-bit data type + representing the number of microseconds since January 1, 2000. It can be + converted into a broken-down time representation by converting the time + into seconds and saving the microseconds elsewhere. + + + Note that in C time is counted from January 1, 1970 so this discrepency + needs to be accounted for in addition to handling the network byte order. + + + + + + + Building libpq Programs
Re: real/float example for testlibpq3
On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote: > Robert Haas writes: > > On Tue, Jun 14, 2022 at 1:40 PM Mark Wong wrote: > >> I've created a function for each data type with the idea that an example > >> for handling a specific data type can be more easily reviewed by looking > >> in a single place. > >> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN > >> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if > >> this is a good way to go. > > > I'm not sure that we want to let these test programs grow very large. > > In particular, I don't think we should let ourselves get sucked into > > adding an example for every data type under the sun -- if that's > > wanted, the solution is perhaps to add documentation for the binary > > formats, not hide impromptu documentation inside a test program. But > > doing this much seems OK to me. > > Yeah, "hiding impromptu documentation inside a test program" is what > this looks like, and I'm not sure that's a reasonable way to go. > > (1) Who's going to think to look in src/test/examples/testlibpq3.c for > documentation of binary formats? > > (2) The useful details are likely to get buried in notational and > portability concerns, as I think your build failure illustrates. > > (3) I bet few if any packagers install these files, so that the new > info would be unavailable to many people. > > I think some new appendix in the main SGML docs would be the appropriate > place if we want to provide real documentation. Ok, I'll leave the testlibpq3.c as it was then. If it's worth keeping any of those changes, then I can remove the timestamp example because of the ntohll() portability since that is trivial. I'll start a new appendix and share again when I have something to show. Regards, Mark -- Mark Wong EDB: http://www.enterprisedb.com
Re: real/float example for testlibpq3
On Wed, Mar 30, 2022 at 01:16:37PM -0400, Greg Stark wrote:
> On Mon, 28 Feb 2022 at 17:50, Tom Lane wrote:
> >
> > Chapman Flack writes:
> > > In the current state of affairs, what's considered the ur-source of that
> > > information?
> >
> > The source code for the type's send/receive functions :-(. One could
> > wish for something better, but no one has stepped up to produce such
> > documentation.
>
> Fwiw the client library I heard of attempting to have good binary mode
> support was the Crystal language client
> https://github.com/will/crystal-pg. I think he was aiming for full
> coverage of the built-in data types. That might make a good reference
> implementation to write up documentation from. He probably uncovered
> some corner cases in development that one might not find from just
> inspection of the server code.
Checking in for quick feedback to see if this refactor makes sense.
I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.
I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.
Regards,
Mark
diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c
index 4f7b791388..68972f17f3 100644
--- a/src/test/examples/testlibpq3.c
+++ b/src/test/examples/testlibpq3.c
@@ -11,21 +11,56 @@
* CREATE SCHEMA testlibpq3;
* SET search_path = testlibpq3;
* SET standard_conforming_strings = ON;
- * CREATE TABLE test1 (i int4, t text, b bytea);
- * INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
- * INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
+ * CREATE SCHEMA testlibpq3;
+ * SET search_path = testlibpq3;
+ * SET standard_conforming_strings = ON;
+ * CREATE TABLE test1 (
+ * i int4
+ * , r real
+ * , bo boolean
+ * , ts timestamp
+ * , t text
+ * , b bytea
+ * );
+ * INSERT INTO test1
+ * VALUES (
+ * 1
+ * , 3.141593
+ * , true
+ * , '2000-01-01 00:00:02.414213'
+ * , 'joe''s place'
+ * , '\000\001\002\003\004'
+ * );
+ * INSERT INTO test1
+ * VALUES (
+ * 2
+ * , 1.618033
+ * , false
+ * , '2000-01-01 00:00:01.465571'
+ * , 'ho there'
+ * , '\004\003\002\001\000'
+ * );
*
* The expected output is:
*
* tuple 0: got
* i = (4 bytes) 1
+ * r = (4 bytes) 3.141593
+ * bo = (1 bytes) 1
* t = (11 bytes) 'joe's place'
* b = (5 bytes) \000\001\002\003\004
*
* tuple 0: got
* i = (4 bytes) 2
+ * r = (4 bytes) 1.618033
+ * bo = (1 bytes) 0
* t = (8 bytes) 'ho there'
* b = (5 bytes) \004\003\002\001\000
+ *
+ * General notes about this example:
+ *
+ * Use PQfnumber to avoid assumptions about field order in result but when
+ * getting the field values we ignore possibility they are null!
*/
#ifdef WIN32
@@ -36,6 +71,7 @@
#include
#include
#include
+#include
#include
#include "libpq-fe.h"
@@ -44,6 +80,13 @@
#include
+/* These macros hopefully make reading calculations for timestamps easier. */
+#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
+#define SECS_PER_DAY 86400
+
+uint64_t ntohll(uint64_t);
+
static void
exit_nicely(PGconn *conn)
{
@@ -51,6 +94,142 @@ exit_nicely(PGconn *conn)
exit(1);
}
+static void
+handle_boolean(PGresult *res, int i)
+{
+ int fnum;
+ char *ptr;
+ int val;
+
+ fnum = PQfnumber(res, "bo");
+ ptr = PQgetvalue(res, i, fnum);
+ val = (int) *ptr;
+ printf(" bo = (%d bytes) %d\n", PQgetlength(res, i, fnum), val);
+}
+
+static void
+handle_bytea(PGresult *res, int i)
+{
+ int j;
+ int fnum;
+ char *ptr;
+ int len;
+
+ fnum = PQfnumber(res, "b");
+ ptr = PQgetvalue(res, i, fnum);
+
+ /*
+* The binary representation of BYTEA is a bunch of bytes, which could
+* include embedded nulls so we have to pay attention to field length.
+*/
+ len = PQgetlength(res, i, fnum);
+ printf(" b = (%d bytes) ", len);
+ for (j = 0; j < len; j++) printf("\\%03o", ptr[j]);
+}
+
+static void
+handle_integer(PGresult *res, int i)
+{
+ int fnum;
+ char *ptr;
+ int val;
+
+ fnum = PQfnumber(res, "i");
+ ptr = PQgetvalue(res, i, fnum);
+
+ /*
+* The binary representation of INT4 is in network byte order, which
+* we'd better coerce to the local byte order.
+*/
+ val = ntohl(*((uint32_t *) ptr));
+
+ printf(" i = (%d bytes) %d\n", PQgetlength(res, i, fnum), val);
+}
+
+static void
+handle_real(PGresult *res, int i)
+{
+ int fnum;
+ char
Re: DBT-5 Stored Procedure Development (2022)
On Mon, May 02, 2022 at 07:14:28AM -0700, Mark Wong wrote: > On Tue, Apr 26, 2022, 10:45 AM Peter Geoghegan wrote: > > > On Tue, Apr 26, 2022 at 10:36 AM Mark Wong wrote: > > > I'm afraid not. I'm guessing that pulling in egen 1.14 would address > > > that. Maybe it would make sense to put that on the top of todo list if > > > this project is accepted... > > > > Wouldn't it be a prerequisite here? I don't actually have any reason > > to prefer the old function-based code to the new stored procedure > > based code. Really, all I'm looking for is a credible implementation > > of TPC-E that I can use to model some aspects of OLTP performance for > > my own purposes. > > > > TPC-C (which I have plenty of experience with) has only two secondary > > indexes (in typical configurations), and doesn't really stress > > concurrency control at all. Plus there are no low cardinality indexes > > in TPC-C, while TPC-E has quite a few. Chances are high that I'd learn > > something from TPC-E, which has all of these things -- I'm really > > looking for bottlenecks, where Postgres does entirely the wrong thing. > > It's especially interesting to me as somebody that focuses on B-Tree > > indexing. I think it could be done in either order. While it's not ideal that the kit seems to work most reliably as-is on RHEL/Centos/etc. 6, I think that could provide some confidence in getting familiar with something on a working platform. The updates to the stored functions/procedures would be the same regardless of egen version. If we get the project slot, we can talk further about what to actually tackle first. Regards, Mark
Re: DBT-5 Stored Procedure Development (2022)
On Tue, Apr 19, 2022 at 05:20:50PM -0700, Peter Geoghegan wrote: > On Tue, Apr 19, 2022 at 11:31 AM Mark Wong wrote: > > As some of tasks proposed are actually in place, one other task could be > > updating egen (the TPC supplied code.) The kit was last developed again > > 1.12 and 1.14 is current as this email. > > As you know, I have had some false starts with using DBT5 on a modern > Linux distribution. Perhaps I gave up too easily at the time, but I'm > definitely still interested. Has there been work on that since? I'm afraid not. I'm guessing that pulling in egen 1.14 would address that. Maybe it would make sense to put that on the top of todo list if this project is accepted... Regards, Mark
Re: DBT-5 Stored Procedure Development (2022)
Hi Mahesh, On Tue, Apr 19, 2022 at 02:01:54PM -0400, Mahesh Gouru wrote: > Dear all, > > Please review the attached for my jerry-rigged project proposal. I am > seeking to continually refactor the proposal as I can! My comments might briefer that they should be, but I need to write this quickly. :) * The 4 steps in the description aren't needed, they already exist. * May 20: I think this should be more about reviewing the TPC-E specification rather than industry research, as we want to try to follow specification guidelines. * June 20: Random data generation and scaling are provided by and already defined by the spec * Aug 01: A report generator already exists, but I think time could be allocated to redoing the raw HTML generation with something like reStructuredText, something that is easier to generate with scripts and convertible into other formats with other tools As some of tasks proposed are actually in place, one other task could be updating egen (the TPC supplied code.) The kit was last developed again 1.12 and 1.14 is current as this email. Regards, Mark
Re: GSoC: Database Load Stress Benchmark (2022)
Hi! On Mon, Apr 18, 2022 at 03:40:23PM +0200, Mohammad Zain Abbas wrote: > Dear concerned, > > I hope you are doing well. > > I am Mohammad Zain Abbas, currently enrolled in Erasmus Mundus (BDMA) > program. I would like you to have a look at my proposal for the "*Database > Load Stress Benchmark" *project. > > Link: > https://docs.google.com/document/d/1TThl7ODGD301GkjITY2k4OU88fZIhc1XvJYGqPCnOns/edit?usp=sharing > > I would appreciate any feedback or guidance that you are able to provide. I think you've covered all the bases here. Good luck! Regards, Mark
Re: trigger example for plsample
On Thu, Apr 07, 2022 at 06:29:53PM -0400, Tom Lane wrote: > Chapman Flack writes: > > v4 looks good to me. > > Pushed with very minor editorialization. Mainly, I undid the > decision to stop printing the function source text, on the > grounds that (1) it falsified the comment immediately above, > and (2) if you have to print it anyway to avoid compiler warnings, > you're just creating confusing inconsistency between the two > handler functions. Sounds good to me, thanks! Regards, Mark
Re: trigger example for plsample
On Thu, Apr 07, 2022 at 10:30:13AM -0400, c...@anastigmatix.net wrote:
> On 2022-04-06 16:44, Mark Wong wrote:
> > I think I've applied all of these suggestions and attached a new patch.
>
> That looks good to me, though I wonder about the pfree(source).
> In the simplest case of a PL that uses no advance compilation or
> augmentation step, the Code Execution block might naturally refer
> to source, so perhaps the example boilerplate shouldn't include
> a pfree that needs to be removed in that case.
>
> In fact, I wonder about the need for any retail pfree()s here. Those
> added in this patch are the only ones in plsample.c. They are small
> allocations, and maybe it would both streamline the example to leave
> out the pfree calls, and be an illustration of best practice in letting
> the memory context machinery handle all the deallocation at once, where
> there isn't a special need to free something, like an especially large
> allocation, at retail.
Thanks, I've attached v4.
I've removed all of the pfree()'s and added an elog(DEBUG1) for source
to quiet a compiler warning about source's lack of use. :) (Was that a
good way?)
Regards,
Mark
diff --git a/src/test/modules/plsample/expected/plsample.out
b/src/test/modules/plsample/expected/plsample.out
index a0c318b6df..a7912c7897 100644
--- a/src/test/modules/plsample/expected/plsample.out
+++ b/src/test/modules/plsample/expected/plsample.out
@@ -6,9 +6,6 @@ AS $$
Example of source with text result.
$$ LANGUAGE plsample;
SELECT plsample_result_text(1.23, 'abc', '{4, 5, 6}');
-NOTICE: source text of function "plsample_result_text":
- Example of source with text result.
-
NOTICE: argument: 0; name: a1; value: 1.23
NOTICE: argument: 1; name: a2; value: abc
NOTICE: argument: 2; name: a3; value: {4,5,6}
@@ -25,12 +22,57 @@ AS $$
Example of source with void result.
$$ LANGUAGE plsample;
SELECT plsample_result_void('{foo, bar, hoge}');
-NOTICE: source text of function "plsample_result_void":
- Example of source with void result.
-
NOTICE: argument: 0; name: a1; value: {foo,bar,hoge}
plsample_result_void
--
(1 row)
+CREATE FUNCTION my_trigger_func() RETURNS trigger AS $$
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+$$ language plsample;
+CREATE TABLE my_table (num integer, description text);
+CREATE TRIGGER my_trigger_func BEFORE INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func();
+CREATE TRIGGER my_trigger_func2 AFTER INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func(8);
+INSERT INTO my_table (num, description)
+VALUES (1, 'first');
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+UPDATE my_table
+SET description = 'first, modified once'
+WHERE num = 1;
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+DROP TRIGGER my_trigger_func ON my_table;
+DROP TRIGGER my_trigger_func2 ON my_table;
+DROP FUNCTION my_trigger_func;
diff --git a/src/test/modules/plsample/plsample.c
b/src/test/modules/plsample/plsample.c
index 6fc33c728c..4e03dd5093 100644
--- a/src/test/modules/plsample/plsample.c
+++ b/src/test/modules/plsample/plsample.c
@@ -19,6 +19,7 @@
#include "catalog/pg_type.h"
#include "commands/event_trigger.h"
#include "commands/trigger.h"
+#include "executor/spi.h"
#include "funcapi.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -29,6 +30,7 @@ PG_MODULE_MAGIC;
PG_FUNCTION_INFO_V1(plsample_call_handler);
static Datum plsample_func_handler(PG_FUNCTION_ARGS);
+static HeapTuple plsample_trigger_handler(PG_FUNCTION_ARGS);
/*
* Handle function, procedure, and trigger calls.
@@ -51,6 +53,7 @@ plsample_call_handler(PG_FUNCTION_ARGS)
* (TriggerData *) fcinfo->context includes the
information of the
* context.
*/
+ retval =
Poin
Re: trigger example for plsample
On Thu, Mar 10, 2022 at 06:36:44PM -0500, Chapman Flack wrote:
> On 03/02/22 15:12, Mark Wong wrote:
>
> > I've attached v2, which reduces the output:
> >
> > * Removing the notices for the text body, and the "compile" message.
> > * Replaced the notice for "compile" message with a comment as a
> > placeholder for where a compiling code or checking a cache may go.
> > * Reducing the number of rows inserted into the table, thus reducing
> > the number of notice messages about which code path is taken.
>
> I think the simplifying assumption of a simple interpreted language allows
> a lot more of this code to go away. I mean more or less that whole first
> PG_TRY...PG_END_TRY block, which could be replaced with a comment saying
> something like
>
> The source text may be augmented here, such as by wrapping it as the
> body of a function in the target language, prefixing a parameter list
> with names like TD_name, TD_relid, TD_table_name, TD_table_schema,
> TD_event, TD_when, TD_level, TD_NEW, TD_OLD, and args, using whatever
> types in the target language are convenient. The augmented text can be
> cached in a longer-lived memory context, or, if the target language uses
> a compilation step, that can be done here, caching the result of the
> compilation.
>
> That would leave only the later PG_TRY block where the function gets
> "executed". That could stay largely as is, but should probably also have
> a comment within it, something like
>
> Here the function (the possibly-augmented source text, or the result
> of compilation if the target language uses such a step) should be
> executed, after binding these values from the TriggerData struct to
> the expected parameters.
>
> That should make the example shorter and clearer, and preserve the output
> tested by the regression test. Trying to show much more than that involves
> assumptions about what the PL's target language syntax looks like, how its
> execution engine works and parameters are bound, and so on, and that is
> likely to just be distracting, IMV.
I think I've applied all of these suggestions and attached a new patch.
Regards,
Mark
diff --git a/src/test/modules/plsample/expected/plsample.out
b/src/test/modules/plsample/expected/plsample.out
index a0c318b6df..a7912c7897 100644
--- a/src/test/modules/plsample/expected/plsample.out
+++ b/src/test/modules/plsample/expected/plsample.out
@@ -6,9 +6,6 @@ AS $$
Example of source with text result.
$$ LANGUAGE plsample;
SELECT plsample_result_text(1.23, 'abc', '{4, 5, 6}');
-NOTICE: source text of function "plsample_result_text":
- Example of source with text result.
-
NOTICE: argument: 0; name: a1; value: 1.23
NOTICE: argument: 1; name: a2; value: abc
NOTICE: argument: 2; name: a3; value: {4,5,6}
@@ -25,12 +22,57 @@ AS $$
Example of source with void result.
$$ LANGUAGE plsample;
SELECT plsample_result_void('{foo, bar, hoge}');
-NOTICE: source text of function "plsample_result_void":
- Example of source with void result.
-
NOTICE: argument: 0; name: a1; value: {foo,bar,hoge}
plsample_result_void
--
(1 row)
+CREATE FUNCTION my_trigger_func() RETURNS trigger AS $$
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+$$ language plsample;
+CREATE TABLE my_table (num integer, description text);
+CREATE TRIGGER my_trigger_func BEFORE INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func();
+CREATE TRIGGER my_trigger_func2 AFTER INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func(8);
+INSERT INTO my_table (num, description)
+VALUES (1, 'first');
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+UPDATE my_table
+SET description = 'first, modified once'
+WHERE num = 1;
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+DROP TRIGGER my_trigger_func ON my_table;
+DROP T
Re: trigger example for plsample
On Fri, Feb 25, 2022 at 06:39:39PM +, Chapman Flack wrote:
> This patch is straightforward, does what it says, and passes the tests.
>
> Regarding the duplication of code between plsample_func_handler and
> plsample_trigger_handler, perhaps that's for the best for now, as 3554 in
> the same commitfest also touches plsample, so merge conflicts may be
> minimized by not doing more invasive refactoring.
>
> That would leave low-hanging fruit for a later patch that could refactor
> plsample to reduce the duplication (maybe adding a validator at the same
> time? That would also duplicate some of the checks in the existing handlers.)
>
> I am not sure that structuring the trigger handler with separate compile and
> execute steps is worth the effort for a simple example like plsample. The main
> plsample_func_handler is not so structured.
>
> It's likely that many real PLs will have some notion of compilation separate
> from
> execution. But those will also have logic to do the compilation only once, and
> somewhere to cache the result of that for reuse across calls, and those kinds
> of
> details might make plsample's basic skeleton more complex than needed.
>
> I know that in just looking at expected/plsample.out, I was a little
> distracted by
> seeing multiple "compile" messages for the same trigger function in the same
> session and wondering why that was.
>
> So maybe it would be simpler and less distracting to assume that the PL
> targeted
> by plsample is one that just has a simple interpreter that works from the
> source text.
I've attached v2, which reduces the output:
* Removing the notices for the text body, and the "compile" message.
* Replaced the notice for "compile" message with a comment as a
placeholder for where a compiling code or checking a cache may go.
* Reducing the number of rows inserted into the table, thus reducing
the number of notice messages about which code path is taken.
I think that reduces the repetitiveness of the output...
Regards,
Mark
diff --git a/src/test/modules/plsample/expected/plsample.out
b/src/test/modules/plsample/expected/plsample.out
index a0c318b6df..a7912c7897 100644
--- a/src/test/modules/plsample/expected/plsample.out
+++ b/src/test/modules/plsample/expected/plsample.out
@@ -6,9 +6,6 @@ AS $$
Example of source with text result.
$$ LANGUAGE plsample;
SELECT plsample_result_text(1.23, 'abc', '{4, 5, 6}');
-NOTICE: source text of function "plsample_result_text":
- Example of source with text result.
-
NOTICE: argument: 0; name: a1; value: 1.23
NOTICE: argument: 1; name: a2; value: abc
NOTICE: argument: 2; name: a3; value: {4,5,6}
@@ -25,12 +22,57 @@ AS $$
Example of source with void result.
$$ LANGUAGE plsample;
SELECT plsample_result_void('{foo, bar, hoge}');
-NOTICE: source text of function "plsample_result_void":
- Example of source with void result.
-
NOTICE: argument: 0; name: a1; value: {foo,bar,hoge}
plsample_result_void
--
(1 row)
+CREATE FUNCTION my_trigger_func() RETURNS trigger AS $$
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+$$ language plsample;
+CREATE TABLE my_table (num integer, description text);
+CREATE TRIGGER my_trigger_func BEFORE INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func();
+CREATE TRIGGER my_trigger_func2 AFTER INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func(8);
+INSERT INTO my_table (num, description)
+VALUES (1, 'first');
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+UPDATE my_table
+SET description = 'first, modified once'
+WHERE num = 1;
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by UPDATE
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+DROP TRIGGER my_trigger_func ON my_table;
+DROP TRIGGER my_trigger_func2 ON my_table;
+DROP FUNCTION my_trigger_func;
diff --git a/src/test/modules/plsample/plsample.c
b/src/test/modules/plsample/plsample.c
index 6fc33c728c..fea266f522 100644
--- a/src/test/modules/plsample/plsample.c
+++ b/src/test/modules/plsample/plsample.c
@@ -19,6 +19,7 @@
#include "catalog/pg_type.h"
#include "commands/event_trigger.h"
#include "commands/trigger.h"
+#include "executor/sp
real/float example for testlibpq3
Hi everyone,
Would adding additional examples to testlibpq3.c on handling more data
types be helpful? I've attached a patch adding how to handle a REAL to
current example.
Regards,
Mark
diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c
index 4f7b791388..0c1739fcbb 100644
--- a/src/test/examples/testlibpq3.c
+++ b/src/test/examples/testlibpq3.c
@@ -11,19 +11,21 @@
* CREATE SCHEMA testlibpq3;
* SET search_path = testlibpq3;
* SET standard_conforming_strings = ON;
- * CREATE TABLE test1 (i int4, t text, b bytea);
- * INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
- * INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
+ * CREATE TABLE test1 (i int4, r real, t text, b bytea);
+ * INSERT INTO test1 values (1, 2.3, 'joe''s place', '\000\001\002\003\004');
+ * INSERT INTO test1 values (2, 4.5 'ho there', '\004\003\002\001\000');
*
* The expected output is:
*
* tuple 0: got
* i = (4 bytes) 1
+ * r = (4 bytes) 2.3
* t = (11 bytes) 'joe's place'
* b = (5 bytes) \000\001\002\003\004
*
* tuple 0: got
* i = (4 bytes) 2
+ * r = (4 bytes) 4.5
* t = (8 bytes) 'ho there'
* b = (5 bytes) \004\003\002\001\000
*/
@@ -62,32 +64,41 @@ show_binary_results(PGresult *res)
int i,
j;
int i_fnum,
+ r_fnum,
t_fnum,
b_fnum;
/* Use PQfnumber to avoid assumptions about field order in result */
i_fnum = PQfnumber(res, "i");
+ r_fnum = PQfnumber(res, "r");
t_fnum = PQfnumber(res, "t");
b_fnum = PQfnumber(res, "b");
for (i = 0; i < PQntuples(res); i++)
{
char *iptr;
+ char *rptr;
char *tptr;
char *bptr;
int blen;
int ival;
+ union {
+ int ival;
+ float fval;
+ } rval;
/* Get the field values (we ignore possibility they are null!)
*/
iptr = PQgetvalue(res, i, i_fnum);
+ rptr = PQgetvalue(res, i, r_fnum);
tptr = PQgetvalue(res, i, t_fnum);
bptr = PQgetvalue(res, i, b_fnum);
/*
-* The binary representation of INT4 is in network byte order,
which
-* we'd better coerce to the local byte order.
+* The binary representation of INT4 and REAL are in network
byte
+* order, which we'd better coerce to the local byte order.
*/
ival = ntohl(*((uint32_t *) iptr));
+ rval.ival = ntohl(*((uint32_t *) rptr));
/*
* The binary representation of TEXT is, well, text, and since
libpq
@@ -102,6 +113,8 @@ show_binary_results(PGresult *res)
printf("tuple %d: got\n", i);
printf(" i = (%d bytes) %d\n",
PQgetlength(res, i, i_fnum), ival);
+ printf(" r = (%d bytes) %f\n",
+ PQgetlength(res, i, r_fnum), rval.fval);
printf(" t = (%d bytes) '%s'\n",
PQgetlength(res, i, t_fnum), tptr);
printf(" b = (%d bytes) ", blen);
diff --git a/src/test/examples/testlibpq3.sql b/src/test/examples/testlibpq3.sql
index 35a95ca347..e3a70cec27 100644
--- a/src/test/examples/testlibpq3.sql
+++ b/src/test/examples/testlibpq3.sql
@@ -1,6 +1,6 @@
CREATE SCHEMA testlibpq3;
SET search_path = testlibpq3;
SET standard_conforming_strings = ON;
-CREATE TABLE test1 (i int4, t text, b bytea);
-INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
-INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
+CREATE TABLE test1 (i int4, r real, t text, b bytea);
+INSERT INTO test1 values (1, 2.3, 'joe''s place', '\000\001\002\003\004');
+INSERT INTO test1 values (2, 4.5, 'ho there', '\004\003\002\001\000');
Re: Time to drop plpython2?
On Wed, Feb 23, 2022 at 07:59:01AM -0800, Mark Wong wrote: > On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote: > > Mark Wong writes: > > > Take 3. :) > > > > > I've upgraded everyone to the v14 buildfarm scripts and made sure the > > > --test passed on HEAD on each one. So I hopefully didn't miss any > > > (other than the one EOL OpenSUSE version that I will plan on upgrading.) > > > > Thanks! > > > > However ... it seems like most of your animals haven't run at all > > in the last couple of days. Did you maybe forget to re-enable > > their cron jobs after messing with them, or something like that? > > Uh oh, more user error. I tested run_build but run_branches wasn't > happy. I'll start working through that... I think I have most of them operational again. I see some animals are still failing on some branches, but still better overall? I discovered that clang for gadwall and pintail got purged when I removed python2, so I disabled those two animals. Regards, Mark
Re: Time to drop plpython2?
On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote: > Mark Wong writes: > > Take 3. :) > > > I've upgraded everyone to the v14 buildfarm scripts and made sure the > > --test passed on HEAD on each one. So I hopefully didn't miss any > > (other than the one EOL OpenSUSE version that I will plan on upgrading.) > > Thanks! > > However ... it seems like most of your animals haven't run at all > in the last couple of days. Did you maybe forget to re-enable > their cron jobs after messing with them, or something like that? Uh oh, more user error. I tested run_build but run_branches wasn't happy. I'll start working through that... Regards, Mark
Re: Time to drop plpython2?
On Mon, Feb 21, 2022 at 03:28:35PM -0500, Tom Lane wrote: > Mark Wong writes: > > On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote: > >> Unfortunately it looks like it wasn't quite enough. All, or nearly all, > >> your > >> animals that ran since still seem to be failing in the same spot... > > > Oops, made another pass for python3 dev libraries. > > You might need to do one more thing, which is manually blow away the cache > files under $BUILDFARM/accache. For example, on demoiselle everything > looks fine in HEAD, but the back branches are failing like this: > > checking for python... (cached) /usr/bin/python > ./configure: line 10334: /usr/bin/python: No such file or directory > configure: using python > ./configure: line 10342: test: : integer expression expected > checking for Python sysconfig module... no > configure: error: sysconfig module not found > > Very recent versions of the buildfarm script will discard accache > automatically after a configure or make failure, but I think the > REL_11 you're running here doesn't have that defense. It'll only > flush accache after a change in the configure script in git. Take 3. :) I've upgraded everyone to the v14 buildfarm scripts and made sure the --test passed on HEAD on each one. So I hopefully didn't miss any (other than the one EOL OpenSUSE version that I will plan on upgrading.) Regards, Mark
Re: Time to drop plpython2?
On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
> Hi,
>
> On 2022-02-19 02:00:28 +0000, Mark Wong wrote:
> > On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> > > There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> > > menagerie. Mark said yesterday that he's working on updating.
> >
> > I've made one pass. Hopefully I didn't make any mistakes. :)
>
> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
> animals that ran since still seem to be failing in the same spot...
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gadwall&dt=2022-02-19%2011%3A22%3A48
>
> checking Python.h usability... no
> checking Python.h presence... no
> checking for Python.h... no
> configure: error: header file is required for Python
>
>
> For that machine (and the other debian based ones) the relevant package likely
> is python3-dev.
>
> For the Red Hat and Suse ones, it's likely python3-devel.
>
>
> I've wondered before if it's worth maintaining a list of packages for
> dependencies for at least the more popular distros. It's annoying to have to
> figure it out everytime one needs to test something.
>
>
> FWIW, here's the recipe I just used to verify the packages necessary for
> Python.h to be found:
>
> $ podman run --rm -it opensuse/leap
> # zypper install -y python3
> # ls -l $(python3 -c "import sysconfig;
> print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
>
> # zypper install -y python3-devel
> # ls -l $(python3 -c "import sysconfig;
> print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
> -rw-r--r-- 1 root root 3221 Jan 4 14:04 /usr/include/python3.6m/Python.h
>
> (Wow, zypper repos are expensive to refresh. And I thought dnf was slow doing
> so, compared to apt.)
Oops, made another pass for python3 dev libraries.
I can't seem to find archived ppc repos OpenSUSE Leap 43.2. I'm
debating whether to disable python or upgrade/rebrand that animal for a
newer SUSE release. I've stopped my cron jobs on this animal for the
time being.
Regards,
Mark
Re: Time to drop plpython2?
Hi everyone,
On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> menagerie. Mark said yesterday that he's working on updating.
I've made one pass. Hopefully I didn't make any mistakes. :)
Regards,
Mark
trigger example for plsample
Hi everyone,
I've adapted the work that Konstantina did for pl/julia as part of her
GSOC project to add an example of handling triggers to plsample. Which
was based from pl/tcl and pl/perl.
One aspect that I'm not sure about is whether the example should be
duplicating code (as it is now) for keeping an example contained within
a single function. The only reason I can come up with is to try to read
through an example with minimal jumping around.
Hoping this is a good start.
Regards,
Mark
diff --git a/src/test/modules/plsample/expected/plsample.out b/src/test/modules/plsample/expected/plsample.out
index a0c318b6df..832db79b5c 100644
--- a/src/test/modules/plsample/expected/plsample.out
+++ b/src/test/modules/plsample/expected/plsample.out
@@ -34,3 +34,280 @@ NOTICE: argument: 0; name: a1; value: {foo,bar,hoge}
(1 row)
+CREATE FUNCTION my_trigger_func() RETURNS trigger AS $$
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+$$ language plsample;
+CREATE TABLE my_table (num integer, description text);
+CREATE TRIGGER my_trigger_func BEFORE INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func();
+CREATE TRIGGER my_trigger_func2 AFTER INSERT OR UPDATE ON my_table
+ FOR EACH ROW EXECUTE FUNCTION my_trigger_func(8);
+INSERT INTO my_table (num, description)
+VALUES (1, 'first'), (2, 'second'), (1, 'first');
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: compile trigger function: my_trigger_func(TD_name, TD_relid, TD_table_name, TD_table_schema, TD_event, TD_when, TD_level, TD_NEW, TD_OLD, args)
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: out
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: compile trigger function: my_trigger_func(TD_name, TD_relid, TD_table_name, TD_table_schema, TD_event, TD_when, TD_level, TD_NEW, TD_OLD, args)
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: out
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: compile trigger function: my_trigger_func(TD_name, TD_relid, TD_table_name, TD_table_schema, TD_event, TD_when, TD_level, TD_NEW, TD_OLD, args)
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: trigger name: my_trigger_func
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered BEFORE
+NOTICE: triggered per row
+NOTICE: out
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: compile trigger function: my_trigger_func(TD_name, TD_relid, TD_table_name, TD_table_schema, TD_event, TD_when, TD_level, TD_NEW, TD_OLD, args)
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+NOTICE: out
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: compile trigger function: my_trigger_func(TD_name, TD_relid, TD_table_name, TD_table_schema, TD_event, TD_when, TD_level, TD_NEW, TD_OLD, args)
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+else
+return "OK"
+end
+
+NOTICE: trigger name: my_trigger_func2
+NOTICE: trigger relation: my_table
+NOTICE: trigger relation schema: public
+NOTICE: triggered by INSERT
+NOTICE: triggered AFTER
+NOTICE: triggered per row
+NOTICE: trigger arg[0]: 8
+NOTICE: out
+NOTICE: source text of function "my_trigger_func":
+if TD_event == "INSERT"
+return TD_NEW
+elseif TD_event == "UPDATE"
+return TD_NEW
+
Re: GSoc Applicant
Hello, On Mon, Apr 05, 2021 at 11:46:36PM +0200, Mohamed Mansour wrote: > Greetings, > > I'm Mohamed Mansour, a Data Engineer at IBM and a Master's degree student > in the Computer Engineering Department - Faculty of Engineering - Cairo > University. > > I would like to apply to google summer of code to work on the following > project: > > Database Load Stress Benchmark > > Kindly find my attached CV and tell me if there is a place for me related > to this project or if you see another project that fits me better, then I > will build the proposal as soon as possible I don't see anything in your CV that suggest you couldn't be successful in this project, but we'd like you to put together a proposal for the projects are you interested in. If this is the only project that is most interesting to you, then please go ahead and submit a draft for the mentors to review and we will offer our comments/suggestios on your proposal. Regards, Mark
Re: [GSoC] Question about Add functionality to pg_top and supporting tools
Hi Lu, On Sat, Mar 27, 2021 at 03:36:25PM +0800, Trafalgar Ricardo Lu wrote: > Hi, > > I am interested in "Add monitoring of pg_stat_statements to pg_systat". I > have read some code of pg_systat and enabled the pg_stat_statements > function. Thanks for your interest! > I noticed that pg_state_statements has many columns so that can't show in a > single view. Should I divided these columns into diffirent view or chose > some of them to show? Dividing the columns up into different views is ok. Some of the views are like that now. For example, the pg_stat_database data is split up between dbblk.c, dbtup.c, and dbxact.c. Regards, Mark
Re: Interest in GSoC 2021 Projects
Hi, On Fri, Feb 26, 2021 at 03:56:17PM +0800, Yixin Shi wrote: > Hi, > > > > I am Yixin Shi, a junior student majoring in Computer Science at University > of Michigan. I notice that the project ideas for GSoC 2021 have been posted > on your webpage and I am quite interested in two of them. I really wish to > take part in the project *Make plsample a more complete procedural language > handler example (2021)* considering both my interest and ability. The > potential mentor for this Project is *Mark Wong*. I am also interested in > the project *Improve PostgreSQL Regression Test Coverage (2021)* and the > potential mentor is *Andreas Scherbaum* and *Stephen Frost*. > > > > I would like to learn more about these two projects but failed to contact > the mentors. How can I contact them? Also, I really hope to join the > project. Are there any suggestions on application? The idea for plsample is to continue providing working code and documentation examples so that plsample can be used as a template for creating additional language handlers. Depending on the individual's comfort level, some effort may need to be put into studying the current handlers for ideas on how to implement the lacking functionality in plsample. Does that help? Regards, Mark
Re: buildfarm animal shoveler failing with "Illegal instruction"
On Thu, Oct 01, 2020 at 09:12:53PM -0400, Tom Lane wrote: > Mark Wong writes: > > I'm getting Tom set up with access too, in case he has time before me to > > get a stack trace to see what's happening... > > tl;dr: it's hard to conclude that this is anything but a compiler bug. > > I was able to reproduce this on shoveler's host, but only when using > the compiler shoveler uses (clang-3.9), not the 6.3 gcc that's also > on there and is of similar vintage. Observations: > > * You don't need any complicated test case; "pg_dump template1" > is enough. > > * Reverting 1ed6b8956's addition of a "postfix operators are not supported > anymore" warning to dumpOpr() makes it go away. This despite the fact > that that code is never reached when dumping template1. (We do enter > dumpOpr, but the oprinfo->dobj.dump test always fails.) > > * Reducing the optimization level to -O1 or -O0 makes it go away. > > * Inserting a debugging fprintf in dumpOpr makes it go away. > > Since clang 3.9 is several years old, maybe we could move shoveler > up to a newer version? Or dial it down to -O1 optimization? There is ayu, same system with clang 4.0, so covered on that front. I went ahead and stopped the jobs to run with clang 3.9. This is also the same system that was running clang 3.8 too. I tried looking for EOL dates, but had trouble finding anything... But I can change the optimization flag if we want it back. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: buildfarm animal shoveler failing with "Illegal instruction"
On Thu, Oct 01, 2020 at 12:12:44PM -0700, Andres Freund wrote: > Hi Mark, > > shoveler has been failing for a while with an odd error. E.g. > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=shoveler&dt=2020-09-18%2014%3A01%3A48 > > Illegal instruction > pg_dumpall: error: pg_dump failed on database "template1", exiting > waiting for server to shut down done > > None of the changes in that time frame look like they're likely causing > illegal instructions to be emitted that weren't before. So I am > wondering if anything changed on that machine around 2020-09-18 > 14:01:48 ? It looks like the last package update was 2020-06-10 06:59:26, according to the apt logs. I'm getting Tom set up with access too, in case he has time before me to get a stack trace to see what's happening... Regards, Mark
Re: doc examples for pghandler
On Fri, Aug 14, 2020 at 02:25:52PM +0900, Michael Paquier wrote:
> On Tue, Aug 11, 2020 at 01:01:10PM -0700, Mark Wong wrote:
> > Ah, right. For the moment I've added some empty conditionals for
> > trigger and event trigger handling.
> >
> > I've created a new entry in the commitfest app. [1] I'll keep at it. :)
>
> Thanks for the patch. I have reviewed and reworked it as the
> attached. Some comments below.
>
> +PGFILEDESC = "PL/Sample - procedural language"
> +
> +REGRESS = create_pl create_func select_func
> +
> +EXTENSION = plsample
> +EXTVERSION = 0.1
>
> This makefile has a couple of mistakes, and can be simplified a lot:
> - make check does not work, as you forgot a PGXS part.
> - MODULES can just be used as there is only one file (forgot WIN32RES
> in OBJS for example)
> - DATA does not need the .control file.
>
> .gitignore was missing.
>
> We could just use 1.0 instead of 0.1 for the version number. That's
> not a big deal one way or another, but 1.0 is more consistent with the
> other modules.
>
> plsample--1.0.sql should complain if attempting to load the file from
> psql. Also I have cleaned up the README.
>
> Not sure that there is a point in having three different files for the
> regression tests. create_pl.sql is actually not necessary as you
> can do the same with CREATE EXTENSION.
>
> The header list of plsample.c was inconsistent with the style used
> normally in modules, and I have extended a bit the handler function so
> as we return a result only if the return type of the procedure is text
> for the source text of the function, tweaked the results a bit, etc.
> There was a family of small issues, like using ALLOCSET_SMALL_SIZES
> for the context creation. We could of course expand the sample
> handler more in the future to check for pseudotype results, have a
> validator, but that could happen later, if necessary.
Thanks for fixing all of that up for me. I did have a couple mental
notes for a couple of the last items. :)
I've attached a small word diff to suggest a few different words to use
in the README, if that sounds better?
Regards,
Mark
--
Mark Wong
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
diff --git a/src/test/modules/plsample/README b/src/test/modules/plsample/README
index 7d44d7b3f2..afe3fa6402 100644
--- a/src/test/modules/plsample/README
+++ b/src/test/modules/plsample/README
@@ -2,5 +2,5 @@ PL/Sample
=
PL/Sample is an example template of procedural-language handler. It is
[-kept-]a [-maximum simple, and-]{+simple implementation, yet+} demonstrates
some of the things that can be
done to build a fully functional procedural-language handler.
Re: doc examples for pghandler
On Mon, Jun 15, 2020 at 04:47:01PM +0900, Michael Paquier wrote:
> On Sun, Jun 14, 2020 at 08:45:17PM -0700, Mark Wong wrote:
> > Sounds good to me. Something more like the attached patch?
>
> That's the idea. I have not gone in details into what you have here,
> but perhaps it would make sense to do a bit more and show how things
> are done in the context of a PL function called in a trigger? Your
> patch removes from the docs a code block that outlined that.
Ah, right. For the moment I've added some empty conditionals for
trigger and event trigger handling.
I've created a new entry in the commitfest app. [1] I'll keep at it. :)
Regards,
Mark
[1] https://commitfest.postgresql.org/29/2678/
--
Mark Wong
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
diff --git a/doc/src/sgml/plhandler.sgml b/doc/src/sgml/plhandler.sgml
index e1b0af7a60..7b2c5624c0 100644
--- a/doc/src/sgml/plhandler.sgml
+++ b/doc/src/sgml/plhandler.sgml
@@ -96,62 +96,12 @@
-This is a template for a procedural-language handler written in C:
-
-#include "postgres.h"
-#include "executor/spi.h"
-#include "commands/trigger.h"
-#include "fmgr.h"
-#include "access/heapam.h"
-#include "utils/syscache.h"
-#include "catalog/pg_proc.h"
-#include "catalog/pg_type.h"
-
-PG_MODULE_MAGIC;
-
-PG_FUNCTION_INFO_V1(plsample_call_handler);
-
-Datum
-plsample_call_handler(PG_FUNCTION_ARGS)
-{
-Datum retval;
-
-if (CALLED_AS_TRIGGER(fcinfo))
-{
-/*
- * Called as a trigger function
- */
-TriggerData*trigdata = (TriggerData *) fcinfo->context;
-
-retval = ...
-}
-else
-{
-/*
- * Called as a function
- */
-
-retval = ...
-}
-
-return retval;
-}
-
-Only a few thousand lines of code have to be added instead of the
-dots to complete the call handler.
-
-
-
-After having compiled the handler function into a loadable module
-(see ), the following commands then
-register the sample procedural language:
-
-CREATE FUNCTION plsample_call_handler() RETURNS language_handler
-AS 'filename'
-LANGUAGE C;
-CREATE LANGUAGE plsample
-HANDLER plsample_call_handler;
-
+A template for a procedural-language handler written as a C extension is
+provided in src/test/modules/plsample. This is a
+working sample demonstrating one way to create a procedural-language
+handler, process parameters, and return a value. A few thousand lines of
+additional code may have to be added to complete a fully functional
+handler.
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 29de73c060..95144d8d7c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -9,6 +9,7 @@ SUBDIRS = \
commit_ts \
dummy_index_am \
dummy_seclabel \
+ plsample \
snapshot_too_old \
test_bloomfilter \
test_ddl_deparse \
diff --git a/src/test/modules/plsample/Makefile b/src/test/modules/plsample/Makefile
new file mode 100644
index 00..757b47c785
--- /dev/null
+++ b/src/test/modules/plsample/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/plsample/Makefile
+
+PGFILEDESC = "PL/Sample - procedural language"
+
+REGRESS = create_pl create_func select_func
+
+EXTENSION = plsample
+EXTVERSION = 0.1
+
+MODULE_big = plsample
+
+OBJS = plsample.o
+
+DATA = plsample.control plsample--0.1.sql
+
+plsample.o: plsample.c
+
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
diff --git a/src/test/modules/plsample/README b/src/test/modules/plsample/README
new file mode 100644
index 00..7ee213700b
--- /dev/null
+++ b/src/test/modules/plsample/README
@@ -0,0 +1,3 @@
+plsample is an example procedural-language handler. It is a simple functional
+template that demonstrates some of the things that need to be done in order to
+build a fully functional procedural-language handler.
diff --git a/src/test/modules/plsample/expected/create_func.out b/src/test/modules/plsample/expected/create_func.out
new file mode 100644
index 00..df2b915a97
--- /dev/null
+++ b/src/test/modules/plsample/expected/create_func.out
@@ -0,0 +1,5 @@
+CREATE FUNCTION plsample_func(a1 NUMERIC, a2 TEXT, a3 INTEGER[])
+RETURNS TEXT
+AS $$
+ This is function's source text.
+$$ LANGUAGE plsample;
diff --git a/src/test/modules/plsample/expected/create_pl.out b/src/test/modules/plsample/expected/create_pl.out
new file mode 100644
index 00..5365391284
--- /dev/null
+++ b/src/test/modules/plsample/expected/create_pl.out
@@ -0,0 +1,8 @@
+CREATE FUNCTION plsample_call_handler()
+RETURNS language_handler
+AS '$libdir/plsample'
+LANGUAGE C;
+CREATE LANGUAGE plsample
+HANDLER plsample_call_handler;
+COMMENT ON LANGUAGE plsample
+IS 'PL/Sampl
Re: doc examples for pghandler
On Sat, Jun 13, 2020 at 01:19:17PM +0900, Michael Paquier wrote:
> On Fri, Jun 12, 2020 at 10:13:41PM -0400, Tom Lane wrote:
> > On second thought, contrib/ is not quite the right place, because we
> > typically expect modules there to actually get installed, meaning they
> > have to have at least some end-user usefulness. The right place for
> > a toy PL handler is probably src/test/modules/; compare for example
> > src/test/modules/test_parser/, which is serving quite the same sort
> > of purpose as a skeleton text search parser.
>
> +1 for src/test/modules/, and if you can provide some low-level API
> coverage through this module, that's even better.
Sounds good to me. Something more like the attached patch?
Regards,
Mark
--
Mark Wong
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
diff --git a/doc/src/sgml/plhandler.sgml b/doc/src/sgml/plhandler.sgml
index e1b0af7a60..7b2c5624c0 100644
--- a/doc/src/sgml/plhandler.sgml
+++ b/doc/src/sgml/plhandler.sgml
@@ -96,62 +96,12 @@
-This is a template for a procedural-language handler written in C:
-
-#include "postgres.h"
-#include "executor/spi.h"
-#include "commands/trigger.h"
-#include "fmgr.h"
-#include "access/heapam.h"
-#include "utils/syscache.h"
-#include "catalog/pg_proc.h"
-#include "catalog/pg_type.h"
-
-PG_MODULE_MAGIC;
-
-PG_FUNCTION_INFO_V1(plsample_call_handler);
-
-Datum
-plsample_call_handler(PG_FUNCTION_ARGS)
-{
-Datum retval;
-
-if (CALLED_AS_TRIGGER(fcinfo))
-{
-/*
- * Called as a trigger function
- */
-TriggerData*trigdata = (TriggerData *) fcinfo->context;
-
-retval = ...
-}
-else
-{
-/*
- * Called as a function
- */
-
-retval = ...
-}
-
-return retval;
-}
-
-Only a few thousand lines of code have to be added instead of the
-dots to complete the call handler.
-
-
-
-After having compiled the handler function into a loadable module
-(see ), the following commands then
-register the sample procedural language:
-
-CREATE FUNCTION plsample_call_handler() RETURNS language_handler
-AS 'filename'
-LANGUAGE C;
-CREATE LANGUAGE plsample
-HANDLER plsample_call_handler;
-
+A template for a procedural-language handler written as a C extension is
+provided in src/test/modules/plsample. This is a
+working sample demonstrating one way to create a procedural-language
+handler, process parameters, and return a value. A few thousand lines of
+additional code may have to be added to complete a fully functional
+handler.
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 29de73c060..95144d8d7c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -9,6 +9,7 @@ SUBDIRS = \
commit_ts \
dummy_index_am \
dummy_seclabel \
+ plsample \
snapshot_too_old \
test_bloomfilter \
test_ddl_deparse \
diff --git a/src/test/modules/plsample/Makefile b/src/test/modules/plsample/Makefile
new file mode 100644
index 00..757b47c785
--- /dev/null
+++ b/src/test/modules/plsample/Makefile
@@ -0,0 +1,20 @@
+# src/test/modules/plsample/Makefile
+
+PGFILEDESC = "PL/Sample - procedural language"
+
+REGRESS = create_pl create_func select_func
+
+EXTENSION = plsample
+EXTVERSION = 0.1
+
+MODULE_big = plsample
+
+OBJS = plsample.o
+
+DATA = plsample.control plsample--0.1.sql
+
+plsample.o: plsample.c
+
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
diff --git a/src/test/modules/plsample/README b/src/test/modules/plsample/README
new file mode 100644
index 00..7ee213700b
--- /dev/null
+++ b/src/test/modules/plsample/README
@@ -0,0 +1,3 @@
+plsample is an example procedural-language handler. It is a simple functional
+template that demonstrates some of the things that need to be done in order to
+build a fully functional procedural-language handler.
diff --git a/src/test/modules/plsample/expected/create_func.out b/src/test/modules/plsample/expected/create_func.out
new file mode 100644
index 00..df2b915a97
--- /dev/null
+++ b/src/test/modules/plsample/expected/create_func.out
@@ -0,0 +1,5 @@
+CREATE FUNCTION plsample_func(a1 NUMERIC, a2 TEXT, a3 INTEGER[])
+RETURNS TEXT
+AS $$
+ This is function's source text.
+$$ LANGUAGE plsample;
diff --git a/src/test/modules/plsample/expected/create_pl.out b/src/test/modules/plsample/expected/create_pl.out
new file mode 100644
index 00..5365391284
--- /dev/null
+++ b/src/test/modules/plsample/expected/create_pl.out
@@ -0,0 +1,8 @@
+CREATE FUNCTION plsample_call_handler()
+RETURNS language_handler
+AS '$libdir/plsample'
+LANGUAGE C;
+CREATE LANGUAGE plsample
+HANDLER plsample_call_handler;
+COMMENT ON LANG
Re: doc examples for pghandler
On Fri, Jun 12, 2020 at 03:10:20PM -0400, Tom Lane wrote: > Mark Wong writes: > > Would some additional procedure language handler code examples in the > > documentation be good to add? I've put some together in the attached > > patch, and can log it to a future commitfest if people think it would > > be a good addition. > > Hmm. The existing doc examples are really pretty laughable, because > there's such a large gap between the offered skeleton and a workable > handler. So I agree it'd be nice to do better, but I'm suspicious of > having large chunks of sample code in the docs --- that's a maintenance > problem, if only because we likely won't notice when we break it. > Also, if somebody is hoping to copy-and-paste such code, it isn't > that nice to work from if it's embedded in SGML. > > I wonder if it'd be possible to adapt what you have here into some > tiny contrib module that doesn't do very much useful, but can at > least be tested to see that it compiles; moreover it could be > copied verbatim to serve as a starting point for a new PL. I do have the code examples in a repo. [1] The 0.4 directory consists of everything the examples show. It would be easy enough to adapt that for contrib, and move some of the content from the doc patch into that. Then touch up the handler chapter to reference the contrib module. Does that sound more useful? [1] https://gitlab.com/markwkm/yappl -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
doc examples for pghandler
Hi everyone,
Would some additional procedure language handler code examples in the
documentation be good to add? I've put some together in the attached
patch, and can log it to a future commitfest if people think it would
be a good addition.
Regards,
Mark
--
Mark Wong
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
diff --git a/doc/src/sgml/plhandler.sgml b/doc/src/sgml/plhandler.sgml
index e1b0af7a60..0287d424cb 100644
--- a/doc/src/sgml/plhandler.sgml
+++ b/doc/src/sgml/plhandler.sgml
@@ -241,4 +241,560 @@ CREATE LANGUAGE plsample
reference page also has some useful details.
+
+The following subsections contain additional examples to help build a
+complete procedural language handler.
+
+
+
+Minimal Example
+
+
+ Here is a complete minimal example that builds a procedural language
+ handler PL/Sample as an extension. Functions
+ can be created and called for PL/Sample but
+ they will not be able to perform any usefule actions.
+
+
+
+ The plsample--0.1.sql file:
+
+CREATE FUNCTION plsample_call_handler()
+RETURNS language_handler
+AS 'MODULE_PATHNAME'
+LANGUAGE C;
+
+CREATE LANGUAGE plsample
+HANDLER plsample_call_handler;
+
+COMMENT ON LANGUAGE plsample IS 'PL/Sample procedural language';
+
+
+
+
+ The control file plsample.control looks like this:
+
+comment = 'PL/Sample'
+default_version = '0.1'
+module_pathname = '$libdir/plsample'
+relocatable = false
+schema = pg_catalog
+superuser = false
+
+ See for more information about writing
+ control files.
+
+
+
+ The following Makefile relies on
+ PGXS.
+
+PGFILEDESC = "PL/Sample - procedural language"
+
+EXTENSION = plsample
+EXTVERSION = 0.1
+
+MODULE_big = plsample
+
+OBJS = plsample.o
+
+DATA = plsample.control plsample--0.1.sql
+
+plsample.o: plsample.c
+
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+
+ See for more information on makefiles with
+ PGXS.
+
+
+
+ Here is the minimal C code in plsample.c that will
+ handle calls to this sample procedural language:
+
+#include <postgres.h>
+#include <fmgr.h>
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(plsample_call_handler);
+
+/*
+ * Handle function, procedure, and trigger calls.
+ */
+Datum
+plsample_call_handler(PG_FUNCTION_ARGS)
+{
+return 0;
+}
+
+
+
+
+ The following sections will continue building upon this example to
+ describe how to add additional functionality to a call handler for a
+ procedural language.
+
+
+
+
+Fetching the source of a function
+
+
+ Additional code is added to plsample.c from to include additional headers for the
+ additional code that fetches the source text of the function. The
+ resulting file now looks like:
+
+#include <postgres.h>
+#include <fmgr.h>
+#include <funcapi.h>
+#include <access/htup_details.h>
+#include <catalog/pg_proc.h>
+#include <catalog/pg_type.h>
+#include <utils/memutils.h>
+#include <utils/builtins.h>
+#include <utils/syscache.h>
+
+MemoryContext TopMemoryContext = NULL;
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(plsample_call_handler);
+
+/*
+ * Handle function, procedure, and trigger calls.
+ */
+
+Datum
+plsample_call_handler(PG_FUNCTION_ARGS)
+{
+HeapTuple pl_tuple;
+Datum pl_datum;
+const char *source;
+bool isnull;
+
+/* Fetch the source of the function. */
+
+pl_tuple = SearchSysCache(PROCOID,
+ObjectIdGetDatum(fcinfo->flinfo->fn_oid), 0, 0, 0);
+if (!HeapTupleIsValid(pl_tuple))
+elog(ERROR, "cache lookup failed for function %u",
+fcinfo->flinfo->fn_oid);
+
+pl_datum = SysCacheGetAttr(PROCOID, pl_tuple, Anum_pg_proc_prosrc,
+&isnull);
+if (isnull)
+elog(ERROR, "null prosrc");
+ReleaseSysCache(pl_tuple);
+
+source = DatumGetCString(DirectFunctionCall1(textout, pl_datum));
+elog(LOG, "source text:\n%s", source);
+
+return 0;
+}
+
+ The variable source containes the source that
+ needs to be interpreted and executed by the procedurual language handler
+ itself, or by an existing inplementation of the programming language that
+ the source text is written with.
+
+
+
+ The following CREATE FUNCTION will set the source text
+ of the function to This is function's source text.:
+
+CREATE FUNCTION plsample_func()
+RETURNS VOID
+AS $$
+ This is function's source text.
+$$ LANGUAGE plsample;
+
+
+
+
+ Calling the function with the following command will log the function's
+ source text because of the elog() statement
Re: GSOC 2020 - Develop Performance Farm Benchmarks and Website (2020)
Hi, On Mon, Mar 09, 2020 at 08:35:10PM +, do w.r. (wrd1e16) wrote: > I am very interested in the Develop Performance Farm Benchmarks and Website > (2020) project as one of the GSOC project. Is it possible to link me up with > Andreas Scherbaum to discuss more and further understand the project? I suggest reaching out on the #gsoc2020-students slack channel. Details on that, and other Postgres specific GSoC information, if you haven't already seen it: https://wiki.postgresql.org/wiki/GSoC Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: [GSoC 2020] Questions About Performance Farm Benchmarks and Website
Hi Kalvin, On Tue, Mar 03, 2020 at 03:07:13AM -0700, Kalvin Eng wrote: > Hi there, > > I am potentially interested in the performance farm project listed here: > https://wiki.postgresql.org/wiki/GSoC_2020#Develop_Performance_Farm_Benchmarks_and_Website_.282020.29 > > I've applied to the pgperffarm mailing list as well, but am waiting for > moderator approval so I thought this list would be the best to ask about > the performance farm code. > > Here are the questions based on the > https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary repo: > >- Why is a front end framework used instead of django templates? I don't have a good answer for this, primarily because my knowledge on the difference is weak... >- Any reason why the server hasn't been containerized? Simply because no effort has been put into it yet. Are you thinking for ease of demoing or evaluating? >- Django 1.11 will no longer be supported in April 2020, is it time to >move to 2.2 LTS? ( >https://www.djangoproject.com/download/#supported-versions) We want to match the same version the community infrastructure uses, so yes, if that's the version they will be on. >- What have been the issues with authentication integration to >postgresql.org? There is a custom authentication module that doesn't work outside of the community infrastructure, and this project has been developed outside of the community infrastructure. We haven't come up with a way to bridge that gap yet. >- Should the client be turned into a package for package managers (e.g. >pypi, DPKG, brew, etc.)? I think that would be a plus. >- The project description mentions refactoring to Python 3, but it seems >like that was completed last GSoC? Yeah, I think that's been squared away... >- Should the performance visualizations be added again? Yes, that would good to have. > I've also looked at past mailing lists for this project, but am interested > in hearing current insights from the community: > >- > > https://www.postgresql-archive.org/GSoC-2019-report-amp-feedback-td6100606.html >- > > https://www.postgresql-archive.org/GSoC-2019-Proposal-Develop-Performance-Farm-Database-and-Website-td6079058.html >- > > https://www.postgresql-archive.org/GSoC-Summery-of-pg-performance-farm-td6034578.html >- > > https://www.postgresql-archive.org/GSOC-18-Performance-Farm-Project-Initialization-Project-td6010380.html >- > > https://www.postgresql-archive.org/GSOC-18-Performance-Farm-Project-td6008120.html >- https://www.postgresql-archive.org/performance-test-farm-td4388584.html Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: [HACKERS] kqueue
On Sat, Jan 25, 2020 at 11:29:11AM +1300, Thomas Munro wrote: > On Thu, Jan 23, 2020 at 9:38 AM Rui DeSousa wrote: > > On Jan 22, 2020, at 2:19 PM, Tom Lane wrote: > >> It's certainly possible that to see any benefit you need stress > >> levels above what I can manage on the small box I've got these > >> OSes on. Still, it'd be nice if a performance patch could show > >> some improved performance, before we take any portability risks > >> for it. > > You might need more than one CPU socket, or at least lots more cores > so that you can create enough contention. That was needed to see the > regression caused by commit ac1d794 on Linux[1]. > > > Here is two charts comparing a patched and unpatched system. > > These systems are very large and have just shy of thousand > > connections each with averages of 20 to 30 active queries concurrently > > running at times including hundreds if not thousand of queries hitting > > the database in rapid succession. The effect is the unpatched system > > generates a lot of system load just handling idle connections where as > > the patched version is not impacted by idle sessions or sessions that > > have already received data. > > Thanks. I can reproduce something like this on an Azure 72-vCPU > system, using pgbench -S -c800 -j32. The point of those settings is > to have many backends, but they're all alternating between work and > sleep. That creates a stream of poll() syscalls, and system time goes > through the roof (all CPUs pegged, but it's ~half system). Profiling > the kernel with dtrace, I see the most common stack (by a long way) is > in a poll-related lock, similar to a profile Rui sent me off-list from > his production system. Patched, there is very little system time and > the TPS number goes from 539k to 781k. > > [1] > https://www.postgresql.org/message-id/flat/CAB-SwXZh44_2ybvS5Z67p_CDz%3DXFn4hNAD%3DCnMEF%2BQqkXwFrGg%40mail.gmail.com Just to add some data... I tried the kqueue v14 patch on a AWS EC2 m5a.24xlarge (96 vCPU) with FreeBSD 12.1, driving from a m5.8xlarge (32 vCPU) CentOS 7 system. I also use pgbench with a scale factor of 1000, with -S -c800 -j32. Comparing pg 12.1 vs 13-devel (30012a04): * TPS increased from ~93,000 to ~140,000, ~ 32% increase * system time dropped from ~ 78% to ~ 70%, ~ 8% decrease * user time increased from ~16% to ~ 23%, ~7% increase I don't have any profile data, but I've attached a couple chart showing the processor utilization over a 15 minute interval from the database system. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: stress test for parallel workers
On Sat, Oct 12, 2019 at 08:41:12AM +1300, Thomas Munro wrote: > On Sat, Oct 12, 2019 at 7:56 AM Tom Lane wrote: > > This matches up with the intermittent infinite_recurse failures > > we've been seeing in the buildfarm. Those are happening across > > a range of systems, but they're (almost) all Linux-based ppc64, > > suggesting that there's a longstanding arch-specific kernel bug > > involved. For reference, I scraped the attached list of such > > failures in the last three months. I wonder whether we can get > > the attention of any kernel hackers about that. > > Yeah, I don't know anything about this stuff, but I was also beginning > to wonder if something is busted in the arch-specific fault.c code > that checks if stack expansion is valid[1], in a way that fails with a > rapidly growing stack, well timed incoming signals, and perhaps > Docker/LXC (that's on Mark's systems IIUC, not sure about the ARM > boxes that failed or if it could be relevant here). Perhaps the > arbitrary tolerances mentioned in that comment are relevant. This specific one (wobbegon) is OpenStack/KVM[2], for what it's worth... "... cluster is an OpenStack based cluster offering POWER8 & POWER9 LE instances running on KVM ..." But to keep you on your toes, some of my ppc animals are Docker within other OpenStack/KVM instance... Regards, Mark [1] https://github.com/torvalds/linux/blob/master/arch/powerpc/mm/fault.c#L244 [2] https://osuosl.org/services/powerdev/ -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: stress test for parallel workers
On Thu, Oct 10, 2019 at 05:34:51PM -0400, Tom Lane wrote: > A nearer-term solution would be to reproduce this manually and > dig into the core. Mark, are you in a position to give somebody > ssh access to wobbegong's host, or another similarly-configured VM? > > (While at it, it'd be nice to investigate the infinite_recurse > failures we've been seeing on all those ppc64 critters ...) Yeah, whoever would like access, just send me your ssh key and login you'd like to use, and I'll get you set up. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Mon, May 20, 2019 at 12:15:49PM -0700, Mark Wong wrote: > On Sun, May 19, 2019 at 02:38:26PM -0700, Andres Freund wrote: > > Hi, > > > > On 2019-05-14 08:31:37 -0700, Mark Wong wrote: > > > Ok, I have this added to everyone now. > > > > > > I think I also have caught up on this thread, but let me know if I > > > missed anything. > > > > I notice > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=demoiselle&dt=2019-05-19%2014%3A22%3A23 > > failed recently, but unfortunately does not appear to have gdb > > installed? Or the buildfarm version is too old? Or ulimits are set > > strictly on a system wide basis? > > I'm on v9 of the build-farm here (have it on my todo list to get > everything up to 10.) Andrew let me know I need to get on v10. I've upgraded demoiselle, and am trying to work through the rest now... Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Sun, May 19, 2019 at 02:38:26PM -0700, Andres Freund wrote: > Hi, > > On 2019-05-14 08:31:37 -0700, Mark Wong wrote: > > Ok, I have this added to everyone now. > > > > I think I also have caught up on this thread, but let me know if I > > missed anything. > > I notice > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=demoiselle&dt=2019-05-19%2014%3A22%3A23 > failed recently, but unfortunately does not appear to have gdb > installed? Or the buildfarm version is too old? Or ulimits are set > strictly on a system wide basis? It looks like I did have gdb on there: opensuse@postgresql-opensuse-p9:~> gdb --version GNU gdb (GDB; openSUSE Leap 15.0) 8.1 I'm on v9 of the build-farm here (have it on my todo list to get everything up to 10.) I hope nothing is overriding my core size ulimit: opensuse@postgresql-opensuse-p9:~> ulimit -c unlimited This animal is using clang. I wonder if gdb is disagreeing with the clang binaries? Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Fri, May 10, 2019 at 05:26:43PM -0400, Andrew Dunstan wrote: > > On 5/10/19 3:35 PM, Tom Lane wrote: > > Andres Freund writes: > >> On 2019-05-10 11:38:57 -0400, Tom Lane wrote: > >>> I am wondering if, somehow, the stack depth limit seen by the postmaster > >>> sometimes doesn't apply to its children. That would be pretty wacko > >>> kernel behavior, especially if it's only intermittently true. > >>> But we're running out of other explanations. > >> I wonder if this is a SIGSEGV that actually signals an OOM > >> situation. Linux, if it can't actually extend the stack on-demand due to > >> OOM, sends a SIGSEGV. The signal has that information, but > >> unfortunately the buildfarm code doesn't print it. p $_siginfo would > >> show us some of that... > >> Mark, how tight is the memory on that machine? Does dmesg have any other > >> information (often segfaults are logged by the kernel with the code > >> IIRC). > > It does sort of smell like a resource exhaustion problem, especially > > if all these buildfarm animals are VMs running on the same underlying > > platform. But why would that manifest as "you can't have a measly two > > megabytes of stack" and not as any other sort of OOM symptom? > > > > Mark, if you don't mind modding your local copies of the buildfarm > > script, I think what Andres is asking for is a pretty trivial addition > > in PGBuild/Utils.pm's sub get_stack_trace: > > > > my $cmdfile = "./gdbcmd"; > > my $handle; > > open($handle, '>', $cmdfile) || die "opening $cmdfile: $!"; > > print $handle "bt\n"; > > + print $handle "p $_siginfo\n"; > > close($handle); > > > > > > > I think we'll need to write that as: > > > print $handle 'p $_siginfo',"\n"; Ok, I have this added to everyone now. I think I also have caught up on this thread, but let me know if I missed anything. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Tue, May 14, 2019 at 10:52:07AM -0400, Tom Lane wrote: > Mark Wong writes: > > Slowly catching up on my collection of ppc64le animals... > > Oh, btw ... you didn't answer my question from before: are these animals > all running on a common platform (and if so, what is that), or are they > really different hardware? If they're all VMs on one machine then it > might be that there's some common-mode effect from the underlying system. Sorry, I was almost there. :) These systems are provisioned with OpenStack. Additionally, a couple more (cardinalfish, devario) are using docker under that. > (Another thing I notice, now, is that these are all Linux variants; > I'd been thinking you had some BSDen in there too, but now I see > that none of those are ppc64. Hm.) Right, the BSDen I have are on different hardware. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Fri, May 10, 2019 at 11:27:07AM -0700, Andres Freund wrote: > Hi, > > On 2019-05-10 11:38:57 -0400, Tom Lane wrote: > > Core was generated by `postgres: debian regression [local] SELECT > > '. > > Program terminated with signal SIGSEGV, Segmentation fault. > > #0 sysmalloc (nb=8208, av=0x3fff916e0d28 ) at malloc.c:2748 > > 2748malloc.c: No such file or directory. > > #0 sysmalloc (nb=8208, av=0x3fff916e0d28 ) at malloc.c:2748 > > #1 0x3fff915bedc8 in _int_malloc (av=0x3fff916e0d28 , > > bytes=8192) at malloc.c:3865 > > #2 0x3fff915c1064 in __GI___libc_malloc (bytes=8192) at malloc.c:2928 > > #3 0x106acfd8 in AllocSetContextCreateInternal > > (parent=0x1000babdad0, name=0x1085508c "inline_function", > > minContextSize=, initBlockSize=, > > maxBlockSize=8388608) at aset.c:477 > > #4 0x103d5e00 in inline_function (funcid=20170, > > result_type=, result_collid=, > > input_collid=, funcvariadic=, > > func_tuple=, context=0x3fffe3da15d0, args=) > > at clauses.c:4459 > > #5 simplify_function (funcid=, result_type=, > > result_typmod=, result_collid=, > > input_collid=, args_p=, > > funcvariadic=, process_args=, > > allow_non_const=, context=) at clauses.c:4040 > > #6 0x103d2e74 in eval_const_expressions_mutator > > (node=0x1000babe968, context=0x3fffe3da15d0) at clauses.c:2474 > > #7 0x103511bc in expression_tree_mutator (node=, > > mutator=0x103d2b10 , > > context=0x3fffe3da15d0) at nodeFuncs.c:2893 > > > > So that lets out any theory that somehow we're getting into a weird > > control path that misses calling check_stack_depth; > > expression_tree_mutator does so for one, and it was called just nine > > stack frames down from the crash. > > Right. There's plenty places checking it... > > > > I am wondering if, somehow, the stack depth limit seen by the postmaster > > sometimes doesn't apply to its children. That would be pretty wacko > > kernel behavior, especially if it's only intermittently true. > > But we're running out of other explanations. > > I wonder if this is a SIGSEGV that actually signals an OOM > situation. Linux, if it can't actually extend the stack on-demand due to > OOM, sends a SIGSEGV. The signal has that information, but > unfortunately the buildfarm code doesn't print it. p $_siginfo would > show us some of that... > > Mark, how tight is the memory on that machine? There's about 2GB allocated: debian@postgresql-debian:~$ cat /proc/meminfo MemTotal:2080704 kB MemFree: 1344768 kB MemAvailable:1824192 kB At the moment it looks like plenty. :) Maybe I should set something up to monitor these things. > Does dmesg have any other > information (often segfaults are logged by the kernel with the code > IIRC). It's been up for about 49 days: debian@postgresql-debian:~$ uptime 14:54:30 up 49 days, 14:59, 3 users, load average: 0.00, 0.34, 1.04 I see one line from dmesg that is related to postgres: [3939350.616849] postgres[17057]: bad frame in setup_rt_frame: 3fffe3d9fe00 nip 3fff915bdba0 lr 3fff915bde9c But only that one time in 49 days up. Otherwise I see a half dozen hung_task_timeout_secs messages around jdb2 and dhclient. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Why is infinite_recurse test suddenly failing?
On Fri, May 03, 2019 at 11:45:33AM -0700, Andres Freund wrote: > Hi, > > On 2019-05-03 10:08:59 -0700, Mark Wong wrote: > > Ok, I think I have gdb installed now... > > Thanks! Any chance you could turn on force_parallel_mode for the other > branches it applies to too? Makes it easier to figure out whether > breakage is related to that, or independent. Slowly catching up on my collection of ppc64le animals... I still need to upgrade the build farm client (v8) on: * dhole * vulpes * wobbegong * cuon * batfish * devario * cardinalfish The following I've enabled force_parallel_mode for HEAD, 11, 10, and 9.6: * buri * urocryon * ayu * shoveler * chimaera * bonito * takin * bufflehead * elasmobranch * demoiselle * cavefish Regards, Mark
Re: Why is infinite_recurse test suddenly failing?
On Thu, May 02, 2019 at 11:45:34AM -0400, Tom Lane wrote: > Andres Freund writes: > > Hm, I just noticed: > >'HEAD' => [ > >'force_parallel_mode = > > regress' > > ] > > Oooh, I didn't see that. > > > on all those animals. So it's not necessarily the case that HEAD and > > backbranch runs are behaving all that identical. Note that isn't a > > recent config change, so it's not an explanation as to why they started > > to fail only recently. > > No, but it does point at another area of the code in which a relevant > change could've occurred. > > While we're looking at this --- Mark, if you could install gdb > on your buildfarm hosts, that would be really handy. I think that's > the only extra thing the buildfarm script needs to extract stack > traces from core dumps. We'd likely already know where the problem > is if we had a stack trace ... Ok, I think I have gdb installed now... Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: [GSoC 2019] Proposal: Develop Performance Farm Database and Website
Hi Ilaria, Edited for bottom posting. :) On Fri, Mar 29, 2019 at 03:01:05PM +0100, Ilaria wrote: > > Am 29.03.2019 um 13:52 schrieb Peter Eisentraut > > : > > > >> On 2019-03-29 13:04, Robert Haas wrote: > >>> On Tue, Mar 26, 2019 at 9:10 AM Ila B. wrote: > >>> I am Ilaria Battiston, an aspiring GSoC student, and I would love to have > >>> a feedback on the first draft of my Google Summer of Code proposal. The > >>> project is "Develop Performance Farm Database and Website”. You can find > >>> any other detail in the attached PDF file :) > >> > >> I think there's probably a very large amount of work to be done in > >> this area. Nobody is going to finish it in a summer. Still, there's > >> probably some useful things you could get done in a summer. I think a > >> lot will depend on finding a good mentor who is familiar with these > >> areas (which I am not). Has anyone expressed an interest? > > > > Moreover, I have a feeling that have been hearing about work on a > > performance farm for many years. Perhaps it should be investigated what > > became of that work and what the problems were getting it to a working > > state. > Hello, > > Thanks for the answer. This project is on the official PostgreSQL project > list of GSoC 2019, and potential mentors are stated there. > > I trust mentors’ judgement on outlining the work and the tasks to be done in > three months, and there is the previous student’s work to use as example if > needed. The project consists in building a database and a website on top of > it for users to browse performance data. > > Let me know whether there are any specific issues you’re concerned about. Hongyuan, our student last summer, put together a summary of his progress in a GitHub issue: https://github.com/PGPerfFarm/pgperffarm/issues/22 We have systems for proofing (from OSUOSL) and you can also see the prototype here: http://140.211.168.111/ For Phase 1, I'd recommend getting familiar with the database schema in place now. Perhaps it can use some tweaking, but I just mean to suggest that it might not be necessary to rebuild it from scratch. In Phase 2, we had some difficulty last year about getting the authentication/authorization completely integrated. I think the main issue was how to integrate this app while using resources outside of the community infrastructure. We may have to continue working around that. Otherwise, I think the rest make sense. Let us know if you have any more questions. Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: A note about recent ecpg buildfarm failures
On Tue, Feb 26, 2019 at 01:25:29PM -0500, Tom Lane wrote: > Since my commits 9e138a401 et al on Saturday, buildfarm members > blobfish, brotula, and wunderpus have been showing core dumps > in the ecpg preprocessor. This seemed inexplicable given what > the commits changed, and even more so seeing that only HEAD is > failing, while the change was back-patched into all branches. > > Mark Wong and I poked into this off-list, and what we find is that > this seems to be a compiler bug. Those animals are all running > nearly the same version of clang (3.8.x / ppc64le). Looking into > the assembly code for preproc.y, the crash is occurring at a branch > that is supposed to jump forward exactly 32768 bytes, but according > to gdb's disassembler it's jumping backwards exactly -32768 bytes, > into invalid memory. It will come as no surprise to hear that the > branch displacement field in PPC conditional branches is 16 bits > wide, so that positive 32768 doesn't fit but negative 32768 does. > Evidently what is happening is that either the compiler or the > assembler is failing to detect the edge-case field overflow and > switch to different coding. So the apparent dependency on 9e138a401 > is because that happened to insert exactly the right number of > instructions in-between to trigger this scenario. It's pure luck we > didn't trip over it before, although none of those buildfarm animals > have been around for all that long. > > Moral: don't use clang 3.8.x on ppc64. I think Mark is going > to upgrade those animals to some more recent compiler version. I've tried clang 3.9 and 4.0 by hand and they seem to be ok. These were the other two readily available versions on Debian stretch. I'll stop those other clang-3.8 animals... Regards, Mark -- Mark Wong 2ndQuadrant - PostgreSQL Solutions for the Enterprise https://www.2ndQuadrant.com/
Re: Odd 9.4, 9.3 buildfarm failure on s390x
On Sun, Sep 30, 2018 at 12:38:46AM +0100, Andrew Gierth wrote:
> >>>>> "Andrew" == Andrew Dunstan writes:
>
> >> What is the size of a C "int" on this platform?
>
> Andrew> 4.
>
> Hmm.
>
> Because int being more than 32 bits is the simplest explanation for this
> difference.
>
> How about the output of this query:
>
> with d(a) as (values ('----'::uuid),
> ('----'::uuid),
> ('3f3e3c3b-3a30-3938-3736-353433a2313e'::uuid))
> select d1.a, d2.a, uuid_cmp(d1.a,d2.a) from d d1, d d2
>order by d1.a asc, d2.a desc;
That also appears to produce the same results:
With 9.4:
postgres=# select version();
version
---
PostgreSQL 9.4.19 on s390x-ibm-linux-gnu, compiled by clang version 5.0.1
(tags/RELEASE_501/final 312548), 64-bit
(1 row)
...
a | a |
uuid_cmp
--+--+-
---- | ---- |
0
---- | ---- |
-2147483648
---- | 3f3e3c3b-3a30-3938-3736-353433a2313e |
-2147483648
---- | ---- |
1
---- | ---- |
0
---- | 3f3e3c3b-3a30-3938-3736-353433a2313e |
-2147483648
3f3e3c3b-3a30-3938-3736-353433a2313e | ---- |
1
3f3e3c3b-3a30-3938-3736-353433a2313e | ---- |
1
3f3e3c3b-3a30-3938-3736-353433a2313e | 3f3e3c3b-3a30-3938-3736-353433a2313e |
0
(9 rows)
Then with HEAD:
postgres=# select version();
version
PostgreSQL 12devel on s390x-ibm-linux-gnu, compiled by clang version 5.0.1
(tags/RELEASE_501/final 312548), 64-bit
(1 row)
...
a | a |
uuid_cmp
--+--+-
---- | ---- |
0
---- | ---- |
-2147483648
---- | 3f3e3c3b-3a30-3938-3736-353433a2313e |
-2147483648
---- | ---- |
1
---- | ---- |
0
---- | 3f3e3c3b-3a30-3938-3736-353433a2313e |
-2147483648
3f3e3c3b-3a30-3938-3736-353433a2313e | ---- |
1
3f3e3c3b-3a30-3938-3736-353433a2313e | ---- |
1
3f3e3c3b-3a30-3938-3736-353433a2313e | 3f3e3c3b-3a30-3938-3736-353433a2313e |
0
(9 rows)
Regards,
Mark
--
Mark Wonghttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: Odd 9.4, 9.3 buildfarm failure on s390x
Hi Andres,
On Fri, Sep 28, 2018 at 03:41:27PM -0700, Andres Freund wrote:
> On 2018-09-28 15:22:23 -0700, Mark Wong wrote:
> > On Fri, Sep 28, 2018 at 11:52:15AM -0700, Andres Freund wrote:
> > > Mark, is there anything odd for specific branches?
> >
> > No... I don't have anything in the config that would be applied to
> > specific branches...
>
> Could you perhaps do some manual debugging on that machine?
>
> Maybe starting with manually running something like:
>
> SELECT uuid_cmp('----'::uuid,
> '----'::uuid);
> SELECT uuid_cmp('----'::uuid,
> '----'::uuid);
> SELECT uuid_cmp('----'::uuid,
> '1113----'::uuid);
> SELECT uuid_cmp('----'::uuid,
> '1110----'::uuid);
>
> on both master and one of the failing branches?
I've attached the output for head and the 9.4 stable branch. It appears
they are returning the same results.
I built them both by:
CC=/usr/bin/clang ./configure --enable-cassert --enable-debug \
--enable-nls --with-perl --with-python --with-tcl \
--with-tclconfig=/usr/lib64 --with-gssapi --with-openssl \
--with-ldap --with-libxml --with-libxslt
What should I try next?
Regards,
Mark
--
Mark Wonghttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
psql (12devel)
Type "help" for help.
postgres=# SELECT uuid_cmp('----'::uuid,
'----'::uuid);
uuid_cmp
-
-2147483648
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'----'::uuid);
uuid_cmp
--
0
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'1113----'::uuid);
uuid_cmp
-
-2147483648
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'1110----'::uuid);
uuid_cmp
--
1
(1 row)
psql (9.4.19)
Type "help" for help.
postgres=# SELECT uuid_cmp('----'::uuid,
'----'::uuid);
uuid_cmp
-
-2147483648
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'----'::uuid);
uuid_cmp
--
0
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'1113----'::uuid);
uuid_cmp
-
-2147483648
(1 row)
postgres=# SELECT uuid_cmp('----'::uuid,
'1110----'::uuid);
uuid_cmp
--
1
(1 row)
Re: Odd 9.4, 9.3 buildfarm failure on s390x
On Fri, Sep 28, 2018 at 11:52:15AM -0700, Andres Freund wrote: > Mark, is there anything odd for specific branches? No... I don't have anything in the config that would be applied to specific branches... Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: [GSoC] Summery of pg performance farm
On Sun, Aug 19, 2018 at 12:20:37PM -0300, Alvaro Herrera wrote: > On 2018-Aug-19, Hongyuan Ma wrote: > > 2. Implementation of the data report related to the list page. Compare each > > metrics whith the previous results. If any of the metrics are a 5% > > improvement( or regression), there is one aspect that is progressive (or > > regressive). This means there may be aspects of "improvement", "regression" > > and "status quo" in one test result.This is the report List page for an > > example: http://140.211.168.111/#/machineInfo/pmJEjJjSk3WREM3Q > > Great stuff! > > I think the visualization that many had in mind was that the numbers > would be displayed in charts there time is the horizontal axis, and the > numerical performance result number is the other axis. That way, we > can see how the results go up or down with commits. > > Individual happy/sad faces for individual commits are not enough to see > the bigger picture. I advised Hongyuan to try something simple here at first. My initial thought was a quick indicator (to not take up too much space on the screen) and to drill down into the individual plant specifics to view more details of history. pgbench is running read-only and read-write tests with scale factors at 10, 1 x memory, and 2 x memory. We could reduce the variations of the tests if folks feel that would make more sense. I thought the current number of variations might be too many things to trend on this page, but we can change that. Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: ppc64le support in 9.3 branch?
On Tue, Mar 27, 2018 at 11:09:37AM +1030, Andrew Dunstan wrote: > On Tue, Mar 27, 2018 at 10:57 AM, Alvaro Herrera > wrote: > > Andrew Dunstan wrote: > > > >> I am in discussions with Mark, he's going to disable the animals from > >> building 9.3. (by setting branches_to_build to 'HEAD_PLUS_LATEST4' > >> instead of 'ALL'). > > > > So once we desupport 9.3, they'll stop building 9.4? :-) > > > > > Then we'll set it back to ALL :-) I'll put it on my calendar. :) Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: GSOC 2018 Proposal review
Hi Garima, On Tue, Mar 20, 2018 at 10:36:05PM +0530, Garima Natani wrote: > Hi All, > > I am looking forward to working with PostgreSQL organization In GSoC 2018. Glad to see you're interested in this project! > I have created Proposal for "Develop Performance Farm Database and Website" > Project. Please, > > Please, can you review it send comments? I think the general idea looks ok. The biggest item I'd adjust is that the performance farm plants (to distinguish them from the build farm animals) shouldn't connect directly to the database. Some of the django web app work needs to be moved up into phase one, because I'd propose the web app should also have an API that the build farm plants should use to transfer results. Good luck! Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: Re: Re: [GSOC 18] Performance Farm Project——Initialization Project
Hi, I have some additional comments to a couple areas... On Wed, Mar 14, 2018 at 05:33:00PM -0400, Dave Page wrote: > On Tue, Mar 13, 2018 at 11:31 PM, Hongyuan Ma wrote: > > At the same time I hope you can help me understand the functional > > requirements of this project more clearly. Here are some of my thoughts > > on PerfFarm: > > > > - I see this comment in the client folder (In line 15 of the "pgperffarm > > \ client \ benchmarks \ pgbench.py" file): > > ''' > > # TODO allow running custom scripts, not just the default > > ''' > > Will PerfFarm have many test items so that the test item search function > > or even the test item sort function is expected to be provided? > > > > I don't know - Tomas or Mark are probably the best folks to ask about that. > I spent some time on the initial web app work, but then ran out of spare > cycles. Yes, there is potential for the number of tests to change over time. (Maybe a test is less relevant, maybe there is a more useful test down he road.) I'm haven't thought too much about sorting and searching functions. But right now there is just pgbench with 6 basic configurations: read/write, read-only, and a low, medium, and high data set size for each of those. > > - What value will be used to determine if a machine's performance is > > improving or declining? > > > > Good question - I think that needs discussion, as it's not necessarily > clear-cut when you think that performance may vary at different numbers of > clients. My thought here is that each test should provide a single metric, defined by the test itself, so expect this to be define in the test results sent to the web site. I do expect that off of the test results should be archived somehow and retrievable. But the idea I have is to have a generic high level view that all of the different tests can share. > > - I see BuildFarm assigning an animal name to each registered machine. Will > > PerfFarm also have this interesting feature? > > > > It was going to, but I forget what we decided on for a naming scheme! > Another discussion item I think - in the code, we can just use "Name" or > similar. Since the buildfarm is composed of animals, I thought plants would be a complimentary scheme? I'm also not sure if this was discussed previously... Regards, Mark
Re: [GSOC 18] Discussion on the datatable
Hi Hongyuan,
On Tue, Mar 06, 2018 at 01:36:23AM +0800, Hongyuan Ma wrote:
> Hi Mark,
> In the past few days I have read some code in pgperffarm.git repository.I
> look forward to discussing the project in detail with you and gradually
> defining the datasheet structure and refining the requirements. Here are some
> of my ideas, if there are any errors or deficiencies, please feel free to
> correct me.
>
>
> To create a datasheet: pg_perf_cate
> Overview:
> pg_perf_cate table is used to store performance test project categories that
> support multi-level categories.
>
>
> Description:
> In line 12 of the "pgperffarm \ client \ benchmarks \ runner.py" file there
> is a line like this:
>
>
> ''
> 'manages runs of all the benchmarks, including cluster restarts etc.'
> ''
>
>
> In my imagination, there may be more items of performance tests than build
> tests. Based on the above comments, I guess, for example, may be there are
> "single instance of performance tests","cluster performance tests", "other
> performance tests" three major categories. Each major category also contains
> their own test sub-categories, such as addition tests and deletion tests and
> so on. In the pg_perf_cate table, the cate_pid field indicates the parent
> category of the current test category. If the pid field of a row of data has
> a value of 0, the row represents the top-level category.
>
>
> Related Functions:
> - Maybe in the navigation bar we can create a category menu to help users
> quickly find their own interest in the test items (similar to the Amazon Mall
> product categories). The cate_order field is used to manage the order of the
> categories in the current level for easy front-end menu display.
> - In the admin back-end need a page which can add or modify categories.
> -
> To create a datasheet: pg_perf_test
> Overview:
> The pg_perf_test table is used to store specific test items, including the
> test item number(test_id), the name of the test item(test_name), the ID of
> the sub-category(cate_id), the description of the test item (test_desc,to be
> discussed), and the person ID(user_id).
>
>
> Description:
> In line 15 of the "pgperffarm \ client \ benchmarks \ pgbench.py" file, I see
> a note like this:
> ''
> # TODO allow running custom scripts, not just the default
> ''
> Now that I want to allow users to run custom test scripts and upload them, I
> think it is necessary to tell others how to run the test again. So I want to
> add a test_desc field that will store the details about this test and tell
> the user how to run this test. But I am not too sure about the storage format
> for the details of the test, perhaps the rich text format or markdown format
> is a suitable choice.
> When this test item is added by the administrator, the user_id field has a
> value of 0. Otherwise, this field corresponds to the user_id field in the
> user table. For this field, I prefer not to use foreign keys.
>
>
> Related Functions:
> - At the front end, each test has its own detail page, on which the test
> related content is presented and a list of test results is listed.
> - In the admin background need a page to manage test items.
> -
> To create a datasheet: pg_perf_test_result
>
>
> Overview:
> The pg_perf_test_result table is used to store test results, including at
> least the result ID(result_id), user ID (user_id,I prefer not to create a
> user-test result association table), test item ID(test_id), test branch
> number(branch_id), system configuration(os_config), pg
> configuration(pg_config), test result details(result_desc) , test
> time(add_time) and other fields.
> Confusion:
> I think compared to other tables, pg_perf_test_result table may be a
> relatively complex one.
> This piece of code can be seen around line 110 of the "pgperffarm \ client \
> benchmarks \ runner.py" file:
> ''
> r ['meta'] = {
> 'benchmark': config ['benchmark'],
> 'date': strftime ("% Y-% m-% d% H:% M:% S.00 + 00",
> gmtime ()),
> 'name': config_name,
> 'uname': uname,
> }
>
>
> with open ('% s / results.json'% self._output, 'w') as f:
> f.write (json.dumps (r, indent = 4))
> ''
> Could you please provide a sample results.json file so that I can better
> understand what information is contained in the uploaded data and design the
> datasheet based on it.
Don't let this distract you too much from finishing your current term.
There will be plenty of time to hammer out the schema.
Here's a brief description of the data that is summarized in a json
object:
The idea is that the json do
Re: [GSOC 18] Performance Farm Project
On Thu, Mar 01, 2018 at 02:31:08PM -0500, Robert Haas wrote: > On Thu, Mar 1, 2018 at 4:56 AM, 马来酸 wrote: > > Hi, my name is Hongyuan Ma. I am a junior student in software engineering in > > Shanghai.I am interested in the idea "Develop Performance Farm Database and > > Website".I started developing web applications since my sophomore year and > > have experience in developing web sites. > > > > I took a look at the buildfarm project and in my opinion, the Performance > > Farm project has three basic functions: > > > > User Authentication > > Upload data (json format or file format) > > Review the data(may provide search, sorting and other functions) > > I might be wrong here, but I think that the problem here has a lot > more to do with deciding what we want to do, figuring out where we're > going to get the machine resources to do it, and making sure that we > don't build something that executes random code off the Internet that > might've been written by bad people. I know the git web reference doesn't provide a whole lot of information, but here is the framework we've been working on for defining/executing the tests (I've updated the description on our gsoc wiki page with this reference): https://git.postgresql.org/gitweb/?p=pgperffarm.git;a=summary And I just realized the pgperffarm mailing list isn't listed on the postgres mailing list page. That would probably help provide some context. I'll see what I can do about that... I've touched base with the pginfra team about hosting the Web site and database so I think I have that aspect covered. Regards, Mark > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise PostgreSQL Company -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: [GSOC 18] Performance Farm Project
Hi Hongyuan, On Thu, Mar 01, 2018 at 05:56:07PM +0800, 马来酸 wrote: > Dear Sir, > Hi, my name is Hongyuan Ma. I am a junior student in software engineering in > Shanghai.I am interested in the idea "Develop Performance Farm Database and > Website".I started developing web applications since my sophomore year and > have experience in developing web sites. > > > I took a look at the buildfarm project and in my opinion, the Performance > Farm project has three basic functions: > User Authentication > Upload data (json format or file format) > Review the data(may provide search, sorting and other functions) > Due to limited information on the idea page, if you can provide more > detailed functional requirements, my thinking will be clearer. > I am willing to initialize this project in the next few days and initially > achieve some basic functions. It's great to see such enthusiasm, but please keep in mind the schedule of the program. We still aren't sure how many students we will be allowed to accept so don't let yourself get too consumed by implementing a solution yet. :) > I plan to use Django 2.x, which requires python 3.5+. Here are some of my > questions about this project. We will want to stick to Django's LTS version, which is currently v1.11 and will work with python 3. I am imagining that having some RESTful services would be a good way to transfer data so my suggestion is to also consider the Django Rest Framework, which may also help with the user management. > Should this project consider the use of front-end framework(vue,react,etc.) > to separate the front-end from the front-end? > Does this project use the UI framework(antDesign,bootstrap,etc.) for better > visual effects? I prefer to keep things simple by not having any javascript, but I am open to being swayed otherwise by the potential consumers of this data. Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
Re: pgbench - allow to specify scale as a size
On Sat, Feb 17, 2018 at 12:22:37PM -0500, Alvaro Hernandez wrote: > > > On 17/02/18 12:17, Tom Lane wrote: > > Alvaro Hernandez writes: > >> On 17/02/18 11:26, Tom Lane wrote: > >>> Fabien COELHO writes: > >>>> Here is a attempt at extending --scale so that it can be given a size. > >>> I do not actually find this to be a good idea. It's going to be > >>> platform-dependent, or not very accurate, or both, and thereby > >>> contribute to confusion by making results less reproducible. > >>> > >>> Plus, what do we do if the backend changes table representation in > >>> some way that invalidates Kaarel's formula altogether? More confusion > >>> would be inevitable. > >> Why not then insert a "few" rows, measure size, truncate the table, > >> compute the formula and then insert to the desired user requested size? > >> (or insert what should be the minimum, scale 1, measure, and extrapolate > >> what's missing). It doesn't sound too complicated to me, and targeting a > >> size is something that I believe it's quite good for user. > > Then you'd *really* have irreproducible results. > > > > regards, tom lane > > You also have irreproducible results today, according to your > criteria. Either you agree on the number of rows but may not agree on > the size (today), or you agree on the size but may not agree on the > number of rows. Right now you can only pick the former, while I think > people would significantly appreciate the latter. If neither is correct, > let's at least provide the choice. What if we consider using ascii (utf8?) text file sizes as a reference point, something independent from the database? I realize even if a flat file size can be used as a more consistent reference across platforms, it doesn't help with accurately determining the final data file sizes due to any architecture specific nuances or changes in the backend. But perhaps it might still offer a little more meaning to be able to say "50 gigabytes of bank account data" rather than "10 million rows of bank accounts" when thinking about size over cardinality. Regards, Mark -- Mark Wonghttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, RemoteDBA, Training & Services
