Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Bruce Momjian writes: > Tom Lane wrote: >> I would have gone the other way on that one, if possible. Seems like >> xlog.h ought to be the lower-level file. > Agreed. Let me work on that. Uh, I just did it. Painful. It would have been a lot easier before the pgrminclude run, because that baked-in a whole lot of bad decisions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Tom Lane wrote: > Bruce Momjian writes: > > Tom Lane wrote: > >> You seem to have entirely missed the point of Alvaro's remark, which is > >> that you've got xlog.h including walsender.h (still) as well as > >> walsender.h including xlog.h. That's broken. > > > Oh, OK, done. xlog.h removed from walsender.h and tested. > > I would have gone the other way on that one, if possible. Seems like > xlog.h ought to be the lower-level file. Agreed. Let me work on that. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] xlogdump 0.4.0
Hi hackers, I'm pleased to announce the latest release of xlogdump. xlogdump is a tool for extracting data from WAL segment files. Here is xlogdump README: https://github.com/snaga/xlogdump/blob/master/xlogdump/README.xlogdump xlogdump was originally developed by Tom Lane and Diogo Biazus around a five years ago, but not be maintained these few years. So, I picked it up and have fixed some problems and enhanced. It has still several issues, but it seems to start working, so it's time to try it again, folks! https://github.com/snaga/xlogdump/downloads xlogdump is now able to be compiled against 8.2~9.0 (it was tough fixes!), so it could be a fun (of course, also useful) stuff for many hackers and users. Please visit the new project repository, and try it. https://github.com/snaga/xlogdump If you have any bug reports and/or comments, please feel free to email me. Thanks in advance, p.s. I really appreciate Tom and Diogo for their previous great work. -- NAGAYASU Satoshi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Bruce Momjian writes: > Tom Lane wrote: >> You seem to have entirely missed the point of Alvaro's remark, which is >> that you've got xlog.h including walsender.h (still) as well as >> walsender.h including xlog.h. That's broken. > Oh, OK, done. xlog.h removed from walsender.h and tested. I would have gone the other way on that one, if possible. Seems like xlog.h ought to be the lower-level file. Some quick mechanical analysis shows that's not the only circular inclusion path in our headers, either: we have storage/proc.h including replication/syncrep.h which includes storage/proc.h. That one appears to be Simon's fault not yours though. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Tom Lane wrote: > Bruce Momjian writes: > > Alvaro Herrera wrote: > >> Hmm, so you included walsender.h into xlog.h? That seems a bit funny > >> considering that walsender.h already includes xlog.h. It seems the > >> reason for this is only the AllowCascadeReplication() definition. Maybe > >> that should go elsewhere instead, for example walsender.h? > > > Moved to walsender.h. Thanks. > > You seem to have entirely missed the point of Alvaro's remark, which is > that you've got xlog.h including walsender.h (still) as well as > walsender.h including xlog.h. That's broken. Oh, OK, done. xlog.h removed from walsender.h and tested. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Bruce Momjian writes: > Alvaro Herrera wrote: >> Hmm, so you included walsender.h into xlog.h? That seems a bit funny >> considering that walsender.h already includes xlog.h. It seems the >> reason for this is only the AllowCascadeReplication() definition. Maybe >> that should go elsewhere instead, for example walsender.h? > Moved to walsender.h. Thanks. You seem to have entirely missed the point of Alvaro's remark, which is that you've got xlog.h including walsender.h (still) as well as walsender.h including xlog.h. That's broken. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [COMMITTERS] pgsql: Remove "fmgr.h" include in cube contrib --- caused crash on a Ge
Alvaro Herrera wrote: > Excerpts from Bruce Momjian's message of vie sep 02 12:20:50 -0300 2011: > > > Wow, that is interesting. So the problem is the inclusion of > > replication/walsender.h in xlog.h. Hard to see how that could cause the > > cube regression tests to fail, but of course, it is. > > Hmm, so you included walsender.h into xlog.h? That seems a bit funny > considering that walsender.h already includes xlog.h. It seems the > reason for this is only the AllowCascadeReplication() definition. Maybe > that should go elsewhere instead, for example walsender.h? Moved to walsender.h. Thanks. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade automatic testing
Peter Eisentraut writes: > On fre, 2011-09-02 at 19:49 -0400, Tom Lane wrote: >> The only one that's problematic is pg_regress.so; contrib modules are >> already installed in $libdir. I still think that installing >> pg_regress.so in $libdir may be the most reasonable solution, assuming >> that the delta involved isn't too great. Yeah, we would have to >> back-patch the changes into every release branch we want to test >> upgrading from, but how risky is that really? The *only* thing it >> affects is the regression tests. > Or maybe make use of dynamic_library_path. That seemed like a promising idea at first, but on reflection I thought probably it's not such a great idea. The problem is, where do you inject the setting? In a temp installation we could put it in postgresql.conf, but for "make installcheck" the only way that seems like it would work at all is to apply it as a database configuration (ALTER DATABASE SET), and that seems problematic. In particular, it would not work for testing pg_dump, since pg_dump doesn't copy those settings. (I know we've talked about making it do so, but we'd certainly not wish to back-patch such a change.) (BTW, this also strikes me as a counterexample for the recently proposed change to make pg_dumpall dump such settings at the end. If you've got datatypes or indexes that depend on a shared library that's found via "ALTER DATABASE SET dynamic_library_path", it will absolutely not work to restore the database contents before that setting is applied.) Anyway, after giving up on that I went back to plan A, namely install regress.so and friends into $libdir. That turns out to be really quite straightforward, though I had to hack pg_regress.c a bit to get its idea of $libdir to match up exactly with the way the backend sees it. (The only reason this matters is that there's one error report in the regression tests where the full expansion of $libdir is reported. Maybe we should just drop that one test case instead of maintaining the infrastructure for replacing @libdir@ in pg_regress.c.) Attached is a draft patch for HEAD. It passes "make check" and "make installcheck" on Unix, but I've not touched the MSVC scripts. Comments? regards, tom lane diff --git a/contrib/dummy_seclabel/Makefile b/contrib/dummy_seclabel/Makefile index 105400f..3a5f968 100644 *** a/contrib/dummy_seclabel/Makefile --- b/contrib/dummy_seclabel/Makefile *** top_builddir = ../.. *** 12,14 --- 12,24 include $(top_builddir)/src/Makefile.global include $(top_srcdir)/contrib/contrib-global.mk endif + + # Build/install only the files needed for regression test support + REGRESSION_MODULES = dummy_seclabel + + .PHONY: regression-modules install-regression-modules + + regression-modules: $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) + + install-regression-modules: $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) + $(INSTALL_SHLIB) $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) '$(DESTDIR)$(pkglibdir)/' diff --git a/contrib/spi/Makefile b/contrib/spi/Makefile index 0c11bfc..83578fb 100644 *** a/contrib/spi/Makefile --- b/contrib/spi/Makefile *** top_builddir = ../.. *** 28,30 --- 28,40 include $(top_builddir)/src/Makefile.global include $(top_srcdir)/contrib/contrib-global.mk endif + + # Build/install only the files needed for regression test support + REGRESSION_MODULES = autoinc refint + + .PHONY: regression-modules install-regression-modules + + regression-modules: $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) + + install-regression-modules: $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) + $(INSTALL_SHLIB) $(addsuffix $(DLSUFFIX), $(REGRESSION_MODULES)) '$(DESTDIR)$(pkglibdir)/' diff --git a/src/test/regress/GNUmakefile b/src/test/regress/GNUmakefile index 90aea6c..96839db 100644 *** a/src/test/regress/GNUmakefile --- b/src/test/regress/GNUmakefile *** EXTRADEFS = '-DHOST_TUPLE="$(host_tuple) *** 41,47 # Build regression test driver ! all: pg_regress$(X) pg_regress$(X): pg_regress.o pg_regress_main.o | submake-libpgport $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@ --- 41,47 # Build regression test driver ! all: pg_regress$(X) regression-modules pg_regress$(X): pg_regress.o pg_regress_main.o | submake-libpgport $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@ *** pg_regress.o: pg_regress.c $(top_builddi *** 53,65 $(top_builddir)/src/port/pg_config_paths.h: $(top_builddir)/src/Makefile.global $(MAKE) -C $(top_builddir)/src/port pg_config_paths.h ! install: all installdirs $(INSTALL_PROGRAM) pg_regress$(X) '$(DESTDIR)$(pgxsdir)/$(subdir)/pg_regress$(X)' installdirs: $(MKDIR_P) '$(DESTDIR)$(pgxsdir)/$(subdir)' ! uninstall: rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/pg_regress$(X)' --- 53,65 $(top_builddir)/src/port/pg_config_paths.h: $(top_builddir)/src/Makefile.global $(MAKE)
[HACKERS] Large C files
FYI, here are all the C files with over 6k lines: - 45133 ./interfaces/ecpg/preproc/preproc.c - 33651 ./backend/parser/gram.c - 17551 ./backend/parser/scan.c 14209 ./bin/pg_dump/pg_dump.c 10590 ./backend/access/transam/xlog.c 9764 ./backend/commands/tablecmds.c 8681 ./backend/utils/misc/guc.c - 7667 ./bin/psql/psqlscan.c 7213 ./backend/utils/adt/ruleutils.c 6814 ./backend/utils/adt/selfuncs.c 6176 ./backend/utils/adt/numeric.c 6030 ./pl/plpgsql/src/pl_exec.c I have dash-marked the files that are computer-generated. It seems pg_dump.c and xlog.c should be split into smaller C files. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore --no-post-data and --post-data-only
On 09/03/2011 04:49 PM, Dimitri Fontaine wrote: Andrew Dunstan writes: Oh, I meant just having it create separate custom format files for each database. As shell scripts all over the world have been doing for years, but it would be nice if it was simply built in. I guess it could be done, although I'm not going to do it :-) I'm more about making somewhat hard things easier than easy things slightly easier :-) Then what about issuing an archive (tar or ar format here) containing one custom file per database plus the globals file, SQL, plus maybe a database listing, and hacking pg_restore so that it knows what to do with such an input ? Bonus points if that supports the current -l and -L options, of course. That's probably a lot of code for a little benefit, at least from my POV, but others might find it useful. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL "low watermark" during base backup
Magnus Hagander writes: > Attached patch implements a "low watermark wal location" in the > walsender shmem array. Setting this value in a walsender prevents > transaction log removal prior to this point - similar to how > wal_keep_segments work, except with an absolute number rather than Cool. The first use case that comes to my mind is when to clean old WAL files when using multiple standby servers. Will it help here? > relative. For now, this is set when running a base backup with WAL > included - to prevent the required WAL to be recycled away while the > backup is running, without having to guestimate the value for > wal_keep_segments. I would have guessed that if you stream WALs in parallel of the backup, and begin streaming before you pg_start_backup(), you don't need anything more. Is that wrong? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore --no-post-data and --post-data-only
Andrew Dunstan writes: >> Oh, I meant just having it create separate custom format files for each >> database. As shell scripts all over the world have been doing for years, >> but it would be nice if it was simply built in. > > I guess it could be done, although I'm not going to do it :-) I'm more about > making somewhat hard things easier than easy things slightly easier :-) Then what about issuing an archive (tar or ar format here) containing one custom file per database plus the globals file, SQL, plus maybe a database listing, and hacking pg_restore so that it knows what to do with such an input ? Bonus points if that supports the current -l and -L options, of course. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sha1, sha2 functions into core?
On Fri, Sep 02, 2011 at 04:27:46PM -0500, Ross J. Reedstrom wrote: > On Fri, Sep 02, 2011 at 02:05:45PM -0500, k...@rice.edu wrote: > > On Fri, Sep 02, 2011 at 09:54:07PM +0300, Peter Eisentraut wrote: > > > On ons, 2011-08-31 at 13:12 -0500, Ross J. Reedstrom wrote: > > > > Hmm, this thread seems to have petered out without a conclusion. Just > > > > wanted to comment that there _are_ non-password storage uses for these > > > > digests: I use them in a context of storing large files in a bytea > > > > column, as a means to doing data deduplication, and avoiding pushing > > > > files from clients to server and back. > > > > > > But I suppose you don't need the hash function in the database system > > > for that. > > > > > > > It is very useful to have the same hash function used internally by > > PostgreSQL exposed externally. I know you can get the code and add an > > equivalent one of your own... > > > Thanks for the support Ken, but Peter's right: the only backend use in > my particular case is to let the backend do the hash calc during bulk > loads: in the production code path, having the hash in two places > doesn't save any work, since the client code has to calculate the hash > in order to test for its existence in the backend. I suppose if the > network cost was negligable, I could just push the files anyway, and > have a before-insert trigger calculate the hash and do the dedup: then > it'd be hidden in the backend completely. But as is, I can do all the > work in the client. > While it is true that it doesn't save any work. My motivation for having it exposed is that "good" hash functions are non-trivial to find. I have dealt with computational artifacts produced by hash functions that seemed at first to be good. We use a very well behaved function within the data- base and exposing it will help prevent bad user hash function implementations. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade automatic testing
Peter Eisentraut wrote: > But if you think about it, it doesn't really test pg_upgrade, it tests > pg_dump. So the test could just as well be moved to src/bin/pg_dump/ > and be labeled "pg_dump smoke test" or whatever. (Minor detail: The bug > fix above involved the --binary-upgrade flag, so it is somewhat > pg_upgrade related.) > > A real pg_upgrade test suite should naturally upgrade across binary > incompatible versions. The real question is how you develop a useful > test input. Most pg_upgrade issues are not bugs of omission or > regression but unexpected corner cases discovered with databases of > nontrivial usage patterns. (The recent one related to upgrade from 8.3 > is an exception.) Because the basic premise of pg_upgrade is, dump and > restore the schema, move over the files, that's it, and the rest of the > code is workarounds for obscure details that are difficult to anticipate > let alone test for. You might want to read my blog entry on why pg_upgrade relies so much on external tools: http://momjian.us/main/blogs/pgblog/2011.html#June_15_2011_2 -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade automatic testing
On fre, 2011-09-02 at 19:49 -0400, Tom Lane wrote: > The only one that's problematic is pg_regress.so; contrib modules are > already installed in $libdir. I still think that installing > pg_regress.so in $libdir may be the most reasonable solution, assuming > that the delta involved isn't too great. Yeah, we would have to > back-patch the changes into every release branch we want to test > upgrading from, but how risky is that really? The *only* thing it > affects is the regression tests. Or maybe make use of dynamic_library_path. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
