== PostgreSQL Weekly News - November 08 2015 == == PostgreSQL Product News ==
pgchronos 1.0.0, an extension providing sql functions and operators for performing set operations on sets of daterange and tstzrange types, released. https://github.com/worden341/pgchronos == PostgreSQL Jobs for November == http://archives.postgresql.org/pgsql-jobs/2015-11/ == PostgreSQL Local == PGConf Silicon Valley 2015 is November 17-18 at the South San Francisco Convention Center. http://www.pgconfsv.com PGBR2015 (The Brazilian PostgreSQL Conference) will take place in Porto Alegre, Rio Grande do Sul, on November 18, 19 and 20. Registration is open. http://pgbr.postgresql.org.br/2015/en/ Postgres Conference China 2015 will be on November 20-22, 2015 in Beijing. http://postgres2015.eventdove.com/ (Chinese) http://postgrescluster2015.eventdove.com/ (English) PGConf.DE will be held on November 26-27, 2015 in Hamburg, Germany, at the Lindner Hotel am Michel. http://2015.pgconf.de/ PostgreSQL@SCaLE is a two day, two track event which takes place on Jan. 21-22, 2016, at Pasadena Convention Center, as part of SCaLE 14X. The CfP is open until Oct 30, 2015. https://www.socallinuxexpo.org/scale/14x/cfp == PostgreSQL in the News == Planet PostgreSQL: http://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm Pacific time. Please send English language ones to da...@fetter.org, German language to p...@pgug.de, Italian language to p...@itpug.org. Spanish language to p...@arpug.com.ar. == Applied Patches == Kevin Grittner pushed: - Add RMV to list of commands taking AE lock. Backpatch to 9.3, where it was initially omitted. Craig Ringer, with minor adjustment by Kevin Grittner http://git.postgresql.org/pg/commitdiff/bf25fb2f93c95e946c5e382eb82ad08ae9d95bd9 Robert Haas pushed: - Fix problems with ParamListInfo serialization mechanism. Commit d1b7c1ffe72e86932b5395f29e006c3f503bc53d introduced a mechanism for serializing a ParamListInfo structure to be passed to a parallel worker. However, this mechanism failed to handle external expanded values, as pointed out by Noah Misch. Repair. Moreover, plpgsql_param_fetch requires adjustment because the serialization mechanism needs it to skip evaluating unused parameters just as we would do when it is called from copyParamList, but params == estate->paramLI in that case. To fix, make the bms_is_member test in that function unconditional. Finally, have setup_param_list set a new ParamListInfo field, paramMask, to the parameters actually used in the expression, so that we don't try to fetch those that are not needed when serializing a parameter list. This isn't necessary for correctness, but it makes the performance of the parallel executor code comparable to what we do for cases involving cursors. Design suggestions and extensive review by Noah Misch. Patch by me. http://git.postgresql.org/pg/commitdiff/1efc7e538204646fec0f790b683012ed9bd8a99f - Correct tiny inaccuracy in strxfrm cache comment. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/0279f62fdca7ff19ef34a836f5201935dc7f627c - shm_mq: Third attempt at fixing nowait behavior in shm_mq_receive. Commit a1480ec1d3bacb9acb08ec09f22bc25bc033115b purported to fix the problems with commit b2ccb5f4e6c81305386edb34daf7d1d1e1ee112a, but it didn't completely fix them. The problem is that the checks were performed in the wrong order, leading to a race condition. If the sender attached, sent a message, and detached after the receiver called shm_mq_get_sender and before the receiver called shm_mq_counterparty_gone, we'd incorrectly return SHM_MQ_DETACHED before all messages were read. Repair by reversing the order of operations, and add a long comment explaining why this new logic is (hopefully) correct. http://git.postgresql.org/pg/commitdiff/4efe26cbd3ef0d85656bf00ac9e5bd41cc8a2a36 - shm_mq: Third attempt at fixing nowait behavior in shm_mq_receive. Commit a1480ec1d3bacb9acb08ec09f22bc25bc033115b purported to fix the problems with commit b2ccb5f4e6c81305386edb34daf7d1d1e1ee112a, but it didn't completely fix them. The problem is that the checks were performed in the wrong order, leading to a race condition. If the sender attached, sent a message, and detached after the receiver called shm_mq_get_sender and before the receiver called shm_mq_counterparty_gone, we'd incorrectly return SHM_MQ_DETACHED before all messages were read. Repair by reversing the order of operations, and add a long comment explaining why this new logic is (hopefully) correct. http://git.postgresql.org/pg/commitdiff/4efe26cbd3ef0d85656bf00ac9e5bd41cc8a2a36 - Improve comments about abbreviation abort. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/ee44cb7566ffafc0144535e1f966f5e3bb7d384b - Pass extra data to bgworkers, and use this to fix parallel contexts. Up until now, the total amount of data that could be passed to a background worker at startup was one datum, which can be a small as 4 bytes on some systems. That's enough to pass a dsm_handle or an array index, but not much else. Add a bgw_extra flag to the BackgroundWorker struct, allowing up to 128 bytes to be passed to a new worker on any platform. Use this to fix a problem I recently discovered with the parallel context machinery added in 9.5: the master assigns each worker an array index, and each worker subsequently assigns itself an array index, and there's nothing to guarantee that the two sets of indexes match, leading to chaos. Normally, I would not back-patch the change to add bgw_extra, since it is basically a feature addition. However, since 9.5 is still in beta and there seems to be no other sensible way to repair the broken parallel context machinery, back-patch to 9.5. Existing background worker code can ignore the bgw_extra field without a problem, but might need to be recompiled since the structure size has changed. Report and patch by me. Review by Amit Kapila. http://git.postgresql.org/pg/commitdiff/64b2e7ad917a9a7814904d0f6dbde52cefbcfa00 - Document interaction of bgworkers with LISTEN/NOTIFY. Thomas Munro and Robert Haas, reviewed by Haribabu Kommi http://git.postgresql.org/pg/commitdiff/dde5f09fad3ac188a155e21667f76825f43a28c8 - pg_size_pretty: Format negative values similar to positive ones. Previously, negative values were always displayed in bytes, regardless of how large they were. Adrian Vondendriesch, reviewed by Julien Rouhaud and myself http://git.postgresql.org/pg/commitdiff/8a1fab36aba7506fcf4559c4ef95fcacdd0b439a - Add sort support routine for the UUID data type. This introduces a simple encoding scheme to produce abbreviated keys: pack as many bytes of each UUID as will fit into a Datum. On little-endian machines, a byteswap is also performed; the abbreviated comparator can therefore just consist of a simple 3-way unsigned integer comparison. The purpose of this change is to speed up sorting data on a column of type UUID. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/a76ef15d9fc9207a0758e8d6f6700dc8c931a934 - Remove unnecessary cast in previous commit. Noted by Kyotaro Horiguchi, who also reviewed the previous patch, but I failed to notice his review before committing. http://git.postgresql.org/pg/commitdiff/cbb82e370da8e80aa9667ffbb395cabb0529ea00 - When completing ALTER INDEX .. SET, add an equals sign also. Jeff Janes http://git.postgresql.org/pg/commitdiff/af9773cf4cac235a32341b11864c4a0a9a22b6c1 - Try to convince gcc that TupleQueueRemap never falls off the end. Without this, MacOS gcc version 4.2.1 isn't convinced. http://git.postgresql.org/pg/commitdiff/8d7396e5092f1bc6433a38ccabaf88c8c681de11 - Modify tqueue infrastructure to support transient record types. Commit 4a4e6893aa080b9094dadbe0e65f8a75fee41ac6, which introduced this mechanism, failed to account for the fact that the RECORD pseudo-type uses transient typmods that are only meaningful within a single backend. Transferring such tuples without modification between two cooperating backends does not work. This commit installs a system for passing the tuple descriptors over the same shm_mq being used to send the tuples themselves. The two sides might not assign the same transient typmod to any given tuple descriptor, so we must also substitute the appropriate receiver-side typmod for the one used by the sender. That adds some CPU overhead, but still seems better than being unable to pass records between cooperating parallel processes. Along the way, move the logic for handling multiple tuple queues from tqueue.c to nodeGather.c; tqueue.c now provides a TupleQueueReader, which reads from a single queue, rather than a TupleQueueFunnel, which potentially reads from multiple queues. This change was suggested previously as a way to make sure that nodeGather.c rather than tqueue.c had policy control over the order in which to read from queues, but it wasn't clear to me until now how good an idea it was. typmod mapping needs to be performed separately for each queue, and it is much simpler if the tqueue.c code handles that and leaves multiplexing multiple queues to higher layers of the stack. http://git.postgresql.org/pg/commitdiff/6e71dd7ce9766582da453f493bc371d64977282f - Remove set-but-not-used variables. Reported by both Peter Eisentraunt and Kevin Grittner. http://git.postgresql.org/pg/commitdiff/fba60e573e092991dde4f757b88384ab1ec6a0ce Tom Lane pushed: - Remove some more dead Alpha-specific code. http://git.postgresql.org/pg/commitdiff/620ac88d6fd39c01bd25c90491333f088d186151 - Code + docs review for unicode linestyle patch. Fix some brain fade in commit a2dabf0e1dda93c8: erroneous variable names in docs, rearrangements that made sentences less clear not more so, undocumented and poorly-chosen-anyway API behaviors of subroutines, bad grammar in error messages, copy-and-paste faults. Albe Laurenz and Tom Lane http://git.postgresql.org/pg/commitdiff/a69b0b2c14c614c129e89ae96d6f624375531760 - Remove obsolete advice about doubling backslashes in regex escapes. Standard-conforming literals have been the default for long enough that it no longer seems necessary to go out of our way to tell people to write regex escapes illegibly. http://git.postgresql.org/pg/commitdiff/fc0b8935213cda555d2b3af2ed08da28ed120e31 - Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me http://git.postgresql.org/pg/commitdiff/d89494166351e1fdac77d87c6af500401deb2422 - Add regression tests for remote execution of extension operators/functions. Rather than relying on other extensions to be available for installation, let's just add some test objects to the postgres_fdw extension itself within the regression script. http://git.postgresql.org/pg/commitdiff/b9f117d6cd3c79780b0c0e57068f0837f7493aa6 - Improve implementation of GEQO's init_tour() function. Rather than filling a temporary array and then copying values to the output array, we can generate the required random permutation in-place using the Fisher-Yates shuffle algorithm. This is shorter as well as more efficient than before. It's pretty unlikely that anyone would notice a speed improvement, but shorter code is better. Nathan Wagner, edited a bit by me http://git.postgresql.org/pg/commitdiff/59464bd6f928ad0da30502cbe9b54baec9ca2c69 - Fix memory leaks in PL/Python. Previously, plpython was in the habit of allocating a lot of stuff in TopMemoryContext, and it was very slipshod about making sure that stuff got cleaned up; in particular, use of TopMemoryContext as fn_mcxt for function calls represents an unfixable leak, since we generally don't know what the called function might have allocated in fn_mcxt. This results in session-lifespan leakage in certain usage scenarios, as for example in a case reported by Ed Behn back in July. To fix, get rid of all the retail allocations in TopMemoryContext. All long-lived allocations are now made in sub-contexts that are associated with specific objects (either pl/python procedures, or Python-visible objects such as cursors and plans). We can clean these up when the associated object is deleted. I went so far as to get rid of PLy_malloc completely. There were a couple of places where it could still have been used safely, but on the whole it was just an invitation to bad coding. Haribabu Kommi, based on a draft patch by Heikki Linnakangas; some further work by me http://git.postgresql.org/pg/commitdiff/8c75ad436f75fc629b61f601ba884c8f9313c9af - Fix erroneous hash calculations in gin_extract_jsonb_path(). The jsonb_path_ops code calculated hash values inconsistently in some cases involving nested arrays and objects. This would result in queries possibly not finding entries that they should find, when using a jsonb_path_ops GIN index for the search. The problem cases involve JSONB values that contain both scalars and sub-objects at the same nesting level, for example an array containing both scalars and sub-arrays. To fix, reset the current stack->hash after processing each value or sub-object, not before; and don't try to be cute about the outermost level's initial hash. Correcting this means that existing jsonb_path_ops indexes may now be inconsistent with the new hash calculation code. The symptom is the same --- searches not finding entries they should find --- but the specific rows affected are likely to be different. Users will need to REINDEX jsonb_path_ops indexes to make sure that all searches work as expected. Per bug #13756 from Daniel Cheng. Back-patch to 9.4 where the faulty logic was introduced. http://git.postgresql.org/pg/commitdiff/b23af458755ed90ae9164857e59d034bad75a672 - Fix enforcement of restrictions inside regexp lookaround constraints. Lookahead and lookbehind constraints aren't allowed to contain backrefs, and parentheses within them are always considered non-capturing. Or so says the manual. But the regexp parser forgot about these rules once inside a parenthesized subexpression, so that constructs like (\w)(?=(\1)) were accepted (but then not correctly executed --- a case like this acted like (\w)(?=\w), without any enforcement that the two \w's match the same text). And in (?=((foo))) the innermost parentheses would be counted as capturing parentheses, though no text would ever be captured for them. To fix, properly pass down the "type" argument to the recursive invocation of parse(). Back-patch to all supported branches; it was agreed that silent misexecution of such patterns is worse than throwing an error, even though new errors in minor releases are generally not desirable. http://git.postgresql.org/pg/commitdiff/a43b4ab1111ca5e5f40a2ddd8e56bf999b9fdad9 - Rename PQsslAttributes() to PQsslAttributeNames(), and const-ify fully. Per discussion, the original name was a bit misleading, and PQsslAttributeNames() seems more apropos. It's not quite too late to change this in 9.5, so let's change it while we can. Also, make sure that the pointer array is const, not only the pointed-to strings. Minor documentation wordsmithing while at it. Lars Kanis, slight adjustments by me http://git.postgresql.org/pg/commitdiff/9042f583429ef880665dd36b7bcad45ba62dea7b - Add "xid <> xid" and "xid <> int4" operators. The corresponding "=" operators have been there a long time, and not having their negators is a bit of a nuisance. Michael Paquier http://git.postgresql.org/pg/commitdiff/c5e86ea932794051d2fe3bd26ce4c03981b0b826 - Update 9.5 release notes through today. http://git.postgresql.org/pg/commitdiff/ad9fad7b68c4209db7d73f59a538ffe619c07872 Peter Eisentraut pushed: - Update spelling of COPY options. The preferred spelling was changed from FORCE QUOTE to FORCE_QUOTE and the like, but some code was still referring to the old spellings. http://git.postgresql.org/pg/commitdiff/7bd099d511d66b55fedced9920357cc5c36811c7 Stephen Frost pushed: - Set include_realm=1 default in parse_hba_line. With include_realm=1 being set down in parse_hba_auth_opt, if multiple options are passed on the pg_hba line, such as: host all all 0.0.0.0/0 gss include_realm=0 krb_realm=XYZ.COM We would mistakenly reset include_realm back to 1. Instead, we need to set include_realm=1 up in parse_hba_line, prior to parsing any of the additional options. Discovered by Jeff McCormick during testing. Bug introduced by 9a08841. Back-patch to 9.5 http://git.postgresql.org/pg/commitdiff/5644419b3de418f81e4461cbaa06c5d4b5da0800 Noah Misch pushed: - Don't connect() to a wildcard address in test_postmaster_connection(). At least OpenBSD, NetBSD, and Windows don't support it. This repairs pg_ctl for listen_addresses='0.0.0.0' and listen_addresses='::'. Since pg_ctl prefers to test a Unix-domain socket, Windows users are most likely to need this change. Back-patch to 9.1 (all supported versions). This could change pg_ctl interaction with loopback-interface firewall rules. Therefore, in 9.4 and earlier (released branches), activate the change only on known-affected platforms. Reported (bug #13611) and designed by Kondo Yuta. http://git.postgresql.org/pg/commitdiff/fed19f312c72778ce7dcbb0670871b22bd06ff85 Andres Freund pushed: - Set replication origin when decoding commit records. By accident the replication origin was not set properly in DecodeCommit(). That's bad because the origin is passed to the output plugins origin filter, and accessible from the output plugin via ReorderBufferTXN->origin_id. Accessing the origin of individual changes worked before the fix, which is why this wasn't notices earlier. Reported-By: Craig Ringer Author: Craig Ringer Discussion: CAMsr+YFhBJLp=qfsz3-j+0p1zlke8znxm2otycn20qrmx38...@mail.gmail.com Backpatch: 9.5, where replication origins where introduced http://git.postgresql.org/pg/commitdiff/f3a764b0dac77451d20ee00d2f5916447a5c4346 == Rejected Patches (for now) == No one was disappointed this week :-) == Pending Patches == Alexander Korotkov sent in another revision of a patch to make the access method API more extensible. David Fetter sent in a patch to add weighted statistics to PostgreSQL. Craig Ringer sent in a patch to add a logical decoding output plugin. Robert Haas and Rajeev Rastogi traded patches to fix an issue that could result in a dangling client backend process. Peter Geoghegan sent in a patch to correct a heap_finish_speculative() error message inaccuracy. Bill Moran sent in a patch to add new configuration options to TOAST. Pavel Stěhule sent in another revision of a patch to add an option psql to execute 1 or more commands, then exit. Amit Kapila sent in another revision of a patch to implement parallel seq scan. Pavel Stěhule and Catalin Iacob traded patches to add a PL/PythonU ereport() function. David Rowley sent in another revision of a patch to speed up timestamptz_out. Kyotaro HORIGUCHI sent in two revisions of a patch to fix some tab completion infelicities in psql. Kaigai Kouhei sent in two more revisions of a patch to add custom scan support to readfuncs.c. Kyotaro HORIGUCHI sent in two revisions of a patch to fix identifier completion of multibyte characters. Christian Marie sent in a patch to add length parameterised dmetaphone functions. Kyotaro HORIGUCHI sent in a patch to allow substitution of ExecScan body for ExecForignScan. Ashutosh Bapat sent in another revision of a patch to allow getting sorted data from a foreign server for merge joins. Michael Paquier sent in a patch to add a space-related constraint on the standby snapshot generated by the bgwriter, so as to not rely entirely on the interval of 15s. Fabien COELHO sent in four more revisions of a patch to extend pgbench expressions with functions. Ildus Kurbangaliev sent in a patch to split the SLRU LWLocks into separate tranches and move them to SLRU Ctl. Robert Haas sent in a patch to add a new flag, consider_parallel, to each RelOptInfo, indicating whether a plan for that relation could conceivably be run inside of a parallel worker. Artur Zakirov sent in two revisions of a patch to improve the use of Hunspell dictionaries. Vitaly Burovoy sent in a patch to ensure that parts extracted from "infinity" timestamps are also infinite. Lars Kanis sent in a doc patch to lowercase pqsslAttribute in SGML-id for consistency. Noah Misch sent in a patch which removes the SlruScanDirCbFindEarliest() test from TruncateMultiXact(), which in turn makes multixact control decisions independent of whether TruncateMultiXact() is successful at unlinking segments. -- Sent via pgsql-announce mailing list (pgsql-announce@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-announce
