Re: [HACKERS] Review : Add hooks for pre- and post-processor executables for COPY and \copy
On Wednesday, February 27, 2013 12:41 AM Heikki Linnakangas wrote: > On 22.02.2013 12:43, Etsuro Fujita wrote: > >> 1. "Broken pipe" is not handled in case of psql "\copy" command; > >> Issue are as follows: > >> Following are verified on SuSE-Linux 10.2. > >> 1) psql is exiting when "\COPY xxx TO" command is > issued > >> and > command/script is not found > >> When popen is called in write mode it is > >> creating valid > file descriptor and when it tries to write to file "Broken pipe" > >>> error > is> coming which is not handled. > >> psql# \copy pgbench_accounts TO PROGRAM > '../compress.sh pgbench_accounts4.txt' > >> 2) When "\copy" command is in progress then > >>> program/command > >> is > killed/"crashed due to any problem" > >> psql is exiting. > > > This is a headache. I have no idea how to solve this. > > I think we can keep it for committer to take a call on this issue. > >>> > >>> Agreed. > > Fortunately this one is easy. We just need to ignore SIGPIPE, by > calling pqsignal(SIGPIPE, SIG_IGN) before popen(). We already do the > same in psql when the query output is piped to a pager, in PageOutput. So are you planning to call the same in do_copy as well; in current patch it is not there. > 5. Following in copy.sgml can be changed to make more meaningful > as > the first line looks little adhoc. > + > + The command that input comes from or that output goes to. > + The command for COPY FROM, which input comes from, must > +write its > output > + to standard output. The command for COPY TO, which output > >>> goes > + to, > must > + read its input from standard input. > + > >>> > >>> I've struggled to make the document more meaningful. > >> > >> To be honest, I am not sure whether introducing pre, post processor > >> terminology is right or not, But again I shall let committer decide > >> about this point. > > > > Agreed. > > I changed the terminology to use terms like "the command specified with > PROGRAM", instead of talking about pre- and post-processsors. > > >> I have one small another doubt that in function parse_slash_copy, > you > >> avoided expand tilde for program case, which I am not sure is the > >> right thing or not. > > > > Sorry, I'm not sure that, too. I'd like to leave this for > committers. > > That seems correct. The shell will do tilde expansion with popen(), we > don't need to do it ourselves. > > There's one oddity in the psql \copy syntax. This is actually present > in earlier versions too, but I think we should fix it now that we > extend the syntax: > > -- Writes to standard output. There's no way to write to a file > called "stdout". > \copy foo to 'stdout' > > I think we should change the interpretation of the above so that it > writes to a file called "stdout". It's possible that there's an > application out there that relies on that to write to stdout, but it's > not hard to fix the application. A small note in the release notes > might be in order. > > Also, I think we should require the command to be quoted in \copy. Ie. > let's forbid this: > > \copy pgbench_accounts to program /bin/cat>/dev/null > > and require that it's written as: > > \copy pgbench_accounts to program '/bin/cat>/dev/null' > > We don't require quoting for filenames, which I find a bit weird too, > but it seems particularly confusing for a shell command. > > Attached is a new version of this patch that I almost committed, but > one thing caught my eye at the last minute: The error reporting is not > very user-friendly. If the program fails after reading/writing some > rows, the reason is printed to the log, but not the user: > > postgres=# copy foo to program '/tmp/midway-crash'; > ERROR: could not execute command "/tmp/midway-crash" > > the log has more details: > > LOG: child process exited with exit code 10 > STATEMENT: copy foo to program '/tmp/midway-crash'; > ERROR: could not execute command "/tmp/midway-crash" > STATEMENT: copy foo to program '/tmp/midway-crash'; > > I think we should arrange for the "child process exited with exit code > 10" to be printed as errdetail to the client. Similarly, with psql > \copy command, the "child process exited with exit code 10" command > shouldn't be printed straight to stderr, but should go through > psql_error. > > I'll try to refactor pclose_check tomorrow to do that. Meanwhile, > please take a look at the attached if you have the time. I rewrote much > of the docs changes, and some comments. If there is semicolon at end, it takes it into account for creating filename. \copy foo to c:\data\foo.out; This problem was fixed in previous patch, but I think handling of quotes has changed this behavior. With Regards, Amit Kapila. -- Sent via pgsql-hackers mailing list
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
2013/2/26 Stephen Frost : > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Dunno, I think that's going to result in a very large chunk of mostly >> duplicative code in psql. regprocedurein() is fairly short because it >> can rely on a ton of code from the parser, but psql won't have that >> luxury. > > Parsing/tokenizing a CSV string inside parens doesn't strike me as all > that difficult, even when handling the space-delimininated varname from > the type. this is not hard task, hard task is correct identification related function see FuncnameGetCandidates() function I am sure, so we don't would to duplicate this function on client side. probably we can simplify searching to search only exact same signature - but still there are problem with type synonyms. And solving this code on client side means code duplication. I prefer some smart searching function on server side - it can be useful for other app too - pgAdmin, plpgsql debugger, ... And in psql we can do switch - for older versions use current code, and for newer "smart" server side function. ??? Regards Pavel > > The hard part would, I think, be the normalization of the > type names into what \df returns, but do we even want to try and tackle > that..?. How much do we care about supporting every textual > representation of the 'integer' type? That's not going to be an issue > for people doing tab-completion or using \df's output. We could also > have it fall-back to trying w/o any arguments for a unique function name > match if the initial attempt w/ the function arguments included doesn't > return any results. > > Thanks, > > Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Strange Windows problem, lock_timeout test request
Zoltan, * Boszormenyi Zoltan (z...@cybertec.at) wrote: > attached is v30, I hope with everything fixed. Making progress, certainly. Given the hack to the API of enable_timeout_after() and the need for timeout_reset_base_time(), I'm just going to voice my objection to the "accumulated wait time on locks" portion again. I still like the idea of a timeout for waiting on relation-level locks, as we acquire those all up-front and we'd be able to just set a timeout at the appropriate point and then release it when we're past acquiring the relation-level locks. Seems like that would be much cleaner. On the other hand, if we're going to go down this route, I'm really starting to wonder if it should be the timeout systems responsibility to keep track of such accumulated time. Other than that.. > - List based enable/disable_multiple_timeouts() That looks good, like the use of foreach(), etc, but I don't like how you've set up delay_ms as a pointer..? Looks to be to allow you to initialize the TimeoutParams structs early in proc.c..? Is there another reason it needs to be a pointer that I'm missing? Why not build the TimeoutParam strcutures in the if() blocks that check if the GUCs are set..? > - separate per-lock and per-statement lock_timeout variants > - modified comments and documentation Thanks. Stephen signature.asc Description: Digital signature
Re: [HACKERS] initdb ignoring options due to bash environment on OS X
On 2/26/13 9:18 PM, Greg Smith wrote: $ otool -L `which initdb` /Users/gsmith/pgwork/inst/latest/bin/initdb: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) Two last bits of trivia and I'll stop talking to myself. You can get the full list of things this dynamic library then links to like this: $ otool -lTv /usr/lib/libSystem.B.dylib | grep "(offset" name /usr/lib/libSystem.B.dylib (offset 24) name /usr/lib/system/libcache.dylib (offset 24) name /usr/lib/system/libcommonCrypto.dylib (offset 24) ... There's a bit more information about all the libraries squashed into this one on OS X at http://0xfe.blogspot.com/2006/03/qa-how-os-x-executes-applications.html It is also worth nothing that otool is part of the developer tools for OS X, not the base system. Features like DYLD_PRINT_LIBRARIES look like they work on any system though. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options due to bash environment on OS X
I have solved my problem after chasing down some library trivia. It's
the fault of my own shell script, but the cause/effect was surprising to
me. I'll go through some troubleshooting library flow on OS X
documentation below since this was new to me and I wrote down notes.
Maybe this will be useful for someone else. The sole upside of this
mess is that I'm almost done with having a OS X based stack/process I
can recommend for reviewers.
My peg development wrapper script defines a function named initdb, and I
was sourcing that in. That's what broke initdb the binary. I think
this was specific to initdb because of how the hand-off is done inside
src/bin/initdb/initdb.c to find "postgres" based on the path to initdb's
argv[0]. I saw that was calling something in /bin/sh on several
operations, like this:
fixing permissions on existing directory
/Users/gsmith/pgwork/data/latest ... ok
creating subdirectories ... ok
dyld: loaded: /bin/sh
dyld: loaded: /usr/lib/libncurses.5.4.dylib
dyld: loaded: /usr/lib/libSystem.B.dylib
[Details on getting that output also below] I'm not exactly sure why
this problem doesn't pop on Linux where I use peg all the time. Is it
because the involved shell is /bin/sh on the Mac? Is it due to linker
implementation differences? System library differences? That level of
detail doesn't seem too important to PostgreSQL development, so I didn't
chase this down to the exactly underlying difference.
Tom started me down the right path with:
On 2/26/13 7:03 PM, Tom Lane wrote:
One of the places where it works as expected for
me is my 10.7.5 laptop. So there's something weird about some library
you're using. What's getting linked into initdb ("otool -L" should
answer that) and where'd you get it from?
$ otool -L `which initdb`
/Users/gsmith/pgwork/inst/latest/bin/initdb:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 159.1.0)
Far as I know that's the OS provided version of that file. But half the
binaries on the system link to that, i.e.:
$ otool -L /bin/bash
/bin/bash:
/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current
version 5.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 159.0.0)
Note how the Mac is linking to two versions at once with the same file
name here. It doesn't use a symlink for that like I'm used to on Linux.
Also, libSystem.B.dylib is a dynamic library (.dy lib) that can pull a
lot more things in. It doesn't seem very useful on its own to know it's
being linked in.
If you want to trace all the dynamic libraries that are pulled in by
something you run from an OS X command line, set this:
$ export DYLD_PRINT_LIBRARIES="yes"
There's more of those options described here, if that's not enough:
https://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/LoggingDynamicLoaderEvents.html#//apple_ref/doc/uid/TP40002077-SW1
You also can override where the system libraries come from, similarly to
LD_LIBRARY_PATH, like this:
export DYLD_LIBRARY_PATH=/Users/gsmith/temp/
I think that only "took" usefully when I started another bash after
setting it.
Once that's done, now you get a useful list of all the libraries pulled in:
$ initdb --version
dyld: loaded: /Users/gsmith/pgwork/inst/latest/bin/initdb
dyld: loaded: /Users/gsmith/temp//libSystem.B.dylib
dyld: loaded: /usr/lib/system/libcache.dylib
dyld: loaded: /usr/lib/system/libcommonCrypto.dylib
dyld: loaded: /usr/lib/system/libcompiler_rt.dylib
dyld: loaded: /usr/lib/system/libcopyfile.dylib
dyld: loaded: /usr/lib/system/libdispatch.dylib
dyld: loaded: /usr/lib/system/libdnsinfo.dylib
dyld: loaded: /usr/lib/system/libdyld.dylib
dyld: loaded: /usr/lib/system/libkeymgr.dylib
dyld: loaded: /usr/lib/system/liblaunch.dylib
dyld: loaded: /usr/lib/system/libmacho.dylib
dyld: loaded: /usr/lib/system/libmathCommon.A.dylib
dyld: loaded: /usr/lib/system/libquarantine.dylib
dyld: loaded: /usr/lib/system/libremovefile.dylib
dyld: loaded: /usr/lib/system/libsystem_blocks.dylib
dyld: loaded: /usr/lib/system/libsystem_c.dylib
dyld: loaded: /usr/lib/system/libsystem_dnssd.dylib
dyld: loaded: /usr/lib/system/libsystem_info.dylib
dyld: loaded: /usr/lib/system/libsystem_kernel.dylib
dyld: loaded: /usr/lib/system/libsystem_network.dylib
dyld: loaded: /usr/lib/system/libsystem_notify.dylib
dyld: loaded: /usr/lib/system/libsystem_sandbox.dylib
dyld: loaded: /usr/lib/system/libunc.dylib
dyld: loaded: /usr/lib/system/libunwind.dylib
dyld: loaded: /usr/lib/system/libxpc.dylib
initdb (PostgreSQL) 9.3devel
The clue for my problem was that after all of that, the one Mac I could
make work here included these lines next:
dyld: loaded: /bin/sh
dyld: loaded: /usr/lib/libncurses.5.4.dylib
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/system/libcache.dylib ...
Making me think about the /bin/sh environment. Looking into the diff
from the working/not work
Re: [HACKERS] initdb ignoring options?
On 2/26/13 6:57 PM, Mark Kirkwood wrote: This might be a red herring - but I recall having problems using --enable-depend on older OSX versions (10.5 I think), so *maybe* worth seeing if leaving that option off and doing a distclean+rebuild changes anything. I toggled off both that and not expliciting enabling enable-thread-safety with no change. For Thom's question, these are all hitting the Postgres I compiled, and yes other tools like pg_ctl seem fine: $ which pg_ctl /Users/gsmith/pgwork/inst/latest/bin/pg_ctl $ pg_ctl --version pg_ctl (PostgreSQL) 9.3devel $ which initdb /Users/gsmith/pgwork/inst/latest/bin/initdb $ initdb --version Database cluster already exists in /Users/gsmith/pgwork/data/latest -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
On 26 February 2013 22:16, Greg Smith wrote: > Here's what I get on the latest repo: > > $ psql --version > psql (PostgreSQL) 9.3devel > $ initdb --help > The files belonging to this database system will be owned by user "gsmith". > This user must also own the server process. > The database cluster will be initialized with locale "en_US.UTF-8" ... What does "which initdb" give you? And is it only initdb with the issue? Have you tested it with pg_ctl? -- Thom -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
Greg Smith writes:
> On 2/26/13 6:01 PM, Tom Lane wrote:
>> Greg Smith writes:
>>> $ initdb --help
>>> The files belonging to this database system will be owned by user "gsmith".
>> Hm. Works as expected for me. What platform is this exactly?
> The broken one is OS X Lion 10.7.5, main build toolchain provided by
> Apple's development tools, and Homebrew used to install the missing
> libraries (zlib etc.) I had been beating my head against the wall
> asking why initdb didn't work for so long today, I didn't even think
> this could be a platform specific issue. I don't have any other OS X
> version installed here to test if this is specific to Lion or not.
[ scratches head... ] One of the places where it works as expected for
me is my 10.7.5 laptop. So there's something weird about some library
you're using. What's getting linked into initdb ("otool -L" should
answer that) and where'd you get it from?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
On 27/02/13 12:41, Greg Smith wrote: On 2/26/13 5:51 PM, Mark Kirkwood wrote: So looks like something odd you are doing - are you using any unusual build options? The unusual part looks to be the build environment or libraries of this Mac I'm trying to use. The build options are the normal boring set: CONFIGURE = '--prefix=/Users/gsmith/pgwork/inst/latest' '--enable-depend' '--enable-thread-safety' '--enable-cassert' '--enable-debug' This might be a red herring - but I recall having problems using --enable-depend on older OSX versions (10.5 I think), so *maybe* worth seeing if leaving that option off and doing a distclean+rebuild changes anything. Cheers Mark -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
On 2/26/13 5:51 PM, Mark Kirkwood wrote: So looks like something odd you are doing - are you using any unusual build options? The unusual part looks to be the build environment or libraries of this Mac I'm trying to use. The build options are the normal boring set: CONFIGURE = '--prefix=/Users/gsmith/pgwork/inst/latest' '--enable-depend' '--enable-thread-safety' '--enable-cassert' '--enable-debug' CC = gcc CPPFLAGS = CFLAGS = -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g CFLAGS_SL = LDFLAGS = -L../../../src/common -Wl,-dead_strip_dylibs LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgport -lpgcommon -lz -lreadline -lm VERSION = PostgreSQL 9.3devel I included the config.log in my last message, here's the simple output of that: cd . && ./config.status --recheck running CONFIG_SHELL=/bin/sh /bin/sh ./configure --prefix=/Users/gsmith/pgwork/inst/latest --enable-depend --enable-thread-safety --enable-cassert --enable-debug --no-create --no-recursion checking build system type... x86_64-apple-darwin11.4.2 checking host system type... x86_64-apple-darwin11.4.2 checking which template to use... darwin checking whether to build with 64-bit integer date/time support... yes checking whether NLS is wanted... no checking for default port number... 5432 checking for block size... 8kB checking for segment size... 1GB checking for WAL block size... 8kB checking for WAL segment size... 16MB checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc supports -Wdeclaration-after-statement... yes checking whether gcc supports -Wendif-labels... yes checking whether gcc supports -Wmissing-format-attribute... yes checking whether gcc supports -Wformat-security... yes checking whether gcc supports -fno-strict-aliasing... yes checking whether gcc supports -fwrapv... yes checking whether gcc supports -fexcess-precision=standard... no checking whether the C compiler still works... yes checking how to run the C preprocessor... gcc -E checking allow thread-safe client libraries... yes checking whether to build with Tcl... no checking whether to build Perl modules... no checking whether to build Python modules... no checking whether to build with GSSAPI support... no checking whether to build with Kerberos 5 support... no checking whether to build with PAM support... no checking whether to build with LDAP support... no checking whether to build with Bonjour support... no checking whether to build with OpenSSL support... no checking whether to build with SELinux support... no checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ld used by GCC... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no checking for ranlib... ranlib checking for strip... strip checking whether it is possible to strip libraries... yes checking for ar... ar checking for a BSD-compatible install... /usr/bin/install -c checking for tar... /usr/bin/tar checking whether ln -s works... yes checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking for a thread-safe mkdir -p... config/install-sh -c -d checking for bison... /usr/bin/bison configure: using bison (GNU Bison) 2.3 checking for flex... /usr/bin/flex configure: using flex 2.5.35 Apple(flex-31) checking for perl... /usr/bin/perl configure: using perl 5.12.3 checking for main in -lm... yes checking for library containing setproctitle... no checking for library containing dlopen... none required checking for library containing socket... none required checking for library containing shl_load... no checking for library containing getopt_long... none required checking for library containing crypt... none required checking for library containing fdatasync... none required checking for library containing gethostbyname_r... no checking for library containing shmget... none required checking for library containing readline... -lreadline checking for inflate in -lz... yes checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking crypt.h usability... no checking crypt.h presence... no checking for crypt.h... no checking dld.h usability... no checking dld.h p
Re: [HACKERS] DBD::Pg PPM?
On Feb 26, 2013, at 3:07 PM, Jan Dubois wrote: >> Hello ActiveStaters, > > The commonly used term is actually Activator. :) Got it! > Yes, that is one reason. We then need to apply several patches to build > things correctly though (to statically link the client libs, deal with > missing SIGALARM on Windows, etc). > > We actually have all those changes, as we include DBD::Pg in all ActivePerl > builds; they would just need to be converted into the distroprefs format > used by the CPAN builders. It will take some time to do it, and hasn't > been a big priority because ActivePerl already contains whatever the latest > version of DBD::Pg was at the time of building AP. So this is really only > about updating DBD::Pg for older releases of AP. Oh, I didn't realize that DBD::Pg was shipped with ActivePerl. That certainly simplifies things. > This is different from DBD::mysql, which is not included in ActivePerl due > to GPL-only licensing. So it was more important to spend the extra time to > get it into PPM. Sure, that makes sense. Thanks for the quick response! David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
Greg Smith writes: > $ initdb --help > The files belonging to this database system will be owned by user "gsmith". Hm. Works as expected for me. What platform is this exactly? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Memory leakage associated with plperl spi_prepare/spi_freeplan
I looked into the problem described here:
http://www.postgresql.org/message-id/5125087d.8090...@deriva.de
The core of the problem is that plperl's plperl_spi_prepare() sets up
input conversion functions for the parameters of a prepared query
using perm_fmgr_info(), which allocates FmgrInfo structs for the
I/O functions in TopMemoryContext and makes their fn_mcxt values point
to TopMemoryContext too. So when domains.c allocates assorted stuff
in fn_mcxt, that stuff lives forever ... but guess what, the prepared
query doesn't. So we have a memory leak on every use of spi_freeplan().
The leak is particularly egregious for this specific test case, where
an 8K-or-so ExprContext is made as a consequence of domain_check_input's
call to CreateStandaloneExprContext(); but it would add up to something
noticeable eventually even with input functions as innocuous as int4in.
Proof is to try this:
create or replace function perlleak(n int) returns void as $$
my ($n) = @_;
while ($n--) {
my $stmt = spi_prepare('select $1', 'int');
spi_freeplan($stmt);
}
$$ language plperl volatile strict;
with repeat count of a million or so.
I'm surprised we've not seen this reported before --- maybe people
don't tend to use spi_freeplan() much in plperl.
I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.
Not being particularly a plperl user, I don't really want to code and
test this. Any takers?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] DBD::Pg PPM?
Hello ActiveStaters, I see that the DBD::Pg build always fails: http://code.activestate.com/ppm/DBD-Pg/ I'm sure this is because PostgreSQL is not installed on any of the PPM build boxes (smokers?). DBD::mysql, on the other hand, builds fine (most of the time): http://code.activestate.com/ppm/DBD-mysql/ Would it be possible to adapt whatever was done for DBD::mysql to allow DBD::Pg to be built? I assume it means that PostgreSQL needs to be installed, or at least its include files and libraries. I've Cc'd pgsql-hackers, as its denizens can provide feedback on what might be needed. There might also be something worth poaching from the abandoned "DBD-Pg PPM Binaries" project. http://pgfoundry.org/projects/dbdpgppm/ Thanks, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] auto_explain WAS: RFC: Timing Events
On 2/26/13 11:19 AM, Robert Haas wrote: On Mon, Feb 25, 2013 at 10:22 PM, Greg Stark wrote: On Mon, Feb 25, 2013 at 8:26 PM, Robert Haas wrote: On Sun, Feb 24, 2013 at 7:27 PM, Jim Nasby wrote: We actually do that in our application and have discovered that random sampling can end up significantly skewing your data. /me blinks. How so? Sampling is a pretty big area of statistics. There are dozens of sampling methods to deal with various problems that occur with different types of data distributions. One problem is if you have some very rare events then random sampling can produce odd results since those rare events will drop out entirely unless your sample is very large whereas less rare events are represented proportionally. There are sampling methods that ensure that x% of the rare events are included even if those rare events are less than x% of your total data set. One of those might be appropriate to use for profiling data when you're looking for rare slow queries amongst many faster queries. I'll grant all that, but it still seems to me like x% of all queries plus all queries running longer than x milliseconds would cover most of the interesting cases. In our specific case, we were capturing statistics about webpage hits; when we took "random" samples and multiplied back out there were some inconsistencies that we couldn't explain. We just turned the sampling off and never really investigated. So it's possible that something in our implementation was flawed. However, randomness can also work against you in strange ways. You could easily get a glut of samples that are skewed in one direction or another. And the problem can potentially be far worse if your "randomness" is actually impacted by some other aspect of the system. For this case it might be good enough. I just wanted to caution about it. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] initdb ignoring options?
Hmm - just did a pull now: $ initdb --version initdb (PostgreSQL) 9.3devel $ initdb --help initdb initializes a PostgreSQL database cluster. Usage: initdb [OPTION]... [DATADIR] ...[snipped rest of help output] So looks like something odd you are doing - are you using any unusual build options? Cheers Mark On 27/02/13 11:16, Greg Smith wrote: Here's a happy initdb on 9.1 providing help: $ psql --version psql (PostgreSQL) 9.1.8 $ /usr/pgsql-9.1/bin/initdb --help initdb initializes a PostgreSQL database cluster. Usage: initdb [OPTION]... [DATADIR] ... Here's what I get on the latest repo: $ psql --version psql (PostgreSQL) 9.3devel $ initdb --help The files belonging to this database system will be owned by user "gsmith". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8" ... initdb --version does the same thing for me: ignores the command line and just goes onward to create a new cluster. I checked a random, not current 9.2 install and saw the same issue existed on that version too: $ psql --version psql (PostgreSQL) 9.2.1 $ initdb --help The files belonging to this database system will be owned by user "gsmith". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8" ... I would like for someone to tell me I'm doing something stupid and this problem isn't really there going back to 9.2. The last potentially related initdb change I found on a quick scan, from around the right time period was the code refactoring of http://www.postgresql.org/message-id/e1teyox-0005w4...@gemulon.postgresql.org (there were a few additional initdb commits around then too) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/PgSQL STRICT
> FWIW, I share Peter's poor opinion of this syntax. I can see the > appeal of not having to write an explicit check of the rowcount > afterwards, but that appeal is greatly weakened by the strange syntax. > (IOW, if you were counting me as a + vote, that was only a vote for > the concept --- on reflection I don't much like this implementation.) I agree with other commentators that it would be useful, but that the word STRICT should be near the word INTO, making it clear that the STRICTness is tied to the variable assignment. I do think we can deal with the "more than one" case once PL/pgSQL INTO actually supports assigning more than one row to a variable, which currently it doesn't. At that time, we'll just have to remember to update the code for STRICT as well. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] initdb ignoring options?
Here's a happy initdb on 9.1 providing help: $ psql --version psql (PostgreSQL) 9.1.8 $ /usr/pgsql-9.1/bin/initdb --help initdb initializes a PostgreSQL database cluster. Usage: initdb [OPTION]... [DATADIR] ... Here's what I get on the latest repo: $ psql --version psql (PostgreSQL) 9.3devel $ initdb --help The files belonging to this database system will be owned by user "gsmith". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8" ... initdb --version does the same thing for me: ignores the command line and just goes onward to create a new cluster. I checked a random, not current 9.2 install and saw the same issue existed on that version too: $ psql --version psql (PostgreSQL) 9.2.1 $ initdb --help The files belonging to this database system will be owned by user "gsmith". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8" ... I would like for someone to tell me I'm doing something stupid and this problem isn't really there going back to 9.2. The last potentially related initdb change I found on a quick scan, from around the right time period was the code refactoring of http://www.postgresql.org/message-id/e1teyox-0005w4...@gemulon.postgresql.org (there were a few additional initdb commits around then too) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/PgSQL STRICT
"Marko Tiikkaja" writes: > If I'm counting correctly, we have four votes for this patch and two votes > against it. > Any other opinions? FWIW, I share Peter's poor opinion of this syntax. I can see the appeal of not having to write an explicit check of the rowcount afterwards, but that appeal is greatly weakened by the strange syntax. (IOW, if you were counting me as a + vote, that was only a vote for the concept --- on reflection I don't much like this implementation.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/PgSQL STRICT
Hi, If I'm counting correctly, we have four votes for this patch and two votes against it. Any other opinions? Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: FILTER for aggregates [was Re: [HACKERS] Department of Redundancy Department: makeNode(FuncCall) division]
On Wed, Feb 13, 2013 at 06:45:31AM -0800, David Fetter wrote: > On Sat, Feb 09, 2013 at 11:59:22PM -0800, David Fetter wrote: > > Folks, > > > > Per suggestions and lots of help from Andrew Gierth, please find > > attached a patch to clean up the call sites for FuncCall nodes, which > > I'd like to expand centrally rather than in each of the 37 (or 38, but > > I only redid 37) places where it's called. The remaining one is in > > src/backend/nodes/copyfuncs.c, which has to be modified for any > > changes in the that struct anyhow. > > > > The immediate purpose is two-fold: to reduce some redundancies, which > > I believe is worth doing in and of itself, and to prepare for adding > > FILTER on aggregates from the spec, and possibly other things in > > the part of the spec. > > > > Cheers, > > David. > > Folks, > > Please find attached two versions of a patch which provides optional > FILTER clause for aggregates (T612, "Advanced OLAP operations"). > > The first is intended to be applied on top of the previous patch, the > second without it. The first is, I believe, clearer in what it's > doing. Rather than simply mechanically visiting every place a > function call might be constructed, it visits a central one to change > the default, then goes only to the places where it's relevant. > > The patches are both early WIP as they contain no docs or regression > tests yet. Docs and regression tests added, makeFuncArgs approached dropped for now, will re-visit later. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate *** a/doc/src/sgml/ref/select.sgml --- b/doc/src/sgml/ref/select.sgml *** *** 595,604 GROUP BY expression [, ...] ! Aggregate functions, if any are used, are computed across all rows making up each group, producing a separate value for each group (whereas without GROUP BY, an aggregate produces a single value computed across all the selected rows). When GROUP BY is present, it is not valid for the SELECT list expressions to refer to ungrouped columns except within aggregate functions or if the --- 595,607 ! In the absence of a FILTER clause, ! aggregate functions, if any are used, are computed across all rows making up each group, producing a separate value for each group (whereas without GROUP BY, an aggregate produces a single value computed across all the selected rows). + When a FILTER clause is present, only those + rows matching the FILTER clause are included. When GROUP BY is present, it is not valid for the SELECT list expressions to refer to ungrouped columns except within aggregate functions or if the *** a/doc/src/sgml/syntax.sgml --- b/doc/src/sgml/syntax.sgml *** *** 1562,1585 sqrt(2) syntax of an aggregate expression is one of the following: ! aggregate_name (expression [ , ... ] [ order_by_clause ] ) ! aggregate_name (ALL expression [ , ... ] [ order_by_clause ] ) ! aggregate_name (DISTINCT expression [ , ... ] [ order_by_clause ] ) ! aggregate_name ( * ) where aggregate_name is a previously defined aggregate (possibly qualified with a schema name), ! expression is ! any value expression that does not itself contain an aggregate ! expression or a window function call, and ! order_by_clause is a optional ! ORDER BY clause as described below. ! The first form of aggregate expression invokes the aggregate ! once for each input row. The second form is the same as the first, since ALL is the default. The third form invokes the aggregate once for each distinct value --- 1562,1587 syntax of an aggregate expression is one of the following: ! aggregate_name (expression [ , ... ] [ order_by_clause ] ) [ FILTER ( WHERE filter_clause ) ] ! aggregate_name (ALL expression [ , ... ] [ order_by_clause ] ) [ FILTER ( WHERE filter_clause ) ] ! aggregate_name (DISTINCT expression [ , ... ] [ order_by_clause ] ) [ FILTER ( WHERE filter_clause ) ] ! aggregate_name ( * ) [ FILTER ( WHERE filter_clause ) ] where aggregate_name is a previously defined aggregate (possibly qualified with a schema name), ! expression is any value expression that ! does not itself contain an aggregate expression or a window ! function call, order_by_clause is a ! optional ORDER BY clause as described below. The ! aggregate_name can also be suffixed ! with FILTER as described below. ! The first form of aggregate expression invokes the aggregate once ! for each input row, or when a FILTER clause is present, each ro
Re: [HACKERS] pg_xlogdump compiler warning
Andres Freund writes: > On 2013-02-25 18:15:48 -0500, Peter Eisentraut wrote: >> compat.c: In function âtimestamptz_to_strâ: >> compat.c:56:9: error: passing argument 1 of âlocaltimeâ from >> incompatible pointer type [-Werror] >> In file included from compat.c:21:0: >> /usr/include/time.h:237:19: note: expected âconst time_t *â but argument >> is of type âpg_time_t *â > 32bit I guess? Yeah, I see the same on a 32-bit machine. > A simple cast should do for now. Patch attached. Pushed along with some other, more cosmetic cleanups. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PGXS contrib builds broken?
Andres Freund writes: > On 2013-02-26 13:57:46 +0100, Andres Freund wrote: >> On 2013-02-25 11:50:46 +0100, Bernd Helmle wrote: >>> Looks like the recent refactoring of code into common/ stopped >>> PGXS builds within the PostgreSQL source tree from working, i get >>> /Users/bernd/pgsql-dev/install/HEAD/include/server/postgres_fe.h:27:32: >>> fatal error: common/fe_memutils.h: No such file or directory >> Oh, yes. It wasn't added to the install target during the src/common >> addition. Patch attached. Pushed, thanks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Dunno, I think that's going to result in a very large chunk of mostly > duplicative code in psql. regprocedurein() is fairly short because it > can rely on a ton of code from the parser, but psql won't have that > luxury. Parsing/tokenizing a CSV string inside parens doesn't strike me as all that difficult, even when handling the space-delimininated varname from the type. The hard part would, I think, be the normalization of the type names into what \df returns, but do we even want to try and tackle that..?. How much do we care about supporting every textual representation of the 'integer' type? That's not going to be an issue for people doing tab-completion or using \df's output. We could also have it fall-back to trying w/o any arguments for a unique function name match if the initial attempt w/ the function arguments included doesn't return any results. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
Stephen Frost writes: > * Andrew Dunstan (and...@dunslane.net) wrote: >> If we're going to mess with this area can I put in a plea to get \ef >> and \sf to handle full parameter specs? I want to be able to c&p >> from the \df output to see the function. But here's what happens: > I was thinking along the same lines. This will involve a bit more code > in psql, but I think that's better than trying to get the backend to do > this work for us, for one thing, we want psql to support multiple major > versions and that could be done by adding code to psql, but couldn't be > done for released versions if we depend on the backend to solve this > matching for us. Dunno, I think that's going to result in a very large chunk of mostly duplicative code in psql. regprocedurein() is fairly short because it can rely on a ton of code from the parser, but psql won't have that luxury. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
* Andrew Dunstan (and...@dunslane.net) wrote: > If we're going to mess with this area can I put in a plea to get \ef > and \sf to handle full parameter specs? I want to be able to c&p > from the \df output to see the function. But here's what happens: I was thinking along the same lines. This will involve a bit more code in psql, but I think that's better than trying to get the backend to do this work for us, for one thing, we want psql to support multiple major versions and that could be done by adding code to psql, but couldn't be done for released versions if we depend on the backend to solve this matching for us. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
On 02/26/2013 02:12 PM, Tom Lane wrote: Stephen Frost writes: * Pavel Stehule (pavel.steh...@gmail.com) wrote: Minimally \ef needs exact specification - you cannot to edit more functions in same time. So we have to be able identify if there are no selected function or if there are more functions. We can write a auxiliary function that returns list of function oids for specified signature - but it is relative much more code and it is hard to implement for older versions - but we can use regproc and regprocedure there. Using regproc and regprocedure is the wrong approach here and will be a pain to maintain as well as a backwards incompatible change to how they behave. We have solved this problem already and what \df does is exactly the right answer. Well, actually I think Pavel's got a point. What about overloaded functions? In \df we don't try to solve that problem, we just print them all: regression=# \df abs List of functions Schema | Name | Result data type | Argument data types | Type +--+--+-+ pg_catalog | abs | bigint | bigint | normal pg_catalog | abs | double precision | double precision| normal pg_catalog | abs | integer | integer | normal pg_catalog | abs | numeric | numeric | normal pg_catalog | abs | real | real| normal pg_catalog | abs | smallint | smallint| normal (6 rows) In \ef you need to select just one of these functions, but \df doesn't have any ability to do that: regression=# \df abs(int) List of functions Schema | Name | Result data type | Argument data types | Type +--+--+-+-- (0 rows) Now, maybe we *should* teach \df about handling parameter types and then \ef can piggyback on it, but that code isn't there now. If we're going to mess with this area can I put in a plea to get \ef and \sf to handle full parameter specs? I want to be able to c&p from the \df output to see the function. But here's what happens: andrew=# \df json_get List of functions Schema | Name | Result data type | Argument data types | Type +--+--+---+ pg_catalog | json_get | json | from_json json, element integer | normal pg_catalog | json_get | json | from_json json, VARIADIC path_elements text[] | normal (2 rows) andrew=# \sf json_get (from_json json, element integer ) ERROR: invalid type name "from_json json" cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Well, actually I think Pavel's got a point. What about overloaded > functions? In \df we don't try to solve that problem, we just print > them all: To be honest, I was reading through that code the other night and could have sworn that I saw us doing some kind of magic on the arguments under \df, but of course I don't see it now. > Now, maybe we *should* teach \df about handling parameter types and > then \ef can piggyback on it, but that code isn't there now. That's definitely the right approach, imv. It should also work if only a function name is provided and it's not overloaded, of course. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] 9.2.3 crashes during archive recovery
Folks, Is there any way this particular issue could cause data corruption without causing a crash? I don't see a way for it to do so, but I wanted to verify. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bugfix: --echo-hidden is not supported by \sf statements
Stephen Frost writes: > * Pavel Stehule (pavel.steh...@gmail.com) wrote: >> Minimally \ef needs exact specification - you cannot to edit more >> functions in same time. So we have to be able identify if there are no >> selected function or if there are more functions. We can write a >> auxiliary function that returns list of function oids for specified >> signature - but it is relative much more code and it is hard to >> implement for older versions - but we can use regproc and regprocedure >> there. > Using regproc and regprocedure is the wrong approach here and will be a > pain to maintain as well as a backwards incompatible change to how they > behave. We have solved this problem already and what \df does is exactly > the right answer. Well, actually I think Pavel's got a point. What about overloaded functions? In \df we don't try to solve that problem, we just print them all: regression=# \df abs List of functions Schema | Name | Result data type | Argument data types | Type +--+--+-+ pg_catalog | abs | bigint | bigint | normal pg_catalog | abs | double precision | double precision| normal pg_catalog | abs | integer | integer | normal pg_catalog | abs | numeric | numeric | normal pg_catalog | abs | real | real| normal pg_catalog | abs | smallint | smallint| normal (6 rows) In \ef you need to select just one of these functions, but \df doesn't have any ability to do that: regression=# \df abs(int) List of functions Schema | Name | Result data type | Argument data types | Type +--+--+-+-- (0 rows) Now, maybe we *should* teach \df about handling parameter types and then \ef can piggyback on it, but that code isn't there now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review : Add hooks for pre- and post-processor executables for COPY and \copy
On 22.02.2013 12:43, Etsuro Fujita wrote: 1. "Broken pipe" is not handled in case of psql "\copy" command; Issue are as follows: Following are verified on SuSE-Linux 10.2. 1) psql is exiting when "\COPY xxx TO" command is issued and command/script is not found When popen is called in write mode it is creating valid file descriptor and when it tries to write to file "Broken pipe" error is> coming which is not handled. psql# \copy pgbench_accounts TO PROGRAM '../compress.sh pgbench_accounts4.txt' 2) When "\copy" command is in progress then program/command is killed/"crashed due to any problem" psql is exiting. This is a headache. I have no idea how to solve this. I think we can keep it for committer to take a call on this issue. Agreed. Fortunately this one is easy. We just need to ignore SIGPIPE, by calling pqsignal(SIGPIPE, SIG_IGN) before popen(). We already do the same in psql when the query output is piped to a pager, in PageOutput. 5. Following in copy.sgml can be changed to make more meaningful as the first line looks little adhoc. + + The command that input comes from or that output goes to. + The command for COPY FROM, which input comes from, must write + its output + to standard output. The command for COPY TO, which output goes + to, must + read its input from standard input. + I've struggled to make the document more meaningful. To be honest, I am not sure whether introducing pre, post processor terminology is right or not, But again I shall let committer decide about this point. Agreed. I changed the terminology to use terms like "the command specified with PROGRAM", instead of talking about pre- and post-processsors. I have one small another doubt that in function parse_slash_copy, you avoided expand tilde for program case, which I am not sure is the right thing or not. Sorry, I'm not sure that, too. I'd like to leave this for committers. That seems correct. The shell will do tilde expansion with popen(), we don't need to do it ourselves. There's one oddity in the psql \copy syntax. This is actually present in earlier versions too, but I think we should fix it now that we extend the syntax: -- Writes to standard output. There's no way to write to a file called "stdout". \copy foo to 'stdout' I think we should change the interpretation of the above so that it writes to a file called "stdout". It's possible that there's an application out there that relies on that to write to stdout, but it's not hard to fix the application. A small note in the release notes might be in order. Also, I think we should require the command to be quoted in \copy. Ie. let's forbid this: \copy pgbench_accounts to program /bin/cat>/dev/null and require that it's written as: \copy pgbench_accounts to program '/bin/cat>/dev/null' We don't require quoting for filenames, which I find a bit weird too, but it seems particularly confusing for a shell command. Attached is a new version of this patch that I almost committed, but one thing caught my eye at the last minute: The error reporting is not very user-friendly. If the program fails after reading/writing some rows, the reason is printed to the log, but not the user: postgres=# copy foo to program '/tmp/midway-crash'; ERROR: could not execute command "/tmp/midway-crash" the log has more details: LOG: child process exited with exit code 10 STATEMENT: copy foo to program '/tmp/midway-crash'; ERROR: could not execute command "/tmp/midway-crash" STATEMENT: copy foo to program '/tmp/midway-crash'; I think we should arrange for the "child process exited with exit code 10" to be printed as errdetail to the client. Similarly, with psql \copy command, the "child process exited with exit code 10" command shouldn't be printed straight to stderr, but should go through psql_error. I'll try to refactor pclose_check tomorrow to do that. Meanwhile, please take a look at the attached if you have the time. I rewrote much of the docs changes, and some comments. - Heikki diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c index d644a46..d1cca1e 100644 --- a/contrib/file_fdw/file_fdw.c +++ b/contrib/file_fdw/file_fdw.c @@ -588,6 +588,7 @@ fileBeginForeignScan(ForeignScanState *node, int eflags) */ cstate = BeginCopyFrom(node->ss.ss_currentRelation, filename, + false, NIL, options); @@ -660,6 +661,7 @@ fileReScanForeignScan(ForeignScanState *node) festate->cstate = BeginCopyFrom(node->ss.ss_currentRelation, festate->filename, + false, NIL, festate->options); } @@ -993,7 +995,7 @@ file_acquire_sample_rows(Relation onerel, int elevel, /* * Create CopyState from FDW options. */ - cstate = BeginCopyFrom(onerel, filename, NIL, options); + cstate = BeginCopyFrom(onerel, filename, false,
Re: [HACKERS] pg_basebackup caused FailedAssertion
Fujii Masao writes:
> In HEAD, when I ran "pg_basebackup -D hoge -X stream",
> I got the following FailedAssertion error:
> TRAP: FailedAssertion("!((wakeEvents & ((1 << 1) | (1 << 2))) != (1 <<
> 2))", File: "pg_latch.c", Line: 234)
> This error happens after the commit 0b6329130e8e4576e97ff763f0e773347e1a88af.
> This assertion error happens when WL_SOCKET_WRITEABLE without
> WL_SOCKET_READABLE is specified in WaitLatchOrSocket(). This
> condition is met when walsender has received CopyDone from the client,
> but the output buffer is not empty. If reaching such condition is legitimate,
> I think that we should get rid of the Assertion check which caused the above
> FailedAssertion error. Thought?
The reason for the assertion is that that case doesn't actually work.
The code that is passing that combination of flags needs to be changed.
Or else you can try to implement the ability to support READABLE only.
But just removing the Assert is 100% wrong.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_basebackup caused FailedAssertion
Hi,
In HEAD, when I ran "pg_basebackup -D hoge -X stream",
I got the following FailedAssertion error:
TRAP: FailedAssertion("!((wakeEvents & ((1 << 1) | (1 << 2))) != (1 <<
2))", File: "pg_latch.c", Line: 234)
This error happens after the commit 0b6329130e8e4576e97ff763f0e773347e1a88af.
This assertion error happens when WL_SOCKET_WRITEABLE without
WL_SOCKET_READABLE is specified in WaitLatchOrSocket(). This
condition is met when walsender has received CopyDone from the client,
but the output buffer is not empty. If reaching such condition is legitimate,
I think that we should get rid of the Assertion check which caused the above
FailedAssertion error. Thought?
Regards,
--
Fujii Masao
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
On Tue, Feb 26, 2013 at 12:02 PM, Peter Eisentraut wrote: > On 2/26/13 11:45 AM, Tom Lane wrote: >> But let's not break the cases that do work. One >> of the functions of contrib/ is to serve as models/skeletons for >> external modules. If we pull out the "useless" PGXS support then we'll >> just be making it that much harder to copy a contrib module and start >> doing something useful. > > Well, this is exactly the problem. Because of this skeleton idea, most > external extension modules do not build unless you set USE_PGXS=1 before > building, because they think that they live in contrib by default, which > is completely bizarre and user-unfriendly. > > We could have an actual example or skeleton whose purpose is to teach > extension authors. The actual contrib module makefiles should just do > their job and don't pretend to teach things that are misguided and/or > don't work. I couldn't agree more. I can't think how much time I've wasted by forgetting (or remembering) to type USE_PGXS=1 over the years, or not realizing that I needed to. I'm sure there are many other people in the same boat. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On 26.02.2013 18:58, Michael Meskes wrote: On Tue, Feb 26, 2013 at 05:13:38PM +0200, Heikki Linnakangas wrote: Any particular reason for ecpg to check that, while the backend doesn't care? I think we should just remove those checks from the ecpg grammar. IIRC this check was added because the check for "COPY FROM STDIN" had to added anyway. Since you left that one in, the patch is fine by me, although I still don't see a reason for it. Just less code to maintain. And it's strange to forbid something in ecpg grammar that the backend accepts, as a matter of principle. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
On 26 February 2013 17:02, Peter Eisentraut wrote: > Well, this is exactly the problem. Because of this skeleton idea, most > external extension modules do not build unless you set USE_PGXS=1 before > building, because they think that they live in contrib by default, which > is completely bizarre and user-unfriendly. repmgr is a popular external extension module. The README actually suggests moving the entire source tree into /contrib as an alternative to setting USE_PGXS=1. That does seem kind of weird, and yet I can understand the train of thought. My advice to others working on external modules would not be to generalise from the example of /contrib, but to generalise from the example of popular external modules. This is particularly important when targeting multiple Postgres versions. -- Regards, Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On Tue, Feb 26, 2013 at 11:50 AM, Heikki Linnakangas wrote: > Yeah, I'd guess that it was an oversight. But it goes back all the way to > Postgres95, so it's a bit too late to change that. I don't see why. We've plugged holes like this before and will do so again in the future, I'm sure. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] auto_explain WAS: RFC: Timing Events
On Mon, Feb 25, 2013 at 10:22 PM, Greg Stark wrote: > On Mon, Feb 25, 2013 at 8:26 PM, Robert Haas wrote: >> On Sun, Feb 24, 2013 at 7:27 PM, Jim Nasby wrote: >>> We actually do that in our application and have discovered that random >>> sampling can end up significantly skewing your data. >> >> /me blinks. >> >> How so? > > Sampling is a pretty big area of statistics. There are dozens of > sampling methods to deal with various problems that occur with > different types of data distributions. > > One problem is if you have some very rare events then random sampling > can produce odd results since those rare events will drop out entirely > unless your sample is very large whereas less rare events are > represented proportionally. There are sampling methods that ensure > that x% of the rare events are included even if those rare events are > less than x% of your total data set. One of those might be appropriate > to use for profiling data when you're looking for rare slow queries > amongst many faster queries. I'll grant all that, but it still seems to me like x% of all queries plus all queries running longer than x milliseconds would cover most of the interesting cases. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
On 2/26/13 11:45 AM, Tom Lane wrote: > But let's not break the cases that do work. One > of the functions of contrib/ is to serve as models/skeletons for > external modules. If we pull out the "useless" PGXS support then we'll > just be making it that much harder to copy a contrib module and start > doing something useful. Well, this is exactly the problem. Because of this skeleton idea, most external extension modules do not build unless you set USE_PGXS=1 before building, because they think that they live in contrib by default, which is completely bizarre and user-unfriendly. We could have an actual example or skeleton whose purpose is to teach extension authors. The actual contrib module makefiles should just do their job and don't pretend to teach things that are misguided and/or don't work. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On Tue, Feb 26, 2013 at 05:13:38PM +0200, Heikki Linnakangas wrote: > COPY foo FROM STDOUT > COPY foo TO STDIN Does this make sense? > Any particular reason for ecpg to check that, while the backend > doesn't care? I think we should just remove those checks from the > ecpg grammar. IIRC this check was added because the check for "COPY FROM STDIN" had to added anyway. Since you left that one in, the patch is fine by me, although I still don't see a reason for it. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On 26.02.2013 18:40, Tom Lane wrote: Heikki Linnakangas writes: On 26.02.2013 18:23, Tom Lane wrote: (I assume the backend will bounce the other cases at some post-grammar stage.) No. All four combinations of FROM/TO and STDIN/STDOUT are accepted: Huh. That seems like an odd decision. If we agree that that behavior is desirable, then your patch is ok as-is, though I do question whether this should be tested in the grammar at all rather than at runtime. I wonder whether this is just an oversight, or if we did it intentionally because people were confused about which combinations to use. It seems like maybe "TO STDIN" could be justified if you thought about stdin of the recipient rather than stdout of the server, but it still seems a bit sloppy thinking. Yeah, I'd guess that it was an oversight. But it goes back all the way to Postgres95, so it's a bit too late to change that. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
Andres Freund writes: > Wait what? So I need to make install before I can compile extensions? > That doesn't seem to be something realistic. You know, any extension that's not in our source tree out there is maintained in a way to target all supported versions of PostgreSQL from the same sources. I understand that we want to continue applying our versioning rules to contribs, and I think that it's a good idea. But now that we're making a real cut decision about that, I think contribs should be documented as NOT extensions. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On Tue, Feb 26, 2013 at 4:34 PM, Heikki Linnakangas wrote: > No. All four combinations of FROM/TO and STDIN/STDOUT are accepted: ... > postgres=# copy foo to stdin; > foo > bar > postgres=# copy foo to stdout; > foo > bar Hm, so STDIN/STDOUT are just noise words and psql uses stdin for input and stdout for output regardless of what's specified? That seems a bit odd. I would have expected to be able to do something like cat > script.sql copy foo to stdin; copy bar to stdout; ^D psql -f script.sql 0>/tmp/foo.data 1>/tmp/bar.data But then I haven't heard any clamoring of demand for such a feature. And if there was it would make sense to implement "copy foo to fd X" and then make stdin an alias for "fd 0" rather than only support two file descriptors. It wouldn't make sense to expend all that effort just to support writing to just stdin. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
Andres Freund writes: > Well, it is broken for xlogdump because it needs a sourcetree arround. > All I personally really want to do is to replace the usual stanza for > pg_xlogdump with something like: > ifdef USE_PGXS > $(error Building pg_xlogdump with PGXS is not supported) > else > ... Seems reasonable. But let's not break the cases that do work. One of the functions of contrib/ is to serve as models/skeletons for external modules. If we pull out the "useless" PGXS support then we'll just be making it that much harder to copy a contrib module and start doing something useful. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
On 2013-02-26 11:33:48 -0500, Tom Lane wrote: > [ Sigh... ] Why this eagerness to fix what isn't broken? Well, it is broken for xlogdump because it needs a sourcetree arround. All I personally really want to do is to replace the usual stanza for pg_xlogdump with something like: ifdef USE_PGXS $(error Building pg_xlogdump with PGXS is not supported) include $(PGXS) else ... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
Heikki Linnakangas writes: > On 26.02.2013 18:23, Tom Lane wrote: >> (I assume >> the backend will bounce the other cases at some post-grammar stage.) > No. All four combinations of FROM/TO and STDIN/STDOUT are accepted: Huh. That seems like an odd decision. If we agree that that behavior is desirable, then your patch is ok as-is, though I do question whether this should be tested in the grammar at all rather than at runtime. I wonder whether this is just an oversight, or if we did it intentionally because people were confused about which combinations to use. It seems like maybe "TO STDIN" could be justified if you thought about stdin of the recipient rather than stdout of the server, but it still seems a bit sloppy thinking. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
On 2013-02-26 17:23:01 +0100, Dimitri Fontaine wrote: > Peter Eisentraut writes: > > I for one wonder why we even have PGXS support in contrib at all. It's > > not documented or tested anywhere, so it might as well not exist. Agreed. I personally think we just should build contrib/ as part of the normal build and be done with it. There's currently the argument that building them separately is interesting because you can do it from a separate sourcetree if you have a precompiled postgres around. But whats the usecase for that? I would much rather designate one example module that is always built with PGXS (possibly only during regress's install?), so we reliably test that. > I think I did about the same comment back when cooking the extension > patch, and the answer then was all about providing PGXS usage examples. > Now if none of the buildfarm animals are actually building our contribs > out of tree, maybe we should just remove those examples. > > The cost of keeping them is that they double-up the Makefile content and > lots of users do think they need their extension's Makefile to be > structured the same. The common effect before the extension availability > was for people to provide extensions that would only build in tree. > > I don't want to kill cleaning up those Makefiles, but I still want to > make a strong correlation in between that point and providing core > maintained extensions. I don't think extensions should have support for > being built in-tree at all. Wait what? So I need to make install before I can compile extensions? That doesn't seem to be something realistic. > My proposal: paint them extension rather than contrib modules, then > cleanup Makefiles so as to stop building them in-tree. Imo painting them extensions is something unrelated. Which doesn't apply to everything in contrib/, there are several modules that don't make sense as extensions (chkpass, oid2name, passwordcheck, pg_archivecleanup, pgbench, ...). Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
Tom Lane writes: > work today --- in particular, this proposal will break building contrib > before the main tree has been installed. True that. > If somebody wants to set up a buildfarm member that occasionally tests > PGXS building of contrib/, that's fine with me. But it isn't, and never > will be, the main build scenario for contrib/ IMO. I guess you just made a final vote on the whole idea of making contrib a set of extensions maintained by the PostgreSQL commiters, then. It might be good to document that position, so that the topic isn't raised again each year? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump compile error
On Wed, Feb 27, 2013 at 1:25 AM, Andres Freund wrote: > Hi, > > On 2013-02-27 00:34:54 +0900, Fujii Masao wrote: >> Hi, >> >> When I compiled pg_xlogdump in HEAD, I got the following error. >> >> make: *** No rule to make target `rmgrdesc.o', needed by `pg_xlogdump'. >> Stop. >> >> $ uname -a >> Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 >> 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64 > > Is that with a clean tree? Or are you perhaps influenced by the bug > fixed in e5bf0c376ed43feaebbe37519a6b8bc8e795f1d2? I.e. that make clean > removed rmgrdesc.c? Thanks! You're right. rmgrdesc.c was accidentally deleted and I could not notice that. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
On 26.02.2013 18:23, Tom Lane wrote: Heikki Linnakangas writes: While looking at Fujita Etsuro's patch to allow copy to/from a shell command, I noticed that the grammar currently allows these: COPY foo FROM STDOUT COPY foo TO STDIN In other words, STDIN and STDOUT can be used completely interchangeably. However, the ecpg grammar is more strict about that: ERROR: COPY TO STDIN is not possible Any particular reason for ecpg to check that, while the backend doesn't care? I think we should just remove those checks from the ecpg grammar. Agreed, but your draft patch doesn't do that completely. It should only make tests that correspond to what the error message says. Sorry, I don't understand what you're saying. Can you elaborate? (I assume the backend will bounce the other cases at some post-grammar stage.) No. All four combinations of FROM/TO and STDIN/STDOUT are accepted: postgres=# copy foo from stdin; Enter data to be copied followed by a newline. End with a backslash and a period on a line by itself. >> foo >> \. postgres=# copy foo from stdout; Enter data to be copied followed by a newline. End with a backslash and a period on a line by itself. >> bar >> \. postgres=# copy foo to stdin; foo bar postgres=# copy foo to stdout; foo bar - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
Dimitri Fontaine writes: > Peter Eisentraut writes: >> I for one wonder why we even have PGXS support in contrib at all. It's >> not documented or tested anywhere, so it might as well not exist. > I think I did about the same comment back when cooking the extension > patch, and the answer then was all about providing PGXS usage examples. > Now if none of the buildfarm animals are actually building our contribs > out of tree, maybe we should just remove those examples. > The cost of keeping them is that they double-up the Makefile content and > lots of users do think they need their extension's Makefile to be > structured the same. The common effect before the extension availability > was for people to provide extensions that would only build in tree. > I don't want to kill cleaning up those Makefiles, but I still want to > make a strong correlation in between that point and providing core > maintained extensions. I don't think extensions should have support for > being built in-tree at all. > My proposal: paint them extension rather than contrib modules, then > cleanup Makefiles so as to stop building them in-tree. [ Sigh... ] Why this eagerness to fix what isn't broken? Leave the Makefiles alone. They're not broken and they provide useful examples, plus a sense of continuity between in-tree and not-in-tree extensions. Any change here will likely break build scenarios that work today --- in particular, this proposal will break building contrib before the main tree has been installed. If somebody wants to set up a buildfarm member that occasionally tests PGXS building of contrib/, that's fine with me. But it isn't, and never will be, the main build scenario for contrib/ IMO. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump compile error
Hi, On 2013-02-27 00:34:54 +0900, Fujii Masao wrote: > Hi, > > When I compiled pg_xlogdump in HEAD, I got the following error. > > make: *** No rule to make target `rmgrdesc.o', needed by `pg_xlogdump'. Stop. > > $ uname -a > Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 > 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64 Is that with a clean tree? Or are you perhaps influenced by the bug fixed in e5bf0c376ed43feaebbe37519a6b8bc8e795f1d2? I.e. that make clean removed rmgrdesc.c? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "COPY foo FROM STDOUT" and ecpg
Heikki Linnakangas writes: > While looking at Fujita Etsuro's patch to allow copy to/from a shell > command, I noticed that the grammar currently allows these: > COPY foo FROM STDOUT > COPY foo TO STDIN > In other words, STDIN and STDOUT can be used completely interchangeably. > However, the ecpg grammar is more strict about that: > ERROR: COPY TO STDIN is not possible > Any particular reason for ecpg to check that, while the backend doesn't > care? I think we should just remove those checks from the ecpg grammar. Agreed, but your draft patch doesn't do that completely. It should only make tests that correspond to what the error message says. (I assume the backend will bounce the other cases at some post-grammar stage.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump
Peter Eisentraut writes: > I for one wonder why we even have PGXS support in contrib at all. It's > not documented or tested anywhere, so it might as well not exist. I think I did about the same comment back when cooking the extension patch, and the answer then was all about providing PGXS usage examples. Now if none of the buildfarm animals are actually building our contribs out of tree, maybe we should just remove those examples. The cost of keeping them is that they double-up the Makefile content and lots of users do think they need their extension's Makefile to be structured the same. The common effect before the extension availability was for people to provide extensions that would only build in tree. I don't want to kill cleaning up those Makefiles, but I still want to make a strong correlation in between that point and providing core maintained extensions. I don't think extensions should have support for being built in-tree at all. My proposal: paint them extension rather than contrib modules, then cleanup Makefiles so as to stop building them in-tree. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_xlogdump compile error
Hi, When I compiled pg_xlogdump in HEAD, I got the following error. make: *** No rule to make target `rmgrdesc.o', needed by `pg_xlogdump'. Stop. $ uname -a Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64 Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] "COPY foo FROM STDOUT" and ecpg
While looking at Fujita Etsuro's patch to allow copy to/from a shell
command, I noticed that the grammar currently allows these:
COPY foo FROM STDOUT
COPY foo TO STDIN
In other words, STDIN and STDOUT can be used completely interchangeably.
However, the ecpg grammar is more strict about that:
ERROR: COPY TO STDIN is not possible
Any particular reason for ecpg to check that, while the backend doesn't
care? I think we should just remove those checks from the ecpg grammar.
- Heikki
*** a/src/interfaces/ecpg/preproc/ecpg.addons
--- b/src/interfaces/ecpg/preproc/ecpg.addons
***
*** 193,207 ECPG: where_or_current_clauseWHERECURRENT_POFcursor_name block
$$ = cat_str(2,mm_strdup("where current of"), cursor_marker);
}
ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listopt_oidscopy_fromcopy_file_namecopy_delimiteropt_withcopy_options addon
! if (strcmp($6, "to") == 0 && strcmp($7, "stdin") == 0)
! mmerror(PARSE_ERROR, ET_ERROR, "COPY TO STDIN is not possible");
! else if (strcmp($6, "from") == 0 && strcmp($7, "stdout") == 0)
! mmerror(PARSE_ERROR, ET_ERROR, "COPY FROM STDOUT is not possible");
! else if (strcmp($6, "from") == 0 && strcmp($7, "stdin") == 0)
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
- ECPG: CopyStmtCOPYselect_with_parensTOcopy_file_nameopt_withcopy_options addon
- if (strcmp($4, "stdin") == 0)
- mmerror(PARSE_ERROR, ET_ERROR, "COPY TO STDIN is not possible");
ECPG: var_valueNumericOnly addon
if ($1[0] == '$')
{
--- 193,201
$$ = cat_str(2,mm_strdup("where current of"), cursor_marker);
}
ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listopt_oidscopy_fromcopy_file_namecopy_delimiteropt_withcopy_options addon
! if (strcmp($6, "from") == 0 &&
! (strcmp($7, "stdin") == 0 || strcmp($7, "stdout") == 0))
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
ECPG: var_valueNumericOnly addon
if ($1[0] == '$')
{
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PGXS contrib builds broken?
On 2013-02-26 13:57:46 +0100, Andres Freund wrote: > On 2013-02-25 11:50:46 +0100, Bernd Helmle wrote: > > Looks like the recent refactoring of code into common/ stopped > > PGXS builds within the PostgreSQL source tree from working, i get > > > > /Users/bernd/pgsql-dev/install/HEAD/include/server/postgres_fe.h:27:32: > > fatal error: common/fe_memutils.h: No such file or directory > > Oh, yes. It wasn't added to the install target during the src/common > addition. Patch attached. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/src/include/Makefile b/src/include/Makefile index 7e31a1e..c553e74 100644 --- a/src/include/Makefile +++ b/src/include/Makefile @@ -17,12 +17,12 @@ all: pg_config.h pg_config_ext.h pg_config_os.h # Subdirectories containing headers for server-side dev -SUBDIRS = access bootstrap catalog commands datatype executor foreign lib libpq mb \ - nodes optimizer parser postmaster regex replication rewrite storage \ - tcop snowball snowball/libstemmer tsearch tsearch/dicts utils \ - port port/win32 port/win32_msvc port/win32_msvc/sys \ - port/win32/arpa port/win32/netinet port/win32/sys \ - portability +SUBDIRS = access bootstrap catalog commands common datatype executor foreign \ + lib libpq mb nodes optimizer parser postmaster regex replication \ + rewrite storage tcop snowball snowball/libstemmer tsearch \ + tsearch/dicts utils port port/win32 port/win32_msvc \ + port/win32_msvc/sys port/win32/arpa port/win32/netinet \ + port/win32/sys portability # Install all headers install: all installdirs -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PGXS contrib builds broken?
On 2013-02-25 11:50:46 +0100, Bernd Helmle wrote: > Looks like the recent refactoring of code into common/ stopped > PGXS builds within the PostgreSQL source tree from working, i get > > /Users/bernd/pgsql-dev/install/HEAD/include/server/postgres_fe.h:27:32: > fatal error: common/fe_memutils.h: No such file or directory Oh, yes. It wasn't added to the install target during the src/common addition. Patch attached. > when trying to build pg_upgrade, pgbench, pg_xlogdump, ... with PGXS. > > I don't see common/fe_memutils getting installed in include/, so i suppose > either some bug between chair and keyboard or some inconsistencies here. pg_xlogdump won't be buildable without a full sourcetree around though, it needs too much backend infrastructure to be built... Thats why I voted somewhere else to remove PGXS support from it and error out instead... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_xlogdump compiler warning
Hi, On 2013-02-25 18:15:48 -0500, Peter Eisentraut wrote: > compat.c: In function ‘timestamptz_to_str’: > compat.c:56:9: error: passing argument 1 of ‘localtime’ from incompatible > pointer type [-Werror] > In file included from compat.c:21:0: > /usr/include/time.h:237:19: note: expected ‘const time_t *’ but argument is > of type ‘pg_time_t *’ > > gcc 4.7.2 32bit I guess? A simple cast should do for now. Patch attached. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/contrib/pg_xlogdump/compat.c b/contrib/pg_xlogdump/compat.c index 99ecad7..0aee756 100644 --- a/contrib/pg_xlogdump/compat.c +++ b/contrib/pg_xlogdump/compat.c @@ -52,7 +52,8 @@ timestamptz_to_str(TimestampTz dt) static char buf[MAXDATELEN + 1]; static char ts[MAXDATELEN + 1]; static char zone[MAXDATELEN + 1]; - pg_time_t result = timestamptz_to_time_t(dt); + /* cast to time_t so we can use localtime() */ + time_t result = (time_t)timestamptz_to_time_t(dt); struct tm *ltime = localtime(&result); strftime(ts, sizeof(zone), "%Y-%m-%d %H:%M:%S", ltime); -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RFC] Extend namespace of valid guc names
On 2013-02-25 21:13:25 -0500, Tom Lane wrote:
> Andres Freund writes:
> > I propose loosening those restrictions to
> > a) allow repeatedly qualified names like a.b.c
>
> If SET allows it, I guess we can allow it here. But is a grammar change
> really all that is needed to make it work from the file?
Seems so. There's no additional validation that I could find
anywhere. And a simple test confirmed it works.
postgres=# SHOW foo.bar.blub;
foo.bar.blub
--
1
(1 row)
Just for reference, thats the grammar for SET/SHOW:
var_name: ColId
{ $$ = $1; }
| var_name '.' ColId
> > b) allow variables to start with a digit from the second level onwards.
>
> That seems like a seriously bad idea. I note that SET does *not* allow
> this; furthermore it seems like a considerable weakening of our ability
> to detect silly typos in config files. Nor did you offer a use-case
> to justify it.
The use-case I had in mind was
bdr.1.dsn = ...
bdr.2.dsn = ...
bdr.3.dsn = ...
bdr.4.dsn = ...
which is what I had used via -c. But I guess it can easy enough be
replaced by node_$i or something.
Any arguments whether we should try to validate -c SET/SHOW,
set_config() and -c the same?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq
On Tuesday, February 26, 2013 1:36 PM Heikki Linnakangas wrote: > On 26.02.2013 09:06, Amit Kapila wrote: > > On Monday, February 25, 2013 11:26 PM Heikki Linnakangas wrote: > >> On 21.02.2013 16:09, Amit Kapila wrote: > >>> On Thursday, February 21, 2013 12:46 AM Heikki Linnakangas wrote: > >> I've committed those patches, with some further changes. If you have > >> the > >> time, please take another look at the committed patches. Thanks! > > > > 1. For pg_dumpall, incase user gives dbname in connection string, for > ex. > > "user=amit dbname=db_regress port=5434" > > it will be ignored and postgres (default database name) will be > used. > > Yeah, that is mentioned in the docs. Sorry, I have missed that part, this just came to me when I was looking at the code. >The other option would be to use > the database name from the connection string for the initial > connection, > ie. the same as specifying a database name with the -l option. This is what come to my mind initially, as if it is allowed with -l option it is better if the same should be allowed through connection string as well. We are ignoring for pg_basebackup as well, but in pg_basebackup there is no way for user to specify database name. With Regards, Amit Kapila. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] GIST_LEAF() causes error in GiST methods union and penalty
Hello all, I tried to use GIST_LEAF() macro in union and penalty methods of GiST and it fails with a terrible error when I run "CREATE INDEX" (in windows postgresql daemon shuts down). On the other hand, GIST_LEAF() works as expected in consistent, distance, and picksplit methods. My leaf nodes contain the indexed column data type whilst the non-leaf nodes have a different (internal) data type that is only known in C level (all types are fixed-length). I don't specify the "STORAGE" parameter in "CREATE OPERATOR CLASS ... USING gist" since the leaves are exactly of the same type as the column. Moreover, compress and decompress don't do anything just return the pointer they received (only compress checks for NULLs since there should be no NULLs in the column in my case).I also noticed in the documentation that in compress we determine if the page is leaf or not by using "if (entry->leafkey)" instead of GIST_LEAF() macro. What is the difference between "if (entry->leafkey)" and GIST_LEAF() and when should each one be used? I believe there are some conventions that GiST takes into account w.r.t. when it calls union and penalty and with what parameters, but I am not sure if they are right. So, I think that "union" is called only in leaf pages so the GistEntryVector parameter contains always leaf enties of my leaf data type (I am calling this an assumption because GIST_LEAF() breaks in union and "if (entry->leafkey)" always returns false which doesn't sound logical and makes me think that entry->leafkey should not be used). Also, the "GISTENTRY *origentry" of "penalty" is always of the non-leaf data type because it contains non-leaf entries whilst "GISTENTRY *newentry" is always of the leaf data type. Are the previous thoughts-assumptions correct? This is very important to me because I have to handle different leaf/non-leaf data types. And one last question, the two parameters of the "same" method are always of the non-leaf data type? Kind regards,Marios Vodas
Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq
On 26.02.2013 09:06, Amit Kapila wrote: On Monday, February 25, 2013 11:26 PM Heikki Linnakangas wrote: On 21.02.2013 16:09, Amit Kapila wrote: On Thursday, February 21, 2013 12:46 AM Heikki Linnakangas wrote: I've committed those patches, with some further changes. If you have the time, please take another look at the committed patches. Thanks! 1. For pg_dumpall, incase user gives dbname in connection string, for ex. "user=amit dbname=db_regress port=5434" it will be ignored and postgres (default database name) will be used. Yeah, that is mentioned in the docs. The other option would be to use the database name from the connection string for the initial connection, ie. the same as specifying a database name with the -l option. Yet another option is to throw an error if database name was given. If we make it an error, then we should make it an error in pg_basebackup and pg_receivexlog too. I'm all ears for opinions on this. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
