[HACKERS] creating/accessing new runtime parameters
I am writing a backend C function whose behavior should depend on the setting of a new runtime parameter that can be set with SET. I have several questions concerning how to implement this, as I can find no information in the documentation. - How can I introduce a new variable (e.g., XXX) to the system so that for example SET XXX=1 will work? - How can I access the value of such a variable from within a backend C function? Thanks for your help. Cheers, Brook ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] creating/accessing new runtime parameters
Tom Lane writes: > The basic thing is to add an appropriate table entry to guc.c. I take it there is not way to do this dynamically, for example to support a dynamically loaded function? All runtime variables are hard-coded into the backend? Thanks for your help. Cheers, Brook ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] creating/accessing new runtime parameters
Andreas Pflug writes: > Maybe you can implement your stuff using a temporary table? Perhaps, but the runtime variable route is much more natural from a user interface perspective, as there are strong parallels with existing variables. I'll see what I can do in that arena first. Thanks for the idea, though. Cheers, Brook ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] creating/accessing new runtime parameters
Joe Conway writes: > I had a patch about 80% complete to do this, but it was rejected. The > comment was that I should use a temp table instead. I still think it > would be useful myself. See this thread: > > http://archives.postgresql.org/pgsql-hackers/2002-12/msg00988.php I'm sorry that was rejected. I can see that there might be error checking issues, but perhaps that simply means some hooks to register validation functions dynamically that could be called during SET. As far as the general utility of it goes, I claim that it could be quite valuable. I am thinking of complex new datatypes (that might be dynamically loaded) that could benefit from having some run-time variables specify some aspect of their behavior. Currently, we have variables controlling the I/O of dates and times, for example. Something analogous would potentially be useful for other datatypes. Without the ability to dynamically add to the set of run-time variables, it is impossible to write a datatype that has this property. In these cases, one really does want run-time variables, not temporary tables, as the use exactly corresponds to the use for existing variables: modifying the behavior of interal functions. Cheers, Brook ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
[HACKERS] psql as an execve(2) interpreter
I would like to use pgsl as an interpreter (in the sense of
execve(2)). In short, if a file begins with the line
#! /path/to/psql -f
it should be interpretable by psql. The normal semantics of execve(2)
ensure that this will work perfectly (indeed a file containing
"#!/path/to/psql -l" works as expected), except for psql's nasty habit
of not interpreting the first line as a comment.
It seems that a simple fix to the following function in
src/bin/psql/input.c would do the trick.
char *
gets_fromFile(FILE *source)
{
PQExpBufferData buffer;
char line[1024];
initPQExpBuffer(&buffer);
while (fgets(line, sizeof(line), source) != NULL)
{
appendPQExpBufferStr(&buffer, line);
if (buffer.data[buffer.len - 1] == '\n')
{
buffer.data[buffer.len - 1] = '\0';
return buffer.data;
}
}
if (buffer.len > 0)
return buffer.data;/* EOF after reading
some bufferload(s) */
/* EOF, so return null */
termPQExpBuffer(&buffer);
return NULL;
}
For example, this feature could be achieved by 1) including a static
variable to differentiate the first from subsequent calls and 2)
discarding the first line (and returning the second) on the first call
if the first line begins with #!.
Thus, I have two questions.
- Is this a feature that would be generally accepted and useful for
the postgresql community (i.e., would it be incorporated into the
code base)?
- Is this the correct solution or are there other portions of the code
that need to be considered?
I appreciate any feedback you can give me on this.
Thank you very much.
Cheers,
Brook
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Re: [HACKERS] psql as an execve(2) interpreter
Tom Lane writes: > Given that # is not a comment introducer in SQL, I would consider > it a bug if it did. I understand that # is not a comment introducer in SQL. I am wondering if it would be sensible to introduce an exception for the first line of a file. To prevent problems the behavior should be controlled by a command line option (-i?) so that it would never have this behavior unless explicitly asked for. I guess you see no value in this and instead would solve the issue with a separate interpreter that has this property? Note that a shell script cannot be an interpreter for execve(2); thus, this would require another binary executable. My own feeling was that psql could be easily taught to have this behavior in a way that would not interfer with any existing applications. I at least can see benefits to having that capability, but perhaps others do not. For example, some of my large database applications are built by running a large collection of scripts (some are shell scripts, some sql, etc.), each of which is responsible for a portion of the task. It would be very handy to execute each member of the collection in a uniform manner, i.e., as a direct execution with execve(2) figuring out which interpreter to use on a script-by-script basis. Currently, that is not possible, but it could be with a small modification to psql or the addition of a completely new interpreter. Thanks for the comments. Cheers, Brook ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] psql as an execve(2) interpreter
Jonah, Thanks for your comments. Jonah H. Harris writes: > I have a lot of shell scripts that run as cron jobs and have considered > this option. However, if you look at it carefully, SQL is totally > different from say perl, php, bash, etc. for scripts which execute from > the shell. Tom is right, it is much more valuable and supportable to > call psql from within a shell script than add the functionality to psql > itself. I, too, have thought a lot about this and I suppose we are reaching different conclusions. I would very much appreciate hearing your logic, as the underlying reasoning you imply is not transparent to me. For what it is worth, here is an outline of part of my thinking. It is true that the nature of SQL as a language is different from other traditional programming languages, as it does not have concepts such as variables and flow control. To my way of thinking, however, that simply dictates what is possible to express in the language. Importantly, I do not see how it influences how one might wish to execute the commands given in the language (or any language for that matter). In my mind the decision about how to execute something is based on what it does and what the larger context is, not what the language can express. Suppose I have a script S in some language L that is interpretable by some interpreter I. The language L might be SQL and the inrepreter I might be psql, but nothing that follows depends on that. Indeed, the language could be perl and the interpreter perl. The fundamental questions are: - What are useful ways to have the interpreter I carry out the instructions contained within S? - What determines why those are useful? - How can those means be achieved? For most scripting languages L (e.g., shell commands, perl, etc.) the prior art has identified two useful means of having the interpreter execute the instructions in S: 1) explicit execution (i.e., execute the interpreter and explicitly pass the appropriate script S to it) and 2) implicit execution (i.e., execute the script and magically have the system invoke the interpreter on it). Interpreting SQL scripts stands out as one exception to this. Why would one choose one method over another? In all cases that I can think of, the decision to use one method over another depends entirely on considerations that are external to the nature of the language L itself. I would venture to say that they are governed primarily by the nature of the external interface one is trying to create. In some cases, depending on what the script actually does, it is much more natural to invoke a script directly. An example would be one that is a wrapper to something else, but must take the responsibility for setting up the environment first. In other cases, the other mechanism is more natural. The decision does not bear on what the _language_ is capable of expressing, but rather on what the particular script is doing and how it fits into a larger external context. In my mind, the same is true for SQL. In some cases it is appropriate to execute the interpreter (i.e., psql) explicitly (that is currently our only option). In other cases it is appropriate to execute it implicitly. I see no fundamental difference between this and any other interpreter. Clearly, an implicit execution mechanism for SQL cannot work quite as transparently as for languages that use the hash mark (#) as a comment introducer. However, supporting this option need not interfere with other uses of the interpreter nor need it be costly. What is required is 1) a command line option that differentiates traditional behavior from the "implicit interpreter" behavior, and 2) a modification of how the first line of the file is handled depending on the mode. No other changes are required; no interaction with the vast majority of the code is needed. Thus, my analysis suggests no fundamental difference between the set of invocations that are useful for the majority of interpreters and the set that would be useful for interpreters of SQL. I also can envision a means of expanding that set for psql that would have no impact on either its normal use or its ongoing maintenance and development. Consequently, I see no compelling reason not to move in this direction. However, I must be missing something obvious, as there seems to be conflicting sentiment. I would be very interested to learn more about what is behind those ideas. Cheers, Brook ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] handling contrib directories as modules not shared libraries
It seems that src/Makefile.shlib has special cases for several directories that build loadable modules rather than shared libraries. The contrib/adminpack is one of the special cases, but none of the other contrib directories are. As a result, they get built as shared libraries (i.e., as libXXX.so rather than XXX.so) and the corresponding *.sql.in files that load them refer to the wrong file and therefore fail. The following patch (against the 8.3 release) fixes this by expanding the set of special cases to include all the contrib directories, not just contrib/adminpack. I only have tested this with the uuid module, but it appears that all of them have the same organization. --- src/Makefile.shlib.orig +++ src/Makefile.shlib @@ -21,7 +21,7 @@ ifneq (,$(findstring src/pl/,$(subdir))) shmodule = yes else -ifneq (,$(findstring contrib/adminpack,$(subdir))) +ifneq (,$(findstring contrib/,$(subdir))) shmodule = yes else shmodule = no Cheers, Brook -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] While we're on the subject of searches...
Over the past few months there've been a number of requests for an interactive type documentation setup like the folks at php.net have. Great to add to the documentation, but I hope the PostgreSQL project doesn't take it so far as to make the primary documentation interactive. A well-thought out, coherent document is _much_ more useful than the skads of random tips that characterize some other projects. The current document is very well-written (though perhaps incomplete). I would hate to see that decline in quality. Cheers, Brook ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [PATCHES] Re: [HACKERS] PostGIS spatial extensions
Projects that are as organized, professional, and value-adding as yours is can surely stand on their own. I compare this to the recently released OpenFTS. If we start including projects of this size we'd explode in size and maintenance overhead. Doesn't this discussion indicate that the time is fast approaching, if not already past, for some type of system for handling installation of 3rd party software? It would seem that two prerequisites would need to be satisfied to do this: - Definition and implementation of the interface to be provided for extensions. Presumably, this would involve defining a well-designed set of public header files and associated libraries at the right level of granularity. For example, encapsulating each type in its own header file with a standardized set of operations defined in a server-side library would be extremely valuable. The library (or libraries) could be used to link the backend and installed for extensions to take advantage of preexisting types when others need to construct new more complex ones. - Definition and implementation of a consistent extension management system for retrieving, compiling, and installing extensions. This could even be used for installing the system itself, thereby making the entire operation of managing the software consistent. I point out that the NetBSD pkgsrc system[1] does the latter in an extremely flexible and well-designed manner, and has been a major foundation for the openpackages project. It even includes 7 distinct packages[2] for different elements of PostgreSQL, not including a number of other packages broken out for different interfaces. The same system could be adopted for managing 3rd party extensions. Having been involved in defining what header files to install, and having been actively involved in developing new types for use in our installation, I can say that external packaging of PostgreSQL, local extension of PostgreSQL, and management of 3rd party software would be greatly enhanced by an effort to address the two prerequisites mentioned above. Cheers, Brook --- [1] http://www.netbsd.org/Documentation/software/packages.html [2] ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/databases/README.html ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Using textin/textout vs. scribbling around
I am tempted to replace all attempts to build text data on your own (with VARDATA, VARHDRSZ, etc.) with proper calls to textin/textout in all places that can't claim to be truly internal (especially all contrib). The background is that the internal format might change sometime when more locale features are implemented (see recent idea). Is this a good idea? I would be in favor of this for the following reasons. One of the great advantages of PostgreSQL is that adding new types is relatively straightforward. In many cases new types could be coded largely in terms of preexisting types. An example I am facing currently involves the problem of constructing basically a TEXT type with rather unusual parsing and output semantics. If a reasonably well-encapsulated version of the TEXT type existed--including header files with the right granularity and with the right functions provided in an installed library--the natural means of providing the new type we need would be to simply define a type with different *in/*out functions implemented over an instance of the TEXT internal data representation. Peter's suggestion appears to be a natural step towards the goal of being able to provide a defined interface that could be used for extensions. The concern that the _external_ format might change seems counter to the effort of providing a stable platform for extending PostgreSQL. If there is a serious possibility that this might occur, and because of that we cannot provide any external interface to the predefined types, then the well-known advantages of composing software modules from well-defined and well-tested components will be largely lost for anyone wishing to rapidly extend the system. Cheers, Brook ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
[HACKERS] undocumented parts of SPI
I'm trying to figure out the SPI and need a bit of help, because the docs do not mention (that I can find) some parts of the interface. - SPI_exec(char * sql, int count): this function seems to execute the query. Is the `count' parameter the maximum number of tuples to return? Does count=0 imply return everything? Return value are the SPI_* flags defined in spi.h? - SPI_processed: number of tuples processed during execution of SPI_exec? Does this equate to N in the INSERT/DELETE/UPDATE N messages that are emitted by psql? - Are there any restrictions on what types of queries may be executed by a trigger using SPI_exec? Thanks for the help. Cheers, Brook
[HACKERS] type design guidance needed
I am working on designing some new datatypes and could use some guidance. Along with each data item, I must keep additional information about the scale of measurement. Further, the relevant scales of measurement fall into a few major families of related scales, so at least a different type will be required for each of these major families. Additionally, I wish to be able to convert data measured according to one scale into other scales (both within the same family and between different families), and these interconversions require relatively large sets of parameters. It seems that there are several alternative approaches, and I am seeking some guidance from the wizards here who have some understanding of the backend internals, performance tradeoffs, and such issues. Possible solutions: 1. Store the data and all the scale parameters within the type. Advantages: All information contained within each type. Can be implemented with no backend changes. No access to ancillary tables required, so processing might be fast. Disadvantages: Duplicate information on the scales recorded in each field of the types; i.e., waste of space. I/O is either cumbersome (if all parameters are required) or they type-handling code has built-in tables for supplying missing parameters, in which case the available types and families cannot be extended by users without recompiling the code. 2. Store only the data and a reference to a compiled-in data table holding the scale parameters. Advantages: No duplicate information stored in the fields. Access to scale data compiled into backend, so processing might be fast. Disadvantages: Tables of scale data fixed at compile time, so users cannot add additional scales or families of scales. Requires backend changes to implement, but these changes are relatively minor since all the scale parameters are compiled into the code handling the type. 3. Store only the data and a reference to a new system table (or tables) holding the scale parameters. Advantages: No duplicate information stored in the fields. Access to scale data _not_ compiled into backend, so users could add scales or families of scales by modifying the system tables. Disadvantages: Requires access to system tables to perform conversions, so processing might be slow. Requires more complex backend changes to implement, including the ability to retrieve information from system tables. Clearly, option 3 is optimal (more flexible, no data duplication) unless the access to system tables by the backend presents too much overhead. (Other suggestions are welcome, especially if I have misjudged the relative merits of these ideas or missed one altogether.) The advice I need is the following: - How much of an overhead is introduced by requiring the backend to query system tables during tuple processing? Is this unacceptable from the outset or is it reasonable to consider this option further? Note that the size of these new tables will not be large (probably less than 100 tuples) if that matters. - How does one access system tables from the backend code? I seem to recall that issuing straight queries via SPI is not necessarily the right way to go about this, but I'm not sure where to look for alternatives. Thanks for your help. Cheers, Brook
Re: [HACKERS] type design guidance needed
It'd be useful to know more about your measurement scales. Evgeni remarks that for his applications, units can be broken down into simple linear combinations of fundamental units --- but if you're doing something like converting between different device-dependent color spaces, I can well believe that that model wouldn't work... Those ideas about linear combinations are great, but I think too simplistic for what I have in mind. I'll give it more thought, though, as I further define the structure of all the interconversions. > - How much of an overhead is introduced by requiring the backend to > query system tables during tuple processing? Is this unacceptable > from the outset or is it reasonable to consider this option further? Assuming that the scale tables are not too large and not frequently changed, the ideal access mechanism seems to be the "system cache" mechanism (cf src/backend/utils/cache/syscache.c, src/backend/utils/cache/lsyscache.c). The cache support allows each backend to keep copies in memory of recently-used rows of a cached table. Updating a cached table requires rather expensive cross- backend signaling, but as long as that doesn't happen often compared to accesses, you win. The only real restriction is that you have to look up cached rows by a unique key that corresponds to an index, but that seems not to be a problem for your application. I have in mind cases in which the system tables will almost never be updated. That is, the table installed initially will serve the vast majority of purposes, but I'd still like the flexibility of updating it when needed. Caches may very well be perfectly appropriate, here; thanks for the pointer. Adding a new system cache is a tad more invasive than the usual sort of user-defined-type addition, but it's certainly not out of the question. Bruce Momjian has done it several times and documented the process, IIRC. Bruce, is that the case? Do you really have it documented? If so, where? Cheers, Brook
Re: make depend (Re: [HACKERS] Coming attractions: VPATH build; make variables issue)
Peter Eisentraut <[EMAIL PROTECTED]> writes:
> What we could do is ship the dependencies (.deps/*.P) in the tarball.
> That would require running an entire build before making a tarball, but it
> would be a nice service to users.
Hm. It might be handy for people not using gcc, since they'd have no
easy way to build dependencies for themselves. Do you have an idea
how much it'd bloat the tarball to do that?
Isn't the basic idea to write Makefile targets to remake dependency
files when they are out of date with code? Won't those targets
involve implicit rules for going, for example, from *.c -> *.d (or
whatever convention you use for dependency files)? Don't these
Makefiles also have a list of srcs to be built, e.g., a make variable
that defines a list of *.c filename?
If so, can't you just implement a depend: target as
${DEPEND_FILES}+=${SRCS:%c=%d}
depend: ${DEPEND_FILES}
.SUFFIXES: .c .d
.c.d:
gcc -M ...
.include "${DEPEND_FILES}"
For gmake users, all the magic happens automatically. Prior to
distribution, just do make depend to get all the *.d files to include
in the tarball. For non-gmake users, all the *.d files already exist
in the source. If they make changes, they can run make depend
manually.
Sorry if this is what you had in mind already, but the discussion
seemed to imply that you can't have it both ways.
Cheers,
Brook
