Re: [HACKERS] Status report on writeable CTEs
2010/7/13 Marko Tiikkaja : > On 7/12/10 9:34 PM +0300, Tom Lane wrote: >> >> Marko Tiikkaja writes: >>> >>> ... So what I'm now thinking of is making the planner plan that as a >>> single >>> Query, and make the planner expand it into multiple PlannedStmts if >>> necessary. This would break the existing planner hooks, but I don't >>> think that's a huge problem. Does anyone see any obvious flaws in this? >> >> How will that interact with the existing rewriter? It sounds a lot >> like a recipe for duplicating functionality ... > > I was thinking that the rewriter would look at the top-level CTEs and > rewrite all non-SELECT queries into a list of Queries and store that list > into the CommonTableExprs. The planner would then use this information and > also expand the rewrite products. Why didn't you choose this strategy? ISTM changing signature of planner() and standard_planner() makes it more difficult to commit this feature unnecessarily. I thought at first that it is possible and better to split the statement into multiple Querys in the rewriter and pass them to the planner, but as you mentioned they should collaborate each other like in ctelevelsup, so to pass a single Query to the planner is probably the answer. However, you can store the sub (ie, in WITH clause) PlannedStmt in the top level PlannedStmt and extract the sub statements in pg_plan_queries() or anywhere in the PortalXXX() then add them to the execution list. Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On 8/3/2010 7:30 PM, Hitoshi Harada wrote: As hackers say, the first to try should be Marko's first design that use the list of tuplestore and DTScanNode. So if he has solid image to reach there, we can wait for him to complete his first compilable version. Then let's take it back and forth. Is it possible? I am currently working on a version that treats all WITH queries like wCTEs. My progress can be seen in my git repo [1], branch "wcte". In its current form, the patch compiles and passes all applicable regression tests but it's still very messy. I'm going to send a cleaner WIP patch to the list the minute I have one, but anyone's free to look at the git repo (and even work on it if they want!). And I concern we might not have concrete consensus about list of features in document form. Does it accept Recursive query? What if x refers to y that refers to x cyclicly? An external design sometimes fix the internal design, and it helps people to review the implementation. If I missed something please point me to the link. A recursive query should be fine as long as 1) it's SELECT-only and 2) it doesn't loop forever. A wCTE can of course refer to the result of the recursive SELECT query with INSERT .. SELECT, UPDATE .. FROM or DELETE .. USING. Cyclic dependencies are out of the scope of this patch; I'm not planning on adding any new features to regular CTEs. [1] http://git.postgresql.org/gitweb?p=users/johto/postgres.git;a=summary Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
2010/7/21 David Fetter : > On Sat, Jul 17, 2010 at 01:15:22AM +0900, Hitoshi Harada wrote: >> 2010/7/17 Marko Tiikkaja : >> > On 7/16/10 6:15 PM +0300, Hitoshi Harada wrote: >> >> 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode >> >> is exsiting one that work with single tuplestore, it might be sane to >> >> modify this so that it accepts tuplestore from Query instead of its >> >> child node. >> > >> > I thought about this, but I don't necessarily like the idea of overloading >> > executor nodes. >> >> Neither do I have good shape for this solution. Maybe it's not good >> idea. But my concern is adding DtScanNode, which looks similar to >> MaterialNode. Of course each purpose is different, but quite big part >> will overlap each other, I think. > > Any ideas as to how to factor this out? Handiest ideas would be in > the form of a patch ;) Yeah, that would be handiest, but I think I must wait for his first compilable patch to modify to try the idea. Current version looks quite messy and can't build. >> >> 2. Use temp table instead of tuplestore list. Since we agreed we need >> >> to execute each plan one by one starting and shutting down executor, >> >> it now looks very simple strategy. >> > >> > I didn't look at this because I thought using a "tuplestore receiver" in >> > the >> > portal logic was simple enough. Any thoughts on how this would work? >> >> It's just deconstructing queries like: >> >> WITH t AS (INSERT INTO x ... RETURING *) >> SELECT * FROM t; >> >> to >> >> CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; >> SELECT * FROM t; >> >> While the second statement is not implemented yet, it will be so simpler. > > So CTAS would get expanded into CTA[row-returning query] ? > Interesting. How much work would that part be? FWIW, this is getting interesting to me these days, and I think this can be separated from wCTE As hackers say, the first to try should be Marko's first design that use the list of tuplestore and DTScanNode. So if he has solid image to reach there, we can wait for him to complete his first compilable version. Then let's take it back and forth. Is it possible? And I concern we might not have concrete consensus about list of features in document form. Does it accept Recursive query? What if x refers to y that refers to x cyclicly? An external design sometimes fix the internal design, and it helps people to review the implementation. If I missed something please point me to the link. Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On Tue, Jul 20, 2010 at 5:13 PM, Merlin Moncure wrote: 2. Use temp table instead of tuplestore list. Since we agreed we need to execute each plan one by one starting and shutting down executor, it now looks very simple strategy. >>> >>> I didn't look at this because I thought using a "tuplestore receiver" in the >>> portal logic was simple enough. Any thoughts on how this would work? >> >> It's just deconstructing queries like: >> >> WITH t AS (INSERT INTO x ... RETURING *) >> SELECT * FROM t; >> >> to >> >> CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; >> SELECT * FROM t; > > Is it acceptable for a wCTE query to manipulate the system catalogs? > Couldn't this cause performance issues in some cases? Yeah, I suspect the performance wouldn't be too hot. I think the idea of writing into a tuplestore and then reading back out from it is the way to go. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On Fri, Jul 16, 2010 at 12:15 PM, Hitoshi Harada wrote: > 2010/7/17 Marko Tiikkaja : >> On 7/16/10 6:15 PM +0300, Hitoshi Harada wrote: >>> >>> 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode >>> is exsiting one that work with single tuplestore, it might be sane to >>> modify this so that it accepts tuplestore from Query instead of its >>> child node. >> >> I thought about this, but I don't necessarily like the idea of overloading >> executor nodes. > > Neither do I have good shape for this solution. Maybe it's not good > idea. But my concern is adding DtScanNode, which looks similar to > MaterialNode. Of course each purpose is different, but quite big part > will overlap each other, I think. > >>> 2. Use temp table instead of tuplestore list. Since we agreed we need >>> to execute each plan one by one starting and shutting down executor, >>> it now looks very simple strategy. >> >> I didn't look at this because I thought using a "tuplestore receiver" in the >> portal logic was simple enough. Any thoughts on how this would work? > > It's just deconstructing queries like: > > WITH t AS (INSERT INTO x ... RETURING *) > SELECT * FROM t; > > to > > CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; > SELECT * FROM t; Is it acceptable for a wCTE query to manipulate the system catalogs? Couldn't this cause performance issues in some cases? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On Sat, Jul 17, 2010 at 01:15:22AM +0900, Hitoshi Harada wrote: > 2010/7/17 Marko Tiikkaja : > > On 7/16/10 6:15 PM +0300, Hitoshi Harada wrote: > >> > >> 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode > >> is exsiting one that work with single tuplestore, it might be sane to > >> modify this so that it accepts tuplestore from Query instead of its > >> child node. > > > > I thought about this, but I don't necessarily like the idea of overloading > > executor nodes. > > Neither do I have good shape for this solution. Maybe it's not good > idea. But my concern is adding DtScanNode, which looks similar to > MaterialNode. Of course each purpose is different, but quite big part > will overlap each other, I think. Any ideas as to how to factor this out? Handiest ideas would be in the form of a patch ;) > >> 2. Use temp table instead of tuplestore list. Since we agreed we need > >> to execute each plan one by one starting and shutting down executor, > >> it now looks very simple strategy. > > > > I didn't look at this because I thought using a "tuplestore receiver" in the > > portal logic was simple enough. Any thoughts on how this would work? > > It's just deconstructing queries like: > > WITH t AS (INSERT INTO x ... RETURING *) > SELECT * FROM t; > > to > > CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; > SELECT * FROM t; > > While the second statement is not implemented yet, it will be so simpler. So CTAS would get expanded into CTA[row-returning query] ? Interesting. How much work would that part be? Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On 7/16/10 7:15 PM +0300, Hitoshi Harada wrote: 2010/7/17 Marko Tiikkaja: I thought about this, but I don't necessarily like the idea of overloading executor nodes. Neither do I have good shape for this solution. Maybe it's not good idea. But my concern is adding DtScanNode, which looks similar to MaterialNode. Of course each purpose is different, but quite big part will overlap each other, I think. The way I see it is that reading from a tuplestore is so simple that we shouldn't be trying to merge together nodes just on that basis. It seems to me that we'd have to add CteScan and WorkTableScan nodes there too and at that point it would become complicated. I didn't look at this because I thought using a "tuplestore receiver" in the portal logic was simple enough. Any thoughts on how this would work? It's just deconstructing queries like: WITH t AS (INSERT INTO x ... RETURING *) SELECT * FROM t; to CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; SELECT * FROM t; That's an idea. Can we somehow avoid name clashes with user-defined temporary tables? Another concern is tuplestore's memory exhausting. Tuplestore holds tuples in memory as far as the estimated memory usage is within work_mem (for *each* not total of all tuplestores!), but if you create dozens of tuplestore (and it's quite possible in wCTE use cases) we will possibly fail into memory overflow problems. That doesn't seem very different from a big SELECT query, except with wCTEs, you actually *know* how many times the work_mem can be used before you run the query and can adjust work_mem accordingly. That said, I personally could live with a separate GUC for just adjusting the work_mem of "wcte tuplestores". Another option would be to unconditionally force the tuplestores to disk, but that sounds painful. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
2010/7/17 Marko Tiikkaja : > On 7/16/10 6:15 PM +0300, Hitoshi Harada wrote: >> >> 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode >> is exsiting one that work with single tuplestore, it might be sane to >> modify this so that it accepts tuplestore from Query instead of its >> child node. > > I thought about this, but I don't necessarily like the idea of overloading > executor nodes. Neither do I have good shape for this solution. Maybe it's not good idea. But my concern is adding DtScanNode, which looks similar to MaterialNode. Of course each purpose is different, but quite big part will overlap each other, I think. >> 2. Use temp table instead of tuplestore list. Since we agreed we need >> to execute each plan one by one starting and shutting down executor, >> it now looks very simple strategy. > > I didn't look at this because I thought using a "tuplestore receiver" in the > portal logic was simple enough. Any thoughts on how this would work? It's just deconstructing queries like: WITH t AS (INSERT INTO x ... RETURING *) SELECT * FROM t; to CREATE TEMP TABLE t AS INSERT INTO x ... RETURING *; SELECT * FROM t; While the second statement is not implemented yet, it will be so simpler. Another concern is tuplestore's memory exhausting. Tuplestore holds tuples in memory as far as the estimated memory usage is within work_mem (for *each* not total of all tuplestores!), but if you create dozens of tuplestore (and it's quite possible in wCTE use cases) we will possibly fail into memory overflow problems. >> I'm not familiar with the long discussion on this feature so not sure >> they are possible, but ISTM they are enough to be discussed (or >> discussed already?). > > We haven't discussed this part of the design yet.. Now is a good time to do > it. Yeah, we should. Anyone has another idea? Or adding DtScanNode for this features is fair enough? Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On 7/16/10 6:15 PM +0300, Hitoshi Harada wrote: Sorry it's not relevant to the topic you post but .. Relevant enough :-) .. it seems you're going to add new executor node called DtScanNode and let it hold tuplestore passed by the Query indicated by index number. However, I suppose there are other ways: 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode is exsiting one that work with single tuplestore, it might be sane to modify this so that it accepts tuplestore from Query instead of its child node. I thought about this, but I don't necessarily like the idea of overloading executor nodes. 2. Use temp table instead of tuplestore list. Since we agreed we need to execute each plan one by one starting and shutting down executor, it now looks very simple strategy. I didn't look at this because I thought using a "tuplestore receiver" in the portal logic was simple enough. Any thoughts on how this would work? I'm not familiar with the long discussion on this feature so not sure they are possible, but ISTM they are enough to be discussed (or discussed already?). We haven't discussed this part of the design yet.. Now is a good time to do it. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
2010/7/13 Marko Tiikkaja : > Hi, > > > I've been working on writeable CTEs during the last couple of months, but > right now it looks like I'm going to miss the first commit fest for 9.1. I > was trying to make it work by expanding all wCTEs to their own Queries > during the rewrite stage (a very crude patch trying to do that for regular > CTEs attached), but I don't think that it's a good way of approaching the > problem. Consider: Sorry it's not relevant to the topic you post but it seems you're going to add new executor node called DtScanNode and let it hold tuplestore passed by the Query indicated by index number. However, I suppose there are other ways: 1. Use MaterialNode instead of adding DtScanNode. Since MaterialNode is exsiting one that work with single tuplestore, it might be sane to modify this so that it accepts tuplestore from Query instead of its child node. 2. Use temp table instead of tuplestore list. Since we agreed we need to execute each plan one by one starting and shutting down executor, it now looks very simple strategy. I'm not familiar with the long discussion on this feature so not sure they are possible, but ISTM they are enough to be discussed (or discussed already?). Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On 7/12/10 9:34 PM +0300, Tom Lane wrote: Marko Tiikkaja writes: ... So what I'm now thinking of is making the planner plan that as a single Query, and make the planner expand it into multiple PlannedStmts if necessary. This would break the existing planner hooks, but I don't think that's a huge problem. Does anyone see any obvious flaws in this? How will that interact with the existing rewriter? It sounds a lot like a recipe for duplicating functionality ... I was thinking that the rewriter would look at the top-level CTEs and rewrite all non-SELECT queries into a list of Queries and store that list into the CommonTableExprs. The planner would then use this information and also expand the rewrite products. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
Marko Tiikkaja writes: > ... So what I'm now thinking of is making the planner plan that as a single > Query, and make the planner expand it into multiple PlannedStmts if > necessary. This would break the existing planner hooks, but I don't > think that's a huge problem. Does anyone see any obvious flaws in this? How will that interact with the existing rewriter? It sounds a lot like a recipe for duplicating functionality ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on writeable CTEs
On 7/12/10 9:07 PM +0300, I wrote: Consider: WITH t AS (SELECT 1), t2 AS (SELECT * FROM t2) VALUES (true); That should of course have been SELECT * FROM t, not t2. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Status report on writeable CTEs
Hi, I've been working on writeable CTEs during the last couple of months, but right now it looks like I'm going to miss the first commit fest for 9.1. I was trying to make it work by expanding all wCTEs to their own Queries during the rewrite stage (a very crude patch trying to do that for regular CTEs attached), but I don't think that it's a good way of approaching the problem. Consider: WITH t AS (SELECT 1), t2 AS (SELECT * FROM t2) VALUES (true); The first big problem I hit is was that when the query for t2 is planned separately, it doesn't have the CTE information. But even if it had that information (we could easily copy it during the rewrite stage), all RTEs referencing CTEs that were expanded would have the wrong levelsup (this is where the patch fails at regression tests). One could probably come up with ways of fixing even that, but I don't think that's the right direction to be heading. So what I'm now thinking of is making the planner plan that as a single Query, and make the planner expand it into multiple PlannedStmts if necessary. This would break the existing planner hooks, but I don't think that's a huge problem. Does anyone see any obvious flaws in this? Any feedback is welcome. I'd also be happy to get some help on this project. Regards, Marko Tiikkaja *** a/src/backend/commands/copy.c --- b/src/backend/commands/copy.c *** *** 1092,1098 DoCopy(const CopyStmt *stmt, const char *queryString) cstate->queryDesc = CreateQueryDesc(plan, queryString, GetActiveSnapshot(), InvalidSnapshot, ! dest, NULL, 0); /* * Call ExecutorStart to prepare the plan for execution. --- 1092,1098 cstate->queryDesc = CreateQueryDesc(plan, queryString, GetActiveSnapshot(), InvalidSnapshot, ! dest, NULL, 0, NIL); /* * Call ExecutorStart to prepare the plan for execution. *** a/src/backend/commands/explain.c --- b/src/backend/commands/explain.c *** *** 367,373 ExplainOnePlan(PlannedStmt *plannedstmt, ExplainState *es, /* Create a QueryDesc requesting no output */ queryDesc = CreateQueryDesc(plannedstmt, queryString, GetActiveSnapshot(), InvalidSnapshot, ! None_Receiver, params, instrument_option); INSTR_TIME_SET_CURRENT(starttime); --- 367,374 /* Create a QueryDesc requesting no output */ queryDesc = CreateQueryDesc(plannedstmt, queryString, GetActiveSnapshot(), InvalidSnapshot, ! None_Receiver, params, instrument_option, ! NIL); INSTR_TIME_SET_CURRENT(starttime); *** *** 692,697 ExplainNode(Plan *plan, PlanState *planstate, --- 693,701 case T_CteScan: pname = sname = "CTE Scan"; break; + case T_DtScan: + pname = sname = "Derived Table Scan"; + break; case T_WorkTableScan: pname = sname = "WorkTable Scan"; break; *** *** 844,849 ExplainNode(Plan *plan, PlanState *planstate, --- 848,854 case T_ValuesScan: case T_CteScan: case T_WorkTableScan: + case T_DtScan: ExplainScanTarget((Scan *) plan, es); break; case T_BitmapIndexScan: *** *** 1565,1570 ExplainScanTarget(Scan *plan, ExplainState *es) --- 1570,1581 objectname = rte->ctename; objecttag = "CTE Name"; break; + case T_DtScan: + /* Assert it's on a non-self-reference CTE */ + Assert(rte->rtekind == RTE_CTE); + Assert(!rte->self_reference); + objectname = rte->ctename; + objecttag = "Derived Table Name"; default: break; } *** a/src/backend/executor/Makefile --- b/src/backend/executor
Re: [HACKERS] Status report: getting plpgsql to use the core lexer
"Kevin Grittner" writes: > ... > We were able to get to much cleaner code by rewriting the parser to > have a "dumb" phase to get the overall structure into an AST, and then > use a tree-walker phase to do all the lookups and type resolution > after we had the rough structure, writing another AST to walk for code > generation. Besides making the code cleaner and easier to maintain, > it helped us give better error messages pointing more accurately to > the source of the problem. I don't know if a similar approach is > feasible in flex/bison, but if it is, refactoring for an extra pass > might be worth the trouble. That's actually what we have in the core parser. plpgsql is trying to take shortcuts, and this whole project is exactly about weaning it away from that. The bottom line is I tried to tackle the sub-projects in the wrong order... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report: getting plpgsql to use the core lexer
Tom Lane wrote: > One problem that wasn't obvious when I started is that if you are > trying to use a reentrant lexer, Bison insists on including its > YYSTYPE union in the call signature of the lexer. Of course, > YYSTYPE means different things to the core grammar and plpgsql's > grammar. I tried to work around that by having an interface layer > that would (among other duties) translate as needed. It turned out > to be a real PITA, not least because you can't include both > definitions into the same C file. The scheme I have has more or > less failed --- I think I'd need *two* interface layers to make it > work without unmaintainable kluges. It would probably be better to > try to adjust the core lexer's API some more so that it does not > depend on the core YYSTYPE, but I'm not sure yet how to get Bison to > play along without injecting an interface layer (and hence wasted > cycles) into the core grammar/lexer interface. > > Another pretty serious issue is that the current plpgsql lexer > treats various sorts of qualified names as single tokens. I had > thought this could be worked around in the interface layer by doing > more lookahead. You can do that, and it mostly works, but it's > mighty tedious. The big problem is that "yytext" gets out of step > --- it will point at the last token the core lexer has processed, > and there's no good way to back it up after lookahead. I spent a > fair amount of time trying to work around that by eliminating uses > of "yytext" in plpgsql, and mostly succeeded, but there are still > some left. (Some of the remaining regression failures are error > messages that point at the wrong token because they rely on yytext.) > > Now, having name lookup happen at the lexical level is pretty bogus > anyhow. The long-term solution here is probably to avoid doing > lookup in the plpgsql lexer and move it into some sort of callback > hook in the main parser, as we've discussed before. I didn't want > to get into that right away, but I'm now thinking it has to happen > before not after refactoring the lexer code. One issue that has to > be surmounted before that can happen is that plpgsql currently > throws away all knowledge of syntactic scope after initial > processing of a function --- the "name stack" is no longer available > when we want to parse individual SQL commands. We can probably > rearrange that design but it's another bit of work I don't have time > for right now. All of this sounds pretty familiar to me. As you may recall, our framework includes a SQL parser which parses the subset of standard SQL we feel is portable enough, and generates Java classes to implement the code in "lowest common denominator" SQL with all procedural code for triggers and stored procedures handled in Java (which runs in our middle tier database service). We use ANTLR, and initially had a three-phase process: lexer, parser, and tree-walkers to generate code. We were doing way too much in the parser phase -- checking for table names, column names, data types, etc. The syntax of SQL forced us to do a lot of scanning forward and remembering where we were (especially to get the FROM clause information so we could process columns in the result list). We were able to get to much cleaner code by rewriting the parser to have a "dumb" phase to get the overall structure into an AST, and then use a tree-walker phase to do all the lookups and type resolution after we had the rough structure, writing another AST to walk for code generation. Besides making the code cleaner and easier to maintain, it helped us give better error messages pointing more accurately to the source of the problem. I don't know if a similar approach is feasible in flex/bison, but if it is, refactoring for an extra pass might be worth the trouble. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report: getting plpgsql to use the core lexer
Alvaro Herrera writes: > Tom Lane wrote: >> ... I spent a fair amount of time trying to work around that >> by eliminating uses of "yytext" in plpgsql, and mostly succeeded, but >> there are still some left. (Some of the remaining regression failures are >> error messages that point at the wrong token because they rely on yytext.) > Just wondering if there are additional regressions not detected due to > pg_regress using the ignore-whitespace option to diff. Good question, but I doubt it --- those messages all use double quotes around the yytext string, and I believe that eg. "foo" and " foo" are different even under --ignore-whitespace. I just finished wiping all that stuff from my work directory, or I'd be able to give you a non-guesswork answer :-( ... but it's not worth regenerating the build for. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report: getting plpgsql to use the core lexer
Tom Lane wrote: > Another pretty serious issue is that the current plpgsql lexer treats > various sorts of qualified names as single tokens. I had thought this > could be worked around in the interface layer by doing more lookahead. > You can do that, and it mostly works, but it's mighty tedious. The big > problem is that "yytext" gets out of step --- it will point at the last > token the core lexer has processed, and there's no good way to back it up > after lookahead. I spent a fair amount of time trying to work around that > by eliminating uses of "yytext" in plpgsql, and mostly succeeded, but > there are still some left. (Some of the remaining regression failures are > error messages that point at the wrong token because they rely on yytext.) Just wondering if there are additional regressions not detected due to pg_regress using the ignore-whitespace option to diff. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Status report: getting plpgsql to use the core lexer
I've spent the past several days working on the project I suggested here: http://archives.postgresql.org/message-id/18653.1239741...@sss.pgh.pa.us of getting rid of plpgsql's private lexer and having it use the core lexer instead. I've run out of time for that and need to go focus on commitfest reviewing, but I thought I'd make a brain dump first. I have a patch that sort of works (attached for amusement purposes); but it doesn't quite pass all the regression tests, and I've concluded that there are a couple of basic mistakes in the way I approached it. One problem that wasn't obvious when I started is that if you are trying to use a reentrant lexer, Bison insists on including its YYSTYPE union in the call signature of the lexer. Of course, YYSTYPE means different things to the core grammar and plpgsql's grammar. I tried to work around that by having an interface layer that would (among other duties) translate as needed. It turned out to be a real PITA, not least because you can't include both definitions into the same C file. The scheme I have has more or less failed --- I think I'd need *two* interface layers to make it work without unmaintainable kluges. It would probably be better to try to adjust the core lexer's API some more so that it does not depend on the core YYSTYPE, but I'm not sure yet how to get Bison to play along without injecting an interface layer (and hence wasted cycles) into the core grammar/lexer interface. Another pretty serious issue is that the current plpgsql lexer treats various sorts of qualified names as single tokens. I had thought this could be worked around in the interface layer by doing more lookahead. You can do that, and it mostly works, but it's mighty tedious. The big problem is that "yytext" gets out of step --- it will point at the last token the core lexer has processed, and there's no good way to back it up after lookahead. I spent a fair amount of time trying to work around that by eliminating uses of "yytext" in plpgsql, and mostly succeeded, but there are still some left. (Some of the remaining regression failures are error messages that point at the wrong token because they rely on yytext.) Now, having name lookup happen at the lexical level is pretty bogus anyhow. The long-term solution here is probably to avoid doing lookup in the plpgsql lexer and move it into some sort of callback hook in the main parser, as we've discussed before. I didn't want to get into that right away, but I'm now thinking it has to happen before not after refactoring the lexer code. One issue that has to be surmounted before that can happen is that plpgsql currently throws away all knowledge of syntactic scope after initial processing of a function --- the "name stack" is no longer available when we want to parse individual SQL commands. We can probably rearrange that design but it's another bit of work I don't have time for right now. The plpgsql code also currently has a bunch of "unreserved keywords" that are only significant in one or two contexts, and are implemented by doing pg_strcasecmp(yytext, "keyword") tests instead of letting the lexer know they exist. This is one of the things that got broken by the above yytext-synchronization issue. What the patch is doing is checking for identifiers spelled to match, but it's really wrong because it can't distinguish quoted and unquoted occurrences of a keyword. I'm inclined to think the right fix is to make an honest-to-goodness concept of reserved and unreserved keywords in plpgsql, as we do in the core grammar. A lot of the words that currently are reserved words probably wouldn't need to be, if we had that infrastructure. Also, the more I messed with this the more I thought it was time to throw away plpgsql's error location tracking and start over. That was all written before we had decent location tracking in the core lexer/parser, and we could do a much better and more consistent job if we relied on the core facilities instead of rolling our own. So there's a lot left to do here, and it has to go on the back burner for now. regards, tom lane binC033irjbRt.bin Description: plpgsql-use-core-lexer-1.patch.gz -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status Report on SE-PostgreSQL
The patch set of SE-PostgreSQL and related stuff were updated (r1467). [1/5] http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1467.patch [2/5] http://sepgsql.googlecode.com/files/sepostgresql-utils-8.4devel-3-r1467.patch [3/5] http://sepgsql.googlecode.com/files/sepostgresql-policy-8.4devel-3-r1467.patch [4/5] http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1467.patch [5/5] http://sepgsql.googlecode.com/files/sepostgresql-tests-8.4devel-3-r1467.patch List of updates: - Documentation is updated based on Robert Haas's suggestions. See, http://archives.postgresql.org/message-id/497c6808.2060...@kaigai.gr.jp - Bugfix: SE-PostgreSQL related functions didn't raise an error when "pgace_feature" option is not "selinux". - Add a launcher program to run testcases with various kind of security context. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status Report on Hot Standby
Robert Treat wrote: On Friday 16 January 2009 19:16:42 Simon Riggs wrote: Bruce asked for 2 more weeks to get patches into shape for commit. Current patch v8e is attached here. Ready for commit? Up to you. My overall opinion is that it's in very good shape. Worth the community including it in this release and spending further time on it. I'm happy to stand by this going forwards. +1 +1 I've been testing several versions of this patch, and overall it looks very good. regards Mark -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status Report on Hot Standby
On Tue, 2009-01-20 at 18:08 -0500, Robert Treat wrote: > > The patch could benefit further from input from other various hackers, > > what couldn't? It's time to put this in a shared repository (of some > > kind) and make further changes to it in a controlled manner. > > > > Yep. I've now got this running on Linux and Solaris and testing so far has > looked good. I've also spoken to a couple other people who have built it and > run it, and everyone has been pretty happy. It'd certainly be nice to see > this get into the main source tree to make it easier for future testing. (For > example, one hurdle on Solaris, I had to get a different version of patch to > handle Simon's diff... ugh!) There'll be a new version out soon, hopefully tomorrow, with some fancy new deferred conflict resolution. Timing depends on passage through testing. I think it will then be fermented in GIT for a while. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status Report on Hot Standby
On Friday 16 January 2009 19:16:42 Simon Riggs wrote: > Bruce asked for 2 more weeks to get patches into shape for commit. > > Current patch v8e is attached here. Ready for commit? Up to you. > > My overall opinion is that it's in very good shape. Worth the community > including it in this release and spending further time on it. I'm happy > to stand by this going forwards. > +1 > The patch could benefit further from input from other various hackers, > what couldn't? It's time to put this in a shared repository (of some > kind) and make further changes to it in a controlled manner. > Yep. I've now got this running on Linux and Solaris and testing so far has looked good. I've also spoken to a couple other people who have built it and run it, and everyone has been pretty happy. It'd certainly be nice to see this get into the main source tree to make it easier for future testing. (For example, one hurdle on Solaris, I had to get a different version of patch to handle Simon's diff... ugh!) -- Robert Treat Conjecture: http://www.xzilla.net Consulting: http://www.omniti.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Status Report on SE-PostgreSQL
I also think it is a good idea to summarize current status of
SE-PostgreSQL, as Simon Riggs doing on his works.
The current revision of SE-PostgreSQL is 1425, available here:
[1/5]
http://sepgsql.googlecode.com/files/sepostgresql-sepgsql-8.4devel-3-r1425.patch
[2/5]
http://sepgsql.googlecode.com/files/sepostgresql-utils-8.4devel-3-r1425.patch
[3/5]
http://sepgsql.googlecode.com/files/sepostgresql-policy-8.4devel-3-r1425.patch
[4/5]
http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1425.patch
[5/5]
http://sepgsql.googlecode.com/files/sepostgresql-tests-8.4devel-3-r1425.patch
We had various kind of comments, feature requests and discussions during
previous/current commit fest, then whole of them are already included.
Currently, we have no open issues here.
As I summarized as follows, we had many discussions about its design
issues mainly, so my patch set has been updated to support them.
I believe we should move to detailed-reviews to merge the feature any
time now, since we should aware of v8.4 schedule.
I really would like folks to help/volunteer reviewing the patches, please!
* CommitFest:Nov
- Simon Riggs requires a new GUC option to turn on/off row-level security
labeling to reduce storage comsumption, then updated as follows:
http://archives.postgresql.org/message-id/492691a8.8030...@ak.jp.nec.com
- Bruce Momjian suggested Row-level database ACLs to be compiled in default.
- Discussions for default compile options: PostgreSQL doesn't prefer compile
time option to turn on/off features, except for platform specific one.
SE-PostgreSQL is indeed platform specific feature. But, it makes other
issue that need mutually-exclusive enhanced security feature.
We concluded it as follows:
- All configurable features should be compiled within a single binary.
- Both of DAC and MAC should be available simultaneously in row-level also.
- DAC is hardwired, and we allow users to choose an enhanced security
feature.
- I updated the patch set to support both of Row-level database ACLs and
an enhanced security feature (SELinux) simultaneously. ('08/12/17)
http://archives.postgresql.org/message-id/4948b6bd.1050...@ak.jp.nec.com
- Robert Haas concerned about Stephen Frost's column-level privileges has
a trouble, so it's unclear whether it can get merged into v8.4.
- I also worked for his patch, then it got being ready for commit:
http://archives.postgresql.org/message-id/20090116045825.gy4...@tamriel.snowman.net
- Alvaro Herrera suggested "static inline" is not preferable.
* CommitFest:Sep
- Peter Eisentraut commented about its design specifications:
http://archives.postgresql.org/message-id/48d03953.6000...@gmx.net
- The hot issues were lack of fine-grained access controls in SQL-level,
and covert channels with row-level controls.
- We finally made agreement to provide platform independent row-level controls,
and explicit documentation about covert channels in PK/FK constraints.
No one didn't want to apply polyinstantiation idea.
- Simon Riggs requires wiki article to introduce SE-PostgreSQL.
http://wiki.postgresql.org/wiki/SEPostgreSQL
- Patch set was updated to support Row-level database ACLs
http://archives.postgresql.org/message-id/48f46606.4080...@ak.jp.nec.com
* CommitFest:Jul
- The patch set got documentation/testcases.
- Peter Eisentraut commented about some of items:
http://archives.postgresql.org/message-id/200807071739.58428.pete...@gmx.net
- Then, these items are updated:
http://archives.postgresql.org/message-id/48773188.6000...@ak.jp.nec.com
* CommitFest:May
- First patch set for v8.4 were proposed.
- Tom Lane gave us various items to be improved.
http://archives.postgresql.org/message-id/3275.1210019...@sss.pgh.pa.us
- I had a presentation at PGcon2008 ottawa.
http://sepgsql.googlecode.com/files/PGCON20080523.pdf
* Prior phase
- First proposal of PGACE security framework, but I didn't know it was
just after the date of feature freeze in v8.3. So, it was suggested
to wait for v8.4 development cycle. ('07/04/17)
- 8.2.x based SE-PostgreSQL announced. ('07/09/04)
- SE-PostgreSQL package got merged into Fedora Project. ('07/11/08)
- 8.3.x based SE-PostgreSQL announced. ('08/03/08)
Thanks,
--
KaiGai Kohei
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status report on 8.3 releaset
Joshua D. Drake wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Thu, 29 Nov 2007 14:01:11 -0800 > Neil Conway <[EMAIL PROTECTED]> wrote: > > > On Thu, 2007-11-29 at 11:26 -0500, Bruce Momjian wrote: > > > I expect these cleanups to continue for at least another week or > > > two. Once they slow we will schedule RC1. > > > > So are there no plans for an additional beta? Given the recent > > addition of changes like > > > > http://archives.postgresql.org/pgsql-committers/2007-11/msg00552.php > > http://archives.postgresql.org/pgsql-committers/2007-11/msg00532.php > > > > I wonder if another beta before RC1 would be warranted. > > I think the email is not quite clear :). There is a Beta 4 coming, the > problem is packagers aren't really available this week so if it comes > it will be next week. Yes, there will be another beta before RC1. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Status report on 8.3 release
"Andreas 'ads' Scherbaum" <[EMAIL PROTECTED]> writes: > i would also like to test another Beta, if we do something about this > problem: > > http://archives.postgresql.org/pgsql-hackers/2007-11/msg00960.php That's already done, it would be in the next beta. You could check out a copy from CVS HEAD if you want to test it now. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support! ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Status report on 8.3 release
On Fri, 30 Nov 2007 11:26:35 +1100 Brendan Jurd wrote: > On Nov 30, 2007 11:10 AM, Andreas 'ads' Scherbaum <[EMAIL PROTECTED]> wrote: > > i would also like to test another Beta, if we do something about this > > problem: > > > > http://archives.postgresql.org/pgsql-hackers/2007-11/msg00960.php > > Hi Andreas, > > Tom's already committed the quote_literal(anyelement) function. > > http://archives.postgresql.org/pgsql-committers/2007-11/msg00530.php Ups, i've overseen this one. Forget my posting and i will keep testing the next beta ;-) Kind regards -- Andreas 'ads' Scherbaum German PostgreSQL User Group ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Status report on 8.3 release
On Nov 30, 2007 11:10 AM, Andreas 'ads' Scherbaum <[EMAIL PROTECTED]> wrote: > i would also like to test another Beta, if we do something about this > problem: > > http://archives.postgresql.org/pgsql-hackers/2007-11/msg00960.php Hi Andreas, Tom's already committed the quote_literal(anyelement) function. http://archives.postgresql.org/pgsql-committers/2007-11/msg00530.php Cheers BJ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Status report on 8.3 release
Hello, On Thu, 29 Nov 2007 17:21:09 -0500 Tom Lane wrote: > Neil Conway <[EMAIL PROTECTED]> writes: > > So are there no plans for an additional beta? > > Yes, there are, but not till we do something about > http://archives.postgresql.org/pgsql-hackers/2007-11/msg01302.php i would also like to test another Beta, if we do something about this problem: http://archives.postgresql.org/pgsql-hackers/2007-11/msg00960.php Kind regards -- Andreas 'ads' Scherbaum German PostgreSQL User Group ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Status report on 8.3 release
Neil Conway <[EMAIL PROTECTED]> writes: > So are there no plans for an additional beta? Yes, there are, but not till we do something about http://archives.postgresql.org/pgsql-hackers/2007-11/msg01302.php regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Status report on 8.3 release
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 29 Nov 2007 14:01:11 -0800 Neil Conway <[EMAIL PROTECTED]> wrote: > On Thu, 2007-11-29 at 11:26 -0500, Bruce Momjian wrote: > > I expect these cleanups to continue for at least another week or > > two. Once they slow we will schedule RC1. > > So are there no plans for an additional beta? Given the recent > addition of changes like > > http://archives.postgresql.org/pgsql-committers/2007-11/msg00552.php > http://archives.postgresql.org/pgsql-committers/2007-11/msg00532.php > > I wonder if another beta before RC1 would be warranted. I think the email is not quite clear :). There is a Beta 4 coming, the problem is packagers aren't really available this week so if it comes it will be next week. At least that is last I heard. Joshua D. Drake > > -Neil > > > > ---(end of > broadcast)--- TIP 2: Don't 'kill -9' the > postmaster > - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTzhHATb/zqfZUUQRAiZlAJ0Tj/Kzn0cFnfm7pB9YFvKQm4txpQCfTCo6 s6FF7Ey5GMISxNmwxTlgi7g= =KypU -END PGP SIGNATURE- ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report on 8.3 release
On Thu, 2007-11-29 at 11:26 -0500, Bruce Momjian wrote: > I expect these cleanups to continue for at least another week or two. > Once they slow we will schedule RC1. So are there no plans for an additional beta? Given the recent addition of changes like http://archives.postgresql.org/pgsql-committers/2007-11/msg00552.php http://archives.postgresql.org/pgsql-committers/2007-11/msg00532.php I wonder if another beta before RC1 would be warranted. -Neil ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] Status report on 8.3 release
I wanted to give everyone an overview of where we are for 8.3. We have addressed almost every major issue for 8.3 but we are getting a steady stream of minor cleanups. These cleanups are vital to keep the quality of Postgres at a high level. It allows us to make major changes in every release but continue to produce a very reliable database system. 8.3 is going to be a watershed release, like 8.0, so it is not surprising we have cleaning up to do. Please keep up that activity. I expect these cleanups to continue for at least another week or two. Once they slow we will schedule RC1. Of course, we are getting near Christmas, so that might affect the final release schedule as well. Based on everything above, it is likely that 8.3 final will not be until the first week of January. I know everyone wanted this to be a short release schedule but the feature set just kept growing, so while we have the normal year-to-year release schedule, we do have a blockbuster release that we can all be proud of. -- Bruce Momjian <[EMAIL PROTECTED]>http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: Release planning (was: Re: [HACKERS] Status report)
Christopher Kings-Lynne wrote: > > We can have a major feature deadline, then a minor feature deadline. I > > particularly liked the idea of 1 July as the major feature deadline, > > then 14 July as major-feature-tweak deadline. That funnels things better > > to cater for the manpower available. > > That being said, your PITR patch still hasn't been comitted and it's > well past 1st July. If I had to drop something from 7.5 right now to > make a release, then it would have to be PITR! Well, the problem is that Simon isn't the reason the patch hasn't been applied. The reason is that we have had other priorities in reviewing patches so it doesn't seem fair to single out that feature for exclusion. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
On Thu, 15 Jul 2004, Christopher Kings-Lynne wrote: We can have a major feature deadline, then a minor feature deadline. I particularly liked the idea of 1 July as the major feature deadline, then 14 July as major-feature-tweak deadline. That funnels things better to cater for the manpower available. That being said, your PITR patch still hasn't been comitted and it's well past 1st July. If I had to drop something from 7.5 right now to make a release, then it would have to be PITR! As Bruce would put it "that would be unfair to Simon", since it isn't his fault that the patch hasn't been reviewed for commit yet ... if, once reviewed, it is found to need work, then it should be put off until 7.6 ... I'm in Josh's camp on this one, in that Nested Xacts should probably be pulled since it still needs work :( Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: Release planning (was: Re: [HACKERS] Status report)
We can have a major feature deadline, then a minor feature deadline. I particularly liked the idea of 1 July as the major feature deadline, then 14 July as major-feature-tweak deadline. That funnels things better to cater for the manpower available. That being said, your PITR patch still hasn't been comitted and it's well past 1st July. If I had to drop something from 7.5 right now to make a release, then it would have to be PITR! Chris ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: Release planning (was: Re: [HACKERS] Status report)
On Wed, 2004-07-14 at 21:02, Bruce Momjian wrote: > Marc G. Fournier wrote: > > The big problem that I see with how this feature freeze/beta/release has > > gone down is that we have *alot* of big items that are/were being worked > > on (ARC, BGWriter, auto_vacuum, PITR, Nested Xacts), and only so much man > > power at the reviewer stage ... we *should* have frozen it all on June > > 1st, got the ready features out the door and released, and then > > concentrated on getting the "almost ready, but not quite" features into > > the next release as quickly as possible ... > > > > Hindsight is 20-20 ... maybe next time we'll learn from it? > > True, 20-20. One thing is that you can't schedule assuming full > knowledge of the future, of course. For example, on May 1, I thought by > June 1 PITR and NT would be done, and that Win32 and tablespaces > wouldn't. What actually happened is that tablespaces made the deadline > (with June adjustments), NT is in but needs more work, and Win32 is > better off now than I thought it would be. And we don't know if PITR is > ready or not because we haven't studied it enough. > I see a couple of issues: - new releases of PostgreSQL require a full initdb. There is little upgrade support as there is with other products. (Not complaining..) - commercial products release around every 18 months...customers do NOT want this to be any quicker...more disruptive upgrades, plus marketing takes time to organise. I have customers that upgrade regularly every 3+ years (!), and take longer term strategy into consideration. - commercial issues often cause products to be delayedMS is said to be late delivering Yukonmarketing-led companies often delay waiting for the right feature setsaled-led companies never do. Delivering early as Oracle often does mean that the received wisdom is never upgrade to a x.0 version, always wait for the x.1 minor version where all the features actually work as advertised. Deadlines are great, but advertise them ahead of time, then stick to them. Every other project I see has a big page saying "how to contribute" and then details of deadlines etc.. We can have a major feature deadline, then a minor feature deadline. I particularly liked the idea of 1 July as the major feature deadline, then 14 July as major-feature-tweak deadline. That funnels things better to cater for the manpower available. Best Regards, Simon Riggs ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 2004-07-13 at 23:03, Marc G. Fournier wrote: > As a community, I don't think we should be 'supporting' anything older > then the last STABLE ... > I agree, but that message isn't clearly stated anywhere. The lists are full of people running very old releases - and everybody keeps having to ask "what release are you running" and the answer is always shocking. Can we set up different lists for the various major versions, so people can see where to go for different types of support? Rather than General, have General7.4 and General7.3...making everybody's first question "what is my release? where should I post this question?" - 95% of people are newbies, whatever the list/project. That will also make it clear that on-list support is available for older releases, but they really should be thinking about upgrade. And different people can choose whether to hang out at the bleeding edge, or legacy support. We could also have different hints and tips by list then. One of the gems (hint 9, I think) is changing in r7.5, so we must either mis-advise people who use the new release or fail to advise those using an older release. The down-rev lists could contain advice about upgrading... Best Regards, Simon Riggs ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > The big problem that I see with how this feature freeze/beta/release has > gone down is that we have *alot* of big items that are/were being worked > on (ARC, BGWriter, auto_vacuum, PITR, Nested Xacts), and only so much man > power at the reviewer stage ... we *should* have frozen it all on June > 1st, got the ready features out the door and released, and then > concentrated on getting the "almost ready, but not quite" features into > the next release as quickly as possible ... > > Hindsight is 20-20 ... maybe next time we'll learn from it? True, 20-20. One thing is that you can't schedule assuming full knowledge of the future, of course. For example, on May 1, I thought by June 1 PITR and NT would be done, and that Win32 and tablespaces wouldn't. What actually happened is that tablespaces made the deadline (with June adjustments), NT is in but needs more work, and Win32 is better off now than I thought it would be. And we don't know if PITR is ready or not because we haven't studied it enough. One problem with pushing 7.5 out June 1 and then coming back to these features is losing momentum. Sure, ideally we could hit them all in 7.6, but realistically it is a lot harder to get things moving once you stop. Look at the PITR patch we had for 7.3 (?) and are only now getting it into the tree. I am not saying we did things right or wrong --- just pointing out the momentum issue. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: Release planning (was: Re: [HACKERS] Status report)
On 7/14/2004 1:13 PM, Bruce Momjian wrote: What are you talking about? Are you suggesting Fujitsu's features are getting more attendtion than ARC for some political reason? You think nested transactions and tablespaces are just press release features? All those features are being developed by the community under community direction and based on community feedback/needs. How is that different from Afilias funding ARC? I was getting a little upset about the suggestion of being "hard-nosed" against the people who want ARC now. Yes, I wanted those features out earlier and this was not because of Afilias or dictated by Afilias, but because there is demand for it from the community. Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: Release planning (was: Re: [HACKERS] Status report)
On Wed, 14 Jul 2004, Bruce Momjian wrote: What are you talking about? Are you suggesting Fujitsu's features are getting more attendtion than ARC for some political reason? You think nested transactions and tablespaces are just press release features? All those features are being developed by the community under community direction and based on community feedback/needs. How is that different from Afilias funding ARC? No, the beef that Jan has, and that I also have, is that we put off the release that was schedualed for June 1st in order to get Nested Xacts into the tree ... hindsight being 20-20, we shouldn't have done that, but should have delayed Nested Xacts for 7.6 ... there *were* enough features in the tree to warrant a release, and features that ppl needed / wanted. Do I believe there were political motivations for postponing the feature freeze? Personally ... no. And I don't believe that the Press Release (and a nice one at that) can really be counted as motivation for the postponement, since the PR was done *after* we decided to push things back a month ... I do believe that there was some pressure from Futjitsu involved, in postponing it, since they'd rather see it in sooner then later ... *but* ... I don't really believe that the pressure is any different then there quite possibly could have been had, say, Jan been "almost finished" ARC and Affilias wanted to see that sooner rather then later ... we all want to see the feature we've either been working on, or funded, released as soon as possible ... The big problem that I see with how this feature freeze/beta/release has gone down is that we have *alot* of big items that are/were being worked on (ARC, BGWriter, auto_vacuum, PITR, Nested Xacts), and only so much man power at the reviewer stage ... we *should* have frozen it all on June 1st, got the ready features out the door and released, and then concentrated on getting the "almost ready, but not quite" features into the next release as quickly as possible ... Hindsight is 20-20 ... maybe next time we'll learn from it? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
Jan Wieck wrote: > On 7/13/2004 10:23 PM, Christopher Kings-Lynne wrote: > > >> I was thinking of something much simpler where Jan would create an ARC > >> patch against 7.4.X and have it either in /contrib for 7.4.X or on our > >> ftp servers, or on a web site. I could create a mechanism so SELECT > >> version() would display Jan's add-on. > > > > I think you guys just need to learn to become comfortable with being > > hard-nosed about this. Just Do Not Care about people who "want" ARC > > right this second. Do you see them calling up Oracle and saying please > > backport the new stuff from 11i-devel so we can use it now please? > > Hmmm ... I wonder what you think who those people are who "want" ARC > right this second, and who "you guys" are on the other hand. > > To fill you in a little, I am the PostgreSQL CORE team member Jan Wieck, > who burned Afilias payroll hours to implement the ARC buffer replacement > strategy. The feature has been completed and fully contributed under the > BSD license way ahead of any possible release schedule. I have had > several requests for backports, but consider the feature too delicate to > make such patch available. Don't worry, it's not one of my goals to get > this into any release, the reason I am personally touched by this is not > because it affects my checking account in any way. > > What touches me here is the fact that the PostgreSQL Open Source Project > under the BSD license seems starting to care a lot more about some press > releases and silly news splashes, than to care about real features > contributed under the terms and conditions of the BSD license by serious > members of the Open Source Community. Which part of the storage manager, > that Futjitsu uses so that their customers don't need ARC, the > background writer or vacuum at all, do you think will be contributed any > time soon? > > Don't get me wrong here, I don't have a problem with someone making > money out of my 8+ years of contributions to this project. But I do have > a problem with those efforts getting in my way of contributing. What are you talking about? Are you suggesting Fujitsu's features are getting more attendtion than ARC for some political reason? You think nested transactions and tablespaces are just press release features? All those features are being developed by the community under community direction and based on community feedback/needs. How is that different from Afilias funding ARC? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: Release planning (was: Re: [HACKERS] Status report)
On 7/13/2004 10:23 PM, Christopher Kings-Lynne wrote: I was thinking of something much simpler where Jan would create an ARC patch against 7.4.X and have it either in /contrib for 7.4.X or on our ftp servers, or on a web site. I could create a mechanism so SELECT version() would display Jan's add-on. I think you guys just need to learn to become comfortable with being hard-nosed about this. Just Do Not Care about people who "want" ARC right this second. Do you see them calling up Oracle and saying please backport the new stuff from 11i-devel so we can use it now please? Hmmm ... I wonder what you think who those people are who "want" ARC right this second, and who "you guys" are on the other hand. To fill you in a little, I am the PostgreSQL CORE team member Jan Wieck, who burned Afilias payroll hours to implement the ARC buffer replacement strategy. The feature has been completed and fully contributed under the BSD license way ahead of any possible release schedule. I have had several requests for backports, but consider the feature too delicate to make such patch available. Don't worry, it's not one of my goals to get this into any release, the reason I am personally touched by this is not because it affects my checking account in any way. What touches me here is the fact that the PostgreSQL Open Source Project under the BSD license seems starting to care a lot more about some press releases and silly news splashes, than to care about real features contributed under the terms and conditions of the BSD license by serious members of the Open Source Community. Which part of the storage manager, that Futjitsu uses so that their customers don't need ARC, the background writer or vacuum at all, do you think will be contributed any time soon? Don't get me wrong here, I don't have a problem with someone making money out of my 8+ years of contributions to this project. But I do have a problem with those efforts getting in my way of contributing. Come again about hard-nosing, please. Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: As Jan points out, its the 'small features that are done' that we've been griping about having to wait for, not the big ones which we know aren't done ... Hmmm... so we do things slightly differently than previously... This upcoming version could be PG version 8.0, We continue with bugfixes on 7.4.x, That still leaves 7.5.x, 7.6.x (etc if needed), for releasing new versions of PG without the "big features". Kind of like an in-between thing, whilst waiting for the major features in the major releases? That would mean we'd have: Version 8.0 as our next main release, Version 9.0 being the version after that with the next "big features" in it. Version 8.x being version 8 plus smaller features, prior to 9.0 Version 8.x.x being version 8.x plus bug fixes. Sounds like it could get hairy if we're not careful, but I reckon the PG Community is mature enough to make the right calls where needed. :) Regards and best wishes, Justin Clift ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
I was thinking of something much simpler where Jan would create an ARC patch against 7.4.X and have it either in /contrib for 7.4.X or on our ftp servers, or on a web site. I could create a mechanism so SELECT version() would display Jan's add-on. I think you guys just need to learn to become comfortable with being hard-nosed about this. Just Do Not Care about people who "want" ARC right this second. Do you see them calling up Oracle and saying please backport the new stuff from 11i-devel so we can use it now please? You learn after a long while in the software industry that if left to themselves the users will make _endless_ demands and they think that they NEED everything NOW. Of course in reality, they don't NEED it NOW - they can wait. And even if they did get it now, then it's buggy and then they complain even louder. You cannot win. The arguments about things "getting more testing" in production is nonsense - we are a major database server - you cannot have users doing the testing for us They have signed on to the PostgreSQL project on "our" terms of development - if they want to sponsor someone to backport something for their own situation, they can do so. Otherwise, they perhaps should have not been using PostgreSQL for their particular app in the first place! It's not worth thinking along the lines of "oh if only we could get this feature out RIGHT NOW, we'd get more users and more people uprading and convertign and better reviews and stuff - but that's a distraction. There will ALWAYS be Just One More Feature that we need to be "really great". Just don't worry about it. Chris ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
God, we still have clients using 7.2 servers, cause they've had no reason yet to upgrade to the latest ... "it works, why upgrade?" is generally the opinion ... Because there's shocking failure-to-restart bugs in 7.2...and if you upgrade to 7.4 you're forced to get new features as well. Chris ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 2004-07-13 at 20:49, Marc G. Fournier wrote: > On Tue, 13 Jul 2004, Tom Lane wrote: > > > We could certainly do something along that line if we had a few people > > willing to be "gatekeepers". We'd have to work harder at making the > > release generation process open and documented though. Right now there > > are plenty of steps that only you, Bruce, or Lamar (respectively) know > > how to do... > > I think we could do it if we restricted it to *just* one release back ... > but I do agree with Peter about ppls motivations for upgrading (bug fixes > vs new features) ... we'd have to have a 'two branch stable', one would be > only bug fixes, the other would be bug fixes+feature backpatch ... would > could get interesting trying to figure out release naming conventions :) FreeBSD does do this. They tag the bug-fix only branches as being SECURITY releases, since typically they only contain fixes for security or reliability related issues. The 5.2.1 release which followed 5.2.0 was an example of one of the few releases the Security team has done. They usually just allow users to use CVSUP to follow the branch. ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: Release planning (was: Re: [HACKERS] Status report)
> However, looking at how the Linux community handles it, I think we can > borrow a lot of concepts from them. I was thinking quite the opposite. The Linux community doesn't exactly have a great track-record for their stable releases. The thoughts behind the process might be good, but do we have examples where it has worked out well? The 2.4 series seems to have been particularly bad for new major issues in their stable releases. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 13 Jul 2004, Bruce Momjian wrote: The nice thing about doing something lke that is those small features would get a degree of testing happening in a live environment ... Of course that last sentence is the downside too --- people don't want to have to retest their setups after a minor release upgrade. Nobody would be required to upgrade to a new minor release either ... nobody is *require* to upgrade to any release, for that matter ... Hell, when we have a client that comes to us with a problem, we don't recommend upgrading unless we find something in the RELEASE NOTES to indicate something has been fixed related to their problem ... it isn't a automatic "well, upgrade to the latest stable first, and then we can help you" ... God, we still have clients using 7.2 servers, cause they've had no reason yet to upgrade to the latest ... "it works, why upgrade?" is generally the opinion ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > On Tue, 13 Jul 2004, Bruce Momjian wrote: > > >> The nice thing about doing something lke that is those small features > >> would get a degree of testing happening in a live environment ... > > > > Of course that last sentence is the downside too --- people don't want > > to have to retest their setups after a minor release upgrade. > > Nobody would be required to upgrade to a new minor release either ... > nobody is *require* to upgrade to any release, for that matter ... > > Hell, when we have a client that comes to us with a problem, we don't > recommend upgrading unless we find something in the RELEASE NOTES to > indicate something has been fixed related to their problem ... it isn't a > automatic "well, upgrade to the latest stable first, and then we can help > you" ... > > God, we still have clients using 7.2 servers, cause they've had no reason > yet to upgrade to the latest ... "it works, why upgrade?" is generally the > opinion ... We have always recommended upgrading to the most recent minor release, at least as a project policy for support. Also, the upgrade is only stop/install/restart so it is quite easy to do, and folks like the fact they don't have to test for new functionality. One idea would be for us to release 7.5 and 7.5.0.1, and allow 7.5.1 to have minor new features. That way, 7.5.0.[1-9] are no new features, and 7.5.[1-9] are minor improvements against 7.5. Of course this does require more project management, and we already don't have enough manpower. I was thinking of something much simpler where Jan would create an ARC patch against 7.4.X and have it either in /contrib for 7.4.X or on our ftp servers, or on a web site. I could create a mechanism so SELECT version() would display Jan's add-on. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 2004-07-13 at 13:03 -0400, Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Jan Wieck wrote: > >> I think in the future we have to force all large features, those that > >> probably need more than 6 months of development time, to be properly > >> #ifdef'd. Then it wouldn't be that big of a deal to release more often. > > > Alvaro started out with ifdef's but it got too confusing and we all > > agreed to just go with a normal patch. He just hits too much code. > > I think the same would be true of almost any really large feature --- > ifdefs all over the code base are just too much of a mess. > > To be honest I think that "releasing more often" isn't necessarily an > appropriate goal for the project anymore. Five or six years ago we were > doing a major (initdb-forcing) release every three or four months, and > that was appropriate at the time, but the project has matured and our > user population has changed. Look at how many people are still using > 7.2 or 7.3. One major release a year may be about right now, because > you can't get people to adopt new major revs faster than that anyway. > > Of course this all ties into the pain of having to dump/reload large > databases, and maybe if we had working pg_upgrade the adoption rate > would be faster, but I'm not at all sure of that. We're playing in > a different league now. Big installations tend to want to do > significant amounts of compatibility testing before they roll out > a new database version. > > regards, tom lane It sounds like your only taking the point of view of people upgrading previous installations. What about new installations? I'm sure there are hundreds, if not thousands of new installations happening every week. These people are going to grab the latest stable version without a doubt. I think releasing more often would result in features getting tested much more. Then the "big installations" can see that a major feature has been in two stable releases (even if the time period was only a year) and feel much more comfortable in upgrading. Why would they have to upgrade more often then necessary anyways? Assuming security exploits are back ported of course. -- Mike Benoit <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 13 Jul 2004, Bruce Momjian wrote: We have always recommended upgrading to the most recent minor release, at least as a project policy for support. Also, the upgrade is only stop/install/restart so it is quite easy to do, and folks like the fact they don't have to test for new functionality. Right, and this made sense when we were dealing with a release every 4 months or so ... now we're talking about one every year, or two ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > On Tue, 13 Jul 2004, Bruce Momjian wrote: > > > I was thinking of something much simpler where Jan would create an ARC > > patch against 7.4.X and have it either in /contrib for 7.4.X or on our > > ftp servers, or on a web site. I could create a mechanism so SELECT > > version() would display Jan's add-on. > > I think this would be a bad direction to take ... I can just see us > flooded with a bunch of 'the patch didn't apply to my source tree' > questions :( > > It might work, I just have my doubts ... I have my doubts too, of course. :-) -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 13 Jul 2004, Lamar Owen wrote: But Tom's assertion is true. We have enough trouble getting patches rolled out; adding parallel branches is just begging for trouble, due to our relatively small resource size. Although, we probably have enough developers at this point to make it happen. Except, we already have parallel branches, else we'd never have made a 7.4.x release ... Bruce I would want to be the patchmeister for the stable branch. Someone else (with Bruce's oversight, or Tom's, or whoever) could do the patchmunging and review for the development tree. But I want a stable hand on patches that go into the stable tree. The BSD's release something like that, with CURRENT, TESTING, and STABLE, right? (I'm not a big BSD user...) We have a CURRENT branch where all the 'innovations' are done (SMP re-writes, etc) ... and a STABLE which is a release with stuff patched down from CURRENT *if* applicable ... periodically, a RELEASE is made along either branch, and, some day, what is currently CURRENT will be re-tag'd as -STABLE, and then CURRENT will become a new branch ... So, for instance, the way we number: 7.4.x would be -STABLE HEAD would be -CURRENT once 7.5 is released, it would surplant 7.4 as -STABLE, previous versions would only ever see 'security related patches' and work towards 7.6 would be -CURRENT ... Now, if we went with a 'long term dev cycle', then we might look at 7.x as being -STABLE, while work towards 8.x would be considered -CURRENT ... As a community, I don't think we should be 'supporting' anything older then the last STABLE ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 13 Jul 2004, Bruce Momjian wrote: I was thinking of something much simpler where Jan would create an ARC patch against 7.4.X and have it either in /contrib for 7.4.X or on our ftp servers, or on a web site. I could create a mechanism so SELECT version() would display Jan's add-on. I think this would be a bad direction to take ... I can just see us flooded with a bunch of 'the patch didn't apply to my source tree' questions :( It might work, I just have my doubts ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
On Wed, 14 Jul 2004, Peter Eisentraut wrote: Marc G. Fournier wrote: Nobody would be required to upgrade to a new minor release either ... nobody is *require* to upgrade to any release, for that matter ... Most people don't have the time to investigate release notes, release policy details, etc. When there are bug fix updates, you use them. When there are feature updates, you use them for the next installation. Anything in between, or more variations added to that, just cause confusion. And frankly, for the examples thrown around that use this kind of policy, I can't see those as being very successful. I don't want to use a "stable" branch of a database system that still changes for random reasons. And I don't want a "current" branch that takes years to finalize. More frequent major releases, to the point that we can stem it, lead to more people getting more features earlier, which is good. Any of the other proposal just make things worse in my mind. 'k, note that I'm not arguing for more (or less) frequent releases ... but, do you have any thoughts on how we can effectively continue with the 'frequent releases' while bringing in the large features like Nested Xacts? We could start looking at 'development branches' for stuff like this, that could be merged up when ready for prime time, but would at least be available in CVS for those wishing to play with them ... ? Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Release planning (was: Re: [HACKERS] Status report)
On Tue, 13 Jul 2004, Tom Lane wrote: We could certainly do something along that line if we had a few people willing to be "gatekeepers". We'd have to work harder at making the release generation process open and documented though. Right now there are plenty of steps that only you, Bruce, or Lamar (respectively) know how to do... I think we could do it if we restricted it to *just* one release back ... but I do agree with Peter about ppls motivations for upgrading (bug fixes vs new features) ... we'd have to have a 'two branch stable', one would be only bug fixes, the other would be bug fixes+feature backpatch ... would could get interesting trying to figure out release naming conventions :) Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > On Tue, 13 Jul 2004, Bruce Momjian wrote: > > > We have always recommended upgrading to the most recent minor release, > > at least as a project policy for support. Also, the upgrade is only > > stop/install/restart so it is quite easy to do, and folks like the fact > > they don't have to test for new functionality. > > Right, and this made sense when we were dealing with a release every 4 > months or so ... now we're talking about one every year, or two ... I meant if they are on 7.4.X they should be on 7.4.3. I don't suggest they have to upgrade 7.2.X to 7.4.X. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: Release planning (was: Re: [HACKERS] Status report)
"Marc G. Fournier" <[EMAIL PROTECTED]> writes: > On Tue, 13 Jul 2004, Lamar Owen wrote: >> But Tom's assertion is true. We have enough trouble getting patches >> rolled out; adding parallel branches is just begging for trouble, due to >> our relatively small resource size. Although, we probably have enough >> developers at this point to make it happen. > Except, we already have parallel branches, else we'd never have made a > 7.4.x release ... The point though is that we expend only very minimal effort on maintaining the stable branches. We only back-patch bug fixes, and 99% of the time the patch is nearly verbatim the same change as we developed to fix the same problem in HEAD. If the code involved has changed enough that a significantly different fix would be required, most of the time we simply don't fix the stable branch. And we spend very nearly zero effort on QA for the stable branch --- there's certainly no significant push to get people to beta-test minor releases. If we did anything much in the way of back-porting features then the level of investment in this would have to rise greatly. In fact, if the proposal is to let people pick and choose which back-ported things they install, then we are not talking about just one stable version but 2^N stable versions for N options. We couldn't possibly test every combination. >> The BSD's release something like that, with CURRENT, TESTING, and STABLE, >> right? (I'm not a big BSD user...) Yeah, but they don't support mix-and-match feature sets. A back release has only one current version. We could certainly do something along that line if we had a few people willing to be "gatekeepers". We'd have to work harder at making the release generation process open and documented though. Right now there are plenty of steps that only you, Bruce, or Lamar (respectively) know how to do... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > Nobody would be required to upgrade to a new minor release either ... > nobody is *require* to upgrade to any release, for that matter ... Most people don't have the time to investigate release notes, release policy details, etc. When there are bug fix updates, you use them. When there are feature updates, you use them for the next installation. Anything in between, or more variations added to that, just cause confusion. And frankly, for the examples thrown around that use this kind of policy, I can't see those as being very successful. I don't want to use a "stable" branch of a database system that still changes for random reasons. And I don't want a "current" branch that takes years to finalize. More frequent major releases, to the point that we can stem it, lead to more people getting more features earlier, which is good. Any of the other proposal just make things worse in my mind. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
On Tuesday 13 July 2004 08:52 am, Jan Wieck wrote: > > I think in the future we have to force all large features, those that > probably need more than 6 months of development time, to be properly > #ifdef'd. Then it wouldn't be that big of a deal to release more often. > > Take my opinion with a grain of salt, as I'm young and don't have much experience with large, super-stable C projects. However, looking at how the Linux community handles it, I think we can borrow a lot of concepts from them. Let's seperate out into two different communities: Stable and Dev. The stable folks only want patches when necessary. They don't want new features - they don't use them. They don't want anything but security or stability patches. They don't want to be forced to upgrade. New major versions are moved into the stable community, but they don't replace old versions. No one ever says "upgrade or else!" Each major version is kept by interested parties and patched as needed. They never die, but are abandoned. The dev community wants to try out all kinds of things. They aren't running production code, so they can risk losing data. They don't have to be so sensitive about backwards compatibility and reliability and security. In the dev community, there is the main branch with all the accepted features. When people feel it is time for a new major version, they spend all their effort stabilizing that main branch. When it is finally stable, they create a new major version and hand it off to the stable community. People keep their own versions of postgresql in the dev community. These compete with each other. For instance, if Tom and others don't feel like a feature should go into the main dev branch, that doesn't mean that Bruce won't put it in his own version and encourage people to develop against that. When Bruce can prove that it is a useful feature and it works and is stable enough, he will work on getting it in the main dev branch. I know this isn't the way things have been done. I know that one of the arguments against proposals like this is that it seperates out our pool of developers. Yes, it will do that. But it will also open up development to more people and more experimentation. Maybe in the end, we will have a stable and dev community that is larger than the entire community we have now. -- Jonathan Gardner [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Release planning (was: Re: [HACKERS] Status report)
On Tuesday 13 July 2004 17:12, Marc G. Fournier wrote: > ... there is alot of stuff that doesn't require a reload of the database > (or initdb) that could very easily be backpatched ... > As Jan points out, its the 'small features that are done' that we've been > griping about having to wait for, not the big ones which we know aren't > done ... Split the feature out into either a patch or a module and put it in contrib until the next major version. Let contrib hold the smaller, non-initdb-forcing patches and such until the next major version rolls them into the mainline. Or even a patches tree parallel to contrib. Either way, the patch or module or whatever wouldn't be in the mainline unless the user needed it. Or maybe we need to rethink what a major version is. But even minor things can force an initdb, although we're better than we have been about this. But Tom's assertion is true. We have enough trouble getting patches rolled out; adding parallel branches is just begging for trouble, due to our relatively small resource size. Although, we probably have enough developers at this point to make it happen. Bruce I would want to be the patchmeister for the stable branch. Someone else (with Bruce's oversight, or Tom's, or whoever) could do the patchmunging and review for the development tree. But I want a stable hand on patches that go into the stable tree. The BSD's release something like that, with CURRENT, TESTING, and STABLE, right? (I'm not a big BSD user...) The Linux kernel has parallel development, and a gatekeeper for each stable release (2.0.x, 2.2.x, 2.4.x, and now 2.6.x). A 2.0.x kernel release happened not too long ago, in fact. We probably could have enough volunteers to do something like that. Gatekeeping a stable release would not be as complex as developing the new release, but, again, I would want an experienced hand on the last stable release where the temptation of backporting 'features' is great. I think a gatekeeper for 7.2.x, for example, would have very little to do except once in a great while if and when a serious bug is found. At that point, that gatekeeper can make a release (the current process is too tied up in people, IMO, and that includes the RPM mechanism (which I have every right to criticize!)). -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: Release planning (was: Re: [HACKERS] Status report)
Marc G. Fournier wrote: > > Note that I'm all for a long (2 year? or even 'indefinite') development > cycle for a major release if we continue on with what has been happening > more often lately, where stuff is back patched to the last stable release > ... there is alot of stuff that doesn't require a reload of the database > (or initdb) that could very easily be backpatched ... > > As Jan points out, its the 'small features that are done' that we've been > griping about having to wait for, not the big ones which we know aren't > done ... > > The nice thing about doing something lke that is those small features > would get a degree of testing happening in a live environment ... Of course that last sentence is the downside too --- people don't want to have to retest their setups after a minor release upgrade. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: Release planning (was: Re: [HACKERS] Status report)
Note that I'm all for a long (2 year? or even 'indefinite') development cycle for a major release if we continue on with what has been happening more often lately, where stuff is back patched to the last stable release ... there is alot of stuff that doesn't require a reload of the database (or initdb) that could very easily be backpatched ... As Jan points out, its the 'small features that are done' that we've been griping about having to wait for, not the big ones which we know aren't done ... The nice thing about doing something lke that is those small features would get a degree of testing happening in a live environment ... On Tue, 13 Jul 2004, Tom Lane wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: Jan Wieck wrote: I think in the future we have to force all large features, those that probably need more than 6 months of development time, to be properly #ifdef'd. Then it wouldn't be that big of a deal to release more often. Alvaro started out with ifdef's but it got too confusing and we all agreed to just go with a normal patch. He just hits too much code. I think the same would be true of almost any really large feature --- ifdefs all over the code base are just too much of a mess. To be honest I think that "releasing more often" isn't necessarily an appropriate goal for the project anymore. Five or six years ago we were doing a major (initdb-forcing) release every three or four months, and that was appropriate at the time, but the project has matured and our user population has changed. Look at how many people are still using 7.2 or 7.3. One major release a year may be about right now, because you can't get people to adopt new major revs faster than that anyway. Of course this all ties into the pain of having to dump/reload large databases, and maybe if we had working pg_upgrade the adoption rate would be faster, but I'm not at all sure of that. We're playing in a different league now. Big installations tend to want to do significant amounts of compatibility testing before they roll out a new database version. regards, tom lane Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: Release planning (was: Re: [HACKERS] Status report)
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> Of course this all ties into the pain of having to dump/reload large > >> databases, and maybe if we had working pg_upgrade the adoption rate > >> would be faster, but I'm not at all sure of that. We're playing in > >> a different league now. Big installations tend to want to do > >> significant amounts of compatibility testing before they roll out > >> a new database version. > > > I think the only downside to a longer release cycle is that features > > developed would take longer to get out to the public. Perhaps we need > > to start thinking of add-ons to existing releases such as an ARC or > > vacuum-delay add-on to the 7.4.X release. > > Mmm ... for people whose pain-point has to do with compatibility > testing, adding features in minor releases would turn them into major > releases, because they'd still have to wonder whether the new version > would break anything. I think our policy of "only bug fixes in stable > releases" is important to maintain, because it makes it easy for people > to upgrade within a stable release series. > > Also, we do not really have the manpower to test multiple concurrently > developed versions ... When I say add-ons, I am thinking of source code patches that are avaiable as options to the standard subreleases. I agree we don't want to change our current subrelease criteria. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: Release planning (was: Re: [HACKERS] Status report)
Bruce Momjian <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Of course this all ties into the pain of having to dump/reload large >> databases, and maybe if we had working pg_upgrade the adoption rate >> would be faster, but I'm not at all sure of that. We're playing in >> a different league now. Big installations tend to want to do >> significant amounts of compatibility testing before they roll out >> a new database version. > I think the only downside to a longer release cycle is that features > developed would take longer to get out to the public. Perhaps we need > to start thinking of add-ons to existing releases such as an ARC or > vacuum-delay add-on to the 7.4.X release. Mmm ... for people whose pain-point has to do with compatibility testing, adding features in minor releases would turn them into major releases, because they'd still have to wonder whether the new version would break anything. I think our policy of "only bug fixes in stable releases" is important to maintain, because it makes it easy for people to upgrade within a stable release series. Also, we do not really have the manpower to test multiple concurrently developed versions ... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: Release planning (was: Re: [HACKERS] Status report)
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > Jan Wieck wrote: > >> I think in the future we have to force all large features, those that > >> probably need more than 6 months of development time, to be properly > >> #ifdef'd. Then it wouldn't be that big of a deal to release more often. > > > Alvaro started out with ifdef's but it got too confusing and we all > > agreed to just go with a normal patch. He just hits too much code. > > I think the same would be true of almost any really large feature --- > ifdefs all over the code base are just too much of a mess. > > To be honest I think that "releasing more often" isn't necessarily an > appropriate goal for the project anymore. Five or six years ago we were > doing a major (initdb-forcing) release every three or four months, and > that was appropriate at the time, but the project has matured and our > user population has changed. Look at how many people are still using > 7.2 or 7.3. One major release a year may be about right now, because > you can't get people to adopt new major revs faster than that anyway. > > Of course this all ties into the pain of having to dump/reload large > databases, and maybe if we had working pg_upgrade the adoption rate > would be faster, but I'm not at all sure of that. We're playing in > a different league now. Big installations tend to want to do > significant amounts of compatibility testing before they roll out > a new database version. Totally agree. I think the only downside to a longer release cycle is that features developed would take longer to get out to the public. Perhaps we need to start thinking of add-ons to existing releases such as an ARC or vacuum-delay add-on to the 7.4.X release. The patch would potentially have to be recreated for every minor release. I would also like to see some psql message that shows the add-ons added to an official release. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: Release planning (was: Re: [HACKERS] Status report)
Bruce Momjian <[EMAIL PROTECTED]> writes: > Jan Wieck wrote: >> I think in the future we have to force all large features, those that >> probably need more than 6 months of development time, to be properly >> #ifdef'd. Then it wouldn't be that big of a deal to release more often. > Alvaro started out with ifdef's but it got too confusing and we all > agreed to just go with a normal patch. He just hits too much code. I think the same would be true of almost any really large feature --- ifdefs all over the code base are just too much of a mess. To be honest I think that "releasing more often" isn't necessarily an appropriate goal for the project anymore. Five or six years ago we were doing a major (initdb-forcing) release every three or four months, and that was appropriate at the time, but the project has matured and our user population has changed. Look at how many people are still using 7.2 or 7.3. One major release a year may be about right now, because you can't get people to adopt new major revs faster than that anyway. Of course this all ties into the pain of having to dump/reload large databases, and maybe if we had working pg_upgrade the adoption rate would be faster, but I'm not at all sure of that. We're playing in a different league now. Big installations tend to want to do significant amounts of compatibility testing before they roll out a new database version. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Release planning (was: Re: [HACKERS] Status report)
On 7/10/2004 11:02 PM, Bruce Momjian wrote: If you get full control of PostgreSQL, you can dictate what will happen. Until then, I will follow the community consensus, which may or may not match your opinion. Marc isn't the only one who didn't like waiting for more features going into 7.5 two months ago. I agree that it is too late now to push things just out. But the mistake made wasn't ours. What this repeated discussion about release plans and schedules (and we have it every release) indicates to me is that we might miss something. As you pointed out elsewhere, a 9-12 month development cycle just isn't enough to get those big features done. But I think that stretching the release cycle to two years and holding back all the smaller features as well isn't a good idea either. I think in the future we have to force all large features, those that probably need more than 6 months of development time, to be properly #ifdef'd. Then it wouldn't be that big of a deal to release more often. Jan --- Marc G. Fournier wrote: On Sat, 10 Jul 2004, Bruce Momjian wrote: > > My point is that it isn't the patch submitters we are waiting on > anymore. It is the backlog of patches that need review/adjustment. "Of course, many of the patches I kept need some adjustment to get applied ..." ... to me, that indicates that even after review, they will have to go back to the submitter to be adjusted, and sent back, and reviewed, and ... Get in what you feel comfortable adding over the next week, the rest can wait until 7.6 ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: Release planning (was: Re: [HACKERS] Status report)
Jan Wieck wrote: > On 7/10/2004 11:02 PM, Bruce Momjian wrote: > > > If you get full control of PostgreSQL, you can dictate what will happen. > > Until then, I will follow the community consensus, which may or may not > > match your opinion. > > Marc isn't the only one who didn't like waiting for more features going > into 7.5 two months ago. I agree that it is too late now to push things > just out. But the mistake made wasn't ours. > > What this repeated discussion about release plans and schedules (and we > have it every release) indicates to me is that we might miss something. > As you pointed out elsewhere, a 9-12 month development cycle just isn't > enough to get those big features done. But I think that stretching the > release cycle to two years and holding back all the smaller features as > well isn't a good idea either. > > I think in the future we have to force all large features, those that > probably need more than 6 months of development time, to be properly > #ifdef'd. Then it wouldn't be that big of a deal to release more often. Alvaro started out with ifdef's but it got too confusing and we all agreed to just go with a normal patch. He just hits too much code. PITR could be done that way though. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report
Justin Clift wrote: Bruce Momjian wrote: If you get full control of PostgreSQL, you can dictate what will happen. Until then, I will follow the community consensus, which may or may not match your opinion. Um, let's take the time to get the features in, otherwise we'll be waiting another year (roughly) to get PITR and others out to end users. We can't affort not having PITR or NT in 7.5; we announced it already everywhere. But it's not a real problem if we need some more time to release, we always tell "we don't release according to a release plan, but if things are mature". Regards, Andreas ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Status report
Bruce Momjian wrote: If you get full control of PostgreSQL, you can dictate what will happen. Until then, I will follow the community consensus, which may or may not match your opinion. Um, let's take the time to get the features in, otherwise we'll be waiting another year (roughly) to get PITR and others out to end users. :) Regards and best wishes, Justin Clift ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report
If you get full control of PostgreSQL, you can dictate what will happen. Until then, I will follow the community consensus, which may or may not match your opinion. --- Marc G. Fournier wrote: > On Sat, 10 Jul 2004, Bruce Momjian wrote: > > > > > My point is that it isn't the patch submitters we are waiting on > > anymore. It is the backlog of patches that need review/adjustment. > > "Of course, many of the patches I kept need some adjustment to get applied > ..." ... to me, that indicates that even after review, they will have to > go back to the submitter to be adjusted, and sent back, and reviewed, and > ... > > Get in what you feel comfortable adding over the next week, the rest can > wait until 7.6 ... > > > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 > -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Status report
On Sat, 10 Jul 2004, Bruce Momjian wrote: My point is that it isn't the patch submitters we are waiting on anymore. It is the backlog of patches that need review/adjustment. "Of course, many of the patches I kept need some adjustment to get applied ..." ... to me, that indicates that even after review, they will have to go back to the submitter to be adjusted, and sent back, and reviewed, and ... Get in what you feel comfortable adding over the next week, the rest can wait until 7.6 ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report
My point is that it isn't the patch submitters we are waiting on anymore. It is the backlog of patches that need review/adjustment. --- Marc G. Fournier wrote: > On Sat, 10 Jul 2004, Bruce Momjian wrote: > > > I am still going through my mailbox, trying to address all the open > > patches so we can move toward beta. Of course, many of the patches I > > kept need some adjustment to get applied (e.g. configuration file > > location) so it is slow going. > > > > However, we still have PITR unapplied, autovacuum unapplied, and nested > > transactions don't have savepoint support, so we are still a while away > > from beta. I think we should start thinking of beta as August 1 rather > > than mid-July. > > And so we fall into the hole we dig for ourselves with each release, where > the beta period takes almost as long as the development period itself ... > originally, this was all supposed to be done for June 1st, but we pushed > it back to July 1st since ppl only needed "another month" ... then, July > 1st, we did the feature freeze, but because so much was left to be > applied, we decided to leave bundling first beta until mid-July ... > > If six extra weeks hasn't been enough, I have little faith that adding > 'yet another 2 weeks' is going to be enough either ... leave them for 7.6 > then ... > > > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 > -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Status report
On Sat, 10 Jul 2004, Bruce Momjian wrote: I am still going through my mailbox, trying to address all the open patches so we can move toward beta. Of course, many of the patches I kept need some adjustment to get applied (e.g. configuration file location) so it is slow going. However, we still have PITR unapplied, autovacuum unapplied, and nested transactions don't have savepoint support, so we are still a while away from beta. I think we should start thinking of beta as August 1 rather than mid-July. And so we fall into the hole we dig for ourselves with each release, where the beta period takes almost as long as the development period itself ... originally, this was all supposed to be done for June 1st, but we pushed it back to July 1st since ppl only needed "another month" ... then, July 1st, we did the feature freeze, but because so much was left to be applied, we decided to leave bundling first beta until mid-July ... If six extra weeks hasn't been enough, I have little faith that adding 'yet another 2 weeks' is going to be enough either ... leave them for 7.6 then ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report
I am still going through my mailbox, trying to address all the open patches so we can move toward beta. Of course, many of the patches I kept need some adjustment to get applied (e.g. configuration file location) so it is slow going. However, we still have PITR unapplied, autovacuum unapplied, and nested transactions don't have savepoint support, so we are still a while away from beta. I think we should start thinking of beta as August 1 rather than mid-July. Plus a bunch of other minor patches :) Chris ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] Status report
I am still going through my mailbox, trying to address all the open patches so we can move toward beta. Of course, many of the patches I kept need some adjustment to get applied (e.g. configuration file location) so it is slow going. However, we still have PITR unapplied, autovacuum unapplied, and nested transactions don't have savepoint support, so we are still a while away from beta. I think we should start thinking of beta as August 1 rather than mid-July. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Status report: regex replacement
Tatsuo Ishii writes: > > UTF-8 seems to be the most popular, but even XML standard requires all > > compliant implementations to deal with at least both UTF-8 and UTF-16. > > I don't think PostgreSQL is going to natively support UTF-16. At FOSDEM it was claimed that Windows natively uses UCS-2, and there are also continuing rumours that the Java Unicode encoding is not quite UTF-8, so there is going to be a certain pressure to support other Unicode encodings besides UTF-8. As for the names, the SQL standard defines most of those. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Status report: regex replacement
Peter Eisentraut <[EMAIL PROTECTED]> writes:
> Tom Lane writes:
>> code is concerned: the regex library actually offers three regex
>> flavors, "advanced", "extended", and "basic", where "extended" matches
>> what we had before ("extended" and "basic" correspond to different
>> levels of the POSIX 1003.2 standard). We just need a way to expose
>> that knob to the user. I am thinking about inventing yet another GUC
>> parameter, say
> Perhaps it should be exposed through different operators. If someone uses
> packages (especially functions) provided externally, they might have a
> hard time coordinating what flavor is required by which part of what he is
> using.
But one could argue the contrary, too: if you've got an
externally-provided package there may be no convenient way to get it to
use, say, ~!#@ in place of ~. GUC variables can come in awfully handy
in scenarios like that.
Also, if one *can* alter the SQL context in which a regexp is used, there
is a solution already provided by Spencer's "regex metasyntax" hack --- see
http://developer.postgresql.org/docs/postgres/functions-matching.html#POSIX-METASYNTAX
That is, one could write something like
foo ~ ('(?b)' || basic_regex_expression)
to force basic_regex_expression to be taken as a BRE and not the
extended syntax. This is a tad uglier than changing the operator name,
perhaps, but it has some advantages too --- for one, the option could be
plugged into the string further upstream than where the SQL syntax is
determined.
Basically I think the flavor-as-GUC-variable approach is orthogonal to
inventing some new operator names. We could do the latter too, but
I don't really see a need for it given the metasyntax feature.
regards, tom lane
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Status report: regex replacement
Tom Lane writes:
> code is concerned: the regex library actually offers three regex
> flavors, "advanced", "extended", and "basic", where "extended" matches
> what we had before ("extended" and "basic" correspond to different
> levels of the POSIX 1003.2 standard). We just need a way to expose
> that knob to the user. I am thinking about inventing yet another GUC
> parameter, say
Perhaps it should be exposed through different operators. If someone uses
packages (especially functions) provided externally, they might have a
hard time coordinating what flavor is required by which part of what he is
using.
By analogy, imagine there was an environment variable that switched all
grep's to egrep's. That would be a complete mess.
--
Peter Eisentraut [EMAIL PROTECTED]
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] Status report: regex replacement
Tatsuo Ishii kirjutas R, 07.02.2003 kell 04:03: > > UTF-8 seems to be the most popular, but even XML standard requires all > > compliant implementations to deal with at least both UTF-8 and UTF-16. > > I don't think PostgreSQL is going to natively support UTF-16. By natively, do you mean "as backend storage" format or "as supported client encoding" ? -- Hannu Krosing <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Status report: regex replacement
On Fri, 7 Feb 2003 00:49, Hannu Krosing wrote: > Tatsuo Ishii kirjutas N, 06.02.2003 kell 17:05: > > > Perhaps we should not call the encoding UNICODE but UTF8 (which it > > > really is). UNICODE is a character set which has half a dozen official > > > encodings and calling one of them "UNICODE" does not make things very > > > clear. > > > > Right. Also we perhaps should call LATIN1 or ISO-8859-1 more precisely > > way since ISO-8859-1 can be encoded in either 7 bit or 8 bit(we use > > this). I don't know what it is called though. > > I don't think that calling 8-bit ISO-8859-1 ISO-8859-1 can confuse > anybody, but UCS-2 (ISO-10646-1), UTF-8 and UTF-16 are all widely used. > > UTF-8 seems to be the most popular, but even XML standard requires all > compliant implementations to deal with at least both UTF-8 and UTF-16. Strong agreement from me, for whatever value you wish to place on my opinion. UTF-8 is a preferable name to UNICODE. The case for distinguishing 7-bit from 8-bit latin1 seems much weaker. Tim -- --- Tim Allen [EMAIL PROTECTED] Proximity Pty Ltd http://www.proximity.com.au/ http://www4.tpg.com.au/users/rita_tim/ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Status report: regex replacement
> > Right. Also we perhaps should call LATIN1 or ISO-8859-1 more precisely > > way since ISO-8859-1 can be encoded in either 7 bit or 8 bit(we use > > this). I don't know what it is called though. > > I don't think that calling 8-bit ISO-8859-1 ISO-8859-1 can confuse > anybody, but UCS-2 (ISO-10646-1), UTF-8 and UTF-16 are all widely used. I just pointed out that ISO-8859-1 is *not* an encoding, but a character set. > UTF-8 seems to be the most popular, but even XML standard requires all > compliant implementations to deal with at least both UTF-8 and UTF-16. I don't think PostgreSQL is going to natively support UTF-16. -- Tatsuo Ishii ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Status report: regex replacement
Tatsuo Ishii kirjutas N, 06.02.2003 kell 17:05: > > Perhaps we should not call the encoding UNICODE but UTF8 (which it > > really is). UNICODE is a character set which has half a dozen official > > encodings and calling one of them "UNICODE" does not make things very > > clear. > > Right. Also we perhaps should call LATIN1 or ISO-8859-1 more precisely > way since ISO-8859-1 can be encoded in either 7 bit or 8 bit(we use > this). I don't know what it is called though. I don't think that calling 8-bit ISO-8859-1 ISO-8859-1 can confuse anybody, but UCS-2 (ISO-10646-1), UTF-8 and UTF-16 are all widely used. UTF-8 seems to be the most popular, but even XML standard requires all compliant implementations to deal with at least both UTF-8 and UTF-16. -- Hannu Krosing <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Status report: regex replacement
> Perhaps we should not call the encoding UNICODE but UTF8 (which it > really is). UNICODE is a character set which has half a dozen official > encodings and calling one of them "UNICODE" does not make things very > clear. Right. Also we perhaps should call LATIN1 or ISO-8859-1 more precisely way since ISO-8859-1 can be encoded in either 7 bit or 8 bit(we use this). I don't know what it is called though. -- Tatsuo Ishii ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Status report: regex replacement
On Thu, 2003-02-06 at 13:25, Tatsuo Ishii wrote: > > I have just committed the latest version of Henry Spencer's regex > > package (lifted from Tcl 8.4.1) into CVS HEAD. This code is natively > > able to handle wide characters efficiently, and so it avoids the > > multibyte performance problems recently exhibited by Wade Klaver. > > I have not done extensive performance testing, but the new code seems > > at least as fast as the old, and much faster in some cases. > > I have tested the new regex with src/test/mb and it all passed. So the > new code looks safe at least for EUC_CN, EUC_JP, EUC_KR, EUC_TW, > MULE_INTERNAL, UNICODE, though the test does not include all possible > regex patterns. Perhaps we should not call the encoding UNICODE but UTF8 (which it really is). UNICODE is a character set which has half a dozen official encodings and calling one of them "UNICODE" does not make things very clear. -- Hannu Krosing <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Status report: regex replacement
> I have just committed the latest version of Henry Spencer's regex > package (lifted from Tcl 8.4.1) into CVS HEAD. This code is natively > able to handle wide characters efficiently, and so it avoids the > multibyte performance problems recently exhibited by Wade Klaver. > I have not done extensive performance testing, but the new code seems > at least as fast as the old, and much faster in some cases. I have tested the new regex with src/test/mb and it all passed. So the new code looks safe at least for EUC_CN, EUC_JP, EUC_KR, EUC_TW, MULE_INTERNAL, UNICODE, though the test does not include all possible regex patterns. -- Tatsuo Ishii ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Status report: regex replacement
Christopher Kings-Lynne kirjutas N, 06.02.2003 kell 03:56: > > > set regex_flavor = advanced > > > set regex_flavor = extended > > > set regex_flavor = basic > > [snip] > > > Any suggestions about the name of the parameter? > > > > Actually I think 'regex_flavor' sounds fine. > > Not more Americanisms in our config files!! :P Maybe support both, like for ANALYZE/ANALYSE ? While at it, could we make another variant - ANALÜÜSI - which would be native for me ;) -- Hannu Krosing <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Status report: regex replacement
"Christopher Kings-Lynne" <[EMAIL PROTECTED]> writes: >> You want regex_flavour? ;-) > Hehe - yeah I don't really care. I have to use 'color' often enough > accessing 100% of the world's programming APIs... > How about regex_type, regex_mode, regex_option, etc.? ;) Well, I used "flavor" in my previous message because Friedl uses that term consistently in his book to refer to the various idiosyncracies of different regex implementations. It seems like a good term to me; the differences among them don't rise to the level of being different languages, yet they're distinctly different. type, mode, option all seem, um, colourless ... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Status report: regex replacement
> "Christopher Kings-Lynne" <[EMAIL PROTECTED]> writes: > >> Actually I think 'regex_flavor' sounds fine. > > > Not more Americanisms in our config files!! :P > > You want regex_flavour? ;-) Hehe - yeah I don't really care. I have to use 'color' often enough accessing 100% of the world's programming APIs... How about regex_type, regex_mode, regex_option, etc.? ;) Chris ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Status report: regex replacement
"Christopher Kings-Lynne" <[EMAIL PROTECTED]> writes: >> Actually I think 'regex_flavor' sounds fine. > Not more Americanisms in our config files!! :P You want regex_flavour? ;-) regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Status report: regex replacement
> > set regex_flavor = advanced > > set regex_flavor = extended > > set regex_flavor = basic > [snip] > > Any suggestions about the name of the parameter? > > Actually I think 'regex_flavor' sounds fine. Not more Americanisms in our config files!! :P Chris ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Status report: regex replacement
On Wed, 5 Feb 2003, Tom Lane wrote:
> 1. There are a couple of minor incompatibilities between the "advanced"
> regex syntax implemented by this package and the syntax handled by our
> old code; in particular, backslash is now a special character within
> bracket expressions. It seems to me that we'd better offer a switch
> to allow backwards compatibility. This is easily done as far as the
> code is concerned: the regex library actually offers three regex
> flavors, "advanced", "extended", and "basic", where "extended" matches
> what we had before ("extended" and "basic" correspond to different
> levels of the POSIX 1003.2 standard). We just need a way to expose
> that knob to the user. I am thinking about inventing yet another GUC
> parameter, say
>
> set regex_flavor = advanced
> set regex_flavor = extended
> set regex_flavor = basic
[snip]
> Any suggestions about the name of the parameter?
Actually I think 'regex_flavor' sounds fine.
Jon
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
[HACKERS] Status report: regex replacement
I have just committed the latest version of Henry Spencer's regex
package (lifted from Tcl 8.4.1) into CVS HEAD. This code is natively
able to handle wide characters efficiently, and so it avoids the
multibyte performance problems recently exhibited by Wade Klaver.
I have not done extensive performance testing, but the new code seems
at least as fast as the old, and much faster in some cases.
Also, we now have a regex flavor that is an exact match for recent
Tcl releases and a close match for recent Perl releases; it sports
back references and lookahead among other niceties.
There's some stuff still to do:
1. There are a couple of minor incompatibilities between the "advanced"
regex syntax implemented by this package and the syntax handled by our
old code; in particular, backslash is now a special character within
bracket expressions. It seems to me that we'd better offer a switch
to allow backwards compatibility. This is easily done as far as the
code is concerned: the regex library actually offers three regex
flavors, "advanced", "extended", and "basic", where "extended" matches
what we had before ("extended" and "basic" correspond to different
levels of the POSIX 1003.2 standard). We just need a way to expose
that knob to the user. I am thinking about inventing yet another GUC
parameter, say
set regex_flavor = advanced
set regex_flavor = extended
set regex_flavor = basic
We could satisfy the immediate need with just a boolean "advanced_regex
= on/off", but it seems forward-looking to allow for the possibility of
more flavors in future. (For one thing, this would offer an easy place
to select a different regex package, in case anyone still wants to play
around with sre or the other alternatives that were mentioned
yesterday.)
Any suggestions about the name of the parameter?
2. Documentation. I've transformed Spencer's manual page into SGML and
added it to func.sgml, but it's starting to look a tad, um, bulky:
http://developer.postgresql.org/docs/postgres/functions-matching.html#FUNCTIONS-POSIX-REGEXP
The regex section now accounts for 1200+ out of func.sgml's 7500 lines.
Should it be split out as an appendix, or is it okay where it is?
3. I've been toying with the idea of getting rid of the special-purpose
matching code for LIKE (see adt/like.c and like_match.c), and
reimplementing LIKE as a front-end to the regex engine; all it would
need is to translate LIKE patterns into regex style, much as we already
do for SQL99's SIMILAR TO patterns. This would reduce the maintenance
needs for LIKE by a great deal. In some preliminary tests here, it
seemed that the special-purpose LIKE code is faster than equivalent
regex matching would be --- but I didn't try the multibyte code path,
nor any but the simplest of patterns. Anyone want to try some more
extensive benchmarking?
4. The new regex code is 8-bit-clean (no dependency on null-terminated
strings), so it'd be feasible to implement regex matching for BYTEA.
Over to you on that one, Joe.
regards, tom lane
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
[HACKERS] status report
With my pg_hba.conf changes done, I am now focusing in the next few days on clearing out my email/patch application backlog. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
