Re: [HACKERS] Why isn't stats_temp_directory automatically created?
I assume we decided we didn't want this. --- Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: Here is the revised patch; If stats_temp_directory indicates the symlink, we pursue the chain of symlinks and create the referenced directory. I looked at this patch a bit. I'm still entirely unconvinced that we should be doing this at all --- if the directory is not there, there's a significant probability that there's something wrong that is beyond the backend's ability to understand or correct. However, even ignoring that objection, the patch is not ready to commit for a number of reasons: * Repeating the operation every time the stats file is written doesn't seem like a particularly good idea; it eats cycles, and if the directory disappears during live operation then there is *definitely* something fishy going on. Can't we fix it so that the work is only done when the path setting changes? (In principle you could do it in assign_pgstat_temp_directory(), but I think something would be needed to ensure that only the stats collector process actually tries to create the directory. Or maybe it would be simplest to try to run the code only when we get a failure from trying to create the stats temp file.) * I don't think the mkdir_p code belongs in fd.c. It looks like you copied-and-pasted it from initdb.c, which isn't any good either; we don't want to maintain multiple copies of this. Maybe a new src/port/ file is indicated. * elog(LOG) is not exactly an adequate response if the final chdir fails --- you have just broken the process beyond recovery. That alone may be sufficient reason to reject the attempt to deal with symlinks. As far as pgstat_temp_directory is concerned, I'm not sure of the point of making the GUC point to a symlink anyway --- if you have a GUC why not just point it where you want the directory to be? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Bruce Momjian br...@momjian.us writes: I assume we decided we didn't want this. I thought the risk/reward ratio was pretty bad. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: I assume we decided we didn't want this. I thought the risk/reward ratio was pretty bad. Yea, the symlink issue killed it for me. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: Here is the revised patch; If stats_temp_directory indicates the symlink, we pursue the chain of symlinks and create the referenced directory. I looked at this patch a bit. I'm still entirely unconvinced that we should be doing this at all --- if the directory is not there, there's a significant probability that there's something wrong that is beyond the backend's ability to understand or correct. However, even ignoring that objection, the patch is not ready to commit for a number of reasons: * Repeating the operation every time the stats file is written doesn't seem like a particularly good idea; it eats cycles, and if the directory disappears during live operation then there is *definitely* something fishy going on. Can't we fix it so that the work is only done when the path setting changes? (In principle you could do it in assign_pgstat_temp_directory(), but I think something would be needed to ensure that only the stats collector process actually tries to create the directory. Or maybe it would be simplest to try to run the code only when we get a failure from trying to create the stats temp file.) My idea was to have it run when it tries to create the temp file and fails. Seems simpler than in the assign hook. * I don't think the mkdir_p code belongs in fd.c. It looks like you copied-and-pasted it from initdb.c, which isn't any good either; we don't want to maintain multiple copies of this. Maybe a new src/port/ file is indicated. Yes, that's what I thought as well. How about src/port/dirmod.c though - it has rename/unlink/symlink now which seem at least remotely related. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Fujii Masao masao.fu...@gmail.com writes: Here is the revised patch; If stats_temp_directory indicates the symlink, we pursue the chain of symlinks and create the referenced directory. I looked at this patch a bit. I'm still entirely unconvinced that we should be doing this at all --- if the directory is not there, there's a significant probability that there's something wrong that is beyond the backend's ability to understand or correct. However, even ignoring that objection, the patch is not ready to commit for a number of reasons: * Repeating the operation every time the stats file is written doesn't seem like a particularly good idea; it eats cycles, and if the directory disappears during live operation then there is *definitely* something fishy going on. Can't we fix it so that the work is only done when the path setting changes? (In principle you could do it in assign_pgstat_temp_directory(), but I think something would be needed to ensure that only the stats collector process actually tries to create the directory. Or maybe it would be simplest to try to run the code only when we get a failure from trying to create the stats temp file.) * I don't think the mkdir_p code belongs in fd.c. It looks like you copied-and-pasted it from initdb.c, which isn't any good either; we don't want to maintain multiple copies of this. Maybe a new src/port/ file is indicated. * elog(LOG) is not exactly an adequate response if the final chdir fails --- you have just broken the process beyond recovery. That alone may be sufficient reason to reject the attempt to deal with symlinks. As far as pgstat_temp_directory is concerned, I'm not sure of the point of making the GUC point to a symlink anyway --- if you have a GUC why not just point it where you want the directory to be? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Hi, On Tue, Apr 21, 2009 at 4:33 PM, Fujii Masao masao.fu...@gmail.com wrote: Here is the revised patch; If stats_temp_directory indicates the symlink, we pursue the chain of symlinks and create the referenced directory. BTW, this patch is useful also as the foundation for improving creation of log_directory. Attached patch fixes the following problem of log_directory by using that fundamental patch. - log_directory is not created when a configuration file is reloaded - creation of log_directory fails if the parent directory of it doesn't exist - if log_directory indicates the symlink, it's not resolved Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center create_log_directory_0422.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Hi, On Mon, Apr 20, 2009 at 1:29 AM, Magnus Hagander mag...@hagander.net wrote: Sorry about the very late response - I've been out of the country and generally busy. Thanks for taking the time to comment! On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander mag...@hagander.net wrote: This does not take into account the effect of symlinks as mentioned by Itakagi Takahiro. I haven't looked at the details, but I don't think it would be that much more work to deal with it - and as he mentions, this is a very common usecase. Okey, I'll revise the patch; create also the directory which is referenced by symlink if not present. Great. Here is the revised patch; If stats_temp_directory indicates the symlink, we pursue the chain of symlinks and create the referenced directory. Also, wouldn't it be better to isolate this to the first time when we try to create the file - then we don't have to export the symbol? You mean having assign_pgstat_temp_directory() create the directory instead of pgstat_start()? In this case, the directory is created automatically not only at the beginning but also when a configuration file is reloaded. This seems to be better behavior. No, I meant creating it when we open the file - in pgstat_write_statsfile(). OK, I changed the patch so. Thanks. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center create_stats_temp_dir_0421.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Fujii Masao wrote: Hi, Hi! Sorry about the very late response - I've been out of the country and generally busy. On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander mag...@hagander.net wrote: This does not take into account the effect of symlinks as mentioned by Itakagi Takahiro. I haven't looked at the details, but I don't think it would be that much more work to deal with it - and as he mentions, this is a very common usecase. Okey, I'll revise the patch; create also the directory which is referenced by symlink if not present. Great. Also, wouldn't it be better to isolate this to the first time when we try to create the file - then we don't have to export the symbol? You mean having assign_pgstat_temp_directory() create the directory instead of pgstat_start()? In this case, the directory is created automatically not only at the beginning but also when a configuration file is reloaded. This seems to be better behavior. No, I meant creating it when we open the file - in pgstat_write_statsfile(). //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Euler Taveira de Oliveira eu...@timbira.com wrote: Fujii Masao escreveu: Is it worth making the patch which creates stats_temp_directory if not present? +1. +1, but AFAIK stats_temp_directory was designed to symlink to a RAM drive. Even if stats_temp_directory exists as a symbolic link, the destination directory might be lost in such a situation. If you try to make servers more robust, you might also need to consider broken symlinks. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Hi, On Tue, Apr 14, 2009 at 10:26 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: Fujii Masao escreveu: Is it worth making the patch which creates stats_temp_directory if not present? +1. Here is the patch. This patch should be added to CommitFest-2009-First?, or committed before 8.4 release? The patch is very small, so I don't think that it'll block 8.4 release. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center Index: src/backend/postmaster/pgstat.c === RCS file: /projects/cvsroot/pgsql/src/backend/postmaster/pgstat.c,v retrieving revision 1.187 diff -c -r1.187 pgstat.c *** src/backend/postmaster/pgstat.c 1 Jan 2009 17:23:46 - 1.187 --- src/backend/postmaster/pgstat.c 15 Apr 2009 06:08:04 - *** *** 111,116 --- 111,117 bool pgstat_track_counts = false; int pgstat_track_functions = TRACK_FUNC_OFF; int pgstat_track_activity_query_size = 1024; + char *pgstat_temp_directory; /* -- * Built from GUC parameter *** *** 589,594 --- 590,600 return 0; /* + * Create temporary statistics directory if not present; ignore errors + */ + mkdir(pgstat_temp_directory, 0700); + + /* * Do nothing if too soon since last collector start. This is a safety * valve to protect against continuous respawn attempts if the collector * is dying immediately at launch. Note that since we will be re-called Index: src/backend/utils/misc/guc.c === RCS file: /projects/cvsroot/pgsql/src/backend/utils/misc/guc.c,v retrieving revision 1.502 diff -c -r1.502 guc.c *** src/backend/utils/misc/guc.c 7 Apr 2009 23:27:34 - 1.502 --- src/backend/utils/misc/guc.c 15 Apr 2009 06:08:18 - *** *** 375,382 char *IdentFileName; char *external_pid_file; - char *pgstat_temp_directory; - int tcp_keepalives_idle; int tcp_keepalives_interval; int tcp_keepalives_count; --- 375,380 Index: src/include/pgstat.h === RCS file: /projects/cvsroot/pgsql/src/include/pgstat.h,v retrieving revision 1.82 diff -c -r1.82 pgstat.h *** src/include/pgstat.h 4 Jan 2009 22:19:59 - 1.82 --- src/include/pgstat.h 15 Apr 2009 06:08:22 - *** *** 593,598 --- 593,599 extern bool pgstat_track_counts; extern int pgstat_track_functions; extern PGDLLIMPORT int pgstat_track_activity_query_size; + extern PGDLLIMPORT char *pgstat_temp_directory; extern char *pgstat_stat_tmpname; extern char *pgstat_stat_filename; -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Fujii Masao wrote: Hi, On Tue, Apr 14, 2009 at 10:26 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: Fujii Masao escreveu: Is it worth making the patch which creates stats_temp_directory if not present? +1. Here is the patch. This patch should be added to CommitFest-2009-First?, or committed before 8.4 release? The patch is very small, so I don't think that it'll block 8.4 release. I think the fix should go into 8.4 - this is a fix for a new feature. However, a couple of comments: This does not take into account the effect of symlinks as mentioned by Itakagi Takahiro. I haven't looked at the details, but I don't think it would be that much more work to deal with it - and as he mentions, this is a very common usecase. Also, wouldn't it be better to isolate this to the first time when we try to create the file - then we don't have to export the symbol? //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Hi, On Wed, Apr 15, 2009 at 5:37 PM, Magnus Hagander mag...@hagander.net wrote: This does not take into account the effect of symlinks as mentioned by Itakagi Takahiro. I haven't looked at the details, but I don't think it would be that much more work to deal with it - and as he mentions, this is a very common usecase. Okey, I'll revise the patch; create also the directory which is referenced by symlink if not present. Also, wouldn't it be better to isolate this to the first time when we try to create the file - then we don't have to export the symbol? You mean having assign_pgstat_temp_directory() create the directory instead of pgstat_start()? In this case, the directory is created automatically not only at the beginning but also when a configuration file is reloaded. This seems to be better behavior. One problem of this is that some directories may be created unexpectedly if stats_temp_directory is specified at multiple locations. It's because assign_hook is called for each setting. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why isn't stats_temp_directory automatically created?
Fujii Masao escreveu: Is it worth making the patch which creates stats_temp_directory if not present? +1. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
