Re: [naviserver-devel] NaviServer 4.99.25 available
thank you
On Wed, Jun 14, 2023 at 1:19 PM Gustaf Neumann wrote:
> Dear Sassy,
>
> You are referring to a change that was released more than 3 years ago
> (4.99.19).
>
> Log file sanitizing works as expected, namely it prints invisible
> characters in a hex representation (in your case tab characters).
> Sanitizing was required by security audits, since otherwise, it would be
> possible to execute code by looking into the log file, one could obfuscate
> the log file, and confuse log file analyzer that alarm when suspect
> activities are noticed, etc.
>
> Certainly, the harm caused by a tab character alone is limited, but when
> debugging and analyze problems, it is often important to distinguish
> between a tab and some spaces.
>
> You can easily replace the hex notation in the log file via "sed", such as
> e.g.
>
>cat log/error.log | sed -En 's/\\x09/\t/p' | more
>
> or the like.
>
> all the best
>
> -gn
> On 13.06.23 20:51, Sassy Natan wrote:
>
> Hi All,
>
> After upgrading my server to the latest version, my log file is broken.
>
> For example I see UTF-8 as special characters.
>
> I check the readme but the sanitizelogfiles 2 doesn't seems to work as
> expected.
> I have in my configuration:
>
> setdebugtrue
> ns_logctl severity "Debug(sql)"on
>
> Any ideas?
>
> Here is example:
> :],is_break_pcols)) as hrs_wo_break
> :FROM wt_payroll_analysis_pp
> : join (select sid,agg_array(case when is_break_pcol=true then 9
> else 0 end) as is_break_pcols
> :\x09\x09\x09from ( select * from wt_et_cols_defs where sid=41
> order by pcol_number )pc_defs group by sid) pdefs using (sid)
> :\x09\x09\x09\x09WHERE sid=41 and employee_id =40599
> :\x09\x09\x09\x09and to_date(pp_year||'-'||pp||'-01','-mm-dd')
> between to_date('2023-6-01','-mm-dd')- interval '6 month'
> :\x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','-mm-dd')
> - interval '1 day' )::date
> :\x09\x09\x09\x09UNION
> :\x09\x09\x09\x09 select
> sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','-mm-dd')
> as day,null2zero(hrs_wo_break)*3600
> :\x09\x09\x09\x09from pp_pa_table
> :\x09\x09\x09\x09) ggg
> :\x09\x09\x09\x09group by sid,employee_id,pp,pp_year
>
>
> Thanks
> Sassy
>
> On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann wrote:
>
>> Dear all,
>>
>> I am glad to announce that the release of NaviServer 4.99.25 is
>> available at SourceForge [1]. This release is mostly a bug-fix
>> release. The forthcoming version 5.0 of NaviServer will contain
>> several new features omitted in this bug-fix branch. In case, you are
>> building NaviServer from the Bitbucket repository, please note that
>> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
>> the NaviServer 4.99 family). New development happens in the "main"
>> branch of the repository, leading to NaviServer 5.*.
>>
>> See below for a summary of the changes.
>>
>> Many thanks to the contributors of this release:
>>
>> Andrew Piskorski
>> Antonio Pisano
>> Brian Fenton
>> Gustaf Neumann
>> Hector Romojaro
>> Joe Oldak
>> Khy Huang
>> Oleg Oleinick
>> Zoran Vasiljevic
>>
>> All the best!
>>
>> -gustaf neumann
>>
>> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>>
>> ===
>> NaviServer 4.99.25, released 2023-05-01
>> ===
>>
>> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>>
>> New Features:
>> -
>>
>> - Added meta-information to configuration values
>>
>>NaviServer can now report, what configuration values provided in
>>the configuration file were actually used, what their default
>>values are, and whether these values were specified or not (using
>>the default values). With this information, administration
>>(e.g. migration) becomes easier. The NaviServer module "nsstats"
>>shows this meta information via the web interface.
>>
>>This functionality is provided via the new option "-filter" for the
>>command "ns_configsection ... section". When the "-filter" option
>>is used, different kinds of information about the parameters is
>>returned from the specified section.
>>
>> "-filter unread":
>>
>> Returns the parameters, which were set during configuration (i.e.,
>> in the configuration file) but which were not read in the startup
>> phase of the server. This option is useful to determine
>> e.g. typographical errors of specified parameter names.
>>
>> "-filter defaulted":
>>
>> Returns the parameters, from which the default values were read
>> (i.e., which were not explicitly set)
>>
>> "-filter defaults":
>>
>> Returns the defaults of the parameter. This is useful for
>> contrasting the actual values with the default
Re: [naviserver-devel] NaviServer 4.99.25 available
Dear Sassy,
You are referring to a change that was released more than 3 years ago
(4.99.19).
Log file sanitizing works as expected, namely it prints invisible
characters in a hex representation (in your case tab characters).
Sanitizing was required by security audits, since otherwise, it would
be possible to execute code by looking into the log file, one could
obfuscate the log file, and confuse log file analyzer that alarm when
suspect activities are noticed, etc.
Certainly, the harm caused by a tab character alone is limited, but when
debugging and analyze problems, it is often important to distinguish
between a tab and some spaces.
You can easily replace the hex notation in the log file via "sed", such
as e.g.
cat log/error.log | sed -En 's/\\x09/\t/p' | more
or the like.
all the best
-gn
On 13.06.23 20:51, Sassy Natan wrote:
Hi All,
After upgrading my server to the latest version, my log file is broken.
For example I see UTF-8 as special characters.
I check the readme but the sanitizelogfiles 2 doesn't seems to work as
expected.
I have in my configuration:
set debug true
ns_logctl severity "Debug(sql)" on
Any ideas?
Here is example:
: ],is_break_pcols)) as hrs_wo_break
: FROM wt_payroll_analysis_pp
: join (select sid,agg_array(case when is_break_pcol=true then
9 else 0 end) as is_break_pcols
: \x09\x09\x09from ( select * from wt_et_cols_defs where sid=41
order by pcol_number )pc_defs group by sid) pdefs using (sid)
: \x09\x09\x09\x09WHERE sid=41 and employee_id =40599
: \x09\x09\x09\x09and
to_date(pp_year||'-'||pp||'-01','-mm-dd') between
to_date('2023-6-01','-mm-dd')- interval '6 month'
: \x09\x09\x09\x09\x09\x09\x09and (
to_date('2023-6-01','-mm-dd') - interval '1 day' )::date
: \x09\x09\x09\x09UNION
: \x09\x09\x09\x09 select
sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','-mm-dd')
as day,null2zero(hrs_wo_break)*3600
: \x09\x09\x09\x09 from pp_pa_table
: \x09\x09\x09\x09) ggg
: \x09\x09\x09\x09group by sid,employee_id,pp,pp_year
Thanks
Sassy
On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann wrote:
Dear all,
I am glad to announce that the release of NaviServer 4.99.25 is
available at SourceForge [1]. This release is mostly a bug-fix
release. The forthcoming version 5.0 of NaviServer will contain
several new features omitted in this bug-fix branch. In case, you are
building NaviServer from the Bitbucket repository, please note that
the release 4.99.25 is in the branch release/4.99 (bug fix branch for
the NaviServer 4.99 family). New development happens in the "main"
branch of the repository, leading to NaviServer 5.*.
See below for a summary of the changes.
Many thanks to the contributors of this release:
Andrew Piskorski
Antonio Pisano
Brian Fenton
Gustaf Neumann
Hector Romojaro
Joe Oldak
Khy Huang
Oleg Oleinick
Zoran Vasiljevic
All the best!
-gustaf neumann
[1]
https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
===
NaviServer 4.99.25, released 2023-05-01
===
132 files changed, 3957 insertions(+), 2068 deletions(-)
New Features:
-
- Added meta-information to configuration values
NaviServer can now report, what configuration values provided in
the configuration file were actually used, what their default
values are, and whether these values were specified or not (using
the default values). With this information, administration
(e.g. migration) becomes easier. The NaviServer module "nsstats"
shows this meta information via the web interface.
This functionality is provided via the new option "-filter" for the
command "ns_configsection ... section". When the "-filter" option
is used, different kinds of information about the parameters is
returned from the specified section.
"-filter unread":
Returns the parameters, which were set during configuration (i.e.,
in the configuration file) but which were not read in the startup
phase of the server. This option is useful to determine
e.g. typographical errors of specified parameter names.
"-filter defaulted":
Returns the parameters, from which the default values were read
(i.e., which were not explicitly set)
"-filter defaults":
Returns the defaults of the parameter. This is useful for
contrasting the actual values with the default values for
Parameters, e.g. in a web based interface.
- ns_set reform (per default deactivated in
Re: [naviserver-devel] NaviServer 4.99.25 available
Hi All,
After upgrading my server to the latest version, my log file is broken.
For example I see UTF-8 as special characters.
I check the readme but the sanitizelogfiles 2 doesn't seems to work as
expected.
I have in my configuration:
setdebugtrue
ns_logctl severity "Debug(sql)"on
Any ideas?
Here is example:
:],is_break_pcols)) as hrs_wo_break
:FROM wt_payroll_analysis_pp
: join (select sid,agg_array(case when is_break_pcol=true then 9
else 0 end) as is_break_pcols
:\x09\x09\x09from ( select * from wt_et_cols_defs where sid=41
order by pcol_number )pc_defs group by sid) pdefs using (sid)
:\x09\x09\x09\x09WHERE sid=41 and employee_id =40599
:\x09\x09\x09\x09and to_date(pp_year||'-'||pp||'-01','-mm-dd')
between to_date('2023-6-01','-mm-dd')- interval '6 month'
:\x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','-mm-dd') -
interval '1 day' )::date
:\x09\x09\x09\x09UNION
:\x09\x09\x09\x09 select
sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','-mm-dd')
as day,null2zero(hrs_wo_break)*3600
:\x09\x09\x09\x09from pp_pa_table
:\x09\x09\x09\x09) ggg
:\x09\x09\x09\x09group by sid,employee_id,pp,pp_year
Thanks
Sassy
On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann wrote:
> Dear all,
>
> I am glad to announce that the release of NaviServer 4.99.25 is
> available at SourceForge [1]. This release is mostly a bug-fix
> release. The forthcoming version 5.0 of NaviServer will contain
> several new features omitted in this bug-fix branch. In case, you are
> building NaviServer from the Bitbucket repository, please note that
> the release 4.99.25 is in the branch release/4.99 (bug fix branch for
> the NaviServer 4.99 family). New development happens in the "main"
> branch of the repository, leading to NaviServer 5.*.
>
> See below for a summary of the changes.
>
> Many thanks to the contributors of this release:
>
> Andrew Piskorski
> Antonio Pisano
> Brian Fenton
> Gustaf Neumann
> Hector Romojaro
> Joe Oldak
> Khy Huang
> Oleg Oleinick
> Zoran Vasiljevic
>
> All the best!
>
> -gustaf neumann
>
> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/
>
> ===
> NaviServer 4.99.25, released 2023-05-01
> ===
>
> 132 files changed, 3957 insertions(+), 2068 deletions(-)
>
> New Features:
> -
>
> - Added meta-information to configuration values
>
>NaviServer can now report, what configuration values provided in
>the configuration file were actually used, what their default
>values are, and whether these values were specified or not (using
>the default values). With this information, administration
>(e.g. migration) becomes easier. The NaviServer module "nsstats"
>shows this meta information via the web interface.
>
>This functionality is provided via the new option "-filter" for the
>command "ns_configsection ... section". When the "-filter" option
>is used, different kinds of information about the parameters is
>returned from the specified section.
>
> "-filter unread":
>
> Returns the parameters, which were set during configuration (i.e.,
> in the configuration file) but which were not read in the startup
> phase of the server. This option is useful to determine
> e.g. typographical errors of specified parameter names.
>
> "-filter defaulted":
>
> Returns the parameters, from which the default values were read
> (i.e., which were not explicitly set)
>
> "-filter defaults":
>
> Returns the defaults of the parameter. This is useful for
> contrasting the actual values with the default values for
> Parameters, e.g. in a web based interface.
>
> - ns_set reform (per default deactivated in 4.99, but activated in 5.*)
>
>The classical implementation for ns_sets uses separately malloced
>storage for every attribute name and attribute value. So, e.g., for
>1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
>malloc/free operations, e.g., for a single db query! Although the
>malloc implementations have improved over the years, these will
>require many lock operations, especially under load, where many
>threads might perform many concurrent malloc operations. One other
>consequence is that the allocated memory will be scattered over
>address space, which has bad implications for CPU caching.
>
>The new implementation uses for one "ns_set" a single Tcl_DString
>keeping all attribute names and attribute values. This reduces the
>malloc operations and improves memory locality, such that cache
>hits will improve.
>
>One caveat of this change is that modules using "ns_set" have to be
>recompiled, since
