Dear all,
on sourceforge is now rc2, tested on
macOS 11.6.2, Rocky Linux 8.5, Ubuntu 20.04, OpenBSD 6.9, FreeBSD 13.1
The handling of collate (as used by the new ns_strcoll function) in
different
OSes is tricky and surprisingly little consistent. Linux seems to have a
very
good support for comparing UTF-8 strings, proving case-folding collation
(probably
using the ICU library (International Components for Unicode [1]), other
ones behave
quite different. Implementing an extra linkage against the libicu (which is
surprisingly complex) for the other platforms seems currently an overkill,
since most it would be probably just needed for non-Linux systems.
For now, the cases, where strcoll() results differ are deactivated in the
regression test for these systems.
Please test if possible.
all the best
-gn
PS: There seems to be as well differences for some versions of OpenSSL
(e.g. openssl 1.1.1d (broken) vs. openssl 1.1.1k (good))
[1] https://icu.unicode.org/
On 05.12.21 14:13, Gustaf Neumann wrote:
Dear all,
on sourceforge is a release candidate for NaviServer 4.99.23 [1].
The change contains one potentially important fix for a memory
leak (which was in the code since 1y+, but turned out to be significant
with a recent OpenACS change) and a couple of new features (including
the improved Unicode support for e.g. emojis, and crypto improvments
like SCRAM, which is already incorporated in OpenACS).
Please test if possible. The release should be in the near future.
Below is a preliminary summary of changes.
all the best
-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/
=======================================
NaviServer 4.99.23, released 2021-12-XX
=======================================
60 files changed, 1014 insertions(+), 270 deletions(-)
New Features:
- Improved hash algorithms for improved security
The new version supports SCRAM (Salted Challenge Response
Authentication Mechanism), which is one of the newer recommended
password hash algorithm to replace the classic salted SHA1
approaches. The classical hash algorithms become easier to attack
via high performance hashing hardware (GPUs). NaviServer supports
now SCRYPT and SCRAM (when compiled with more recent versions of
OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)
The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
the pseudorandom function (PRF) and with dkLen == output length of
HMAC == output length of the digest function.
Here is an example of using pbkdf2_hmac for the hash function of
SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:
::ns_crypto::pbkdf2_hmac \
-digest sha256 \
-iterations 15000 \
-secret $password \
-salt $salt]
OpenACS supports already switching to from salted SHA1 to SCAM (or
SCRYPT) via configuration variable.
- Better Unicode support, including emojis requiring 4-byte UTF-8 characters.
Earlier versions of NaviServer and the nsdb* database drivers
assumed on a few places that Tcl-internal UTF-8 is also valid UTF-8
for external sources, which is often, but not always true. Now, the
proper export functions are everywhere called.
The new code was tested with Emojis up to Unicode 13 (many thanks
to Wolfgang Winkler)
This change effects as well the database driver module "nsdbpg".
- ns_trim enhancements:
The new option "-prefix ..." can be used to strip a string
(such as ">> ") from every line starting with it.
- extended time unit support (added "w" for weeks and "y" for years)
- Added an experimental global parameter "nocache" to ease to
experiment with horizontal scaling. As a consequence, "ns_cache
eval" becomes a dummy operation.
- Added an experimental command "ns_baseunit"
ns_baseunit ?-size size? ?-time time?
Convert from memory units or from time units to its base value
using the NaviServer internal converters, which are used the same
way for various commands. The base unit for a memory size is a
byte, for a time value a second). This command is necessary to
provide Tcl-level commands calculating with these units to support
uniform interfaces (e.g., calculating cache partition sizes base on
values such as 2MB).
Either "-size" or "-time" has to be specified.
% ns_baseunit -size 10KB
10240
ns_baseunit -time 2.5h
9000
- Added an experimental command "ns_strcoll"
ns_strcoll ?-locale locale? string1 string2"
This command compares lexicographically string1 with string2
according to the current locale collation and returns an integer
greater than, equal to, or less than 0, depending on whether
string1 is greater than, equal to, or less than string2. The
command is necessary in cases, where e.g., the sorting order from
the database (normally based on local collation) is different from
default Tcl sorting order to provide a uniform interface with same
sorting orders. The name is derived from the baseline POSIX
function call.
The command is suitable for usage in the lsort command:
% set l {Bor Bar Bär}
% lsort -command ns_strcoll $l
Bar Bär Bor
% lsort $l
Bar Bor Bär
Performance Improvements:
- Increase scalability on DB operations by reducing DB pool locks
On high load servers, the total number of locks and busy locks on
the DB pools might become quite high. The new code caches these
statistics in the handles (which are per thread, requiring no
locks) and transfers the aggregated values on handle closes or
statistics calls.
- Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
newer. Versions up to at least Tcl 8.5 are known to crash in case
two threads create interpreters at the same time. These crashes
were hard to reproduce, but serializing interpreter creation
helped. Since all our major servers are running since several years
without problems with this parameter turned on the default is now
set to "true", when NaviServer is compiled with Tcl 8.6 or newer.
Bug Fixes:
- Fixed memory leak in "nsv_dict get" operations.
Documentation improvements:
---------------------------
- Improved the following man pages:
doc/src/manual/admin-tuning.man
doc/src/naviserver/commandlist.man
doc/src/naviserver/ns_adp_include.man
doc/src/naviserver/ns_baseunit.man
doc/src/naviserver/ns_cache.man
doc/src/naviserver/ns_charsets.man
doc/src/naviserver/ns_choosecharset.man
doc/src/naviserver/ns_connchan.man
doc/src/naviserver/ns_cookie.man
doc/src/naviserver/ns_cookiecharset.man
doc/src/naviserver/ns_crypto.man
doc/src/naviserver/ns_encodingforcharset.man
doc/src/naviserver/ns_encodingfortype.man
doc/src/naviserver/ns_formfieldcharset.man
doc/src/naviserver/ns_ictl.man
doc/src/naviserver/ns_job.man
doc/src/naviserver/ns_register.man
doc/src/naviserver/ns_schedule.man
doc/src/naviserver/ns_setformencoding.man
doc/src/naviserver/ns_shutdown.man
doc/src/naviserver/ns_sleep.man
doc/src/naviserver/ns_sockcallback.man
doc/src/naviserver/ns_sockopen.man
doc/src/naviserver/ns_strcoll.man
doc/src/naviserver/ns_time.man
doc/src/naviserver/ns_urlcharset.man
doc/src/naviserver/ns_valid_utf8.man
doc/src/naviserver/textutil-cmds.man
Configuration Changes:
----------------------
- Ease configuration via environment variables
This feature is useful to manage many NaviServer instances with
mostly identical configurations without having to provide multiple
configuration files (e.g. for Docker setups, or clusters).
The sample configuration for of OpenACS (openacs-config.tcl)
contains now a Tcl dictionary with default values:
set defaultConfig {
hostname localhost
ipaddress 127.0.0.1
httpport 8000
httpsport ""
server "openacs"
serverroot /var/www/$server
logroot $serverroot/log/
homedir /usr/local/ns
bindir $homedir/bin
db_name $server
db_user $server
db_host localhost
db_port ""
}
These configuration values (keys of the dict) can be overridden by
environment variables prefixed with "oacs_" followed by the
parameter name. One can change the default port specified in the
configuration file for plain HTTP connections by e.g., providing it
via environment variables:
oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....
Code Changes:
- Extended regression test
- Code Cleanup
. Do not declare reserved C identifiers
- Improved comments, fixed typos
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
