Thanks for all the work on this Gustaf. Looks like a great release! Regards, Dave
On Sun, 4 Nov 2018 at 17:54, Gustaf Neumann <neum...@wu.ac.at> wrote: > > Dear all, > > on sourceforge is now the release of NaviServer 4.99.17 [1]. > Below is a summary of the changes. This release was tested > under Windows 8.1 (32bit) and 10 (64bit) many thanks to > Oleg Oleinik), various Linux versions (e.g. Ubuntu 18.04, > FC 27, including Raspberry PI Raspbian GNU/Linux 9.4 > with ARM processor), and OmniOS and OpenBSD, with > Tcl 8.5, 8.6 and 8.7 (alpha). Many thanks to all contributors! > > all the best > > -gustaf > [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.17/ > > > ======================================= > NaviServer 4.99.17, released 2018-11-04 > ======================================= > > 421 files changed, 17805 insertions(+), 10275 deletions(-) > > New Features: > ------------- > > - Added brotli support for delivering static content > --------------------------------------------------- > The brotli compression format (RFC 7932) achieves better results for > compression for typical web content compared to e.g. gzip and is > supported by most current browsers. The definition of static > content delivery is exactly the same for "brotli" as for static > "gzip" content. > > New configuration options for ns/fastpath: > * "brotli_static": should static brotli delivery be checked? > * "brotli_refresh": should outdated brotli compressed files be refreshed? > * "brotli_cmd": OS-level command with options used for brotli compression > > - "ns_conn" additions: > * New subcommand "ns_conn acceptedcompression" to return accepted > compressions for the current connection (gzip or brotli) > > * New subcommands "ns_conn currentaddr" and "ns_conn currentport" > to refer to the address and port of the currently open server side > of the socket. > > > - New feature for GDPR: add mask IP addresses optionally in access log > -------------------------------------------------------------------- > This feature is similar to "anonip" IP anonymizer of the Swiss > privacy foundation): When this feature is activated, all IP > addresses are masked in the log file, such that the host-specific > (= person- specific) information is masked out and the IP-address > does not match a particular individual anymore. Still, with the > masking turned on, reverse DNS lookup and/or geolocation is > possible. > > The option can be configured via the following parameters in the > ".../module/nslog" section of the config file of NaviServer: > * "masklogaddr": boolean value to turn feature on/off (default off) > * "maskipv4": mask for IPv4 addresses (default 255.255.255.0) > * "maskipv6": mask for IPv6 addresses (default ff:ff:ff:ff::) > > > - New atomic "nsv_set" commands/options > ------------------------------------- > > * Obtain (old) value from nsv ARRAY and set it to a new value set foo > > set foo [nsv_set -reset ARRAY KEY NEWVALUE] > > After the operation, ARRAY(KEY) has a new value. > The operation is similar to "nsv_set ARRAY KEY NEWVALUE", > but the variant with "-reset" returns the old value. > Similar to GETSET in REDIS. > > * Obtain a value from nsv ARRAY and unset it (no new value is provided). > > set foo [nsv_set -reset ARRAY KEY] > > The operations returns the old value of ARRAY(KEY) or empty, if > it does not exist. After the operation, the ARRAY(KEY) is unset. > > * Set a default value for a nsv ARRAY > > nsv_set -default ARRAY KEY DEFAULTVALUE > > Do nothing when ARRAY(KEY) has already a value. > Return an error, when DEFAULTVALUE is not provided. > Similar to SETNX in REDIS. > > > - Extended functionality of "ns_http" > ----------------------------------- > > * Changed return value for "ns_http run" and "ns_http wait". > Now the commands return attribute value pairs (Tcl dict). > Previously, the commands returned different results, > depending on the options > > - "ns_http run" returned the name of the queued command (which > was completely useless, this the queued command was already > terminated) > - "ns_http wait" returned sometimes the body of the reply (when > the reply was not spooled) or empty (spooled) or "1", when the > option "-R" was specified > > Now the result is always a dict containing "status" (HTTP status > code), "time" (elapsed time), "headers" (ns_set of the reply > header fields, and either the "body" (reply body) or "file" (the > name of the spooled file). The values in the dict might mirror > the result of other (optional) output variables, which continue > to work. > > Since the results were irregular and not documented in the man > pages, the backward compatibility should be high. Also, the > regression test of NaviServer was not effected by this change. > > > * Support for asynchronous “ns_http” tasks: > > When the new flag "-donecallback” is provided, then a “ns_http” > task is started in the background and does not require an > “ns_http wait” anymore. Instead, when it finishes it executes > the provided donecallback which receives as additional arguments > the Tcl result (in form of an integer) and the result dict, > returned otherwise from e.g. “ns_http run”. > > As a consequence, requests of the form > ns_http queue … -donecallback … > differ from client requests without the callback > in the following points: > > a) the command does not return a handle for “ns_http wait” > b) The command is not automatically cleaned up (canceled) > at the end of a connection requests > c) Typically, donecallbacks are executed in a different > thread than it was started. > > Some more options, which were previously only valid on > “ns_http wait” (such as e.g. spoolsize) are now valid > for “ns_http queue” as well. > > The new flag allows a developer to start many requests in > parallel without having to start many threads or complex > synchronization tasks. > > * New option "-body_file" for "ns_http wait" (similar to "ns_http > queue" which can be used to specify a filename for the > downloaded content) > > * New option "-outputfile" to "ns_http run" and "ns_http wait": > by specifying the option "-outputfile" the received file will be > always written to the disk, no matter how large it is. > > * New subcommand "ns_http stats": returns a dict (flat list of > attributes and values). The list contains "task", "url", > "requestlength", "sent", "replylength" and "received". > > > - Support resolving a hostname against multiple IP addresses > ---------------------------------------------------------- > > * In case, one domain name has multiple IP addresses registered, > previous versions of NaviServer tried only the first returned > address and returned an error, when this failed. This happened > when using client functions (e.g. ns_http). > > * host names with multiple IP addresses are nowadays quite common, > e.g. "localhost" having an IPv4 and an IPv6 address registered. In > previous versions, when e.g. localhost has as first address the IPv6 > address, but IPv6 was not active, it returned an error. > > * resolving against IP addresses continues to work as always. > > > - Extended crypto functionality > ----------------------------- > > * Added HMAC-based Extract-and-Expand Key Derivation Function (HKDF) > ::ns_crypto::md hkdf -digest sha256 .... > (requires OpenSSL 1.1.0 or newer) > > * Added support for elliptic curve cryptography > . ns_crypto::eckey import ?-string string? ?-encoding encoding? > . ns_crypto::eckey priv ?-pem pem? ?-encoding encoding? > . ns_crypto::eckey pub ?-pem pem? ?-encoding encoding? > . ns_crypto::eckey sharedsecret ?-pem pem? ?-encoding encoding? pubkey > . ns_crypto::eckey generate ?-name name? ?-pem pem? > > * Added support for getting an arbitrary number of random bytes > based on crypto support on OpenSSL: > . ns_crypto::randombytes ?-encoding encoding? bytes > > * Added support for Authenticated Encryption with Associated Data > (AEAD) scheme, which provides confidentiality, integrity, and > authenticity. > > . ns_crypto::aead::encrypt string ?-aad aad? ?-cipher cipher? > ?-encoding encoding? ?-iv iv? ?-key key? input" > . ns_crypto::aead::decrypt string ?-aad aad? ?-cipher cipher? > ?-encoding encoding? ?-iv iv? ?-key key? ?-tag tag? input" > > * Added "-encoding hex|base64url|base64" option for output encoding > for the crypto functions. Previously, all these functions returned the > result in "hex" format. If the option is not specified, the result > is the same as before (hex). The encoding option was added to the > following functions: > - ns_crypto::hmac string > - ns_crypto::hmac get > - ns_crypto::md string > - ns_crypto::md get > - ns_crypto::md vapidsign > - ns_crypto::enc > > > - "ns_connchan" improvements > -------------------------- > > * Callback suspension: a Tcl socket callback function might now > return the value "2" to flag that the callback will be > suspended, but it keeps the socket open. Previously, the Tcl > callback could return just "0" (callback is cancelled, socket is > closed) or "1" (continue callback handling, keep socket open). > This can be used to suspend callback handling on bottlenecks and > continue later, when situation improves. > > * "ns_connchan write": The function returns now the number of > bytes sent, which might be less than the length of the data to > be sent. > > * "ns_connchan list": the function returns now two more fields per > entry: a) cmd name of the callback (eases debugging, when > callbacks are changed dynamically) b) conditions used for > registering the callback. > > * The "ns_connchan" callbacks are now listed via the nsstats > interface as well (with proper callback info). > > > - Added Possibility to add per-server "initcmds" to config file > ------------------------------------------------------------- > > This new feature provides an easy means to add command which > should be executed after server initialization into a configuration > files. By using the "initcmds" one can start multiple instances > of NaviServer differing just in the initialization command > by providing different config files. The "initcmds" approach > has the advantage over e.g. "ns_atserverstart" that it is > executed in a state when the server is fully initialized. > > > > - Improved readability for config files > ------------------------------------- > > The "ns_section" command has new an optional last argument for the > parameters of its configuration section. Instead of writing e.g. > > ns_section ns/parameters > ns_param serverlog $logroot/error.log > ns_param pidfile $logroot/nsd.pid > ns_param home $homedir > #... > > one can write now > > ns_section ns/parameters { > ns_param serverlog $logroot/error.log > ns_param pidfile $logroot/nsd.pid > ns_param home $homedir > #... > } > > This block notation has two advantages: a) it shows clearly, where > the section ends and b) when using an editor with automatic > indentation, the log file looks nicer without manual indentation > work. This change is fully backward compatible, old style config > files continue to work. > > > - Further new commands and features > --------------------------------- > > * "ns_hash": provide a interface for the HashStringKey() function > (very useful for e.g. cache partitioning) > > * "ns_reflow_text". > ns_reflow_text ?-width width? ?-offset offset? ?-prefix prefix? > ?--? text > Tcl-based solutions are surprisingly slow on largish input. > > * "ns_base64urlendcode", "ns_base64urldedcode": URL-save variants of > base64, > used in various new RFCs for passing base64 values in HTTP requests. > > * "ns_uuid": runtime efficient version of a version 4 UUID > according to RFC 4122: A Universally Unique IDentifier (UUID) > URN Namespace > > * Improved results of [ns_cache_stats -contents ...]. The command > returns now a tcl list instead of a string, including reuse count. > > - Improved results of "ns_striphtml": > * Added resolving of all HTML4 entities (including etc.) > * Stripped as well HTML comments. > > > Performance Improvements: > ------------------------- > > - Check for epoch updates on blue-prints as well in cleanup phase > (deallocate trace) to reduce latency for client requests. > > > Bug Fixes: > ---------- > > - Don't try to serve redirected files in case the connection is > already closed (will lead only to confusing messages) > > - Ns_IsBinaryMimeType(): When a chartype is specified, never assume > the mimetype is binary. > > - ns_getformfile: Don't rely on global variables > (see https://sourceforge.net/p/naviserver/bugs/80/) > Before this change, Depending on configured values, > ns_getformfile could have missed uploaded files. > > - Added missing fetch commands for obtaining handles (on obviously > seldom used subcommands "ns_db setexception|sp_setparam") > > - ns_http: > * Timeout and connection behavior show works now consistent > for HTTP and HTTPS URLs. > * Fixed a potential bug (infinite loop) when requests > run into timeouts. > > - "ns_set print": align with documented behavior > > - "ns_parseurl": make results more robust, when a URL with a colon > in the path was given, but no port provided (detected a port > incorrectly) > > - "ns_parseformfile": handle also www-form-urlencoded (in > addition to multipart/form-data as before) > > - Windows: > * Make sure to create temporary files correctly under windows (many > thanks to Oleg Oleinik) > * Incorrect handle was used to perform operations on the > process in Ns_WaitForProcess (many thanks to Oleg Oleinik) > > > Documentation improvements: > --------------------------- > > - Updated several man pages > * admin-db.man: > * admin-install.man > * admin-maintenance.man > * adp-overview.man > * commandlist.man > * main-features.man > * main-history.man > * ns_accesslog.man > * ns_adp.man > * ns_adp_argc.man > * ns_adp_argv.man > * ns_adp_bind_args.man > * ns_adp_eval.man > * ns_adp_flush.man > * ns_adp_info.man > * ns_adp_safeeval.man > * ns_atexit.man > * ns_atexit.man > * ns_atshutdown.man > * ns_atsignal.man > * ns_base64.man > * ns_base64encode.man > * ns_conn.man > * ns_connchan.man > * ns_cookie.man > * ns_db.man > * ns_driver.man > * ns_fmttime.man > * ns_getcontent.man > * ns_getform.man > * ns_hmac.man > * ns_hotp.man > * ns_http.man > * ns_httpget.man > * ns_httpopen.man > * ns_ictl.man > * ns_job.man > * ns_kill.man > * ns_md.man > * ns_memoize.man > * ns_mutex.man > * ns_parsequery.man > * ns_perm.man > * ns_proxy.man > * ns_queryexists.man > * ns_queryget.man > * ns_querygetall.man > * ns_quotehtml.man > * ns_rand.man > * ns_register.man > * ns_return.man > * ns_rlimit.man > * ns_schedule_proc.man > * ns_sendmail.man > * ns_server.man > * ns_set.man > * ns_set.man: > * ns_setexpires.man > * ns_sockopen.man > * ns_sockselect.man > * ns_striphtml.man > * ns_thread.man > * ns_time.man > * ns_tmpnam.man > * ns_totp.man > * ns_urlspace.man > * ns_uuencode.man > * ns_writer.man > * nscgi.man > * nsperm.man > * nssock.man > * nsv.man > * returnstatus-cmds.man > * tcl-lib-file.man > * tcl-lib-nstrace.man > * tcl-libraries.man > * tcl-overview.man > > > Tcl API Changes: > ---------------- > > - "ns_conn isconnected": returns now false, when the connection is > already closed. Otherwise we have to add a new subcommand > "ns_conn closed", which seems somewhat odd (isconnected true and > closed true?) > > - "ns_cache_stats" returns now hitrate in form of a float instead of > an integer. > > > C API Changes: > -------------- > > - Ns_SockTimedConnect2() returns now Ns_ReturnCode to be able to > distinguish between timeouts and errors > > - Ns_SockConnectError(): new call to provide consistent error messages > and to set Tcl error info consistently > > > Configuration Changes: > ---------------------- > > - Use same initialization for "enablelclpages" when configured over > "ns/server/$server" or over "ns/server/$server/adp" to avoid subtle > differences on global variables. > > - Extended sample config files: > > * nsd-config.tcl > . Added parameter "masklogaddr" for "nslog" section > . Added parameter for "gzip" and "brotli" static delivery > . Added parameter for "initcmds" > . Added setting for recommended environment variables > HOME, LANG and RANDFILE. RANDFILE is sometimes needed > to avoid surprises with permissions, when OpenSSL > crypto functions are called from the server > > * openacs-config.tcl > . Switched to PostgreSQL's conninfo notation in the datasource > to hint that more such parameters are possible as well > (e.g. connections via SSL) > . Added comments and examples for configuring OpenACS caches > . Added comments how to configure EmailDeliveryMode via config file > . Added comments how to configure IncludeCallingInfo of OpenACS API > browser > . Added comments how to configure WithDeprecatedCode > . Added example for activating more intense SQL logging > . Added subsite-based error pages > . Added section indicating how to configure for nsstats module > . Added parameter settings for "reuseport" > . Added parameter "masklogaddr" for "nslog" section > . Added parameter for "brotli" static delivery > . Preconfigure customized error pages based on subsites > > - Config files changed to block notation (see above) > > - Improved handling of configuration errors: > * Provided better log message, when configured value is out of range > * Updated config values when range check corrects the value to get the > finally configured value via introspection. > > > Code Changes: > ------------- > > - Extended regression test: > * tests/adp.test > * tests/compat.test > * tests/http.test > * tests/ns_base64.test > * tests/ns_base64encode.test > * tests/ns_cache.test > * tests/ns_conn.test > * tests/ns_crypto.test > * tests/ns_nsv.test > * tests/ns_parseurl.test > * tests/ns_reflow_text.test > * tests/ns_server.test > * tests/ns_striphtml.test > > > - Added version tag of NaviServer to "starting" message in log file > to ease analysis in case of problems. > > - Implemented deprecated commands as Tcl proc and complain on > its usage > . ns_adp_eval > . ns_adp_safeeval > > - Improved Platform support: > * Extended list of platforms for manual provisioning of __PRIPTR_PREFIX > in case it is not defined. > * Do not rely on implementation defined print/scan format "%p", > but use instead standardized integer types. > * Windows: > . Improved type-cleanness (many thanks to Oleg Oleinik) > . Support to compile with MSVC 2017 (many thanks to Oleg Oleinik) > . When compiling with VS 2012, use built-in support for > intptr_t and uintptr_t. > . Added support to run regression test under Windows > (many thanks to Oleg Oleinik) > > - Changed preset encodings (when not specified in config file) for > files with extensions .htm, .html, and .adp from iso8859-1 to > utf-8. > > - Improved debugging support: > > * Added compiler-flag NS_MUTEX_NAME_DEBUG for ease locating > anonymous mutexes. > * Provided names for all remaining mutexes to make it easier > to pinpoint bottlenecks due to mutexes > * Improve debugging output in various messages, when debugging > options are activated. > > > - Minor update of mime types based on actual IANA media types. > > - Provided compatibility with Tcl 8.7.2 (removed CONST and VOID macros) > - Further improved robustness by avoiding potential NULL dereferences > found be static code analyzers (mostly error cases) > - Improved configure scripts > - Make sure, we have enough space for the terminating null character > - Added "pure" specifier for improved code generation > > - Various code smell removals > * Prefer boolean type over int > * Improved type cleanness > * Reduced implicit conversions > * Reduced number of return statements before end of function > * Don't pass implementation-defined NULL after the last typed argument > to a variadic function > * Removed dead assignments > * Added "const" declarations > * Fixed typos, improved comments > > > Modules: > -------- > > - nsdbbdb (many thanks to Brendan Graves): > * Changed to work with FreeTDS 1.00.86 > * Updated outdated paths. > > - nsdbpg: > * Added sample configuration to include hints how to use SSL > connections to the database > * Provided compatibility with Tcl 8.7.2 > * Minor code cleanups > > - nssmtpd, nsdns, nstftpd, nsradiusd, nsdbi, nsdbipg: > * Brought code in sync with the current code base (including IPv6, 8.7.2 > compat) > * Improved spelling > > - nsphp (many thanks to Anthony Bennett): > * Refactored for newer versions of NaviServer > * Improved README and spelling > > - nsstats: > * Added results to "ns_connchan list" to per-server info > * Added optional query parameter "raw=0|1" to switch between > raw and prettified numbers > * Improved number prettification > * Added optional query parameter "max=..." for controlling the number of > per-cache entries shown (default 50) > * Improved cache statistics: added hits/req, saved/req, saved/hit; dropped > saved/KB > * Improved mutex statistics: added locks/req, potential locks/req, > potential locks per second > * Added summative statistics to mutex locks page > * Made background locks (locks, not included in per-request statistics) > configurable via NaviServer config file > * Various other small changes and beautifications (use more CSS) > * Bumped version to 1.8 > > - nsoracle: > * Fixed potential crash when oci_error_p is called with dbh == NULL > (many thanks to Oleg Oleinik) > * Some more code cleanup and improved spellings. > > - websocket: > * Fixed potential bug: make sure that upgrade header is sent as a single > block. > * More minor code cleanup. > > - revproxy: > * Fixed potential blocking conditions when send buffer of OS overflows. > With "-sendtimeout 0" a call to "ns_connchan write" might write less > bytes than desired; In this case, callbacks are suspended and continue > when the situation improves. > * Added new switches "-sendtimeout" and "-receivetimeout" to > revproxy::upstream > * Added "-backend_reply_callback" to ::revproxy::upstream to allow proxy > the > modification of backend header fields per request. > * Bumped version number to 0.11 > > - letsencrypt: > * Modernized code: use built-in ns_base64urlencode (requires 4.99.17) > * Changed order of certificates in pem file (private key first) > * Don't hard-code OpenSSL dir on OS (many thanks to Stefan Sobernig) > > - nswebpush: > * New module for implementing Web Push via NaviServer > (Many thanks to Georg Prohaska) > * For details, see: https://bitbucket.org/naviserver/nswebpush/src/default/ > > _______________________________________________ > naviserver-devel mailing list > naviserver-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/naviserver-devel >
_______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/naviserver-devel
