Re: Voting: std.logger
On Friday, 8 August 2014 at 09:16:11 UTC, Robert burner Schadek wrote: could you elaborate please? Currently I use the version statements in two template functions. I'm not sure why one would brand this negatively as a leak into the library. For example we don't reinstatiate templates if they are instantiated in an imported module. But that module might have been compiled with different settings. Thank you for taking the time, but I found several conceptional problems with that POC. The worst being that if I make the LogLevel inside the TestLogger anything other than a enum it fails, as it is used in the template constraint. That would mean the LogLevel is fixed at CT. Well if your test logger only knows it's log level at runtime, then you obviously can do the comparison only at runtime. You can solve this by checking whether minLogLevel is a compile time constant in the log constraints adding another overload for runtime configurable log levels.
Re: Voting: std.logger
Moved to: https://github.com/burner/phobos/pull/2 I did some simple benchmark, logging Hello world 1_000_000 times with the default logger. (DMD with release settings). Comparing the version before the thread-safety changes with the one afterwards. The timings are: 6.67s and 6.66s - so it is -0.15% slower now. :p Yes, I'm just trying to confuse you. Within margin of error the performance stays the same. The default logger is using lockingTextWriter() for optimum performance. I compared it with a logger I wrote a while ago and which I ported to std.experimental.logger.core.Logger, but extends it by honoring the system locale for terminal output: Assuming LC_ALL=de_DE.iso88591 it * formats the message using Phobos' formattedWrite(...) * converts UTF-8 to UTF-16 * passes it to ICU for normalization (to combine accents) * transcodes the NFC UTF-16 with ICU to system local * uses POSIX write() to dump it to the terminal With all the overhead it still takes only 2.26s for the same 1_000_000 messages. If I use a UTF-8 terminal and skip string conversion it will be 2.03s. So on the one hand it means string transcoding accounts for ~10% and on the other that there is 228% overhead with the default logger. :) Some of which might be low hanging fruit. But that should not be part of this review (which is about API). It was just an observation I wanted to share. -- Marco
Re: Voting: std.logger
Am Fri, 19 Sep 2014 15:12:34 + schrieb Robert burner Schadek rburn...@gmail.com: PRs are most welcome Ok, I'm working on it. From the looks of it I will iterate a bit on the code and create one massive pull request with a lot of smaller commits to make it comprehensible. -- Marco
Re: Voting: std.logger
On Friday, 19 September 2014 at 11:48:28 UTC, Marco Leise wrote: By the way, does anyone else use std.experimental.logger yet? Dicebot? No, I provide purely management service here, aggregating reviews/opinions of other community members.
Re: Voting: std.logger
Seeing how MultiLogger passes on the payload to its child
loggers by ref, I tried to make it const, so that no Logger
implementation can correct any of the values and spoil it
for the others in the list.
Then I noticed, that isn't possible because the payloads
contain indirections and one logger takes payloads apart into
separate variables (FileLogger) while others build payloads
from separate variables. Catch 22. You cannot have:
protected void beginLogMsg(string file, int line, string funcName,
string prettyFuncName, string moduleName, LogLevel logLevel,
Tid threadId, SysTime timestamp, Logger logger)
@trusted
{
static if (isLoggingActive)
{
header = LogEntry(file, line, funcName, prettyFuncName,
moduleName, logLevel, threadId, timestamp, null, logger);
}
}
and
override void writeLogMsg(const ref LogEntry payload)
{
this.beginLogMsg(payload.file, payload.line, payload.funcName,
payload.prettyFuncName, payload.moduleName, payload.logLevel,
payload.threadId, payload.timestamp, payload.logger);
…
}
.
Also I wonder if Tid is the correct information to pass in.
It is actually just an MBox from std.concurrency. The real
thread handle is the Thread, which also contains its name,
which might be more useful for logging. What do you think?
--
Marco
Re: Voting: std.logger
On Saturday, 20 September 2014 at 10:24:30 UTC, Marco Leise wrote: Also I wonder if Tid is the correct information to pass in. It is actually just an MBox from std.concurrency. The real thread handle is the Thread, which also contains its name, which might be more useful for logging. What do you think? See also https://github.com/D-Programming-Language/phobos/pull/2482 For std.log I think Tid more useful because it clearly denotes execution context while thread ID is more of an implementation detail (considering message-passing is promoted as a standard D thing)
Re: Voting: std.logger
Ok, then here are my thread-safety changes for std.logger: https://github.com/burner/logger/pull/19 -- Marco
Re: Voting: std.logger
On Friday, 19 September 2014 at 05:25:18 UTC, Marco Leise wrote: WAT? So next round? Hell no! before every voting there is a review and I guess you started that. I will address the thread-safety sufficiently in the next incarnation. Thank you for taking the time
Re: Voting: std.logger
On Tuesday, 16 September 2014 at 21:22:37 UTC, Robert burner Schadek wrote: On Wednesday, 10 September 2014 at 16:54:14 UTC, Dicebot wrote: so the current version has one template args log functions that take __LINE__ and friends as normal parameter. I think that was the last complaint, please correct me if I'm wrong. So next round? Once you feel that thread safety concerns are addressed please write me an e-mail. I am currently on vacation with irregular internet access so may miss regular NG post.
Re: Voting: std.logger
Am Fri, 19 Sep 2014 09:26:59 + schrieb Robert burner Schadek rburn...@gmail.com: On Friday, 19 September 2014 at 05:25:18 UTC, Marco Leise wrote: WAT? So next round? Hell no! before every voting there is a review and I guess you started that. :) I will address the thread-safety sufficiently in the next incarnation. Ok, thank you. If you get stuck or need a another review, just drop me a mail. 4 eyes see more than 2. I'd even do the changes I had in mind myself and create a pull request if you don't mind. By the way, does anyone else use std.experimental.logger yet? Dicebot? Thank you for taking the time You're welcome. I'm really looking forward to this project. This is the kind of component that makes libraries mix and match and helps D grow. -- Marco
Re: Voting: std.logger
On 9/19/14, 2:26 AM, Robert burner Schadek wrote: On Friday, 19 September 2014 at 05:25:18 UTC, Marco Leise wrote: WAT? So next round? Hell no! before every voting there is a review and I guess you started that. I will address the thread-safety sufficiently in the next incarnation. Thank you for taking the time Nice, thanks! -- Andrei
Re: Voting: std.logger
On Friday, 19 September 2014 at 11:48:28 UTC, Marco Leise wrote: Ok, thank you. If you get stuck or need a another review, just drop me a mail. 4 eyes see more than 2. I'd even do the changes I had in mind myself and create a pull request if you don't mind. PRs are most welcome Thank you for taking the time You're welcome. I'm really looking forward to this project. This is the kind of component that makes libraries mix and match and helps D grow.
Re: Voting: std.logger
Am Tue, 16 Sep 2014 21:22:36 +
schrieb Robert burner Schadek rburn...@gmail.com:
On Wednesday, 10 September 2014 at 16:54:14 UTC, Dicebot wrote:
so the current version has one template args log functions that
take __LINE__ and friends as normal parameter.
I think that was the last complaint, please correct me if I'm
wrong.
WAT?
So next round?
Hell no!
Am Mon, 15 Sep 2014 22:33:45 +
schrieb Robert burner Schadek rburn...@gmail.com:
again, the idea of std.logger is not to give you everything,
because nobody knows what that even is, the idea is to make it
possible to do everything and have it understandable later and
use transparently
I understand that part Robert, and already made use of that
flexibility. But you use this to play down any concerns about
the thread safety of the infrastructure you wrote and finally
get std.logger accepted and merged.
the threading behavior has been clarified in the api docs.
You can't just clarify it in the docs.
It requires actual code to work both ways.
the (a)synchronicity guarantees is part of the concrete Logger
impl. the Logger api does not force synchronize or asynchronize
behavior, it allows both to be implemented by every subclass of
Logger.
All access to global state has to be synchronized before we
can safely do so, and std.logger doesn't even attempt to in its
current state! Some examples:
SITUATION:
isLoggingEnabled(LogLevel ll, LogLevel loggerLL, LogLevel
globalLL, lazy bool condition)
{
…
return ll = globalLL
ll = loggerLL
globalLL != LogLevel.off
loggerLL != LogLevel.off
condition
}
@property LogLevel globalLogLevel() @trusted @nogc
{
return globalLogLevelImpl();
}
private ref LogLevel globalLogLevelImpl() @trusted @nogc
{
static __gshared LogLevel ll = LogLevel.all;
return ll;
}
is called like this:
isLoggingEnabled(stdlog.logLevel, stdlog.logLevel,
globalLogLevel,
condition);
Inside `isLoggingEnabled`, we can expect condition to be
evaluated in the context of the calling thread, but the three
log level variables passed in create a race condition.
Imagine another thread sets `stdlog.logLevel` from warning to
error during a logging call. Inside `isLoggingEnabled` you'd
now get:
return LogLevel.warning = globalLL
LogLevel.warning = LogLevel.error
globalLL != LogLevel.off
loggerLL != LogLevel.off
condition
This will unconditionally return false of course. The `stdlog`
is now at log level warning AND error at the same time. WAT?
EFFECT:
Every blue moon a log message will be swallowed by std.logger.
SITUATION:
private Mutex __stdloggermutex;
static this() {
__stdloggermutex = new Mutex;
}
@property ref Logger stdlog() @trusted
{
static __gshared bool once;
static __gshared Logger logger;
static __gshared ubyte[__traits(classInstanceSize,
FileLogger)] buffer;
__stdloggermutex.lock();
scope(exit) __stdloggermutex.unlock();
if (!once)
{
once = true;
logger = emplace!FileLogger(buffer, stderr,
globalLogLevel()); }
return logger;
}
Every thread will now create its own thread local mutex to
protect access to global state.
EFFECT:
** This protects exactly nothing. **
Write instead:
__gshared private Mutex __stdloggermutex;
shared static this() {
__stdloggermutex = new Mutex;
}
If you need help with multi-threading please ask either here or
on IRC. I have found that we have some people in the community
that can explain even the fine details of atomic fences.
SITUATION:
We set the global log level through `globalLogLevel`:
@property void globalLogLevel(LogLevel ll) @trusted
{
if (stdlog !is null)
{
stdlog.logLevel = ll;
}
globalLogLevelImpl = ll;
}
What you tried here, was to set the global log level in case
we don't have `stdlog` initialized already, because during its
creation it will pick up the global log level.
Now `globalLogLevelImpl = ll;` will never be executed, because
inside the `stdlog` property function, it is initialized and
thus `stdlog !is null` will always be true.
Unless the user sets `stdlog` to null, which I assume is
invalid, since it creates fuzzy semantics: The first time
`stdlog` is encountered to be null it is set to a FileLogger,
any other time it stays null.
EFFECT:
There is no synchronization around the access to the stdlog.
D as far as I know doesn't require that machine word
reads/writes are atomic, so in theory you could get a
`stdlog` where one half is an old value and the other a new
one. That won't happen on ARM and x86, but I thought I'd
mention. Much more importantly though another thread could
change `stdlog` between `stdlog !is null` and
`stdlog.logLevel = ll;`. Assuming you want to protect all
of the global state with
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 16:54:14 UTC, Dicebot wrote: so the current version has one template args log functions that take __LINE__ and friends as normal parameter. I think that was the last complaint, please correct me if I'm wrong. So next round?
Re: Voting: std.logger
On 9/9/14, 12:38 PM, Robert burner Schadek wrote: On Saturday, 30 August 2014 at 02:18:30 UTC, Dicebot wrote: ... have noticed that all logging functions have file/line/function data as template parameters. This will create insane symbol bloat. While I can understand desire to use some nicer variadic argument API providing at least one log function that it simplistic but moves all compile-time data to run-time default argument is absolutely necessary for me to consider this viable. I do not consider that a problem. The benefit is much higher than the cost of the bigger binary. This has already been a long conversation on some previous thread. There may be a miscommunication here. In short: void fun(int x, string f = __FILE__)(double y); can be replaced with: void fun(int x)(double y, string f = __FILE__); Both work and the second produces fewer instantiations. Andrei
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 07:41:49 UTC, Andrei Alexandrescu wrote: There may be a miscommunication here. In short: void fun(int x, string f = __FILE__)(double y); can be replaced with: void fun(int x)(double y, string f = __FILE__); Both work and the second produces fewer instantiations. Andrei But void fun(int l = __LINE__, A...)(A...); cannot be replaced by void fun(A...)(A..., int l = __LINE__); anyway thanks for reading and for trying to help
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 08:47:47 UTC, Robert burner Schadek wrote: But void fun(int l = __LINE__, A...)(A...); cannot be replaced by void fun(A...)(A..., int l = __LINE__); anyway thanks for reading and for trying to help And this is why I am asking for separate `logRaw` overload that takes pre-formatted string, for those who care.
Re: Voting: std.logger
On 10/09/14 10:47, Robert burner Schadek wrote: But void fun(int l = __LINE__, A...)(A...); cannot be replaced by void fun(A...)(A..., int l = __LINE__); Could we modify the compiler to allow that? Just for the special identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to specify a value for that parameter then. -- /Jacob Carlborg
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 11:39:52 UTC, Jacob Carlborg wrote: On 10/09/14 10:47, Robert burner Schadek wrote: But void fun(int l = __LINE__, A...)(A...); cannot be replaced by void fun(A...)(A..., int l = __LINE__); Could we modify the compiler to allow that? Just for the special identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to specify a value for that parameter then. IMO that is overkill, adding another another method that only takes one string as parameter is fine here
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 11:39:52 UTC, Jacob Carlborg wrote: On 10/09/14 10:47, Robert burner Schadek wrote: But void fun(int l = __LINE__, A...)(A...); cannot be replaced by void fun(A...)(A..., int l = __LINE__); Could we modify the compiler to allow that? Just for the special identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to specify a value for that parameter then. This is much desired compiler enhancement in my opinion (this template instance bloat is really bad as it impacts not only symbol bloat but also compile times) but trying to get something that works right here and now.
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 12:14:09 UTC, Dicebot wrote: This is much desired compiler enhancement in my opinion (this template instance bloat is really bad as it impacts not only symbol bloat but also compile times) but trying to get something that works right here and now. killing this special overload, once the compiler does this, should be completely transparent
Re: Voting: std.logger
Jacob Carlborg wrote in message news:lupda8$nl1$1...@digitalmars.com... Could we modify the compiler to allow that? Just for the special identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to specify a value for that parameter then. IIRC Andrei has a bugzilla open for this.
Re: Voting: std.logger
On 09/10/2014 03:22 PM, Daniel Murphy wrote:
Jacob Carlborg wrote in message news:lupda8$nl1$1...@digitalmars.com...
Could we modify the compiler to allow that? Just for the special
identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to
specify a value for that parameter then.
IIRC Andrei has a bugzilla open for this.
Why? I cannot remember a compiler version where the following did not work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 15:41:49 UTC, Timon Gehr wrote:
Why? I cannot remember a compiler version where the following
did not work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
For every instantiation of of fun there will be a new symbol even
when A are the same, because of __LINE__. For Dicebot that is a
nogo
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 15:41:49 UTC, Timon Gehr wrote:
Why? I cannot remember a compiler version where the following
did not work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
For every instantiation of of fun there will be a new symbol even
when A are the same, because of __LINE__. For Dicebot that is a
nogo
Re: Voting: std.logger
Timon Gehr wrote in message news:luprft$29v6$1...@digitalmars.com...
Why? I cannot remember a compiler version where the following did not
work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
Hmm, do it does. Maybe I was thinking of this:
https://issues.dlang.org/show_bug.cgi?id=2599
Re: Voting: std.logger
Robert burner Schadek wrote in message
news:wsanssfvnomcwtnqy...@forum.dlang.org...
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
For every instantiation of of fun there will be a new symbol even when A
are the same, because of __LINE__. For Dicebot that is a nogo
Not in that example there won't.
Re: Voting: std.logger
On 9/10/14, 1:47 AM, Robert burner Schadek wrote:
On Wednesday, 10 September 2014 at 07:41:49 UTC, Andrei Alexandrescu wrote:
There may be a miscommunication here. In short:
void fun(int x, string f = __FILE__)(double y);
can be replaced with:
void fun(int x)(double y, string f = __FILE__);
Both work and the second produces fewer instantiations.
Andrei
But
void fun(int l = __LINE__, A...)(A...); cannot be replaced by
void fun(A...)(A..., int l = __LINE__);
anyway thanks for reading and for trying to help
One possibility is to have the first function be a one-liner that
forwards to another. That way there will be less code bloating.
void fun(uint l = __LINE__, A...)(A... as) {
funImpl(l, as);
}
private void funImpl(A...)(uint line, A...) {
... bulk of the code goes here ...
}
Andrei
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 16:34:06 UTC, Andrei
Alexandrescu wrote:
One possibility is to have the first function be a one-liner
that forwards to another. That way there will be less code
bloating.
void fun(uint l = __LINE__, A...)(A... as) {
funImpl(l, as);
}
private void funImpl(A...)(uint line, A...) {
... bulk of the code goes here ...
}
Those are already small functions AFAIK (I was speaking about
symbol bloat, not code bloat). It does not help with compilation
issue though - each logging call creates a totally new template
instance which means allocating new object for DMD internal
representation and running semantic phase for it. And mature
applications can have thousands of logging calls. I have yet to
run tests to see actual impact but it concerns me from the pure
DMD internals point of view.
Re: Voting: std.logger
On Wednesday, 10 September 2014 at 15:41:49 UTC, Timon Gehr wrote:
On 09/10/2014 03:22 PM, Daniel Murphy wrote:
Jacob Carlborg wrote in message
news:lupda8$nl1$1...@digitalmars.com...
Could we modify the compiler to allow that? Just for the
special
identifiers __LINE__, __FILE__ and similar. It wouldn't be
possible to
specify a value for that parameter then.
IIRC Andrei has a bugzilla open for this.
Why? I cannot remember a compiler version where the following
did not work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
This is new to me - I can definitely remember trying it and
failing ~ 1-2 years ago.
Re: Voting: std.logger
Am 10.09.2014 17:41, schrieb Timon Gehr:
On 09/10/2014 03:22 PM, Daniel Murphy wrote:
Jacob Carlborg wrote in message news:lupda8$nl1$1...@digitalmars.com...
Could we modify the compiler to allow that? Just for the special
identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to
specify a value for that parameter then.
IIRC Andrei has a bugzilla open for this.
Why? I cannot remember a compiler version where the following did not work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
Except that it unfortunately doesn't do what is intended:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(10,11); }
output:
10 11
expected:
1011 3
Re: Voting: std.logger
On 09/10/2014 08:18 PM, Sönke Ludwig wrote:
Am 10.09.2014 17:41, schrieb Timon Gehr:
On 09/10/2014 03:22 PM, Daniel Murphy wrote:
Jacob Carlborg wrote in message news:lupda8$nl1$1...@digitalmars.com...
Could we modify the compiler to allow that? Just for the special
identifiers __LINE__, __FILE__ and similar. It wouldn't be possible to
specify a value for that parameter then.
IIRC Andrei has a bugzilla open for this.
Why? I cannot remember a compiler version where the following did not
work:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(1,2,3); }
I.e. there is absolutely no issue here.
Except that it unfortunately doesn't do what is intended:
import std.stdio;
void fun(A...)(A args, int l = __LINE__){ writeln(args, ,l); }
void main(){ fun(10,11); }
output:
10 11
expected:
1011 3
Oops! Touché! Thanks. While the code indeed works, there actually _is_
an issue here. :o)
(It is even more embarrassing seeing that I have actually built my own
implementation of IFTI and it actually matches DMD's behaviour in this
case.)
Re: Voting: std.logger
On 09/10/2014 06:54 PM, Dicebot wrote:
On Wednesday, 10 September 2014 at 16:34:06 UTC, Andrei Alexandrescu wrote:
One possibility is to have the first function be a one-liner that
forwards to another. That way there will be less code bloating.
void fun(uint l = __LINE__, A...)(A... as) {
funImpl(l, as);
}
private void funImpl(A...)(uint line, A...) {
... bulk of the code goes here ...
}
Those are already small functions AFAIK (I was speaking about symbol
bloat, not code bloat). It does not help with compilation issue though -
each logging call creates a totally new template instance which means
allocating new object for DMD internal representation and running
semantic phase for it. And mature applications can have thousands of
logging calls. I have yet to run tests to see actual impact but it
concerns me from the pure DMD internals point of view.
Another possibility would be to fix this bug, but the mechanism isn't
particularly elegant nor efficient:
https://issues.dlang.org/show_bug.cgi?id=13455 :o)
Re: Voting: std.logger
On 9/10/14, 11:46 AM, Timon Gehr wrote: Oops! Touché! Thanks. While the code indeed works, there actually _is_ an issue here. :o) Please bugzillize. Thanks! -- Andrei
Re: Voting: std.logger
On Monday, 8 September 2014 at 22:54:36 UTC, Dicebot wrote: nogc is not possible because of custom toString that won't fix, but for default types it is nogc. It should be possible to provide custom implementation that throws ad Error for those cases (and thus fits the requirements) and `toString` has sink-based overload. Are there any reason why it doesn't help? catching an Exception from formattedWrite just to throw an Error and thus allowing nothrow is just silly IMO. sink-based overloads are nice but we don't write the toString methods of the user and so can not be sure that they are nogc.
Re: Voting: std.logger
On Saturday, 30 August 2014 at 02:18:30 UTC, Dicebot wrote: I have likely missed several points but overall it seem pretty clear to me that all requests / concerns have not been addressed and this proposal is not yet ready for another round of review. I made the stdlog creating thread-safe and on stack. I think that was the last point that was mentioned. Also x-post from GitHub PR of my personal nitpick: ... have noticed that all logging functions have file/line/function data as template parameters. This will create insane symbol bloat. While I can understand desire to use some nicer variadic argument API providing at least one log function that it simplistic but moves all compile-time data to run-time default argument is absolutely necessary for me to consider this viable. I do not consider that a problem. The benefit is much higher than the cost of the bigger binary. This has already been a long conversation on some previous thread.
Re: Voting: std.logger
On Saturday, 6 September 2014 at 19:41:54 UTC, Kevin Lamonte wrote: On Wednesday, 3 September 2014 at 13:13:31 UTC, Ola Fosheim Grøstad wrote: Since we are talking about performance, I did some tests and found to my surprise that ~95% of the time consumed in a log call is Clock.currTime's call to clock_gettime(). I submitted a report for it (https://issues.dlang.org/show_bug.cgi?id=13433), but it also brings up how to expose it in the std.logger API. that is with all likelihood a syscall, so there goes your performance The API automatically grabs thisTid and Clock.currTime during construction of the LogEntry (plus it should also grab Thread.getThis and Fiber.getThis). Should this behavior be modifiable by clients, by subclasses, or neither? If so, how? yes, I will move some of it into beginLogMsg
Re: Voting: std.logger
On Tuesday, 9 September 2014 at 19:38:16 UTC, Robert burner Schadek wrote: Also x-post from GitHub PR of my personal nitpick: ... have noticed that all logging functions have file/line/function data as template parameters. This will create insane symbol bloat. While I can understand desire to use some nicer variadic argument API providing at least one log function that it simplistic but moves all compile-time data to run-time default argument is absolutely necessary for me to consider this viable. I do not consider that a problem. The benefit is much higher than the cost of the bigger binary. This has already been a long conversation on some previous thread. This unfortunately makes it almost unusable in absence of --gc-sections supporting compiler for lower level domains. Probably I am missing some data though, can you link the discussion thread? This PR dicussion is so long now that it hangs my browser when uncollapsing threads :(
Re: Voting: std.logger
On Saturday, 30 August 2014 at 02:11:49 UTC, Dicebot wrote: On Tuesday, 26 August 2014 at 21:04:28 UTC, Robert burner Schadek wrote: Jakob Ovrum The *API* must support minimal dynamic memory allocation for the normal execution path. However, theoretical *implementation* changes that reduce memory allocation are not a deal-breaker. This seems to be addressed but though it is desired to verify it via @nogc unittest block which uses stub no-op logger (to verify that nothing in between allocates). One place were GC allocations is unavoidable is core.d:1628 - it will always create FileLogger first and allow assigning custom one later. Is there any way to circumvent it? API must specify a strong stance on threading, whatever the form it takes Does not seem to be addressed at all. At least I see no mentions of it in core.d documentation and logger instance itself is plain __gshared thing. $ grep -R -n shared std/experimental/logger/ std/experimental/logger/core.d:1625:static __gshared Logger logger; std/experimental/logger/core.d:1635:static __gshared LogLevel ll = LogLevel.all; Does not seem enough for sure. I working on this This one might seem unnecessarily strict, but I think the fact that Logger is a class and not an interface smells of poor design. I might still be convinced that having it as a class is justified, but my current stance is that it must be an interface. Neither does seem to be addressed nor I can find any comment on why it is not going to be addressed. Every Logger has to have a LogLevel, interfaces won't work there Francesco Cattoglio As far as I undestood, there's no way right now to do logging without using the GC. And that means it is currently impossible to log inside destructor calls. That is a blocker in my book. First part partially addressed - missing @nogc @nothrow logger implementation out of the box. Considering this request does not go very well with current language implementation it may be ignored for now but official stance about it must be clearly documented. at least for logf nothrow will not work because of a wrong formatting string or args. log can not be nothrow because custom toString for structs and class are allowed. nogc is not possible because of custom toString that won't fix, but for default types it is nogc. Byron Heads Logger should include a shared Logger, or include it in the interface for outside libraries to handle the implementation. There will be libraries that thread internally and will need to support shared logging. Is not addressed. See Jakob Ovrun
Re: Voting: std.logger
On Saturday, 30 August 2014 at 02:16:55 UTC, Dicebot wrote: == Martin Nowak == Support duck-typing for the log functions. Logger should be a concept and log functions should be free-standing UFCS functions that take any `isLogger!T`. To support a global `defaultLog` variable, you could add a Logger interface and a loggerObject shim. See http://dlang.org/phobos/std_range.html#inputRangeObject for this a pattern. Neither seem to be addressed nor countered. Overly complicated IMO
Re: Voting: std.logger
Am Mon, 08 Sep 2014 11:17:48 + schrieb Robert burner Schadek rburn...@gmail.com: On Saturday, 30 August 2014 at 02:16:55 UTC, Dicebot wrote: == Martin Nowak == Support duck-typing for the log functions. Logger should be a concept and log functions should be free-standing UFCS functions that take any `isLogger!T`. To support a global `defaultLog` variable, you could add a Logger interface and a loggerObject shim. See http://dlang.org/phobos/std_range.html#inputRangeObject for this a pattern. Neither seem to be addressed nor countered. Overly complicated IMO This may sound surprising, but I believe if we want to make Phobos consistent and give no incentive to roll your own stuff, we should do this for a lot of APIs. Without going into depth (but we could) there are good reasons to use classes and there are good reasons to use duck typing structs. Another API where this mixed scheme would apply are streams. By using function templates with `if (isLogger!T)` and an abstract class Logger, it will only get instantiated once for all derived classes reducing template bloat, while allowing custom instantiations for logger structs to avoid virtual function calls or GC issues. So I agree with Martin. It is a great way to bring the two camps together without major casualties. -- Marco
Re: Voting: std.logger
Am Mon, 08 Sep 2014 11:06:42 + schrieb Robert burner Schadek rburn...@gmail.com: Francesco Cattoglio As far as I undestood, there's no way right now to do logging without using the GC. And that means it is currently impossible to log inside destructor calls. That is a blocker in my book. First part partially addressed - missing @nogc @nothrow logger implementation out of the box. […] at least for logf nothrow will not work because of a wrong formatting string or args. log can not be nothrow because custom toString for structs and class are allowed. nogc is not possible because of custom toString that won't fix, but for default types it is nogc. It is fairly obvious that the next GC implementation needs to allow allocations during a sweep. Maybe we should just assume that it already works ? -- Marco
Re: Voting: std.logger
On Monday, 8 September 2014 at 12:36:29 UTC, Marco Leise wrote: This may sound surprising, but I believe if we want to make Phobos consistent and give no incentive to roll your own stuff, we should do this for a lot of APIs. Without going into depth (but we could) there are good reasons to use classes and there are good reasons to use duck typing structs. Another API where this mixed scheme would apply are streams. By using function templates with `if (isLogger!T)` and an abstract class Logger, it will only get instantiated once for all derived classes reducing template bloat, while allowing custom instantiations for logger structs to avoid virtual function calls or GC issues. So I agree with Martin. It is a great way to bring the two camps together without major casualties. I think the template bloat argument is invalid as __LINE__ and friends are passed as template arguments to allow write and writef type logging. Anyway I will try to make them free standing
Re: Voting: std.logger
On Monday, 8 September 2014 at 13:20:27 UTC, Robert burner
Schadek wrote:
On Monday, 8 September 2014 at 12:36:29 UTC, Marco Leise wrote:
I think the template bloat argument is invalid as __LINE__ and
friends are passed as template arguments to allow write and
writef type logging.
Anyway I will try to make them free standing
The biggest problem I have currently with this that you, or at
least I, can not override the free standing function.
void log(L)(ref L logger) if(isLogger!L) { ... } will match always
and if I create void log(L)(ref L logger) if(isMySpecialLogger!L)
{ ... }
both match and thats a nogo
Re: Voting: std.logger
Am Mon, 08 Sep 2014 13:37:02 +
schrieb Robert burner Schadek rburn...@gmail.com:
On Monday, 8 September 2014 at 13:20:27 UTC, Robert burner
Schadek wrote:
On Monday, 8 September 2014 at 12:36:29 UTC, Marco Leise wrote:
I think the template bloat argument is invalid as __LINE__ and
friends are passed as template arguments to allow write and
writef type logging.
You are right, this benefit of classes doesn't apply here.
Anyway I will try to make them free standing
The biggest problem I have currently with this that you, or at
least I, can not override the free standing function.
void log(L)(ref L logger) if(isLogger!L) { ... } will match always
and if I create void log(L)(ref L logger) if(isMySpecialLogger!L)
{ ... }
both match and thats a nogo
Ok, no matter what the outcome is, I'll see if I can write a
simple file logger that I can use in RAII struct dtors (where
neither allocations nor throwing seem to be an issue) and that
has a fallback to writing to stderr. I wrote earlier that I
would want a fallback logger if writing via the network fails
or the disk is full, but maybe this logic can be implemented
inside a logger implementation. I haven't actually tried your
API yet!
--
Marco
Re: Voting: std.logger
On Monday, 8 September 2014 at 14:49:06 UTC, Marco Leise wrote: Ok, no matter what the outcome is, I'll see if I can write a simple file logger that I can use in RAII struct dtors (where neither allocations nor throwing seem to be an issue) and that has a fallback to writing to stderr. I wrote earlier that I would want a fallback logger if writing via the network fails or the disk is full, but maybe this logic can be implemented inside a logger implementation. I haven't actually tried your API yet! That should be a no-brainer just have a look at FileLogger and start from there
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 03:05:42 UTC, Kevin Lamonte
wrote:
On Tuesday, 2 September 2014 at 14:53:17 UTC, Dicebot wrote:
This is exactly what I call theoretical speculations. Please
provide specific list like this:
1) some method signature needs to be changed
I propose the following API changes (+ changes on default
implementation):
protected LogEntry beginLogMsg(string file, int line,
string funcName,
string prettyFuncName, string moduleName, LogLevel
logLevel,
Tid threadId, SysTime timestamp, Logger logger)
@trusted
{
static if (isLoggingActive())
{
return LogEntry(file, line, funcName,
prettyFuncName,
moduleName, logLevel, threadId, timestamp,
null, logger);
}
}
/** Logs a part of the log message. */
protected void logMsgPart(MsgRange msgAppender,
const(char)[] msg)
{
static if (isLoggingActive())
{
msgAppender.put(msg);
}
}
/** Signals that the message has been written and no more
calls to
$(D logMsgPart) follow. */
protected void finishLogMsg(ref LogEntry entry, MsgRange
msgAppender)
{
static if (isLoggingActive())
{
entry.msg = msgAppender.data;
this.writeLogMsg(entry);
}
}
...with the corresponding changes to logImpl/logImplf to create
msgAppender as a local function variable, and the elimination
of header and msgAppender as Logger class variables.
The benefit to this change is that Logger (including stdlog)
becomes thread-safe, as well as any subclass of Logger that
only implements writeLogMsg().
The only problem with that change is that it will always require
a buffer. FileLogger currently doesn't require a buffer and is
already thread-safe.
stdlog will not be thread-safe by default by this change only
syncing the writeLogMsg function will.
Re: Voting: std.logger
On Monday, 8 September 2014 at 11:06:44 UTC, Robert burner Schadek wrote: First part partially addressed - missing @nogc @nothrow logger implementation out of the box. Considering this request does not go very well with current language implementation it may be ignored for now but official stance about it must be clearly documented. at least for logf nothrow will not work because of a wrong formatting string or args. log can not be nothrow because custom toString for structs and class are allowed. nogc is not possible because of custom toString that won't fix, but for default types it is nogc. It should be possible to provide custom implementation that throws ad Error for those cases (and thus fits the requirements) and `toString` has sink-based overload. Are there any reason why it doesn't help?
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 13:13:31 UTC, Ola Fosheim Grøstad wrote: If you accept slightly out of sync logging then you can have thread local buffers and on x86 use the command RDTSC which gives you a (good enough) timer value so you can merge the buffers from threads later. It takes roughly 20-30 cycles which I presume is better than CAS instructions, or you can just write directly to a global counter without CAS and accept that it jitters? Since we are talking about performance, I did some tests and found to my surprise that ~95% of the time consumed in a log call is Clock.currTime's call to clock_gettime(). I submitted a report for it (https://issues.dlang.org/show_bug.cgi?id=13433), but it also brings up how to expose it in the std.logger API. The API automatically grabs thisTid and Clock.currTime during construction of the LogEntry (plus it should also grab Thread.getThis and Fiber.getThis). Should this behavior be modifiable by clients, by subclasses, or neither? If so, how?
Re: Voting: std.logger
On Saturday, 6 September 2014 at 19:41:54 UTC, Kevin Lamonte wrote: The API automatically grabs thisTid and Clock.currTime during construction of the LogEntry (plus it should also grab Thread.getThis and Fiber.getThis). Should this behavior be modifiable by clients, by subclasses, or neither? If so, how? I think maybe we should start with creating a high performance inlined (for ldc/gdc) in-memory multi-threaded binary reference logger and then extend the interface in ways that does not make it noticeably slower using the reference logger as the baseline. (Noticeably 100%?) When logging to an external logging service you might want the logging service do the time-keeping so you don't get merged logs from multiple servers that are out of sync. In that case collecting absolute time locally is kinda pointless (although you might want to submit serial numbers and relative time between logging events from the same server).
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 15:10:35 UTC, Ola Fosheim Grøstad wrote: On Tuesday, 2 September 2014 at 14:53:17 UTC, Dicebot wrote: If you are going to speak more about abstract performance I am going to simply ignore any of your further posts on topic. I am not abstract. These are very concrete requirements: 1. You have to be able to determine types and formatting at compile time otherwise you cannot do binary logging. 2. You have to be able to determine types and formatting at compile time otherwise you cannot minimize the probability of introducing run-time errors by turning on full logging. Ok, this is much more specific. With a similar concerns in mind I have proposed to add a `log` overload that doesn't have variadic arguments and takes a single pre-formatted string (with file/line/module as default run-time arguments). With a custom formatting function (compile-time or run-time at users choice) it should fit these requirements. Does that sounds feasible to you?
Re: Voting: std.logger
On Friday, 5 September 2014 at 18:24:01 UTC, Dicebot wrote: Ok, this is much more specific. With a similar concerns in mind I have proposed to add a `log` overload that doesn't have variadic arguments and takes a single pre-formatted string (with file/line/module as default run-time arguments). With a custom formatting function (compile-time or run-time at users choice) it should fit these requirements. Does that sounds feasible to you? P.S. Reason why compile-time format checking can't be added to base API is rather simple - it needs to be at least somewhat similar to one of writefln
Re: Voting: std.logger
On Friday, 5 September 2014 at 18:27:12 UTC, Dicebot wrote: P.S. Reason why compile-time format checking can't be added to base API is rather simple - it needs to be at least somewhat similar to one of writefln Hm. My experience with Python for server use tells me that the problem with dynamic languages isn't the primarily the main execution paths, but the exceptional ones. Having to update a server because a request erroneously fails due to a logging statement (or typos in asserts or any other kind of debugging statement) is annoying. Upon further reflection I think it is important to require logging to be a fail-safe transparent operation (conceptually close to an annotation). I don't think monitoring statements should be able to cause runtime errors at all.
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 22:34:30 UTC, Kevin Lamonte wrote: Sounds like a candidate for an attribute, just prefix a function or function call with @trace(level)? I've got a feature request in for just that: https://issues.dlang.org/show_bug.cgi?id=13406 While thinking about it I realize that it's actually very easy to generalize @trace into the equivalent of Common Lisp :before, :after, and :around methods: @scope(scopeFn) . (It would work even better if scope(success) and scope(failure) exposed what they are returning/throwing.) In the meantime Log4D has a (barely tested) mixin. This can already be implemented in a library if mixin of an implementation idiom is used. I think it fits D style better (having attributes modify actual code flow is unprecedented)
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 10:14:27 UTC, Ola Fosheim Grøstad
wrote:
On Tuesday, 2 September 2014 at 06:24:45 UTC, Kevin Lamonte
wrote:
I see a difference between signalling state, tracing execution
and logging state. I guess one roughly can say that:
- signalling is for coordination of subsystems
- logging state is for tracking effects on the database
- tracing is for detecting logic failure after a crash
I've written my own ideas about logging vs tracing over at
https://github.com/klamonte/log4d/docs/philosophy.md . In a
nutshell, logging means the engineered messages targeting
non-developers that are part of the application deliverable and
follow the narrative flow, while tracing is the automated
messages targeting the developers that follow the structural flow.
std.logger provides an acceptable interface for logging and an
absolute minimal interface to tracing, with just the single
LogLevel.trace and trace/tracef functions. Most other systems
provide at least two (debug+trace) or three (fine/finer/finest)
levels between INFO and everything, and some provide
log.entering()/log.exiting() functions that could provide for
what you are describing.
Sounds interesting. I'll have a look at log4d later. But it is
not fully typesafe then? The way I see it you should log a
tuple of values and a reference to a type-safe formatting
expression, but only apply the formatting expression when you
need to so you don't burden the performance thread with
unnecessary work.
Well, sort of. It's not CTFE-like typesafe where the compiler
catches it, but it also isn't possible (AFAIK) to get it wrong
either. PatternLayout's format specifiers don't perform
conversions on client-controlled input, they are straight
substitutions of the LogEntry fields (+ some extras) into the
final emitted string.
It sounds like what you would like is a trace function that
doesn't feature a format string at all, but formatting would
instead be applied at I/O time by a Logger subclass. How about
this?
1. Add a fnArgs field of type Object [] to LogEntry
2, Add something like the following:
void traceFn(int line = __LINE__,
string file = __FILE__, string funcName = __FUNCTION__,
string prettyFuncName = __PRETTY_FUNCTION__,
string moduleName = __MODULE__)(lazy Object [] fnArgs)
@trusted
{
static if (isLoggingActive!LogLevel.trace)
{
if (isLoggingEnabled(LogLevel.trace)
ll = stdlog.logLevel
stdlog.logLevel = globalLogLevel
globalLogLevel != LogLevel.off
stdlog.logLevel != LogLevel.off)
{
auto entry = LogEntry(file, line, funcName,
prettyFuncName,
moduleName, LogLevel.trace, thisTid,
Clock.currTime, null, this, fnArgs);
this.writeLogMsg(entry);
}
}
}
This could also be split into traceFnEnter, traceFnExitSuccess,
and traceFnExitFailure with LogEntry.args set to indicate which
one (, , ! for example) and a mixin provided so that
client code could get it all in a one-liner.
If this would meet your needs, I wouldn't mind it myself. Save
log.trace() for generic messages the developer want to see
between the traceFnEnter/traceFnExitX messages.
Re: Voting: std.logger
Another API change: LogEntry must have a Thread reference, either in addition to or in replacement of the Tid reference it has currently.
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 11:39:59 UTC, Kevin Lamonte wrote: I've written my own ideas about logging vs tracing over at https://github.com/klamonte/log4d/docs/philosophy.md Nice writeup! It is kind of interesting that there are so many different takes on a mundane task such as logging, monitoring, debug-tracing etc. Reminds me of reflections on how many types of NULL you need to cover all the different semantics that NULL can express (was it 5 or 6?). It sounds like what you would like is a trace function that doesn't feature a format string at all, but formatting would instead be applied at I/O time by a Logger subclass. Yes, either that or no formatting at all. If all formatting strings are literals and resolve at compile time then you can extract them from the source and create a big compile time map that convert them into numeric values at compile time and convert the types into numeric values as well. If the API supports full compile time reflection that should be a possibility. High performance logging should just be a series of MOV instructions or SSE equivalents that completes in ~8-40 cycles for a circular buffer with 2^N size. With increasing availability of memory on cloud servers this becomes more and more attractive IMO (you can log a lot in 50MB) It important is that you exploit the fact that the values are already in registers because you usually log values that have recently been computed and that you spend a minimal amount of execution time on registering them, perhaps even writing directly to full memory cache lines to avoid cache pollution (using special SSE commands). If you accept slightly out of sync logging then you can have thread local buffers and on x86 use the command RDTSC which gives you a (good enough) timer value so you can merge the buffers from threads later. It takes roughly 20-30 cycles which I presume is better than CAS instructions, or you can just write directly to a global counter without CAS and accept that it jitters? I personally don't care about enter/exit so much. I care about: 1. tracking the state of the system configuration at the point of failure 2. the paths of execution before the incident 3. the events that led up to it (in order to reproduce the failure). This could also be split into traceFnEnter, traceFnExitSuccess, and traceFnExitFailure with LogEntry.args set to indicate which one (, , ! for example) and a mixin provided so that client code could get it all in a one-liner. Sounds like a candidate for an attribute, just prefix a function or function call with @trace(level)?
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 13:13:31 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 3 September 2014 at 11:39:59 UTC, Kevin Lamonte wrote: This could also be split into traceFnEnter, traceFnExitSuccess, and traceFnExitFailure with LogEntry.args set to indicate which one (, , ! for example) and a mixin provided so that client code could get it all in a one-liner. Sounds like a candidate for an attribute, just prefix a function or function call with @trace(level)? I've got a feature request in for just that: https://issues.dlang.org/show_bug.cgi?id=13406 While thinking about it I realize that it's actually very easy to generalize @trace into the equivalent of Common Lisp :before, :after, and :around methods: @scope(scopeFn) . (It would work even better if scope(success) and scope(failure) exposed what they are returning/throwing.) In the meantime Log4D has a (barely tested) mixin.
Re: Voting: std.logger
On Wednesday, 3 September 2014 at 22:34:30 UTC, Kevin Lamonte wrote: I've got a feature request in for just that: https://issues.dlang.org/show_bug.cgi?id=13406 Interesting! I am not 100% convinced that scope(failure/success) is the way to go since it will cause potentially a lot of extra work when unwinding the stack in case of an exception? Or maybe the stack unwinder could to the tracing directly without using the scope construct. Another option would be to only trace the landing-pad (catch statement) for exceptions, so you would get something like exception X caught in functionname(), but not sure how to do it without loosing information. You might need a counter for each enter/exit or something like that and let the stack-unwinder count down.
Re: Voting: std.logger
On Thursday, 4 September 2014 at 04:53:36 UTC, Ola Fosheim Grøstad wrote: You might need a counter for each enter/exit or something like that and let the stack-unwinder count down. I meant: you might need to increment a stack specific counter, but that does not sound practical. I guess it is better to have a more complex stack unwinder. Co-routines create some problems here. You probably need to trace that they yield.
Re: Voting: std.logger
On Monday, 1 September 2014 at 10:43:34 UTC, Ola Fosheim Grøstad wrote: I guess the most robust solution is to use shared memory and fork, when the child dies you soup up the log and upload it to a logging-server. I'm used to a centralized system taking logs on a continuous basis, with normal, I'm happy messages coming through in a regular interval. When the application dies, it already has had its messages emitted and sucked up by the collection system, and if its heartbeat messages aren't seen then people are prompted to investigate anyway. It's on the main server (e.g. syslog or LogStash) to handle storage space issues like log rotation. I am also interested in lazy formatting, meaning you log a reference to the immutable formatting string and the parameters, but wait with the formatting until the result is needed. log4d does this, it saves the Logger+LogEntry references and only applies PatternLayouts at the end, BUT it does so with the risk that additional fields specified in the conversionPattern might be wrong since they were not generated/evaluated in the context of the original log call. Thread ID (%t) is the main culprit (since it is not in LogEntry, PatternLayout has to get it), but also time between log calls and time since application startup (%r/%R). But it sounds like you want std.logger to not apply formatting even in its infof/errorf/etc functions. That might be a problem for things like the number of rows in a result set, the current time, or the remote system hostname. For example, by the time the logging infrastructure evaluates the number of rows, the result set is long gone and could throw an exception. I would argue that this kind of lazy evaluation would be fine if it was not enabled by default.
Re: Voting: std.logger
On Monday, 1 September 2014 at 18:57:25 UTC, Ola Fosheim Grøstad wrote: On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote: You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business. Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario. +1 While its useful for the standard library to provide stubs, these mean very little without a default implementation. And the latter shall be quite generic to cover most of the use cases. Rolling one's own loggers shall be done only if the world is not enough.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 06:53:30 UTC, eles wrote: Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario. +1 While its useful for the standard library to provide stubs, these mean very little without a default implementation. And the latter shall be quite generic to cover most of the use cases. Rolling one's own loggers shall be done only if the world is not enough. I disagree and it was declared among goals of this module from the very beginning that it won't go that way, won't even attempt to do it. If you have some good ideas about better default implementation - make pull request after it is merged into std.experimental. Right now it is not in the scope of the review and I will simply ignore all comments that are related purely to implementation. However, if there any API issues that will likely block the implementation you want - those are very important to hear about. This is #1 priority right now.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 06:24:45 UTC, Kevin Lamonte wrote: I'm used to a centralized system taking logs on a continuous basis, with normal, I'm happy messages coming through in a regular interval. When the application dies, it already has had its messages emitted and sucked up by the collection system, and if its heartbeat messages aren't seen then people are prompted to investigate anyway. It's on the main server (e.g. syslog or LogStash) to handle storage space issues like log rotation. Yes, I think we are discussing many different things at the same time here and it would be a good idea to sort out the different use cases since that affects functionality. I have not thought about heartbeats/keep-alive etc as logging, but it is reasonable to view them as such. I see a difference between signalling state, tracing execution and logging state. I guess one roughly can say that: - signalling is for coordination of subsystems - logging state is for tracking effects on the database - tracing is for detecting logic failure after a crash ? log4d does this, it saves the Logger+LogEntry references and only applies PatternLayouts at the end, BUT it does so with the risk that additional fields specified in the conversionPattern might be wrong since they were not generated/evaluated in the context of the original log call. Thread ID (%t) is the main culprit (since it is not in LogEntry, PatternLayout has to get it), but also time between log calls and time since application startup (%r/%R). Sounds interesting. I'll have a look at log4d later. But it is not fully typesafe then? The way I see it you should log a tuple of values and a reference to a type-safe formatting expression, but only apply the formatting expression when you need to so you don't burden the performance thread with unnecessary work. But it sounds like you want std.logger to not apply formatting even in its infof/errorf/etc functions. That might be a problem for things like the number of rows in a result set, the current time, or the remote system hostname. For example, by the time the logging infrastructure evaluates the number of rows, the result set is long gone and could throw an exception. I think you should log the values as a tuple, but not do the string-conversion, but it is more important for tracing execution than for logging. I guess formatting high level info/error is acceptable, but for tracing I am only interested in very terse value/type info along with an indicator of context. Performance and trouble-free type safe logging is much more important than nice formatting IMO. Traced execution will primarily be used for log analysis after a crash. You can use this on game clients, game servers, web servers etc… E.g.: log configuration + trace last 5 events - crash - compress and upload for analysis.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 07:10:29 UTC, Dicebot wrote: into std.experimental. Right now it is not in the scope of the review and I will simply ignore all comments that are related purely to implementation. Configuration of logging is part of the API. Conversion of objects to log info affects the API. The API affects performance. You need to design the API with a reference model in mind. Without it the API has no value. That's why you need a reference implementation in order to evaluate the API design. However, if there any API issues that will likely block the implementation you want - those are very important to hear about. This is #1 priority right now. I am concerned about performance, formatting and type safety. You need to: 1. define the use cases you want to cover 2. list the requirements 3. define a model Only then does it make sense to define the API. If D is to have the upper hand as a system level language then the logging must be performant. For performant logging you might not want to do string-formatting at all in the engine you are monitoring.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 10:25:03 UTC, Ola Fosheim Grøstad wrote: However, if there any API issues that will likely block the implementation you want - those are very important to hear about. This is #1 priority right now. I am concerned about performance, formatting and type safety. You need to: 1. define the use cases you want to cover 2. list the requirements 3. define a model Only then does it make sense to define the API. If D is to have the upper hand as a system level language then the logging must be performant. For performant logging you might not want to do string-formatting at all in the engine you are monitoring. Sorry but it doesn't work that way. If you are concerned about those cases it is you who must do necessary research and provided specific list of requirements / changes. No one else is going to do it. While your raise important concerns it doesn't have any practical application right now and I can't use it in any way as part of review process. We need details (see the responses of other voters). Pure theoretical speculations won't help.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 13:08:02 UTC, Dicebot wrote: While your raise important concerns it doesn't have any practical application right now and I can't use it in any way as part of review process. We need details (see the responses of other voters). Pure theoretical speculations won't help. Uhm, it isn't theoretical. A low performance string based stdio-logger is not useful in a high performance server where you have short spikes with idle time between the spikes. A coarse grained logger logs state on the application level and I don't need library support for that since it only happens in a handful of locations that I control myself. A fine grained logger logs state on the framework/library level and I don't want to use a logger that spends time on turning ints into strings when it is supposed to be handling requests and sits idle a few milliseconds later. The phobos design lacks a performance oriented focus and si too scripty for a system level langauge. You need benchmarking from the get go. Performance does not happen later as a QoI issue because performance depends on the model the API implies. Fine grained logging must be performant.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 13:58:24 UTC, Ola Fosheim Grøstad wrote: On Tuesday, 2 September 2014 at 13:08:02 UTC, Dicebot wrote: While your raise important concerns it doesn't have any practical application right now and I can't use it in any way as part of review process. We need details (see the responses of other voters). Pure theoretical speculations won't help. Uhm, it isn't theoretical. A low performance string based stdio-logger is not useful in a high performance server where you have short spikes with idle time between the spikes. A coarse grained logger logs state on the application level and I don't need library support for that since it only happens in a handful of locations that I control myself. A fine grained logger logs state on the framework/library level and I don't want to use a logger that spends time on turning ints into strings when it is supposed to be handling requests and sits idle a few milliseconds later. The phobos design lacks a performance oriented focus and si too scripty for a system level langauge. You need benchmarking from the get go. Performance does not happen later as a QoI issue because performance depends on the model the API implies. Fine grained logging must be performant. This is exactly what I call theoretical speculations. Please provide specific list like this: 1) some method signature needs to be changed 2) design decision that makes GC allocation unavoidable for specific use case 3) example logger implementation that doesn't fit into existing API (show how) If you are going to speak more about abstract performance I am going to simply ignore any of your further posts on topic. Sorry but there is no other way.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 14:53:17 UTC, Dicebot wrote: If you are going to speak more about abstract performance I am going to simply ignore any of your further posts on topic. I am not abstract. These are very concrete requirements: 1. You have to be able to determine types and formatting at compile time otherwise you cannot do binary logging. 2. You have to be able to determine types and formatting at compile time otherwise you cannot minimize the probability of introducing run-time errors by turning on full logging. This is very important use scenarios: * Performant fine grained logging (as close to zero overhead as you get) in libraries and frameworks is essential for fixing cloud-based servers where you typically cannot use regular strategies. I don't control frameworks, so it is important that they use standard lib logging. * Performant fine grained logging (as close to zero overhead as you get) in client side frameworks is essential for fixing online game clients that runs on heterogenous hardware since you don't get to run a debugger on all configurations. I don't know enough about what the limitations for advanced compile time reflection is, but it has been claimed in this thread that the current design does not make it possible to establish this at compile time. Then I have to conclude that the current design cannot be safe enough or performant enough to be useful in libraries and frameworks and give D an advantage in the server-market... Risks: 1. If D libraries and frameworks starts using and under-performing logger because it is the official D logger library, then you cannot ship products with fine grained logging based on these framworks. This is a lost opportunity. 2. If phobos includes under-performing libraries then you risk having a new split and have two standard libraries and/or two incompatible logging frameworks.
Re: Voting: std.logger
On Tuesday, 2 September 2014 at 14:53:17 UTC, Dicebot wrote:
This is exactly what I call theoretical speculations. Please
provide specific list like this:
1) some method signature needs to be changed
I propose the following API changes (+ changes on default
implementation):
protected LogEntry beginLogMsg(string file, int line, string
funcName,
string prettyFuncName, string moduleName, LogLevel
logLevel,
Tid threadId, SysTime timestamp, Logger logger)
@trusted
{
static if (isLoggingActive())
{
return LogEntry(file, line, funcName, prettyFuncName,
moduleName, logLevel, threadId, timestamp, null,
logger);
}
}
/** Logs a part of the log message. */
protected void logMsgPart(MsgRange msgAppender, const(char)[]
msg)
{
static if (isLoggingActive())
{
msgAppender.put(msg);
}
}
/** Signals that the message has been written and no more
calls to
$(D logMsgPart) follow. */
protected void finishLogMsg(ref LogEntry entry, MsgRange
msgAppender)
{
static if (isLoggingActive())
{
entry.msg = msgAppender.data;
this.writeLogMsg(entry);
}
}
...with the corresponding changes to logImpl/logImplf to create
msgAppender as a local function variable, and the elimination of
header and msgAppender as Logger class variables.
The benefit to this change is that Logger (including stdlog)
becomes thread-safe, as well as any subclass of Logger that only
implements writeLogMsg().
Re: Voting: std.logger
On Monday, 1 September 2014 at 04:32:42 UTC, Kevin Lamonte wrote: Does this logger already exist, could I take a look at it? Not in D AFAIK, circular in-memory logging is a common technique for fixing servers though. If not, if someone writes an appender for writing to the database, you could accomplish this goal with log4d using a buffer appender that triggers on fatal. I guess the most robust solution is to use shared memory and fork, when the child dies you soup up the log and upload it to a logging-server. I am also interested in lazy formatting, meaning you log a reference to the immutable formatting string and the parameters, but wait with the formatting until the result is needed.
Re: Voting: std.logger
On Sunday, 31 August 2014 at 01:09:33 UTC, Ola Fosheim Grøstad wrote: I've got some questions: How does logging interact with pure? You need to be able to log in pure functions. Weakly pure function can take logger as an argument (so I doubt it is useful in practice). Strongly pure functions can't be logged in non-debug statement pretty much by D definition of purity. This may be or may not be an issue but is definitely goes out of the scope of this Phobos proposal. If you have any specific ideas how to address it, please create a separate thread. Does the logger implementation flush() in the case of a crash? (Does it trap all crashes in a way that ensures that logging buffers are written to disk?) Current implementations use basic std.stdio facilities and those flush upon writing a newline symbol - flush happens after each log call. More efficient buffered implementation can be provided later, this shouldn't affect the API. Is logf() needed? Can't you somehow detect that the string is an immutable string literal with string formatting characters? Unreliable, not KISS. I think it is a bad idea.
Re: Voting: std.logger
On Monday, 1 September 2014 at 16:30:46 UTC, Dicebot wrote: purity. This may be or may not be an issue but is definitely goes out of the scope of this Phobos proposal. If you have any […] Current implementations use basic std.stdio facilities and those flush upon writing a newline symbol - flush happens after each log call. More efficient buffered implementation can be provided later, this shouldn't affect the API. […] Unreliable, not KISS. I think it is a bad idea. If the standard library does not provide the following from the get go: 1. general usefulness 2. performance 3. KISS in terms of interface (not in language semantics) then people will be better off rolling their own. Creating a simple logger is not difficult, the challenge is in creating a performant, generally useful one with legible syntax and full CT type safety.
Re: Voting: std.logger
On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote: You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here *are expected
Re: Voting: std.logger
On Monday, 1 September 2014 at 16:45:29 UTC, Ola Fosheim Grøstad wrote: If the standard library does not provide the following from the get go: 1. general usefulness 2. performance 3. KISS in terms of interface (not in language semantics) then people will be better off rolling their own. Creating a simple logger is not difficult, the challenge is in creating a performant, generally useful one with legible syntax and full CT type safety. You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business. And no, magic identification of format string is neither language KISS nor interface KISS.
Re: Voting: std.logger
On Monday, 1 September 2014 at 16:52:16 UTC, Dicebot wrote: You are totally misunderstanding goals of std.logger - people as _expected_ to roll their own Loggers. std.logger is here only to provide standard API for those to integrate with each other and few trivial common implementation as examples. Rest is dub business. Oh, I understand the intentions perfectly well, but the default should be performant, multithreaded, and cover the most common use scenario. Without a performant reference implementation that can be used for performance testing it isn't convincing. You also need to account for issues such as multi-threading, throttling etc. Meaning, you might need to have queues and a merger-thread, defer formatting etc etc. And no, magic identification of format string is neither language KISS nor interface KISS. Formatting should be typesafe and preferably configurable with custom formatters, so it should resolve at compile-time one way or the other. You don't want an exceptional path to have a runtime error triggered by a log() statement, i.e. you don't want the possibility that turning on logging can introduce bugs. That should be the most important requirement.
Re: Voting: std.logger
Am Sun, 31 Aug 2014 01:09:32 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: I've got some questions: How does logging interact with pure? You need to be able to log in pure functions. How do you come to that conclusion? Purity is a synonym for _not_ having side effects. That said - as usual - debug statements allow you to punch a hole into purity. […] Is logf() needed? Can't you somehow detect that the string is an immutable string literal with string formatting characters? 1) The first argument does not need to be a literal. 2) Checking the first argument for formatting chars slows the system down. 3) If you want to log a regular string, e.g. an incoming HTTP request or something that contains formatting symbols, log() would throw an exception about a missing second argument. This in turn could become a DOS vulnerability. Other than that, you could create an additional log function that only accepts compile-time known formatting strings as a CT argument and verifies the runtime argument types at the same time. -- Marco
Re: Voting: std.logger
On Tuesday, 29 July 2014 at 06:09:25 UTC, Andrei Alexandrescu wrote: 4. Replace defaultLogger with theLog. Logger is a word, but one that means lumberjack so it doesn't have the appropriate semantics. The use is generally acceptable as a nice play on words and as a disambiguator between the verb to log and the noun log. When we actually want to talk about the current log in an application, we should, however, call it the log. This is negotiable. FWIW: https://www.google.com/search?q=logger Here's the results I see: 8 results related to programming and log files 1 Wikipedia disambiguation page 1 sports-related result (La Crosse Loggers)
Re: Voting: std.logger
On Sunday, 31 August 2014 at 06:11:56 UTC, Marco Leise wrote: Am Sun, 31 Aug 2014 01:09:32 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: How does logging interact with pure? You need to be able to log in pure functions. How do you come to that conclusion? Purity is a synonym for _not_ having side effects. That said - as usual - debug statements allow you to punch a hole into purity. 1. ~90% of all functions are weakly pure, if you cannot log execution in those functions then logging becomes a liability. 2. If you define logging in a weakly pure function as tracing of execution rather than logging of state, then you can allow memoization too. 3. You don't normally read back the log in the same execution, state is thus not preserved through logging within a single execution. It has traits which makes it less problematic than general side effects that change regular global variables.
Re: Voting: std.logger
Am Sun, 31 Aug 2014 08:52:58 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: On Sunday, 31 August 2014 at 06:11:56 UTC, Marco Leise wrote: Am Sun, 31 Aug 2014 01:09:32 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: How does logging interact with pure? You need to be able to log in pure functions. How do you come to that conclusion? Purity is a synonym for _not_ having side effects. That said - as usual - debug statements allow you to punch a hole into purity. 1. ~90% of all functions are weakly pure, if you cannot log execution in those functions then logging becomes a liability. 2. If you define logging in a weakly pure function as tracing of execution rather than logging of state, then you can allow memoization too. Ok, here is the current state: Logging is not a first-class language feature with special semantics. Drop your pure keywords on those 90% of functions or only log in debug. 3. You don't normally read back the log in the same execution, state is thus not preserved through logging within a single execution. It has traits which makes it less problematic than general side effects that change regular global variables. I still don't see it fly even theoretically. The stdlog will be an interface with an arbitrary implementation behind it. A file logger will eventually hit a disk full state and throw an exception. Since pure implies that a function call can be elided such a change of execution path cannot work. It is much like discussing the strictness of transitive const. If you need to cache values of initialize on first access, you just have to drop const. -- Marco
Re: Voting: std.logger
On Sunday, 31 August 2014 at 09:34:51 UTC, Marco Leise wrote: Am Sun, 31 Aug 2014 08:52:58 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: On Sunday, 31 August 2014 at 06:11:56 UTC, Marco Leise wrote: Am Sun, 31 Aug 2014 01:09:32 + schrieb Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com: I still don't see it fly even theoretically. The stdlog will be an interface with an arbitrary implementation behind it. A file logger will eventually hit a disk full state and throw an exception. Why would that be the sole outcome? There are several strategies to cope with that, maybe through a special logger. For example: - start writing over the old logs transparently (because, usually, the most important logs are the most recent ones) - simply fake logging, but not logging anything more (keeping all history and simply discard anything that comes after the disk is full) These could be solved easily, by catching the exception and either processing it (the first strategy), either by ignoring it (the second strategy). But it matters to have the functionality by default.
Re: Voting: std.logger
On Sunday, 31 August 2014 at 09:34:51 UTC, Marco Leise wrote: Ok, here is the current state: Logging is not a first-class language feature with special semantics. Drop your pure keywords on those 90% of functions or only log in debug. Then maybe the language lacks features that allow you to escape purity in a safe manner. But maybe you should only allow trace() and not log() in pure functions. Tracing execution and logging state. I still don't see it fly even theoretically. The stdlog will be an interface with an arbitrary implementation behind it. An interface can require specific properties of the implementation. file logger will eventually hit a disk full state and throw an exception. Since pure implies that a function call can be elided such a change of execution path cannot work. Not sure what you mean by this. The logger I am most interested in writes to a circular buffer and uploads the log to a database on a crash so that the source of the crash can be identified. I am only interested in in logging execution, not preserved state without execution. It is not uncommon to have loggers that writes to a fixed size preallocated area that behaves like a circular buffer (e.g. retain at most 1GB and 3 months old logging-data)
Re: Voting: std.logger
On Sunday, 31 August 2014 at 09:56:29 UTC, Ola Fosheim Grøstad wrote: The logger I am most interested in writes to a circular buffer and uploads the log to a database on a crash so that the source of the crash can be identified. I am only interested in in logging execution, not preserved state without execution. Does this logger already exist, could I take a look at it? If not, if someone writes an appender for writing to the database, you could accomplish this goal with log4d using a buffer appender that triggers on fatal.
Re: Voting: std.logger
On 8/30/14, 5:18 AM, Dicebot wrote: I have likely missed several points but overall it seem pretty clear to me that all requests / concerns have not been addressed and this proposal is not yet ready for another round of review. How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei
Re: Voting: std.logger
On Saturday, 30 August 2014 at 02:11:49 UTC, Dicebot wrote: API must specify a strong stance on threading, whatever the form it takes Does not seem to be addressed at all. At least I see no mentions of it in core.d documentation and logger instance itself is plain __gshared thing. I'm not a voter as far as I know, but I suggest that Logger is formally documented to be not-thread-safe, and that it is the responsibility of subclasses to provide thread safety. Actual Logger's can do so easily by overloading beginLogMsg/logMsgPart/finishLogMsg and not using a shared class variable between calls. Loggers could also remain thread-local but overload writeLogMsg to dispatch the result to thread-safe I/O. The latter is the path I have chosen currently for log4d (https://github.com/klamonte/log4d). (That may change in the future if there is memory pressure from too many thread-local logger instances.) We need to hammer out how this will work inside libraries. If my app is using multiple libraries I need to know I have full control of how they log and where (), and if I write libraries I need to include logging that will not affect performance or added dependencies. This is a task for a fuller backend, not the std.logger frontend. std.logger should be mechanism, interfacing multiple libraries together requires policy. Logger should include a shared Logger, or include it in the interface for outside libraries to handle the implementation. There will be libraries that thread internally and will need to support shared logging. Is not addressed. stdlog is global and the default implementation is thread-safe by way of FileLogger mutex-locking its writes. It should either be documented that stdlog is always global and it is up to subclasses or logging backends to make stdlog thread-safe, or Logger should be made thread-safe by default by eliminating it's header and msgAppender fields. I'm not sure either way which is better. I don't like the idea that libraries A, B, and C will just blindly use stdlog and hope for the best. If they are logging (and not tracing, for which I hope D evolves a @trace attribute) then they should be logging by category/name, which is a job for something like log4d. But Logger's don't have names. Perhaps the better solution would be a convention for libraries to always use a static Logger Logger.getLogger(string category) function, for which the default simply ignores category and returns stdlog. Logger.setGetLoggerFunction(...) could be provided for backends to change this behavior on application startup, assuming they can guarantee that they can call this function before various libraries have gotten references to stdlog.
Re: Voting: std.logger
On Saturday, 30 August 2014 at 02:11:49 UTC, Dicebot wrote: Jakob Ovrum The *API* must support minimal dynamic memory allocation for the normal execution path. However, theoretical *implementation* changes that reduce memory allocation are not a deal-breaker. This seems to be addressed but though it is desired to verify it via @nogc unittest block which uses stub no-op logger (to verify that nothing in between allocates). One place were GC allocations is unavoidable is core.d:1628 - it will always create FileLogger first and allow assigning custom one later. Is there any way to circumvent it? Yes - split the `stdlog` property into a getter and a setter. Then if the setter is called first (with a non-null reference), the getter never gets to allocate the default instance. I pointed this out in a line comment before, but I guess it disappeared due to the name change... Also, as I said in the line comment, it doesn't need to GC-allocate, it can allocate in global memory or TLS using emplace (which of them is appropriate depends on the threading model, I guess...). If Andrei's reference-counting suggestion is implemented, then depending on the details, it's possible that the default logger could be allocated on the C heap instead of the GC heap as well, managed by RC.
Re: Voting: std.logger
On Saturday, 30 August 2014 at 09:54:36 UTC, Andrei Alexandrescu wrote: On 8/30/14, 5:18 AM, Dicebot wrote: I have likely missed several points but overall it seem pretty clear to me that all requests / concerns have not been addressed and this proposal is not yet ready for another round of review. How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation
Re: Voting: std.logger
On Sat, Aug 30, 2014 at 01:08:05PM +, Dicebot via Digitalmars-d wrote: On Saturday, 30 August 2014 at 09:54:36 UTC, Andrei Alexandrescu wrote: On 8/30/14, 5:18 AM, Dicebot wrote: I have likely missed several points but overall it seem pretty clear to me that all requests / concerns have not been addressed and this proposal is not yet ready for another round of review. How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation Quality of Implementation. T -- The peace of mind---from knowing that viruses which exploit Microsoft system vulnerabilities cannot touch Linux---is priceless. -- Frustrated system administrator.
Re: Voting: std.logger
On Saturday, 30 August 2014 at 13:35:57 UTC, H. S. Teoh via Digitalmars-d wrote: How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation Quality of Implementation. And how can one break issues into quality of implementation? :O I have literally no idea what Andrei means.
Re: Voting: std.logger
On Sat, Aug 30, 2014 at 01:37:15PM +, Dicebot via Digitalmars-d wrote: On Saturday, 30 August 2014 at 13:35:57 UTC, H. S. Teoh via Digitalmars-d wrote: How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation Quality of Implementation. And how can one break issues into quality of implementation? :O I have literally no idea what Andrei means. I'm guessing he means, break them down into must-solve-before-merge issues and stuff-to-be-fixed-later issues. T -- Acid falls with the rain; with love comes the pain.
Re: Voting: std.logger
On Saturday, 30 August 2014 at 13:37:17 UTC, Dicebot wrote: On Saturday, 30 August 2014 at 13:35:57 UTC, H. S. Teoh via Digitalmars-d wrote: How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation Quality of Implementation. And how can one break issues into quality of implementation? :O I have literally no idea what Andrei means. Implementation issues can be fixed later as long as the API allows for it, so we should focus on the latter.
Re: Voting: std.logger
On 8/30/14, 4:42 PM, H. S. Teoh via Digitalmars-d wrote: On Sat, Aug 30, 2014 at 01:37:15PM +, Dicebot via Digitalmars-d wrote: On Saturday, 30 August 2014 at 13:35:57 UTC, H. S. Teoh via Digitalmars-d wrote: How do the remaining issues break down into QoI that can be deferred to a future release? -- Andrei What is QoI? I am not familiar with this abbreviation Quality of Implementation. And how can one break issues into quality of implementation? :O I have literally no idea what Andrei means. I'm guessing he means, break them down into must-solve-before-merge issues and stuff-to-be-fixed-later issues. Correct. (I'm on vacation with scarce online access.) -- Andrei
Re: Voting: std.logger
On Saturday, 30 August 2014 at 20:53:58 UTC, Andrei Alexandrescu wrote: Correct. (I'm on vacation with scarce online access.) -- Andrei With API stability in mind defining official stance on multi-threading in most important part. Rest can be added with small to none breaking changes.
Re: Voting: std.logger
I've got some questions: How does logging interact with pure? You need to be able to log in pure functions. Does the logger implementation flush() in the case of a crash? (Does it trap all crashes in a way that ensures that logging buffers are written to disk?) Is logf() needed? Can't you somehow detect that the string is an immutable string literal with string formatting characters?
Re: Voting: std.logger
On Tuesday, 26 August 2014 at 21:04:28 UTC, Robert burner Schadek wrote: On Tuesday, 26 August 2014 at 18:23:31 UTC, Dicebot wrote: I will compare changelist against list of requirements from voters this weekend and if all seems to be addressed will start a new round of review/voting. sounds good! just to note, I will be mostly offline for a good week starting friday. Ok, going through the list of No voters. Jakob Ovrum The *API* must support minimal dynamic memory allocation for the normal execution path. However, theoretical *implementation* changes that reduce memory allocation are not a deal-breaker. This seems to be addressed but though it is desired to verify it via @nogc unittest block which uses stub no-op logger (to verify that nothing in between allocates). One place were GC allocations is unavoidable is core.d:1628 - it will always create FileLogger first and allow assigning custom one later. Is there any way to circumvent it? API must specify a strong stance on threading, whatever the form it takes Does not seem to be addressed at all. At least I see no mentions of it in core.d documentation and logger instance itself is plain __gshared thing. $ grep -R -n shared std/experimental/logger/ std/experimental/logger/core.d:1625:static __gshared Logger logger; std/experimental/logger/core.d:1635:static __gshared LogLevel ll = LogLevel.all; Does not seem enough for sure. This one might seem unnecessarily strict, but I think the fact that Logger is a class and not an interface smells of poor design. I might still be convinced that having it as a class is justified, but my current stance is that it must be an interface. Neither does seem to be addressed nor I can find any comment on why it is not going to be addressed. Andrei Alexandrescu Minimal logging level must be selected statically in addition to the current dynamic selection. Static settings preclude dynamic settings. This is not negotiable. Seems to be addressed. All logging code must be rigorously eliminated if below the static logging level. More precisely, there must be a front-end optimization that eliminates all code dedicated to a lazy variable that's not used by a generic function. This would be a fantastic redeeming of the lazy keyword, which has been of marginal utility until now. The challenge here is cooperating with someone on the compiler team to make sure that front-end improvement gets implemented, and writing unit tests that make sure there's no regression later. This is not negotiable. Seems to be addressed. The suffix notations must be replaced with overloads. The only acceptable suffix is f for formatting. Everything else must be achieved via overloads with the help of template constraints. Robert's answer http://goo.gl/FehDVh suggests he didn't consider using template constraints. We can't let that slip become a feature of the library for millenia to come. Seems to be addressed. Replace defaultLogger with theLog. Logger is a word, but one that means lumberjack so it doesn't have the appropriate semantics. The use is generally acceptable as a nice play on words and as a disambiguator between the verb to log and the noun log. When we actually want to talk about the current log in an application, we should, however, call it the log. This is negotiable. Addressed with a name of stdlog. I'm still hoping for RefCounted as the storage for the class backend. I realize users can do their own management but log objects are unlikely to contain cycles and other liabilities for reference counting, and at some point if we want to use reference counting where appropriate we got to start somewhere with a few solid precedents. This is negotiable, but I plan to fight for it. Can't see any traces of RefCounted in sources though I may have missed change of mind in PR discussion (sorry it is too huge to pay regular attention) Casey However, I wanted to address the suffix notation as I suggested it. What I was going for was consistency with the write/writef method signatures to keep things consistent. I felt it would be good to make the two similar since they do similar things. Obsolete with overload-based resolution Francesco Cattoglio As far as I undestood, there's no way right now to do logging without using the GC. And that means it is currently impossible to log inside destructor calls. That is a blocker in my book. First part partially addressed - missing @nogc @nothrow logger implementation out of the box. Considering this request does not go very well with current language implementation it may be ignored for now but official stance about it must be
Re: Voting: std.logger
I have likely missed several points but overall it seem pretty clear to me that all requests / concerns have not been addressed and this proposal is not yet ready for another round of review. Also x-post from GitHub PR of my personal nitpick: ... have noticed that all logging functions have file/line/function data as template parameters. This will create insane symbol bloat. While I can understand desire to use some nicer variadic argument API providing at least one log function that it simplistic but moves all compile-time data to run-time default argument is absolutely necessary for me to consider this viable.
Re: Voting: std.logger
== David Nadlinger == I agree. For this reason, I also vote for no (1 as well as 2), as the current conditional logging support doubles the size of the API for shaving a grand total of 3 characters off the invocation in a rather infrequent use case. Addressed. Wow, upon further code review I discovered that Logger actually overrides opCmp/opEquals to be based on the name (?!). This leads to the following gem: .. Fixed. == Martin Nowak == Get rid of the 8 different suffixes. I only see the need for log and logf, why is the rest needed? Addressed. Support duck-typing for the log functions. Logger should be a concept and log functions should be free-standing UFCS functions that take any `isLogger!T`. To support a global `defaultLog` variable, you could add a Logger interface and a loggerObject shim. See http://dlang.org/phobos/std_range.html#inputRangeObject for this a pattern. Neither seem to be addressed nor countered.
Re: Voting: std.logger
On Wednesday, 27 August 2014 at 06:14:10 UTC, Marco Leise wrote: I really think this is the case where you should roll your own FileNoThrowingLogger and go with it. *Me* or everyone who needs to log something in a dtor? I really believe that necessity to log something in dtor is currently an indicator of design issues in the application. D class destructors are unreliable to the point of being almost useless, I'd be very careful about actually caring if those run at all. In a long term this is something much better to be fixed in GC implementation than constantly hacked in stdlib modules. Or is this maybe the other language change (besides not generating code for unused lambdas) that should be pushed with std.log, because otherwise it will never be solved ? I don't know, but no logging in dtors is a serious and hard to sell limitation. Not the author's fault though. I agree that current situation is bad but I disagree that std.log is a proper place to solve it. It is a part of much bigger issue and adding few local workarounds does not make situation much better.
Re: Voting: std.logger
Am Wed, 27 Aug 2014 01:09:21 + schrieb Dicebot pub...@dicebot.lv: On Wednesday, 27 August 2014 at 00:09:15 UTC, Marco Leise wrote: As far as I know, exactly this is not possible with the current GC implementation. The exception you catch there has just been allocated somewhere deeper in the log function. But all GC allocations in a GC invoked dtor cause MemoryErrors and program abortion/crashes. :( In a perfect world I'd imagine you can set up a fallback logger. So if the disk is full an exception is thrown by e.g. std.stdio.File, which is passed as an error level log message to the fallback logger, which might write to stderr: ERROR: Could not write the following message to logXYZ: message The reason was: Disk full I don't think it will help either. The very moment exception is allocated inside std.stdio.File your program will crash, it won't get to fallback. Only solution is to implement your logger as @nothrow thing by using only C functions internally instead of std.stdio - something that feels overly limited for a general use case. Exactly, I just needed someone else to speak it out. :) I really think this is the case where you should roll your own FileNoThrowingLogger and go with it. *Me* or everyone who needs to log something in a dtor? In a long term this is something much better to be fixed in GC implementation than constantly hacked in stdlib modules. Or is this maybe the other language change (besides not generating code for unused lambdas) that should be pushed with std.log, because otherwise it will never be solved ? I don't know, but no logging in dtors is a serious and hard to sell limitation. Not the author's fault though. -- Marco
Re: Voting: std.logger
On Tuesday, 26 August 2014 at 19:39:26 UTC, Marco Leise wrote: Am Tue, 26 Aug 2014 18:23:30 + schrieb Dicebot pub...@dicebot.lv: On Tuesday, 26 August 2014 at 15:44:19 UTC, Robert burner Schadek wrote: BTW: Someone else mentioned it before: Logging in destructors would be a requirement for me, too. It would be a pity to have forbidden spaces for logging. I very much like the fact that use of printk() is so much ubiquitous.
