Re: [go-nuts] how to design log package to avoid allocations

['Axel Wagner' via golang-nuts]

IMO, there really isn't a super good answer. The simple answer is: You need
to delay the actual `fmt.Sprintf` call as long as possible. Which generally
means, that the interface you consume (to allow the user to direct and
configure logging) will need to reflect formatting and all kinds of things
that the user actually doesn't want to deal with. So it's hard to find a
good tradeoff between API simplicity and usability and not allocating in
this case.

Another, maybe unforseen problem, is that you don't want logging to
interfere with production. It's a painful lesson to learn, that synchronous
logging can take down a production service with ease. If you want to be
prepared for that and reduce the runtime overhead of logging, you will have
to make it asynchronous. But that's fundamentally at odds with the above
goal of delaying formatting - the user will usually not expect arguments to
loggers to be used after the logging call returns. Buffering a formatted
string avoids that of course.

Lastly, if you want runtime-configurability of logging and conditionally
discard messages, you will likely reach for interfaces and thus limit the
usefulness of escape analysis and inlining. So, by trying to allocate less,
you might end up allocating more, because the compiler has to put things on
the heap to be safe.

FWIW, I've blogged about this
 a
couple of years ago. The opinions expressed there are probably
controversial (and I haven't looked at it in a while - I might not even
agree with them myself anymore), but it might help open up even more
questions or give you some ideas. Also, there are of course many, many
logging APIs already existing and I'm sure the people who wrote and use
them will have strong opinions about those as well :) Personally, I've kind
of given up on the idea of a panacea log API.

On Mon, Mar 9, 2020 at 4:27 PM Vasiliy Tolstov  wrote:

> Hi! I have some logging package and after memory profiling saw that for
> disabled log levels i'm allocate memory for message.
> For example i'm send in logger only Info level. So i want to avoid
> Debug/Trace stuff.
> but memory profiling says that for call log.Tracef("my message %v", msg)
> i'm allocate memory.
> I don't want to guard all calls to check enabled log level to
> avoid allocation. How can i do zero allocation log in such case?
> Thanks!
>
> --
> Vasiliy Tolstov,
> e-mail: v.tols...@selfip.ru
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CACaajQswqK3DGHESCFd4pCdxndFpOFyKXhO25UDKgEsxYeK3SA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfF6kRp0V4LUGxfu3qDu4K7mpz%3DdhA76EU_qd1DfveZk-Q%40mail.gmail.com.

Re: [go-nuts] how to design log package to avoid allocations

[andrey mirtchovski]

to avoid allocations you have to hint at the type of what you're going
to print. for example see/use zerolog: https://github.com/rs/zerolog

On Mon, Mar 9, 2020 at 10:36 AM 'Axel Wagner' via golang-nuts
 wrote:
>
> IMO, there really isn't a super good answer. The simple answer is: You need 
> to delay the actual `fmt.Sprintf` call as long as possible. Which generally 
> means, that the interface you consume (to allow the user to direct and 
> configure logging) will need to reflect formatting and all kinds of things 
> that the user actually doesn't want to deal with. So it's hard to find a good 
> tradeoff between API simplicity and usability and not allocating in this case.
>
> Another, maybe unforseen problem, is that you don't want logging to interfere 
> with production. It's a painful lesson to learn, that synchronous logging can 
> take down a production service with ease. If you want to be prepared for that 
> and reduce the runtime overhead of logging, you will have to make it 
> asynchronous. But that's fundamentally at odds with the above goal of 
> delaying formatting - the user will usually not expect arguments to loggers 
> to be used after the logging call returns. Buffering a formatted string 
> avoids that of course.
>
> Lastly, if you want runtime-configurability of logging and conditionally 
> discard messages, you will likely reach for interfaces and thus limit the 
> usefulness of escape analysis and inlining. So, by trying to allocate less, 
> you might end up allocating more, because the compiler has to put things on 
> the heap to be safe.
>
> FWIW, I've blogged about this a couple of years ago. The opinions expressed 
> there are probably controversial (and I haven't looked at it in a while - I 
> might not even agree with them myself anymore), but it might help open up 
> even more questions or give you some ideas. Also, there are of course many, 
> many logging APIs already existing and I'm sure the people who wrote and use 
> them will have strong opinions about those as well :) Personally, I've kind 
> of given up on the idea of a panacea log API.
>
> On Mon, Mar 9, 2020 at 4:27 PM Vasiliy Tolstov  wrote:
>>
>> Hi! I have some logging package and after memory profiling saw that for 
>> disabled log levels i'm allocate memory for message.
>> For example i'm send in logger only Info level. So i want to avoid 
>> Debug/Trace stuff.
>> but memory profiling says that for call log.Tracef("my message %v", msg) i'm 
>> allocate memory.
>> I don't want to guard all calls to check enabled log level to avoid 
>> allocation. How can i do zero allocation log in such case?
>> Thanks!
>>
>> --
>> Vasiliy Tolstov,
>> e-mail: v.tols...@selfip.ru
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CACaajQswqK3DGHESCFd4pCdxndFpOFyKXhO25UDKgEsxYeK3SA%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfF6kRp0V4LUGxfu3qDu4K7mpz%3DdhA76EU_qd1DfveZk-Q%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAK4xykU0Ra_c0p3nGY0j%3DfgPdvWDBbRDm1Sg7j6o3md-CjWJtA%40mail.gmail.com.

Re: [go-nuts] how to design log package to avoid allocations

[Vasiliy Tolstov]

пн, 9 мар. 2020 г. в 19:41, andrey mirtchovski :

> to avoid allocations you have to hint at the type of what you're going
> to print. for example see/use zerolog: https://github.com/rs/zerolog


Tanks, I saw it. But mostly i want to avoid typing hint

-- 
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACaajQuANkcEisvyh3ad3s%2B0MP1dpR9vaoh7-CTjWF_G5SOYCA%40mail.gmail.com.

Re: [go-nuts] how to design log package to avoid allocations

[Vasiliy Tolstov]

пн, 9 мар. 2020 г. в 19:36, Axel Wagner :

> IMO, there really isn't a super good answer. The simple answer is: You
> need to delay the actual `fmt.Sprintf` call as long as possible. Which
> generally means, that the interface you consume (to allow the user to
> direct and configure logging) will need to reflect formatting and all kinds
> of things that the user actually doesn't want to deal with. So it's hard to
> find a good tradeoff between API simplicity and usability and not
> allocating in this case.
>
> Another, maybe unforseen problem, is that you don't want logging to
> interfere with production. It's a painful lesson to learn, that synchronous
> logging can take down a production service with ease. If you want to be
> prepared for that and reduce the runtime overhead of logging, you will have
> to make it asynchronous. But that's fundamentally at odds with the above
> goal of delaying formatting - the user will usually not expect arguments to
> loggers to be used after the logging call returns. Buffering a formatted
> string avoids that of course.
>
>
Thanks! So i think that most simple case is check each time log level , and
output log only if level satisfied.
-- 
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACaajQshiJaz%2BAJ_Tm5VbTW8FDfP0QyLvzj_KCGDTAM7RpiPrg%40mail.gmail.com.

Re: [go-nuts] how to design log package to avoid allocations

[Marcin Romaszewicz]

I'm using Uber's zap logger in production systems (
https://github.com/uber-go/zap). It is designed to emit structured JSON
logs and do the minimal amount of allocations possible. It's a completely
different interface from Go's log package, but that is where the efficiency
comes from. I'd highly recommend it, and the JSON logs are easy to ingest
into services like Datadog or Sumologic.

-- Marcin

On Mon, Mar 9, 2020 at 9:52 AM Vasiliy Tolstov  wrote:

>
>
> пн, 9 мар. 2020 г. в 19:36, Axel Wagner :
>
>> IMO, there really isn't a super good answer. The simple answer is: You
>> need to delay the actual `fmt.Sprintf` call as long as possible. Which
>> generally means, that the interface you consume (to allow the user to
>> direct and configure logging) will need to reflect formatting and all kinds
>> of things that the user actually doesn't want to deal with. So it's hard to
>> find a good tradeoff between API simplicity and usability and not
>> allocating in this case.
>>
>> Another, maybe unforseen problem, is that you don't want logging to
>> interfere with production. It's a painful lesson to learn, that synchronous
>> logging can take down a production service with ease. If you want to be
>> prepared for that and reduce the runtime overhead of logging, you will have
>> to make it asynchronous. But that's fundamentally at odds with the above
>> goal of delaying formatting - the user will usually not expect arguments to
>> loggers to be used after the logging call returns. Buffering a formatted
>> string avoids that of course.
>>
>>
> Thanks! So i think that most simple case is check each time log level ,
> and output log only if level satisfied.
> --
> Vasiliy Tolstov,
> e-mail: v.tols...@selfip.ru
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CACaajQshiJaz%2BAJ_Tm5VbTW8FDfP0QyLvzj_KCGDTAM7RpiPrg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LsSkRBtJfR9CAhsy1OJnaGTwFMe89oq4VowzQ7hTsJwtQ%40mail.gmail.com.