Re: Proposal about new default SSL log format
On Tue, Jul 06, 2021 at 12:19:56PM +0200, Tim Düsterhus wrote: > Willy, > > On 7/6/21 12:12 PM, Willy Tarreau wrote: > > A few points first, that are needed to address various concerns. The > > goal here is to defined an HTTPS log format because that's what the > > vast majority of users are dealing with day-to-day. For specific usages, > > everyone already redefines log formats. But for a very basic setup, right > > now it's true that httplog is a bit limited, and all it shows to help guess > > there was SSL involved is to append this '~' after the frontend's name. > > It was not clear to me that this log format is meant to apply to HTTPS > requests, because the example given by Remi does not include the HTTP verb > and neither the request URI (unless I need glasses). I thought it was a > format for TLS errors or something like this. At least it was my understanding that it was mostly aimed at HTTPS given that the discussion we had on the subject started around this :-) > Is this a mistake in the examples? Or is HAProxy going to emit multiple log > lines: One for the TLS connection and one for each HTTP request passing > through this TLS connection? There was a discussion around logging at various layers, I don't remember if it was on the list or in a github issue, but I remember having been involved in this some time ago. I think that it makes sense to ultimately be able to emit error logs at various levels (i.e. when being certain that no other event will happen) but it's particularly confusing to log successes at various levels. Thus typically seeing TLS handshake failures is important but their success should not be logged as all the info will instead be available with the traffic logs. There are already enough users complaining about noise caused by client-aborted connections :-) Willy
Re: Proposal about new default SSL log format
Willy, On 7/6/21 12:12 PM, Willy Tarreau wrote: A few points first, that are needed to address various concerns. The goal here is to defined an HTTPS log format because that's what the vast majority of users are dealing with day-to-day. For specific usages, everyone already redefines log formats. But for a very basic setup, right now it's true that httplog is a bit limited, and all it shows to help guess there was SSL involved is to append this '~' after the frontend's name. It was not clear to me that this log format is meant to apply to HTTPS requests, because the example given by Remi does not include the HTTP verb and neither the request URI (unless I need glasses). I thought it was a format for TLS errors or something like this. Is this a mistake in the examples? Or is HAProxy going to emit multiple log lines: One for the TLS connection and one for each HTTP request passing through this TLS connection? However, it's also clear that most users will not violently migrate from httplog to httpslog, and it's important to keep a smooth enough transition. This also means not to change stuff that would be relevant to httplog as well (e.g. delimitors, time format etc). We can (and should) have discussions about what to change in future log formats, but let's not use the https one as a bootstrap for passing everyone's missing field. Instead, let's focus on the SSL-specific stuff that users are always missing from HTTP logs, and try to establish a reasonable list that will always be there and suit most users without adding too much for others, and that will require limited adaptations to parsers. Agree. Best regards Tim Düsterhus
Re: Proposal about new default SSL log format
Hi Rémi, [ I warned you that this was going to open a pandora box :-) ] On Fri, Jul 02, 2021 at 04:26:48PM +0200, Remi Tricot-Le Breton wrote: > Some work in ongoing to ease connection error and SSL handshake error > logging. > This will rely on some new sample fetches that could be added to a custom > log-format string. > In order to ease SSL logging and debugging, we will also add a new default > log > format for SSL connections. Now is then the good time to find the best > format > for everyone. > The proposed format looks like the HTTP one to which the SSL specific > information is added. But if anybody sees a missing information that could > be > beneficial for everybody, feel free to tell it, nothing is set in stone yet. A few points first, that are needed to address various concerns. The goal here is to defined an HTTPS log format because that's what the vast majority of users are dealing with day-to-day. For specific usages, everyone already redefines log formats. But for a very basic setup, right now it's true that httplog is a bit limited, and all it shows to help guess there was SSL involved is to append this '~' after the frontend's name. However, it's also clear that most users will not violently migrate from httplog to httpslog, and it's important to keep a smooth enough transition. This also means not to change stuff that would be relevant to httplog as well (e.g. delimitors, time format etc). We can (and should) have discussions about what to change in future log formats, but let's not use the https one as a bootstrap for passing everyone's missing field. Instead, let's focus on the SSL-specific stuff that users are always missing from HTTP logs, and try to establish a reasonable list that will always be there and suit most users without adding too much for others, and that will require limited adaptations to parsers. > The format would look like this : > >>> Jul 1 18:11:31 haproxy[143338]: 127.0.0.1:37740 > [01/Jul/2021:18:11:31.517] \ > ssl_frontend~ ssl_frontend/s2 0/0/0/7/+7 \ > 0/0/0/0 2750 1/1/1/1/0 0/0 TLSv1.3 TLS_AES_256_GCM_SHA384 Like Aleks, I think that we could have a single version+cipher field, as not all combinations are permitted anyway. > Field Format Extract from the example > above > 1 process_name '[' pid ']:' haproxy[143338]: > 2 client_ip ':' client_port 127.0.0.1:37740 > 3 '[' request_date ']' [01/Jul/2021:18:11:31.517] > 4 frontend_name ssl_frontend~ > 5 backend_name '/' server_name ssl_frontend/s2 > 6 TR '/' Tw '/' Tc '/' Tr '/' Ta* > 0/0/0/7/+7 > 7 *conn_status '/' SSL hsk error '/' SSL vfy '/' SSL CA vfy*0/0/0/0 Be careful above not to reproduce the errors from the past, namely using a designation that could apply to both sides. "SSL hsk error" etc could be valid for either frontend connection, backend connection or both. We must absolutely be clear and explicit here about what this is. The same will be true for the versions and ciphers. I suspect that all of these will mostly be of interest for the front side but I could be wrong (particularly for errors). But it's probable that discussing the backend side already enters a new sub-topic in fact. Willy
Re: [PATCH] MEDIUM: stats: include disabled proxies that hold active sessions to stats
Hi Marno, On Mon, Jun 28, 2021 at 07:32:19AM +, Marno Krahmer wrote: > Hello, > > I would like to add a path to HAProxy. > This patch is supposed to change how stats are handled for disabled proxies. (...) It's been one week without comments, my concerns about the rare but possible special corner cases are now addressed, I've merged it. Thanks! Willy
Re: Proposal about new default SSL log format
Hello Aleksandar, On 03/07/2021 13:19, Aleksandar Lazic wrote: Hi Remi. How about to combine ssl_version/ssl_ciphers in one line. Yes why not. It would be helpful to see also the backend status. Maybe add a 14th and 15th line with following fields *backend_name '/' conn_status '/' SSL hsk error '/' SSL vfy '/' SSL CA vfy* *backend_name '/' ssl_version '/' ssl_ciphers* The backend name is already included in the log line, I'll avoid duplicating information. I had in the past several issues with the backend where the backend CA wasn't in the CA File which was quite difficult to debug. Backend related errors won't be raised in the log line since it is emitted on the fronted side of the connection but this backend error problem will also be treated after this. +1 to the suggestion from Илья Шипицин to use iso8601 which is already in haproxy since 2019/10/01:2.1-dev2. I haven't found sub second format parameter in strftime call therefore I assume the strftime call have this ".00" as fix value. ``` strftime(iso_time_str, sizeof(iso_time_str), "%Y-%m-%dT%H:%M:%S.00+00:00", &tm) ``` In the timeofday_as_iso_us function where the strftime call happens, the time zone and microsecond precision are set in the time string just after the strftime call (see the 'offset' variable and the utoa_pad call). Maybe another option is to use TAI for timestamps. https://en.wikipedia.org/wiki/International_Atomic_Time https://cr.yp.to/proto/utctai.html http://www.madore.org/~david/computers/unix-leap-seconds.html Thanks Rémi Jm2c Alex Thanks Rémi
ratio spam/useful message
Hello, Lately, the ratio spam/useful message on the ML has been quite high. Could we maybe force the registration on the mailing list or set moderation for new users? Regards, -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Bid Writing, Fundraising and Volunteering Workshops
NFP WORKSHOPS Affordable Training Courses Bid Writing: The Basics Do you know the most common reasons for rejection? Are you gathering the right evidence? Are you making the right arguments? Are you using the right terminology? Are your numbers right? Are you learning from rejections? Are you assembling the right documents? Do you know how to create a clear and concise standard funding bid? Are you communicating with people or just excluding them? Do you know your own organisation well enough? Are you thinking through your projects carefully enough? Do you know enough about your competitors? Are you answering the questions funders will ask themselves about your application? Are you submitting applications correctly? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95.00 CLICK ON DATE TO BOOK YOUR PLACE MON 05 JUL 2021 MON 19 JUL 2021 MON 02 AUG 2021 MON 23 AUG 2021 MON 06 SEP 2021 MON 20 SEP 2021 MON 04 OCT 2021 MON 18 OCT 2021 Bid Writing: Advanced Are you applying to the right trusts? Are you applying to enough trusts? Are you asking for the right amount of money? Are you applying in the right ways? Are your projects the most fundable projects? Are you carrying out trust fundraising in a professional way? Are you delegating enough work? Are you highly productive or just very busy? Are you looking for trusts in all the right places? How do you compare with your competitors for funding? Is the rest of your fundraising hampering your bids to trusts? Do you understand what trusts are ideally looking for? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95.00 CLICK ON DATE TO BOOK YOUR PLACE TUE 06 JUL 2021 TUE 20 JUL 2021 TUE 03 AUG 2021 TUE 24 AUG 2021 TUE 07 SEP 2021 TUE 21 SEP 2021 TUE 05 OCT 2021 TUE 19 OCT 2021 Recruiting and Managing Volunteers Where do you find volunteers? How do you find the right volunteers? How do you attract volunteers? How do you run volunteer recruitment events? How do you interview volunteers? How do you train volunteers? How do you motivate volunteers? How do you involve volunteers? How do you recognise volunteers? How do you recognise problems with volunteers? How do you learn from volunteer problems? How do you retain volunteers? How do you manage volunteers? What about volunteers and your own staff? What about younger, older and employee volunteers? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 07 JUL 2021 WED 08 SEP 2021 Legacy Fundraising Why do people make legacy gifts? What are the ethical issues? What are the regulations? What are the tax issues? What are the statistics? What are the trends? How can we integrate legacy fundraising into our other fundraising? What are the sources for research? How should we set a budget? How should we evaluate our results? How should we forecast likely income? Should we use consultants? How should we build a case for support? What media and marketing channels should we use? What about in memory giving? How should we setup our admin systems? What are the common problems & pitfalls? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 21 JUL 2021 WED 22 SEP 2021 Major Donor Fundraising Major Donor Characteristics, Motivations and Requirements. Researching and Screening Major Donors. Encouraging, Involving and Retaining Major Donors. Building Relationships with Major Donors. Major Donor Events and Activities. Setting Up Major Donor Clubs. Asking For Major Gifts. Looking After and Reporting Back to Major Donors. Delivering on Major Donor Expectations. Showing Your Appreciation to Major Donors. Fundraising Budgets and Committees. ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 04 AUG 2021 WED 06 OCT 2021 Corporate Fundraising Who are these companies? Why do they get involved? What do they like? What can you get from them? What can you offer them? What are the differences between donations, sponsorship, advertising and cause related marketing? Are companies just like trusts? How do you find these companies? How do you research them? How do you contact them? How do you pitch to them? How do you negotiate with them? When should you say no? How do you draft contracts? How do you manage the relationships? What could go wrong? What are the tax issues? What are the legal considerations? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 25 AUG 2021 WED 20 OCT 2021 Feedback From Past Attendees I must say I was really impressed with the course and the content. My knowledge and confidence has increased hugely. I got a lot from your course and a lot of pointers! I can say after years of fundraising I learnt so much from your bid writing course. It was a very informative day and for someone who has not written bids before I am definitely more confident to get involved with them. I found the workshops very helpful. It is a whole new area for me but the information
