Re: HAProxy-1.8 sometimes sends a shorter html when using multithread function
Hellow Willy, # Sorry I post this message directly to you, # so i resend to the mailing list. I tested the patches you suggested, and got the following results. TestA haproxy-1.8.30 + patch(1 + 2 + 3) Shorter HTML: 0 HTTP 408: 2 TestB haproxy-1.8.30 + patch(2 only) Shorter HTML: 0 HTTP 408: 3 TestC haproxy-2.4.0 Shorter HTML: 0 HTTP 408: 0 haproxy-2.4.0 ran without no issue, but I found the bytes_read column of haproxy-2.4.0 was diffrent from that of haproxy-1.8.30 and haproxy-2.0.22 in the normal cases. Values of bytes_read: haproxy-1.8.30: 4864 haproxy-2.0.22: 4837 (Also 2.3.X) haproxy-2.4.0 : 4832 I don't think the difference of bytes_read is related to the shorter html issue and the HTTP 408 issue, I just noticed so report it. For now, I've ran TestA,B,C once. I am planning to do 2 more tests on each case. >> I also tried haproxy-2.0.22 and I didn't find >> messages about shorter htmls but the number of >> log messages is smaller than the number of HTTP >> accesses. > >This should definitely not happen, but it obviously depends >where they are counted. > >For example some aborted idempotent requests between haproxy >and the server on a reused connection could lead to a silent >close to let the client try again. > >In this case it is guaranteed by the compliance with the >HTTP protocol that the server has not done anything that >cannot be retried, and all haproxy can see there is a >failure to process (while in fact it could just be caused >by an abort in flight). > >So if you're missing very few logs it could be an explanation. >Or you may simply be losing logs since they're sent as datagrams >only in 2.0. Yes, difference of the number of logs was 3. Your explanation hit the point, thank you. Kazu On Tue, May 18, 2021 at 6:52 PM Willy Tarreau wrote: > Hello, > > On Mon, May 17, 2021 at 09:47:10AM +0900, Kazuhiro Takenaka wrote: > > Hello > > > > This is my first post to this mailing list. > > I am not good at using English, so feel free to ask me > > if my text is hard to understand. > > Rest assured that the majority of participants here (me included) do > not have English as their native language, so you're not special on > this point. And I had absolutely no problem understanding all your > explanations, your English is better than you seem to think :-) > > > I noticed haproxy-1.8 sometimes sent incomplete htmls to > > clients when running haproxy with the attached haproxy.cfg > > that uses multithread function. > > > > # I also attached content01.html and check.html that > > # are deployed on backend servers. > > # content01.html is used in the confirmnation test > > # described below, check.html is for health check > > # purpose. > > > > In this case, the client receives a shorter html. > > > > I confirmed haoproxy's log message corresponded to the > > shorter html, the following line is one of them. > > > > 2021-03-23T15:54:46.869626+09:00 lb01 [err] haproxy[703957]: > > 192.168.1.73:60572 [23/Mar/2021:15:54:46.868] vs_http01 > rs_http01web/web01 > > 0/0/0/0/0 200 2896 - - SD-- 12/12/7/2/0 0/0 "GET > > /content01.html?x=170026897 HTTP/1.1" > > So it indicates an immediate close or error that is being detected on > the server side just after two TCP segments. This can have some > importance because depending how the server's TCP stack is tuned, it > is very likely that it will only send two segments before growing its > window, leaving a short blank period after them which can allow haproxy > to detect an error. > > (...) > > The following list is the frequency of abnormal access > > when a total of 10 million accesses are made in 20 parallel > > curl processes. > > > > status_code bytes_read occurence > > 200 4344 1 > => exactly 3 segments > > 200 2896 9 > => exactly 2 segments > > 200 1448 6 > => exactly 1 segment > > It makes me think about a server which closes with a TCP reset after > the data. It then becomes extremely timing-dependent and could explain > why the size is a multiple of the number of segments. > > > 408 2162 > > > > The following line is the log messages > > in the case of 408. > > > > 2021-03-23T16:02:42.444084+09:00 lb01 [err] haproxy[703957]: > > 192.168.1.73:37052 [23/Mar/2021:16:02:32.472] vs_http01 > vs_http01/ > > -1/-1/-1/-1/1 408 212 - - cR-- 14/14/0/0/0 0/0 "" > > So the client's request didn't make it into haproxy. > > > When I first met this behavior, I used haproxy-1.8.25 shipped > > with RHEL8.3. So I obtained haproxy-1.8.30 from http://git.haproxy.org/ > > and built it, ran the test and got the result of the same sort. > > > > This behavior didn't happen without using multithread function. > > So that definitely indicates a race condition somewhere. > > > Next, I tried on haproxy-2.0.0 and confirmed it ran normally > > without this behavior. > > > > Then I picked up several versions of haproxy > > between 1.8.0 and 2.0.0 and built them, test them > > and fo
Re: Termination Code 'CC' + HTTP status?
Willy, On 5/21/21 6:48 PM, Willy Tarreau wrote: I'd like your advice on a few log entries that confuse me. I am seeing HTTP 2.0 requests dying with a termination code of 'CC', i.e.: C : the TCP session was unexpectedly aborted by the client. C : the proxy was waiting for the CONNECTION to establish on the server. The server might at most have noticed a connection attempt. Thus my understanding is that the client is "simply gone". However HAProxy *also logs* a HTTP status of 503 with 0 bytes sent to the client. For requests that never even established a connection to the server and that were aborted by the client I would not have expected any status code to be logged at all, after all the server could not generate one and there is no client that would be able to receive it. I'd say "it depends". If a connection attempt was made, and your servers are flooded with aborted connections coming from haproxy, I'm pretty sure you'll come here and ask why haproxy doesn't produce a log that permits to identify the client and frontend involved in this. Yes, it absolutely makes sense that this connection attempt is logged in HAProxy. It's just that the status code I disagree with. The 503 is just a byproduct of the aborted connection. That can sound strange but what happens here is that the status was set after it was noticed the processing ended without being able to connect to any server. I agree that a 4xx would be more suitable here, and this problem was already raised in the past regarding the fact that *certain* aborts or errors need closer inspection to figure where the problem was. I believe in some cases a status code of 0 (zero) or -1 is logged, but I'm not 100% sure there. I would consider such a status to be appropriate as well, because no HTTP response was actually sent to the client. We've already noted that significant improvements in close/abort need to be made to 1) process them better and 2) report them more accurately. This is a tedious task which *will* inevitably cause regressions, but it's the price to pay to improve that. Shall I file a feature request on GitHub that the 'CC' case does not result in a 5xx, but a 4xx or something clearly invalid? For the time being, status codes are here for a reason and are more s/status/termination/? accurate than a status code (status which could also come from a server by the way). There used to be some corner cases of combined client+server errors reporting server errors, but these faded away with the muxes and I don't think there are still any (or only in tricky cases). Thus I'd encourage you to check the termination flags to be certain. ack, thank you. Best regards Tim Düsterhus Developer WoltLab GmbH -- WoltLab GmbH Nedlitzer Str. 27B 14469 Potsdam Tel.: +49 331 96784338 duester...@woltlab.com www.woltlab.com Managing director: Marcel Werk AG Potsdam HRB 26795 P
Re: Termination Code 'CC' + HTTP status?
On Tue, May 25, 2021 at 10:30:27AM +0200, Tim Düsterhus, WoltLab GmbH wrote: > > The 503 is just a byproduct of the aborted connection. That can sound > > strange but what happens here is that the status was set after it was > > noticed the processing ended without being able to connect to any server. > > I agree that a 4xx would be more suitable here, and this problem was > > already raised in the past regarding the fact that *certain* aborts or > > errors need closer inspection to figure where the problem was. > > I believe in some cases a status code of 0 (zero) or -1 is logged, but I'm > not 100% sure there. I would consider such a status to be appropriate as > well, because no HTTP response was actually sent to the client. You make a good point here. Actually I remember that historically we used to report status -1 when nothing was returned to a client, but it was never officially documented, there used to be only one example in the doc (which is still there by the way). That's probably something we should improve to officially translate a client abort before the server responds. It's still not 100% trivial but should already be easier to deal with than the CD vs SD mess. > > We've already noted that significant improvements in close/abort need to > > be made to 1) process them better and 2) report them more accurately. > > This is a tedious task which *will* inevitably cause regressions, but > > it's the price to pay to improve that. > > Shall I file a feature request on GitHub that the 'CC' case does not result > in a 5xx, but a 4xx or something clearly invalid? Yes, why not. This will save us from forgetting about it. > > > For the time being, status codes are here for a reason and are more > > s/status/termination/? Yes of course :-) Willy
Re: Termination Code 'CC' + HTTP status?
Willy, On 5/25/21 11:02 AM, Willy Tarreau wrote: We've already noted that significant improvements in close/abort need to be made to 1) process them better and 2) report them more accurately. This is a tedious task which *will* inevitably cause regressions, but it's the price to pay to improve that. Shall I file a feature request on GitHub that the 'CC' case does not result in a 5xx, but a 4xx or something clearly invalid? Yes, why not. This will save us from forgetting about it. Okay, here it is: https://github.com/haproxy/haproxy/issues/1266 Best regards Tim Düsterhus Developer WoltLab GmbH -- WoltLab GmbH Nedlitzer Str. 27B 14469 Potsdam Tel.: +49 331 96784338 duester...@woltlab.com www.woltlab.com Managing director: Marcel Werk AG Potsdam HRB 26795 P
Four New Workshops
NFP WORKSHOPS Affordable Training Courses Four New Workshops ONLINE VIA ZOOM 10.00 TO 12.30 COST £95.00 CLICK ON DATE FOR MORE DETAILS OR TO BOOK YOUR PLACE Fundraising OverviewFRI 25 JUN 2021 Finding & Researching Trusts FRI 09 JUL 2021 Donor Databases FRI 23 JUL 2021 Fundraising RegulationsFRI 06 AUG 2021 Bid Writing: The Basics Do you know the most common reasons for rejection? Are you gathering the right evidence? Are you making the right arguments? Are you using the right terminology? Are your numbers right? Are you learning from rejections? Are you assembling the right documents? Do you know how to create a clear and concise standard funding bid? Are you communicating with people or just excluding them? Do you know your own organisation well enough? Are you thinking through your projects carefully enough? Do you know enough about your competitors? Are you answering the questions funders will ask themselves about your application? Are you submitting applications correctly? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95.00 CLICK ON DATE TO BOOK YOUR PLACE MON 24 MAY 2021 MON 07 JUN 2021 MON 21 JUN 2021 MON 05 JUL 2021 MON 19 JUL 2021 Bid Writing: Advanced Are you applying to the right trusts? Are you applying to enough trusts? Are you asking for the right amount of money? Are you applying in the right ways? Are your projects the most fundable projects? Are you carrying out trust fundraising in a professional way? Are you delegating enough work? Are you highly productive or just very busy? Are you looking for trusts in all the right places? How do you compare with your competitors for funding? Is the rest of your fundraising hampering your bids to trusts? Do you understand what trusts are ideally looking for? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95.00 CLICK ON DATE TO BOOK YOUR PLACE TUE 25 MAY 2021 TUE 08 JUN 2021 TUE 22 JUN 2021 TUE 06 JUL 2021 TUE 20 JUL 2021 Legacy Fundraising Why do people make legacy gifts? What are the ethical issues? What are the regulations? What are the tax issues? What are the statistics? What are the trends? How can we integrate legacy fundraising into our other fundraising? What are the sources for research? How should we set a budget? How should we evaluate our results? How should we forecast likely income? Should we use consultants? How should we build a case for support? What media and marketing channels should we use? What about in memory giving? How should we setup our admin systems? What are the common problems & pitfalls? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE THU 27 MAY 2021 WED 21 JUL 2021 Major Donor Fundraising Major Donor Characteristics, Motivations and Requirements. Researching and Screening Major Donors. Encouraging, Involving and Retaining Major Donors. Building Relationships with Major Donors. Major Donor Events and Activities. Setting Up Major Donor Clubs. Asking For Major Gifts. Looking After and Reporting Back to Major Donors. Delivering on Major Donor Expectations. Showing Your Appreciation to Major Donors. Fundraising Budgets and Committees. ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 09 JUN 2021 Corporate Fundraising Who are these companies? Why do they get involved? What do they like? What can you get from them? What can you offer them? What are the differences between donations, sponsorship, advertising and cause related marketing? Are companies just like trusts? How do you find these companies? How do you research them? How do you contact them? How do you pitch to them? How do you negotiate with them? When should you say no? How do you draft contracts? How do you manage the relationships? What could go wrong? What are the tax issues? What are the legal considerations? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 23 JUN 2021 Recruiting and Managing Volunteers Where do you find volunteers? How do you find the right volunteers? How do you attract volunteers? How do you run volunteer recruitment events? How do you interview volunteers? How do you train volunteers? How do you motivate volunteers? How do you involve volunteers? How do you recognise volunteers? How do you recognise problems with volunteers? How do you learn from volunteer problems? How do you retain volunteers? How do you manage volunteers? What about volunteers and your own staff? What about younger, older and employee volunteers? ONLINE VIA ZOOM 10.00 TO 12.30 COST £95 CLICK ON DATE TO BOOK YOUR PLACE WED 07 JUL 2021 Fundraising Overview The workshop is for people who wish to move into fundraising, have just started in fundraising, have responsibilities for fundraisers or who are in fundraising and want to know whether they could be doing it better. We examine each of the different areas of fundraising – public donors, major donors, legacies, trusts and corporates. We explain h
Old Github Issue
Hi. I wanted to cleanup some old issues but was not able due to the fact that I'm not sure if the bugs are still valid, especially for 1.8/1.9 and previous versions. https://github.com/haproxy/haproxy/issues?page=10&q=is%3Aissue+is%3Aopen It would be nice when someone with more knowledge then I can take a look and close the not relevant of fixed issues. Regards Alex
Re: Termination Code 'CC' + HTTP status?
On Tue, May 25, 2021 at 11:09:22AM +0200, Tim Düsterhus, WoltLab GmbH wrote: > Okay, here it is: > > https://github.com/haproxy/haproxy/issues/1266 Thank you! Willy
Your $120,000 Nonprofit Ad Grant Is Waiting
Have You Secured Your $120,000 Per Year Ad Grant For... Nonprofit Execs & Board Members CHARITABLE ORGANIZATIONS VISIT HERE TO SECURE YOUR GRANT https://jimmylarose.lt.acemlnd.com/Prod/link-tracker?redirectUrl=aHR0cHMlM0ElMkYlMkZjYXVzZWluc3BpcmVkbWVkaWEuY29tJTJGZ2V0LWdvb2dsZS1hZC1ncmFudC10b2RheSUyRg==&a=26605550&account=jimmylarose%2Eactivehosted%2Ecom&email=vbeBh1HJZ6KtT6pdCg2Rk3b7ImIKGlCdLGgtNBedJm0%3D&s=b4e69d94bf077ab3e7944b1f14cd3206&i=120A288A16A610 VISIT HERE TO SECURE YOUR GRANT Dear Nonprofit Execs, As a nonprofit organization, you likely face many challenges day-to-day that for-profit companies don't necessarily have to deal with. From fundraising, to volunteer coordination, there are a myriad of important tasks that nonprofits have to juggle on a daily basis, often with limited resources. Fortunately, being a registered 501(c)3 organization does come with some benefits, especially when it comes to marketing opportunities. While that may seem surprising to some, there are actually a variety of ways in which nonprofit organizations can advertise effectively without paying exorbitant fees, or spending a great deal of time. By taking advantage of Google's popular Ad Grants program, nonprofit organizations can gain access to a $10,000 monthly ad credit to utilize on Google's Search Network. This ad credit can help your nonprofit advertise your important mission to qualified individuals using the world's largest search engine. What Is The Google Ad Grants Program? Google first launched their Ad Grants program in 2003 in order to level the playing field, and give nonprofit organizations the same opportunity as for-profit companies to advertise on Google's competitive platform. Since then, the Google Ad Grants program has become extremely popular among nonprofit organizations, and is being utilized by over one hundred thousand nonprofits nationwide. Although Google frequently updates their policies regarding Ad Grant maintenance, nonprofits of all shapes and sizes are consistently seeing value in the program. Cause Inspired Secures $120,000 Ad Grants for Nonprofits Who Is Cause Inspired? Cause Inspired was one of Google's first Certified Professional Agencies, and has continued to cement themselves as leaders in the nonprofit advertising industry. Their team of digital marketing experts has dedicated themselves to helping nonprofit organizations expand the impact of their unique missions with the Google Ad Grant. Their nonprofit focused strategies, paired with their deep understanding of Google's machine learning technology, ensure that your advertising credit will be well spent. Cause Inspired began working with nonprofit organizations because they wanted to create a way in which they could extend their valuable digital marketing capabilities to nonprofits across the world. Through decades of experience working with for-profit companies, their team noticed that nonprofits were often taken advantage of, and being charged unnecessarily high prices for marketing services. Cause Inspired was created specifically to help nonprofits reap the many benefits of the Google Ad Grants program without charging expensive management fees. The Cause Inspired team is truly passionate about expanding the reach of nonprofit's missions through online efforts, and working with them to help accomplish their important goals. Why Is Cause Inspired The Best Solution For Nonprofits? In order to help nonprofits maximize the value of their Google Ad Grant, Cause Inspired has developed their own process, called 'Insight,' to ensure that their client's ads are as effective as possible. Insight involves combining human inspiration with Google's artificial intelligence (AI) technology. Their Certified Professionals understand and train one of the most complicated AIs, using it to give nonprofits insight and access to their target audiences. Inspired by the unique missions of their clients, Cause Inspired takes what is instinctively human-inspiration, love and empathy-and uses AI to solve some of the most complicated problems that nonprofits face. Cause Inspired not only has the passion and dedication that it takes to obtain and manage Google Ad Grants for nonprofits, but their team also has the experience needed to navigate Google's challenging softwares for tracking website data, and creating ad campaigns. In fact, Cause Inspired is the largest Google Ad Gant service provider, and their team is trusted by over 500 nonprofits with varying missions and scopes. Due to their size, high ethical standards, and consistent use of best practices, Cause Inspired has developed a direct relationship with Google's Ad Grants team. With this direct line to Google, they are able to get answers quickly, and keep their client's ads at peak performance. When Google creates new policies, Cause Inspired understands their exact purpose and what Google hopes to achieve. This allows them to optimize accounts in a way that meets and ex
