Re: mod_md 1.1.0 repeating on error

[Steffen]

Luca says: executing scripts in response to an event 


I hope you are aware of the MDNotifyCmd, which is executing a script 
on the event that a renew is OK.  I am not a coder: for me looks it 
not that difficult  to make an Else statement when it is NOT_OK.  To 
notify on error  is much more needed then notify on all fine.


Luca says: get all the feedback from a broader users community 
(the one testing latest 2.4.x versions)


We have quite a base of early adopters, my 
comments/testing/requests/behaviors are mostly based on the feedback 
from them who trying to run on the latest versions. Sadly enough there 
are users that are stopping, mostly for them it is not that easy to 
understand, special on errors. Also some do not see an added value 
over for example certbot or letsencrypt-win-simple, a typical feedback 
is  at www.apachelounge.com/viewtopic.php?p=35905#35905 .

Time for us to advertise the add value over them.

Steffen


On Wednesday 13/12/2017 at 14:19, Luca Toscano  wrote:


Hi Steffen,


2017-12-12 19:20 GMT+01:00 Steffen :


Your advise is that I stop testing/ helping ? Please answer this 
question.
If yes, then I have to  remove the test mod_md download from Apache 
Lounge and cannot give anymore support  there to the users I 
represent. Do not under estimate the many users that are testing via 
me, not only windows.



Please don't ask these questions, we all value the ton of time that 
you are spending in giving user feedback for mod_md, nobody will ever 
ask you to stop trying to improve httpd :)






Am 12.12.2017 um 16:19 schrieb Steffen :


>>> All that said: I do not want to make more changes to mod_md before 
a release. If we find a
>>> serious error, sure. But otherwise I'd rather enhance the 
documentation for now.


>> Looks here you are bikeshedding my reports/requests (again). What 
is serious, an introduction with too much user questions is not good. 
Consider me as a user in this case.


>> I spend tons of hours in testing mod_md (I am not paid).   My only 
goal was to get quality and a usable function set  for our users, and 
not to frustrate you. Looks you are only interested now to get it in 
httpd as quickly as possible, I can only guess the reason for your 
pressing.



To be completely honest, the above sentences are not really nice to 
read, but I do think that there was no intention to send strong 
statements, only to suggest improvements. From my point of view, both 
of you are bringing up valid points:


1) Stefan is saying that we should not concentrate on all the things 
that can be improved before this first release, but on serious error 
conditions that could indicate a broken behavior for our users. The 
MDNotifyCmd endless retry (if I got the context correctly) is surely 
something to improve in the future, but it is such a difficult subject 
(executing scripts in response to an event and figure out what is an 
error and what is not) to get right straight away, that it might be 
enough for the moment (in my opinion) to release now and then get all 
the feedback from a broader users community (the one testing latest 
2.4.x versions).


2) Steffen is trying to avoid confusion for our users when dealing 
with a directive that might raise some questions due to also to the 
fact that the docs do not state exactly what should be the right 
behavior.


Thanks to both for all the work that you are doing, I hope to see some 
consensus soon :)


Luca

Re: mod_md 1.1.0 repeating on error

[Luca Toscano]

Hi Steffen,

2017-12-12 19:20 GMT+01:00 Steffen :
>
>
> Your advise is that I stop testing/ helping ? Please answer this question.
> If yes, then I have to  remove the test mod_md download from Apache Lounge
> and cannot give anymore support  there to the users I represent. Do not
> under estimate the many users that are testing via me, not only windows.
>

Please don't ask these questions, we all value the ton of time that you are
spending in giving user feedback for mod_md, nobody will ever ask you to
stop trying to improve httpd :)


>  Am 12.12.2017 um 16:19 schrieb Steffen :
> >>
> >>> All that said: I do not want to make more changes to mod_md before a
> release. If we find a
> >>> serious error, sure. But otherwise I'd rather enhance the
> documentation for now.
> >>
> >> Looks here you are bikeshedding my reports/requests (again). What is
> serious, an introduction with too much user questions is not good. Consider
> me as a user in this case.
> >>
> >> I spend tons of hours in testing mod_md (I am not paid).   My only goal
> was to get quality and a usable function set  for our users, and not to
> frustrate you. Looks you are only interested now to get it in httpd as
> quickly as possible, I can only guess the reason for your pressing.
>

To be completely honest, the above sentences are not really nice to read,
but I do think that there was no intention to send strong statements, only
to suggest improvements. From my point of view, both of you are bringing up
valid points:

1) Stefan is saying that we should not concentrate on all the things that
can be improved before this first release, but on serious error conditions
that could indicate a broken behavior for our users. The MDNotifyCmd
endless retry (if I got the context correctly) is surely something to
improve in the future, but it is such a difficult subject (executing
scripts in response to an event and figure out what is an error and what is
not) to get right straight away, that it might be enough for the moment (in
my opinion) to release now and then get all the feedback from a broader
users community (the one testing latest 2.4.x versions).

2) Steffen is trying to avoid confusion for our users when dealing with a
directive that might raise some questions due to also to the fact that the
docs do not state exactly what should be the right behavior.

Thanks to both for all the work that you are doing, I hope to see some
consensus soon :)

Luca

Re: mod_md 1.1.0 repeating on error

[Steffen]

Help !
Your reaction makes me very sad that you accuse me. 

Other dev’s are slilence about mod_md, as you said before: they are shoulder 
clapping etc. And you said: You got to be kiddding me! Afterwords maybe better 
that I was also more silent. But on the other-hand we did not discovered all 
the errors and improvements. 

Marketing wise is that the first release not raises too much questions and have 
dev’s for support.  So other dev’s please go testing it. 

 I am very serious in the mail and I do not see any personal offense. Only the 
sentence with  bikeshedding can maybe trigger, but that sentence I begin with 
“Looks”, so you can answer there normal. Please let me know where more you feel 
offense, maybe better off -list. 

You ask what I want here: I want the best possible module. As I said I want 
quality and a good function set for our users. Looks you have an other goal at 
the moment and emotional. Looks you want to stall issues to get it quick in 
httpd (this is no offense), fine but say it, then can I shut my mouth on the 
list etc. and wait till the experimental module is released. 

Your advise is that I stop testing/ helping ? Please answer this question. 
If yes, then I have to  remove the test mod_md download from Apache Lounge and 
cannot give anymore support  there to the users I represent. Do not under 
estimate the many users that are testing via me, not only windows. 

Last question on this list about mod_md:
What  is your opinion about the MDNotifyCMD on error. 

Still some design decisions are not clear to me. 

Cheers and drying my tears from your mail. 

Sorry that I triggered you in some way. 

Steffen


> Op 12 dec. 2017 om 18:07 heeft Stefan Eissing  
> het volgende geschreven:
> 
> There is so much personal offence and mal-attribution in this mail, I am not 
> sure where you get that from. I ask you to take a deep breath and consider 
> what you want here. Just because I do what I feel is right does not make me 
> your enemy. Give me a break.
> 
> -Stefan
> 
> 
 Am 12.12.2017 um 16:19 schrieb Steffen :
>>> 
>>> - No, I will not change the endless retry behaviour.
>> 
>> I still not see the reason for retries on all the errors, I am not aware of 
>> other modules doing it this way.
>> 
>> But Ok, when mdnotifycmd on error is added (see other post feature request). 
>> Then I  vote a  SUPER +1  for an experimental release.
>> 
>> User are going to be happy with mdnotifycmd on error, had already requests. 
>> Now it only triggered when it is OK, which is not logical.
>> 
>> See for a mdnotifycmd script see : 
>> https://www.apachelounge.com/viewtopic.php?p=36054#36054 
>> 
>> 
>>> - Yes, I expect people to check their server logs, somehow. Either manual 
>>> or with some tools.
>>>   There is more than mod_md that can have trouble and every module 
>>> implementing its own
>>>   solution is not a good service for users.
>> 
>> Be aware that most users only check there logs when there is an issue. Agree 
>> a standard for every module (the retry is not standard behavior), but mod_md 
>> log  messages must be more clear to see what is going on, see also next 
>> point.
>>> - Yes, I think some log levels need adjustment, e.g. when LE cannot be 
>>> reached at all.
>> 
>> Let me know which one you pick. We had mentions on GIT about this.
>>> - Yes, I think there should be a high level NOTICE log entry when a 
>>> certificate could not
>>>   be renewed and the existing one only last a couple of days.
>> 
>> Agree, plus a mdnotifycmd should be nice.
>> 
>>> All that said: I do not want to make more changes to mod_md before a 
>>> release. If we find a
>>> serious error, sure. But otherwise I'd rather enhance the documentation for 
>>> now.
>> 
>> Looks here you are bikeshedding my reports/requests (again). What is 
>> serious, an introduction with too much user questions is not good. Consider 
>> me as a user in this case. 
>> 
>> I spend tons of hours in testing mod_md (I am not paid).   My only goal was 
>> to get quality and a usable function set  for our users, and not to 
>> frustrate you. Looks you are only interested now to get it in httpd as 
>> quickly as possible, I can only guess the reason for your pressing.
>> 
>>> If you want to add some Windows specific advice to the mod_md XML, please 
>>> do so.
>> 
>> At the moment no advice. On Apache Lounge users can follow the dev at 
>> http://www.apachelounge.com/viewtopic.php?t=7786 
>> 
>> 
>> PS.
>> For windows there is still testing and work (like cmake) to be done. I did 
>> not tested completely the .dsp/mak etc.  I see in trunk there are still 
>> changes on the way, so when voting starts for 2.4.30 then I shall test all 
>> again.
>> 
>> 
>> Regards,
>> 
>> Steffen
>> 
>> 
>>> On Tuesday 12/12/2017 at 14:49, Stefan Eissing wrote:
>>> 
>>> 
 Am 12.12.2017 um 14:37 schrieb Steffen >>> 

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

There is so much personal offence and mal-attribution in this mail, I am not 
sure where you get that from. I ask you to take a deep breath and consider what 
you want here. Just because I do what I feel is right does not make me your 
enemy. Give me a break.

-Stefan


> Am 12.12.2017 um 16:19 schrieb Steffen :
> 
>> - No, I will not change the endless retry behaviour.
> 
> I still not see the reason for retries on all the errors, I am not aware of 
> other modules doing it this way.
> 
> But Ok, when mdnotifycmd on error is added (see other post feature request). 
> Then I  vote a  SUPER +1  for an experimental release.
> 
> User are going to be happy with mdnotifycmd on error, had already requests. 
> Now it only triggered when it is OK, which is not logical.
> 
> See for a mdnotifycmd script see : 
> https://www.apachelounge.com/viewtopic.php?p=36054#36054 
> 
> 
>> - Yes, I expect people to check their server logs, somehow. Either manual or 
>> with some tools.
>>There is more than mod_md that can have trouble and every module 
>> implementing its own
>>solution is not a good service for users.
> 
> Be aware that most users only check there logs when there is an issue. Agree 
> a standard for every module (the retry is not standard behavior), but mod_md 
> log  messages must be more clear to see what is going on, see also next point.
>> - Yes, I think some log levels need adjustment, e.g. when LE cannot be 
>> reached at all.
> 
> Let me know which one you pick. We had mentions on GIT about this.
>> - Yes, I think there should be a high level NOTICE log entry when a 
>> certificate could not
>>be renewed and the existing one only last a couple of days.
> 
> Agree, plus a mdnotifycmd should be nice.
> 
>> All that said: I do not want to make more changes to mod_md before a 
>> release. If we find a
>> serious error, sure. But otherwise I'd rather enhance the documentation for 
>> now.
> 
> Looks here you are bikeshedding my reports/requests (again). What is serious, 
> an introduction with too much user questions is not good. Consider me as a 
> user in this case. 
> 
> I spend tons of hours in testing mod_md (I am not paid).   My only goal was 
> to get quality and a usable function set  for our users, and not to frustrate 
> you. Looks you are only interested now to get it in httpd as quickly as 
> possible, I can only guess the reason for your pressing.
> 
>> If you want to add some Windows specific advice to the mod_md XML, please do 
>> so.
> 
> At the moment no advice. On Apache Lounge users can follow the dev at 
> http://www.apachelounge.com/viewtopic.php?t=7786 
> 
> 
> PS.
> For windows there is still testing and work (like cmake) to be done. I did 
> not tested completely the .dsp/mak etc.  I see in trunk there are still 
> changes on the way, so when voting starts for 2.4.30 then I shall test all 
> again.
> 
> 
> Regards,
> 
> Steffen
>  
> 
> On Tuesday 12/12/2017 at 14:49, Stefan Eissing wrote:
>> 
>> 
>>> Am 12.12.2017 um 14:37 schrieb Steffen >> >:
>>> 
>>> The curl error was just to show you the debug log entries which you asked.
>>> 
>>> This curl error we discussed by mail already in the very beginning (mod_md 
>>> does not work with curl openssl on Windows).
>>> 
>>> 1.1.0 is working fine so far.
>>> 
>>> I am only testing rare cases (you asked to test).
>> 
>> I see. I did not understand you before. I suspected you had a real error 
>> with v1.1.0.
>> 
>> As to your design proposals:
>> - No, I will not change the endless retry behaviour.
>> - Yes, I expect people to check their server logs, somehow. Either manual or 
>> with some tools.
>>There is more than mod_md that can have trouble and every module 
>> implementing its own
>>solution is not a good service for users.
>> - Yes, I think some log levels need adjustment, e.g. when LE cannot be 
>> reached at all.
>> - Yes, I think there should be a high level NOTICE log entry when a 
>> certificate could not
>>be renewed and the existing one only last a couple of days.
>> 
>> All that said: I do not want to make more changes to mod_md before a 
>> release. If we find a
>> serious error, sure. But otherwise I'd rather enhance the documentation for 
>> now.
>> 
>> If you want to add some Windows specific advice to the mod_md XML, please do 
>> so.
>> 
>> Cheers,
>> 
>> Stefan
>> 
> 
> 

Re: mod_md 1.1.0 repeating on error

[Steffen]

- No, I will not change the endless retry behaviour.
I still not see the reason for retries on all the errors, I am not 
aware of other modules doing it this way.



But Ok, when mdnotifycmd on error is added (see other post feature 
request). Then I  vote a  SUPER +1  for an experimental release.



User are going to be happy with mdnotifycmd on error, had already 
requests. Now it only triggered when it is OK, which is not logical.



See for a mdnotifycmd script see : 
https://www.apachelounge.com/viewtopic.php?p=36054#36054




- Yes, I expect people to check their server logs, somehow. Either 
manual or with some tools.
  There is more than mod_md that can have trouble and every module 
implementing its own

  solution is not a good service for users.
Be aware that most users only check there logs when there is an issue. 
Agree a standard for every module (the retry is not standard 
behavior), but mod_md log  messages must be more clear to see what is 
going on, see also next point.


- Yes, I think some log levels need adjustment, e.g. when LE cannot be 
reached at all.

Let me know which one you pick. We had mentions on GIT about this.


- Yes, I think there should be a high level NOTICE log entry when a 
certificate could not

  be renewed and the existing one only last a couple of days.


Agree, plus a mdnotifycmd should be nice.



All that said: I do not want to make more changes to mod_md before a 
release. If we find a
serious error, sure. But otherwise I'd rather enhance the 
documentation for now.
Looks here you are bikeshedding my reports/requests (again). What is 
serious, an introduction with too much user questions is not good. 
Consider me as a user in this case.



I spend tons of hours in testing mod_md (I am not paid).   My only 
goal was to get quality and a usable function set  for our users, and 
not to frustrate you. Looks you are only interested now to get it in 
httpd as quickly as possible, I can only guess the reason for your 
pressing.




If you want to add some Windows specific advice to the mod_md XML, 
please do so.
At the moment no advice. On Apache Lounge users can follow the dev at 
http://www.apachelounge.com/viewtopic.php?t=7786


PS.
For windows there is still testing and work (like cmake) to be done. I 
did not tested completely the .dsp/mak etc.  I see in trunk there are 
still changes on the way, so when voting starts for 2.4.30 then I 
shall test all again.



Regards,

Steffen



On Tuesday 12/12/2017 at 14:49, Stefan Eissing  wrote:





Am 12.12.2017 um 14:37 schrieb Steffen :

The curl error was just to show you the debug log entries which you 
asked.


This curl error we discussed by mail  already in the very beginning 
(mod_md does not work with curl openssl on Windows).


1.1.0 is working  fine so far.

I am only testing rare cases  (you asked to test).


I see. I did not understand you before. I suspected you had a real 
error with v1.1.0.


As to your design proposals:
- No, I will not change the endless retry behaviour.
- Yes, I expect people to check their server logs, somehow. Either 
manual or with some tools.
   There is more than mod_md that can have trouble and every module 
implementing its own

   solution is not a good service for users.
- Yes, I think some log levels need adjustment, e.g. when LE cannot be 
reached at all.
- Yes, I think there should be a high level NOTICE log entry when a 
certificate could not

   be renewed and the existing one only last a couple of days.

All that said: I do not want to make more changes to mod_md before a 
release. If we find a
serious error, sure. But otherwise I'd rather enhance the 
documentation for now.


If you want to add some Windows specific advice to the mod_md XML, 
please do so.


Cheers,

Stefan

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

> Am 12.12.2017 um 14:37 schrieb Steffen :
> 
> The curl error was just to show you the debug log entries which you asked.
> 
> This curl error we discussed by mail  already in the very beginning (mod_md 
> does not work with curl openssl on Windows).
> 
> 1.1.0 is working  fine so far.
> 
> I am only testing rare cases  (you asked to test).

I see. I did not understand you before. I suspected you had a real error with 
v1.1.0.

As to your design proposals:
- No, I will not change the endless retry behaviour.
- Yes, I expect people to check their server logs, somehow. Either manual or 
with some tools.
  There is more than mod_md that can have trouble and every module implementing 
its own
  solution is not a good service for users.
- Yes, I think some log levels need adjustment, e.g. when LE cannot be reached 
at all.
- Yes, I think there should be a high level NOTICE log entry when a certificate 
could not
  be renewed and the existing one only last a couple of days.

All that said: I do not want to make more changes to mod_md before a release. 
If we find a
serious error, sure. But otherwise I'd rather enhance the documentation for now.

If you want to add some Windows specific advice to the mod_md XML, please do so.

Cheers,

Stefan

> Steffen
>  
> On Tuesday 12/12/2017 at 14:21, Stefan Eissing wrote:
>> *without* introducing new ones, I meant. Please provide a log.
>> 
>>> Am 12.12.2017 um 14:21 schrieb Stefan Eissing 
>>> :
>>> 
>>> 
>>> 
 Am 12.12.2017 um 14:17 schrieb Steffen :
 
 To be clear : As I said the curl error I have introduced (by my self), so 
 I know exactly what is wrong.
>>> 
>>> Ah, that was not clear to me.
>>> 
>>> So, what is the error happening with you introducing new ones? Is there 
>>> nothing to see in the logs or did I miss it?
>>> 
 Your reply shows me that you want to keep the endless retry loop. I the 
 worst case a user can end with a non working SSL because a certificate is 
 not renewed.
 
 Why is it retried again and again ? Looks all hard errors, except when LE 
 is temporary down.
 
 I think it should be fixed. No every one is constantly look at the 
 error.log.
 
 
 What I like:
 
 Use MDNotifyCmd for the first error AH10057 . 
 Now the MDNotifyCmd is only triggered when it is ok, seems logical to also 
 notify when there is some wrong.
 
 
 On Tuesday 12/12/2017 at 13:58, Stefan Eissing wrote: 
> 
> 
>> Am 12.12.2017 um 13:47 schrieb Steffen :
>> 
>> It was happening before 1.1.0, but i did not give it attention, seen it 
>> in several situations which all I unfortunate cannot recall (see the 
>> retries as example https://github.com/icing/mod_md/issues/52and 
>> https://github.com/icing/mod_md/issues/62 ).
>> 
>> It is a more serious issue then I thought before. 
>> 
>> I think we must first fix this, otherwise it is a bad introduction to 
>> our users. This because Windows community first-time users learned that 
>> they are dealing with it and are dealing with all kind of (try) errors, 
>> most users stopped using it. As said in an other post mod_md is not that 
>> easy to start with.
>> 
>> Also when the loglevel is on the default Warn, users see hardly what is 
>> happening. I advise our users to use LogLevel info md:trace2 ssl:notice
>> 
>> The Endless Retry loop Tested now in the following situations, tested 
>> during renew and no new certificate is generated, httpd running fine 
>> with the old certificate which was still valid.
>> 
>> 1 - Mis-configuration like below.
>> 2 - ACME CA service down (cause Letsencrypt down)
>> 3 - ACME CA service not reachable (cause local network, or OS 
>> failure/misconfig)
>> 4 - Error response (Get/Post errors)when accessing Letsencrypt, 
>> dependency issue like curl, mod_ssl.
>> 5 - mod_md/mod_ssl faults
>> 6 - Should be more
>> 
>> 
>> 2) 3) Both can be that Letsencrypt is temp down maybe retry there, but 
>> hard to tell if the cause is temp LE-Down, issue local or OS misconfig.
>> 
>> 4) Is a good example: Error response from LE, which happens quite some 
>> situations, Curl issues, Rate-Limits, mod_md faults etc.
>> 
>> Below I introduced a Curl issue:
>> 
>> ...
>> [md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog run, 
>> auto drive 2 mds
>> [md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: 
>> md(apachelounge.nl): state=2, driving
>> [md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run 
>> staging
>> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
>> staging started, state=2, can_http=0, can_https=1, 
>> challenges='tls-sni-01'
>> [md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
>> staging/apachelounge.nl (D:/servers/apacheS/md/staging/apa

Re: mod_md 1.1.0 repeating on error

[Steffen]

The curl error was just to show you the debug log entries which you 
asked.


This curl error we discussed by mail  already in the very beginning 
(mod_md does not work with curl openssl on Windows).


1.1.0 is working  fine so far.

I am only testing rare cases  (you asked to test).

Steffen


On Tuesday 12/12/2017 at 14:21, Stefan Eissing  wrote:

*without* introducing new ones, I meant. Please provide a log.



Am 12.12.2017 um 14:21 schrieb Stefan Eissing 
:






Am 12.12.2017 um 14:17 schrieb Steffen :

To be clear :  As I said the curl error I have introduced (by my 
self), so I know exactly what is wrong.


Ah, that was not clear to me.

So, what is the error happening with you introducing new ones? Is 
there nothing to see in the logs or did I miss it?




Your reply shows me that you want to keep the endless retry loop. I 
the worst case a user can end with a non working SSL because a 
certificate is not renewed.


Why is it retried again and again ?  Looks all hard errors, except 
when LE is temporary down.


I think it should be fixed. No every one is constantly look at the 
error.log.



What I like:

Use MDNotifyCmd for the first error AH10057 .
Now the MDNotifyCmd is only triggered when it is ok, seems logical to 
also notify when there is some wrong.



On Tuesday 12/12/2017 at 13:58, Stefan Eissing wrote:






Am 12.12.2017 um 13:47 schrieb Steffen :

It was happening before 1.1.0, but i did not give it attention, seen 
it in several situations which all I unfortunate cannot recall (see 
the retries as example https://github.com/icing/mod_md/issues/52and 
https://github.com/icing/mod_md/issues/62 ).


It is a more serious issue then I thought before.

I think we must first fix this, otherwise it is a bad introduction to 
our users. This because Windows community first-time users learned 
that they are dealing with it and are dealing with all kind of (try) 
errors, most users stopped using it. As said in an other post mod_md 
is not that easy to start with.


Also when the loglevel is on the default Warn, users see hardly what 
is happening. I advise our users to use LogLevel info md:trace2 
ssl:notice


The Endless Retry loop Tested now in the following situations, tested 
during renew and no new certificate is generated, httpd running fine 
with the old certificate which was still valid.


1 - Mis-configuration like below.
2 - ACME CA service down (cause Letsencrypt down)
3 - ACME CA service not reachable (cause local network, or OS 
failure/misconfig)
4 - Error response (Get/Post errors)when accessing Letsencrypt, 
dependency issue like curl, mod_ssl.

5 - mod_md/mod_ssl faults
6 - Should be more


2) 3) Both can be that Letsencrypt is temp down maybe retry there, but 
hard to tell if the cause is temp LE-Down, issue local or OS 
misconfig.


4) Is a good example: Error response from LE, which happens quite some 
situations, Curl issues, Rate-Limits, mod_md faults etc.


Below I introduced a Curl issue:

...
[md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog 
run, auto drive 2 mds
[md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: 
md(apachelounge.nl): state=2, driving
[md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run 
staging
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
staging started, state=2, can_http=0, can_https=1, 
challenges='tls-sni-01'
[md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
staging/apachelounge.nl 
(D:/servers/apacheS/md/staging/apachelounge.nl)
[md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
(specific information not available): request 10 failed(60): Peer 
certificate cannot be authenticated with given CA certificates


Ok, this needs to be logged at ERROR level, so users do not have to 
mess with LogLevel to see what is going on.


As for the reason, this seems to indicate that the curl client finds 
no way to verify the Let's Encrypt server certificate. Can you verify 
that the "curl.exe" can connect to 
"https://acme-v01.api.letsencrypt.org/directory"; and retrieve the JSON 
there *without* you giving it the '-k' or '--insecure' option? And 
where does your curl.exe/libcurl come from? Did you build it yourself?




[md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
(specific information not available): req sent
[md:error] [pid 7508:tid 1052] (20014)Internal error (specific 
information not available): apachelounge.nl: setup 
ACME(https://acme-v01.api.letsencrypt.org/directory)
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal 
error (specific information not available): apachelounge.nl: ACME, 
ACME staging
[md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error 
(specific information not available): apac

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

*without* introducing new ones, I meant. Please provide a log.

> Am 12.12.2017 um 14:21 schrieb Stefan Eissing :
> 
> 
> 
>> Am 12.12.2017 um 14:17 schrieb Steffen :
>> 
>> To be clear :  As I said the curl error I have introduced (by my self), so I 
>> know exactly what is wrong.
> 
> Ah, that was not clear to me.
> 
> So, what is the error happening with you introducing new ones? Is there 
> nothing to see in the logs or did I miss it?
> 
>> Your reply shows me that you want to keep the endless retry loop. I the 
>> worst case a user can end with a non working SSL because a certificate is 
>> not renewed.
>> 
>> Why is it retried again and again ?  Looks all hard errors, except when LE 
>> is temporary down.
>> 
>> I think it should be fixed. No every one is constantly look at the error.log.
>> 
>> 
>> What I like:
>> 
>> Use MDNotifyCmd for the first error AH10057 . 
>> Now the MDNotifyCmd is only triggered when it is ok, seems logical to also 
>> notify when there is some wrong.
>> 
>> 
>> On Tuesday 12/12/2017 at 13:58, Stefan Eissing wrote: 
>>> 
>>> 
 Am 12.12.2017 um 13:47 schrieb Steffen :
 
 It was happening before 1.1.0, but i did not give it attention, seen it in 
 several situations which all I unfortunate cannot recall (see the retries 
 as example https://github.com/icing/mod_md/issues/52and 
 https://github.com/icing/mod_md/issues/62 ).
 
 It is a more serious issue then I thought before. 
 
 I think we must first fix this, otherwise it is a bad introduction to our 
 users. This because Windows community first-time users learned that they 
 are dealing with it and are dealing with all kind of (try) errors, most 
 users stopped using it. As said in an other post mod_md is not that easy 
 to start with.
 
 Also when the loglevel is on the default Warn, users see hardly what is 
 happening. I advise our users to use LogLevel info md:trace2 ssl:notice
 
 The Endless Retry loop Tested now in the following situations, tested 
 during renew and no new certificate is generated, httpd running fine with 
 the old certificate which was still valid.
 
 1 - Mis-configuration like below.
 2 - ACME CA service down (cause Letsencrypt down)
 3 - ACME CA service not reachable (cause local network, or OS 
 failure/misconfig)
 4 - Error response (Get/Post errors)when accessing Letsencrypt, dependency 
 issue like curl, mod_ssl.
 5 - mod_md/mod_ssl faults
 6 - Should be more
 
 
 2) 3) Both can be that Letsencrypt is temp down maybe retry there, but 
 hard to tell if the cause is temp LE-Down, issue local or OS misconfig.
 
 4) Is a good example: Error response from LE, which happens quite some 
 situations, Curl issues, Rate-Limits, mod_md faults etc.
 
 Below I introduced a Curl issue:
 
 ...
 [md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog run, 
 auto drive 2 mds
 [md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: 
 md(apachelounge.nl): state=2, driving
 [md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run staging
 [md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
 staging started, state=2, can_http=0, can_https=1, challenges='tls-sni-01'
 [md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
 staging/apachelounge.nl (D:/servers/apacheS/md/staging/apachelounge.nl)
 [md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
 https://acme-v01.api.letsencrypt.org/directory
 [md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
 https://acme-v01.api.letsencrypt.org/directory
 [md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
 (specific information not available): request 10 failed(60): Peer 
 certificate cannot be authenticated with given CA certificates
>>> 
>>> Ok, this needs to be logged at ERROR level, so users do not have to mess 
>>> with LogLevel to see what is going on.
>>> 
>>> As for the reason, this seems to indicate that the curl client finds no way 
>>> to verify the Let's Encrypt server certificate. Can you verify that the 
>>> "curl.exe" can connect to "https://acme-v01.api.letsencrypt.org/directory"; 
>>> and retrieve the JSON there *without* you giving it the '-k' or 
>>> '--insecure' option? And where does your curl.exe/libcurl come from? Did 
>>> you build it yourself?
>>> 
 [md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
 (specific information not available): req sent
 [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
 not available): apachelounge.nl: setup 
 ACME(https://acme-v01.api.letsencrypt.org/directory)
 [md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal error 
 (specific information not available): apachelounge.nl: ACME, ACME staging
 [md:debug] [pid 7508:tid 1052] md_re

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

> Am 12.12.2017 um 14:17 schrieb Steffen :
> 
> To be clear :  As I said the curl error I have introduced (by my self), so I 
> know exactly what is wrong.

Ah, that was not clear to me.

So, what is the error happening with you introducing new ones? Is there nothing 
to see in the logs or did I miss it?
 
> Your reply shows me that you want to keep the endless retry loop. I the worst 
> case a user can end with a non working SSL because a certificate is not 
> renewed.
> 
> Why is it retried again and again ?  Looks all hard errors, except when LE is 
> temporary down.
> 
> I think it should be fixed. No every one is constantly look at the error.log.
> 
> 
> What I like:
> 
> Use MDNotifyCmd for the first error AH10057 . 
> Now the MDNotifyCmd is only triggered when it is ok, seems logical to also 
> notify when there is some wrong.
> 
>  
> On Tuesday 12/12/2017 at 13:58, Stefan Eissing wrote: 
>> 
>> 
>>> Am 12.12.2017 um 13:47 schrieb Steffen :
>>> 
>>> It was happening before 1.1.0, but i did not give it attention, seen it in 
>>> several situations which all I unfortunate cannot recall (see the retries 
>>> as example https://github.com/icing/mod_md/issues/52and 
>>> https://github.com/icing/mod_md/issues/62 ).
>>> 
>>> It is a more serious issue then I thought before. 
>>> 
>>> I think we must first fix this, otherwise it is a bad introduction to our 
>>> users. This because Windows community first-time users learned that they 
>>> are dealing with it and are dealing with all kind of (try) errors, most 
>>> users stopped using it. As said in an other post mod_md is not that easy to 
>>> start with.
>>> 
>>> Also when the loglevel is on the default Warn, users see hardly what is 
>>> happening. I advise our users to use LogLevel info md:trace2 ssl:notice
>>> 
>>> The Endless Retry loop Tested now in the following situations, tested 
>>> during renew and no new certificate is generated, httpd running fine with 
>>> the old certificate which was still valid.
>>> 
>>> 1 - Mis-configuration like below.
>>> 2 - ACME CA service down (cause Letsencrypt down)
>>> 3 - ACME CA service not reachable (cause local network, or OS 
>>> failure/misconfig)
>>> 4 - Error response (Get/Post errors)when accessing Letsencrypt, dependency 
>>> issue like curl, mod_ssl.
>>> 5 - mod_md/mod_ssl faults
>>> 6 - Should be more
>>> 
>>> 
>>> 2) 3) Both can be that Letsencrypt is temp down maybe retry there, but hard 
>>> to tell if the cause is temp LE-Down, issue local or OS misconfig.
>>> 
>>> 4) Is a good example: Error response from LE, which happens quite some 
>>> situations, Curl issues, Rate-Limits, mod_md faults etc.
>>> 
>>> Below I introduced a Curl issue:
>>> 
>>> ...
>>> [md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog run, 
>>> auto drive 2 mds
>>> [md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: md(apachelounge.nl): 
>>> state=2, driving
>>> [md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run staging
>>> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
>>> staging started, state=2, can_http=0, can_https=1, challenges='tls-sni-01'
>>> [md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
>>> staging/apachelounge.nl (D:/servers/apacheS/md/staging/apachelounge.nl)
>>> [md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
>>> https://acme-v01.api.letsencrypt.org/directory
>>> [md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
>>> https://acme-v01.api.letsencrypt.org/directory
>>> [md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
>>> (specific information not available): request 10 failed(60): Peer 
>>> certificate cannot be authenticated with given CA certificates
>> 
>> Ok, this needs to be logged at ERROR level, so users do not have to mess 
>> with LogLevel to see what is going on.
>> 
>> As for the reason, this seems to indicate that the curl client finds no way 
>> to verify the Let's Encrypt server certificate. Can you verify that the 
>> "curl.exe" can connect to "https://acme-v01.api.letsencrypt.org/directory"; 
>> and retrieve the JSON there *without* you giving it the '-k' or '--insecure' 
>> option? And where does your curl.exe/libcurl come from? Did you build it 
>> yourself?
>> 
>>> [md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
>>> (specific information not available): req sent
>>> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
>>> not available): apachelounge.nl: setup 
>>> ACME(https://acme-v01.api.letsencrypt.org/directory)
>>> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal error 
>>> (specific information not available): apachelounge.nl: ACME, ACME staging
>>> [md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error 
>>> (specific information not available): apachelounge.nl: staging done
>>> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
>>> not available): AH10056: processing apache

Re: mod_md 1.1.0 repeating on error

[Steffen]

To be clear :  As I said the curl error I have introduced (by my 
self), so I know exactly what is wrong.


Your reply shows me that you want to keep the endless retry loop. I 
the worst case a user can end with a non working SSL because a 
certificate is not renewed.


Why is it retried again and again ?  Looks all hard errors, except 
when LE is temporary down.


I think it should be fixed. No every one is constantly look at the 
error.log.



What I like:

Use MDNotifyCmd for the first error AH10057 .
Now the MDNotifyCmd is only triggered when it is ok, seems logical to 
also notify when there is some wrong.




On Tuesday 12/12/2017 at 13:58, Stefan Eissing  wrote:





Am 12.12.2017 um 13:47 schrieb Steffen :

It was happening before 1.1.0, but i did not give it attention, seen 
it in several situations which all I unfortunate cannot recall (see 
the retries  as example https://github.com/icing/mod_md/issues/52and 
https://github.com/icing/mod_md/issues/62 ).


It is a more serious issue then I thought before.

I think we must first fix this, otherwise it is a bad introduction to 
our users. This because Windows community first-time users learned 
that they  are dealing with it and are dealing with all kind of (try) 
errors, most users stopped using it.  As said in an other post mod_md 
is not that easy to start with.


Also when the loglevel is on the default Warn, users see hardly what 
is happening. I advise our users to use  LogLevel info md:trace2 
ssl:notice


The Endless Retry loop Tested now in the following situations, tested 
during renew and no new certificate is generated, httpd running fine 
with the old certificate which was still valid.


1 - Mis-configuration like below.
2 - ACME CA service  down (cause Letsencrypt down)
3 - ACME CA service not reachable (cause local network, or OS 
failure/misconfig)
4 - Error response (Get/Post errors)when accessing Letsencrypt, 
dependency issue like curl, mod_ssl.

5 - mod_md/mod_ssl faults
6 - Should be more


2) 3) Both can be that Letsencrypt is temp down maybe retry there, but 
hard to tell if the cause is temp LE-Down, issue local or OS 
misconfig.


4) Is a good example: Error response from LE, which happens quite  
some situations, Curl issues, Rate-Limits, mod_md faults  etc.


Below I introduced a Curl issue:

...
[md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog 
run, auto drive 2 mds
[md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: 
md(apachelounge.nl): state=2, driving
[md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run 
staging
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
staging started, state=2, can_http=0, can_https=1, 
challenges='tls-sni-01'
[md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
staging/apachelounge.nl 
(D:/servers/apacheS/md/staging/apachelounge.nl)
[md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
(specific information not available): request 10 failed(60): Peer 
certificate cannot be authenticated with given CA certificates


Ok, this needs to be logged at ERROR level, so users do not have to 
mess with LogLevel to see what is going on.


As for the reason, this seems to indicate that the curl client finds 
no way to verify the Let's Encrypt server certificate. Can you verify 
that the "curl.exe" can connect to 
"https://acme-v01.api.letsencrypt.org/directory"; and retrieve the JSON 
there *without* you giving it the '-k' or '--insecure' option? And 
where does your curl.exe/libcurl come from? Did you build it yourself?




[md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
(specific information not available): req sent
[md:error] [pid 7508:tid 1052] (20014)Internal error (specific 
information not available): apachelounge.nl: setup 
ACME(https://acme-v01.api.letsencrypt.org/directory)
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal 
error (specific information not available): apachelounge.nl: ACME, 
ACME staging
[md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error 
(specific information not available): apachelounge.nl: staging done
[md:error] [pid 7508:tid 1052] (20014)Internal error (specific 
information not available): AH10056: processing apachelounge.nl
[md:info] [pid 7508:tid 1052] AH10057: apachelounge.nl: encountered 
error for the 6. time, next run in  0:02:40 hours

...

Maybe a little solution:  starting httpd, mod_md checks if LE is 
reachable without error.


No, I think checking external servers on every httpd restart is a good 
idea.




And a solution for the below one can be: make a check that 443 and/or 
80 is used.


Still my questions:

Does the retry stop ?


The retry does not stop, but it uses longer and longer retry 
intervals. Exa

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

And btw. what is the Windows OS version that your server runs on?

And since you had mod_md running before, what did change in relation to
Windows and the curl you use?

> Am 12.12.2017 um 13:58 schrieb Stefan Eissing :
> 
> 
> 
>> Am 12.12.2017 um 13:47 schrieb Steffen :
>> 
>> It was happening before 1.1.0, but i did not give it attention, seen it in 
>> several situations which all I unfortunate cannot recall (see the retries  
>> as example https://github.com/icing/mod_md/issues/52and 
>> https://github.com/icing/mod_md/issues/62 ).
>> 
>> It is a more serious issue then I thought before. 
>> 
>> I think we must first fix this, otherwise it is a bad introduction to our 
>> users. This because Windows community first-time users learned that they  
>> are dealing with it and are dealing with all kind of (try) errors, most 
>> users stopped using it.  As said in an other post mod_md is not that easy to 
>> start with.
>> 
>> Also when the loglevel is on the default Warn, users see hardly what is 
>> happening. I advise our users to use LogLevel info md:trace2 ssl:notice
>> 
>> The Endless Retry loop Tested now in the following situations, tested during 
>> renew and no new certificate is generated, httpd running fine with the old 
>> certificate which was still valid.
>> 
>> 1 - Mis-configuration like below.
>> 2 - ACME CA service  down (cause Letsencrypt down)
>> 3 - ACME CA service not reachable (cause local network, or OS 
>> failure/misconfig)
>> 4 - Error response (Get/Post errors)when accessing Letsencrypt, dependency 
>> issue like curl, mod_ssl.
>> 5 - mod_md/mod_ssl faults
>> 6 - Should be more
>> 
>> 
>> 2) 3) Both can be that Letsencrypt is temp down maybe retry there, but hard 
>> to tell if the cause is temp LE-Down, issue local or OS misconfig.
>> 
>> 4) Is a good example: Error response from LE, which happens quite  some 
>> situations, Curl issues, Rate-Limits, mod_md faults  etc.
>> 
>> Below I introduced a Curl issue:
>> 
>> ...
>> [md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog run, auto 
>> drive 2 mds
>> [md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: md(apachelounge.nl): 
>> state=2, driving
>> [md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run staging
>> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
>> staging started, state=2, can_http=0, can_https=1, challenges='tls-sni-01'
>> [md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
>> staging/apachelounge.nl (D:/servers/apacheS/md/staging/apachelounge.nl)
>> [md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
>> https://acme-v01.api.letsencrypt.org/directory
>> [md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
>> https://acme-v01.api.letsencrypt.org/directory
>> [md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
>> (specific information not available): request 10 failed(60): Peer 
>> certificate cannot be authenticated with given CA certificates
> 
> Ok, this needs to be logged at ERROR level, so users do not have to mess with 
> LogLevel to see what is going on.
> 
> As for the reason, this seems to indicate that the curl client finds no way 
> to verify the Let's Encrypt server certificate. Can you verify that the 
> "curl.exe" can connect to "https://acme-v01.api.letsencrypt.org/directory"; 
> and retrieve the JSON there *without* you giving it the '-k' or '--insecure' 
> option? And where does your curl.exe/libcurl come from? Did you build it 
> yourself?
> 
>> [md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
>> (specific information not available): req sent
>> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
>> not available): apachelounge.nl: setup 
>> ACME(https://acme-v01.api.letsencrypt.org/directory)
>> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal error 
>> (specific information not available): apachelounge.nl: ACME, ACME staging
>> [md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error 
>> (specific information not available): apachelounge.nl: staging done
>> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
>> not available): AH10056: processing apachelounge.nl
>> [md:info] [pid 7508:tid 1052] AH10057: apachelounge.nl: encountered error 
>> for the 6. time, next run in 0:02:40 hours
>> ...
>> 
>> Maybe a little solution:  starting httpd, mod_md checks if LE is reachable 
>> without error.
> 
> No, I think checking external servers on every httpd restart is a good idea.
> 
>> And a solution for the below one can be: make a check that 443 and/or 80 is 
>> used.
>> 
>> Still my questions:
>> 
>> Does the retry stop ?
> 
> The retry does not stop, but it uses longer and longer retry intervals. 
> Exactly to recover from errors with the ACME server that are recoverable, 
> e.g. server/internet down. Your local certificate store not able to verify 
> the LE server will not recover itself, however.
> 
>> Whe

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

> Am 12.12.2017 um 13:47 schrieb Steffen :
> 
> It was happening before 1.1.0, but i did not give it attention, seen it in 
> several situations which all I unfortunate cannot recall (see the retries  as 
> example https://github.com/icing/mod_md/issues/52and 
> https://github.com/icing/mod_md/issues/62 ).
> 
> It is a more serious issue then I thought before. 
> 
> I think we must first fix this, otherwise it is a bad introduction to our 
> users. This because Windows community first-time users learned that they  are 
> dealing with it and are dealing with all kind of (try) errors, most users 
> stopped using it.  As said in an other post mod_md is not that easy to start 
> with.
> 
> Also when the loglevel is on the default Warn, users see hardly what is 
> happening. I advise our users to use  LogLevel info md:trace2 ssl:notice
> 
> The Endless Retry loop Tested now in the following situations, tested during 
> renew and no new certificate is generated, httpd running fine with the old 
> certificate which was still valid.
> 
> 1 - Mis-configuration like below.
> 2 - ACME CA service  down (cause Letsencrypt down)
> 3 - ACME CA service not reachable (cause local network, or OS 
> failure/misconfig)
> 4 - Error response (Get/Post errors)when accessing Letsencrypt, dependency 
> issue like curl, mod_ssl.
> 5 - mod_md/mod_ssl faults
> 6 - Should be more
> 
> 
> 2) 3) Both can be that Letsencrypt is temp down maybe retry there, but hard 
> to tell if the cause is temp LE-Down, issue local or OS misconfig.
> 
> 4) Is a good example: Error response from LE, which happens quite  some 
> situations, Curl issues, Rate-Limits, mod_md faults  etc.
> 
> Below I introduced a Curl issue:
> 
> ...
> [md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog run, auto 
> drive 2 mds
> [md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: md(apachelounge.nl): 
> state=2, driving
> [md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run staging
> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: staging 
> started, state=2, can_http=0, can_https=1, challenges='tls-sni-01'
> [md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
> staging/apachelounge.nl (D:/servers/apacheS/md/staging/apachelounge.nl)
> [md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
> https://acme-v01.api.letsencrypt.org/directory
> [md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
> https://acme-v01.api.letsencrypt.org/directory
> [md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
> (specific information not available): request 10 failed(60): Peer certificate 
> cannot be authenticated with given CA certificates

Ok, this needs to be logged at ERROR level, so users do not have to mess with 
LogLevel to see what is going on.

As for the reason, this seems to indicate that the curl client finds no way to 
verify the Let's Encrypt server certificate. Can you verify that the "curl.exe" 
can connect to "https://acme-v01.api.letsencrypt.org/directory"; and retrieve 
the JSON there *without* you giving it the '-k' or '--insecure' option? And 
where does your curl.exe/libcurl come from? Did you build it yourself?

> [md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
> (specific information not available): req sent
> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
> not available): apachelounge.nl: setup 
> ACME(https://acme-v01.api.letsencrypt.org/directory)
> [md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal error 
> (specific information not available): apachelounge.nl: ACME, ACME staging
> [md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error (specific 
> information not available): apachelounge.nl: staging done
> [md:error] [pid 7508:tid 1052] (20014)Internal error (specific information 
> not available): AH10056: processing apachelounge.nl
> [md:info] [pid 7508:tid 1052] AH10057: apachelounge.nl: encountered error for 
> the 6. time, next run in  0:02:40 hours
> ...
> 
> Maybe a little solution:  starting httpd, mod_md checks if LE is reachable 
> without error.

No, I think checking external servers on every httpd restart is a good idea.

> And a solution for the below one can be: make a check that 443 and/or 80 is 
> used.
> 
> Still my questions:
> 
> Does the retry stop ?

The retry does not stop, but it uses longer and longer retry intervals. Exactly 
to recover from errors with the ACME server that are recoverable, e.g. 
server/internet down. Your local certificate store not able to verify the LE 
server will not recover itself, however.

> When does it happen, on what errors ?

On any error where signup/renew is necessary and could not complete.

> 
> 
> Steffen
> 
>  
> On Tuesday 12/12/2017 at 10:18, Stefan Eissing wrote:
>> Can you switch to "LogLevel md:debug" for a while and send me the details? 
>> Did this start on the v1.1.0 or before that?
>> 
>>> Am 11.12.2017 um 16:09 schr

Re: mod_md 1.1.0 repeating on error

[Steffen]

It was happening before 1.1.0, but i did not give it attention, seen 
it in several situations which all I unfortunate cannot recall (see 
the retries  as example https://github.com/icing/mod_md/issues/52 and 
https://github.com/icing/mod_md/issues/62 ).


It is a more serious issue then I thought before.


I think we must first fix this, otherwise it is a bad introduction to 
our users. This because Windows community first-time users learned 
that they  are dealing with it and are dealing with all kind of (try) 
errors, most users stopped using it.  As said in an other post mod_md 
is not that easy to start with.



Also when the loglevel is on the default Warn, users see hardly what 
is happening. I advise our users to use  LogLevel info md:trace2 
ssl:notice


The Endless Retry loop Tested now in the following situations, tested 
during renew and no new certificate is generated, httpd running fine 
with the old certificate which was still valid.


1 - Mis-configuration like below.
2 - ACME CA service  down (cause Letsencrypt down)
3 - ACME CA service not reachable (cause local network, or OS 
failure/misconfig)
4 - Error response (Get/Post errors)when accessing Letsencrypt, 
dependency issue like curl, mod_ssl.

5 - mod_md/mod_ssl faults
6 - Should be more


2) 3) Both can be that Letsencrypt is temp down maybe retry there, but 
hard to tell if the cause is temp LE-Down, issue local or OS 
misconfig.


4) Is a good example: Error response from LE, which happens quite  
some situations, Curl issues, Rate-Limits, mod_md faults  etc.


Below I introduced a Curl issue:


...

[md:debug] [pid 7508:tid 1052] mod_md.c(762): AH10055: md watchdog 
run, auto drive 2 mds
[md:debug] [pid 7508:tid 1052] mod_md.c(691): AH10052: 
md(apachelounge.nl): state=2, driving
[md:debug] [pid 7508:tid 1052] md_reg.c(884): apachelounge.nl: run 
staging
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(690): apachelounge.nl: 
staging started, state=2, can_http=0, can_https=1, 
challenges='tls-sni-01'
[md:debug] [pid 7508:tid 1052] md_store_fs.c(690): purge 
staging/apachelounge.nl 
(D:/servers/apacheS/md/staging/apachelounge.nl)
[md:debug] [pid 7508:tid 1052] md_acme.c(144): get directory from 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_acme.c(407): req: POST 
https://acme-v01.api.letsencrypt.org/directory
[md:debug] [pid 7508:tid 1052] md_curl.c(258): (20014)Internal error 
(specific information not available): request 10 failed(60): Peer 
certificate cannot be authenticated with given CA certificates
[md:debug] [pid 7508:tid 1052] md_acme.c(425): (20014)Internal error 
(specific information not available): req sent
[md:error] [pid 7508:tid 1052] (20014)Internal error (specific 
information not available): apachelounge.nl: setup 
ACME(https://acme-v01.api.letsencrypt.org/directory)
[md:debug] [pid 7508:tid 1052] md_acme_drive.c(912): (20014)Internal 
error (specific information not available): apachelounge.nl: ACME, 
ACME staging
[md:debug] [pid 7508:tid 1052] md_reg.c(891): (20014)Internal error 
(specific information not available): apachelounge.nl: staging done
[md:error] [pid 7508:tid 1052] (20014)Internal error (specific 
information not available): AH10056: processing apachelounge.nl
[md:info] [pid 7508:tid 1052] AH10057: apachelounge.nl: encountered 
error for the 6. time, next run in  0:02:40 hours

...

Maybe a little solution:  starting httpd, mod_md checks if LE is 
reachable without error.


And a solution for the below one can be: make a check that 443 and/or 
80 is used.


Still my questions:

Does the retry stop ?
When does it happen, on what errors ?


Steffen



On Tuesday 12/12/2017 at 10:18, Stefan Eissing  wrote:
Can you switch to "LogLevel md:debug" for a while and send me the 
details? Did this start on the v1.1.0 or before that?




Am 11.12.2017 um 16:09 schrieb Steffen :


Running 1.1.0 with the new naming.

When mod_md encounters an error it looks like it is going in a endless 
loop:



[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 1. time, next run in  0:00:05 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 2. time, next run in  0:00:10 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 3. time, next run in  0:00:20 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 4. time, next run in  0:00:40 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 5. time, next run in  0:01:20 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 6. time, next run in  0:02:40 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 7. time, next run in  0:05:20 hours
[md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered 
error for the 8. time, next run in  0:10:40 hours

...
...
...

Above is during renew and

Re: mod_md 1.1.0 repeating on error

[Stefan Eissing]

Can you switch to "LogLevel md:debug" for a while and send me the details? Did 
this start on the v1.1.0 or before that?

> Am 11.12.2017 um 16:09 schrieb Steffen :
> 
> 
> Running 1.1.0 with the new naming.
> 
> When mod_md encounters an error it looks like it is going in a endless loop:
> 
> 
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 1. time, next run in  0:00:05 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 2. time, next run in  0:00:10 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 3. time, next run in  0:00:20 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 4. time, next run in  0:00:40 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 5. time, next run in  0:01:20 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 6. time, next run in  0:02:40 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 7. time, next run in  0:05:20 hours
> [md:info] [pid 10372:tid 1964] AH10057: apachelounge.nl: encountered error 
> for the 8. time, next run in  0:10:40 hours
> ...
> ...
> ...
> 
> Above is during renew and using port 444..
> 
> Apache is running fine  because the certificate is still valid.
> 
> Does it stop ?
> 
> When does it happen,  on what errors ? Above happens when: (20014)Internal 
> error (specific information not available): AH10056: processing 
> apachelounge.nl.
> 
> What to do.  Stopping on above retries can be tricky because when the ACME CA 
> service is temp down or not reachable we do want maybe a retry.  A reachable 
> error/down error is different then a configuration error causing it like in 
> above case..
> 
> 
> 
> 
> 
> 
> 
> 
>