Hi Stephane,
Here's how I look at it:
Section 8.1 is about a user agent noting a new HSTS host. If the
connection had an underlying error (e.g. self-signed cert), the user
agent will not note that host as using HSTS.
Section 11.3 is about when the user agent connects to a host that it
previously
[I'm not subscribed to the websec working group so please copy me when
replying.]
I don't know how to read section 11.3 of RFC 6797. It says "If all
four of the following conditions are true... [self-signed
certificates...] ...then secure connections to that site will fail,
per the HSTS design."