Re: [TLS] Time to first byte vs time to last byte

2024-03-13 Thread Ben Smyth
On Sat, 9 Mar 2024 at 10:23, Bas Westerbaan  wrote:

> Given that especially for the web, CDNs used much higher initcwnds,
>
>
> Please, let us not assume every website is behind a CDN.
>

Isn't that assumption reasonable? At least for global websites --- without
CDN performance sucks.
___
TLS mailing list
TLS@ietf.org
https://www.ietf.org/mailman/listinfo/tls


Re: [TLS] Time to first byte vs time to last byte

2024-03-09 Thread Bas Westerbaan
>
> Given that especially for the web, CDNs used much higher initcwnds,


Please, let us not assume every website is behind a CDN.



> let's focus on Figure 10. Based on Fig 10, 50-100KB of data over a PQ
> connection, the TTLB would be 10-15% slower for 1Mbps and 200ms RTT. At
> higher speeds, this percentage is much less (1-1.5% based on Fig 9b), but
> let's focus on the slow link.
>
> If we consider the same case for handshake, then the PQ handshake slowdown
> is 30-35% which definitely looks like a very impactful slowdown. A 10-15%
> for the TTLB is much less, but someone could argue that even that is a
> significant slowdown. Note we are still in a slow link, so even the
> classical conn transferring 72KB is probably suffering. To quantify that I
> looked at my data from these experiments. A classical connection TTLB for
> 50-100KB of data at 1Mbps and 200ms RTT and 0% loss was ~1.25s. This is not
> shown in the paper because I only included text about the 10% loss case.
> 1.25s for a 72KB page to start getting rendered on a browser over a
> classical conn vs 1.25*1.15=1.44s for a PQ one. I am not sure any user
> waiting for 1.25s will close the browser at 1.44s.
>
> Btw, the Google PageSpeed Insights TTFB metric which includes (DNS lookup,
> redirects and more) considers 0.8s - 1.8s as "Needs improvement". In our
> experiments, the handshake time for 1Mbps and 200ms RTT amounted to 436ms
> and 576ms for the classical and PQ handshakes respectively. I am not sure
> the extra 140ms (30-35% slowdown) for the PQ handshake would even throw the
> Google PageSpeed Insights TTFB metric to the "Needs improvement" category.
>
>
>
> -Original Message-
> From: Martin Thomson 
> Sent: Thursday, March 7, 2024 10:26 PM
> To: Kampanakis, Panos ; David Benjamin <
> david...@chromium.org>; Deirdre Connolly ; Rob
> Sayre 
> Cc: TLS@ietf.org; Childs-Klein, Will 
> Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Panos,
>
> I realize that TTLB might correlate well for some types of web content,
> but it's important to recognize that lots of web content is badly bloated
> (if you can tolerate the invective, this is a pretty good look at the
> situation, with numbers:
> https://infrequently.org/series/performance-inequality/).
>
> I don't want to call out your employer's properties in particular, but at
> over 3M and with relatively few connections, handshakes really don't play
> much into page load performance.  That might be typical, but just being
> typical doesn't mean that it's a case we should be optimizing for.
>
> The 72K page I linked above looks very different.  There, your paper shows
> a 20-25% hit on TTLB.  TTFB is likely more affected due to the way
> congestion controllers work and the fact that you never leave slow start.
>
> Cheers,
> Martin
>
> On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote:
> > Thx Deirdre for bringing it up.
> >
> > David,
> >
> > ACK. I think the overall point of our paper is that application
> > performance is more closely related to PQ TTLB than PQ TTFB/handshake.
> >
> > Snippet from the paper
> >
> > *> Google’s PageSpeed Insights [12] uses a set of metrics to measure
> > the user experience and webpage performance. The First Contentful
> > Paint (FCP), Largest Contentful Paint (LCP), First Input Delay (FID),
> > Interaction to Next Paint (INP), Total Blocking Time (TBT), and
> > Cumulative Layout Shift (CLS) metrics include this work’s TTLB along
> > with other client-side, browser application-specific execution delays.
> > The PageSpeed Insights TTFB metric measures the total time up to the
> > point the first byte of data makes it to the client. So, PageSpeed
> > Insights TTFB is like this work’s TTFB/TLS handshake time with
> > additional network delays like DNS lookup, redirect, service worker
> > startup, and request time.*
> >
> > Specifically about the Web, TTLB (as defined in the paper) is directly
> > related to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics
> > in Google’s PageSpeed Insights. We don’t want to declare that TTLB is
> > the ultimate metric, but intuitively, I think it is a better indicator
> > when it comes to application performance than TTFB.
> >
> > That does not intend to underestimate the importance of the studies on
> > handshake performance which was crucial to identify the best
> > 

Re: [TLS] Time to first byte vs time to last byte

2024-03-08 Thread Kampanakis, Panos
Hi Martin,

I think we are generally in agreement, but I want to push back on the argument 
that the PQ slowdown for a page transferring 72KB is going to be the problem. I 
will try to quantify this below (look for [72KBExample]). 

Btw, if you have any stats on Web content size distribution, I am interested. 
Other than averages, I could not find any data on how Web content size looks 
today.

Note that our paper not bashing TTFB as a metric, we are just saying TTFB is 
more relevant for use-cases that send little data, which is not the case for 
most applications today. Snippet from the Conclusion of the paper 
> Connections that transfer <10-20KB of data will probably be more impacted by 
> the new data-heavy handshakes  
This study picked data sizes based on public data on Web sizes (HTTP Archive) 
and other data for other cloud uses. Of course, if we reached a world where 
most use-cases (Web connections, IoT sensor measurement conns, cloud conns) 
were typically sending <50KB, then the TTFB would become more relevant. I am 
not sure we are there or we will ever be. Even the page you referenced (thx, I 
did not know of it) argues " ~100KiB of HTML/CSS/fonts and ~300-350KiB of JS." 
from 2021. 

[72KBExample] 
I think your 20-25% for a 72KB example page probably came from reading Fig 4b 
which includes an extra RTT due to initcwnd=10. Given that especially for the 
web, CDNs used much higher initcwnds, let's focus on Figure 10. Based on Fig 
10, 50-100KB of data over a PQ connection, the TTLB would be 10-15% slower for 
1Mbps and 200ms RTT. At higher speeds, this percentage is much less (1-1.5% 
based on Fig 9b), but let's focus on the slow link. 

If we consider the same case for handshake, then the PQ handshake slowdown is 
30-35% which definitely looks like a very impactful slowdown. A 10-15% for the 
TTLB is much less, but someone could argue that even that is a significant 
slowdown. Note we are still in a slow link, so even the classical conn 
transferring 72KB is probably suffering. To quantify that I looked at my data 
from these experiments. A classical connection TTLB for 50-100KB of data at 
1Mbps and 200ms RTT and 0% loss was ~1.25s. This is not shown in the paper 
because I only included text about the 10% loss case. 1.25s for a 72KB page to 
start getting rendered on a browser over a classical conn vs 1.25*1.15=1.44s 
for a PQ one. I am not sure any user waiting for 1.25s will close the browser 
at 1.44s. 

Btw, the Google PageSpeed Insights TTFB metric which includes (DNS lookup, 
redirects and more) considers 0.8s - 1.8s as "Needs improvement". In our 
experiments, the handshake time for 1Mbps and 200ms RTT amounted to 436ms and 
576ms for the classical and PQ handshakes respectively. I am not sure the extra 
140ms (30-35% slowdown) for the PQ handshake would even throw the Google 
PageSpeed Insights TTFB metric to the "Needs improvement" category. 



-Original Message-
From: Martin Thomson  
Sent: Thursday, March 7, 2024 10:26 PM
To: Kampanakis, Panos ; David Benjamin 
; Deirdre Connolly ; Rob Sayre 

Cc: TLS@ietf.org; Childs-Klein, Will 
Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi Panos,

I realize that TTLB might correlate well for some types of web content, but 
it's important to recognize that lots of web content is badly bloated (if you 
can tolerate the invective, this is a pretty good look at the situation, with 
numbers: https://infrequently.org/series/performance-inequality/).

I don't want to call out your employer's properties in particular, but at over 
3M and with relatively few connections, handshakes really don't play much into 
page load performance.  That might be typical, but just being typical doesn't 
mean that it's a case we should be optimizing for.

The 72K page I linked above looks very different.  There, your paper shows a 
20-25% hit on TTLB.  TTFB is likely more affected due to the way congestion 
controllers work and the fact that you never leave slow start.

Cheers,
Martin

On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote:
> Thx Deirdre for bringing it up.
>
> David,
>
> ACK. I think the overall point of our paper is that application 
> performance is more closely related to PQ TTLB than PQ TTFB/handshake.
>
> Snippet from the paper
>
> *> Google’s PageSpeed Insights [12] uses a set of metrics to measure 
> the user experience and webpage performance. The First Contentful 
> Paint (FCP), Largest Contentful Paint (LCP), First Input Delay (FID), 
> Interaction to Next Paint (INP), Total Blocking Time (TBT), and 
> Cumulative Layout Shift (CLS) metrics include this work’s TTLB along 
> with other client-side, browse

Re: [TLS] Time to first byte vs time to last byte

2024-03-07 Thread Martin Thomson
Hi Panos,

I realize that TTLB might correlate well for some types of web content, but 
it's important to recognize that lots of web content is badly bloated (if you 
can tolerate the invective, this is a pretty good look at the situation, with 
numbers: https://infrequently.org/series/performance-inequality/).

I don't want to call out your employer's properties in particular, but at over 
3M and with relatively few connections, handshakes really don't play much into 
page load performance.  That might be typical, but just being typical doesn't 
mean that it's a case we should be optimizing for.

The 72K page I linked above looks very different.  There, your paper shows a 
20-25% hit on TTLB.  TTFB is likely more affected due to the way congestion 
controllers work and the fact that you never leave slow start.

Cheers,
Martin

On Fri, Mar 8, 2024, at 13:56, Kampanakis, Panos wrote:
> Thx Deirdre for bringing it up. 
> 
> David,
> 
> ACK. I think the overall point of our paper is that application 
> performance is more closely related to PQ TTLB than PQ TTFB/handshake. 
> 
> Snippet from the paper
> 
> *> Google’s PageSpeed Insights [12] uses a set of metrics to measure 
> the user experience and webpage performance. The First Contentful Paint 
> (FCP), Largest Contentful Paint (LCP), First Input Delay (FID), 
> Interaction to Next Paint (INP), Total Blocking Time (TBT), and 
> Cumulative Layout Shift (CLS) metrics include this work’s TTLB along 
> with other client-side, browser application-specific execution delays. 
> The PageSpeed Insights TTFB metric measures the total time up to the 
> point the first byte of data makes it to the client. So, PageSpeed 
> Insights TTFB is like this work’s TTFB/TLS handshake time with 
> additional network delays like DNS lookup, redirect, service worker 
> startup, and request time.*
> 
> Specifically about the Web, TTLB (as defined in the paper) is directly 
> related to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics 
> in Google’s PageSpeed Insights. We don’t want to declare that TTLB is 
> the ultimate metric, but intuitively, I think it is a better indicator 
> when it comes to application performance than TTFB. 
> 
> That does not intend to underestimate the importance of the studies on 
> handshake performance which was crucial to identify the best performing 
> new KEMs and signatures. It also does not intend to underestimate the 
> importance of slimming down PQ TLS 1.3 handshakes as much as possible.
> 
> Side note about Rob’s point: 
> We have not collected QUIC TTLB data yet, but I want to say that the 
> paper’s TTLB experimental results could more or less be extended to 
> QUIC be subtracting one RTT. OK, I don’t have experimental measurements 
> to prove it yet. So I will only make this claim and stop until I have 
> more data.
> 
> 
> 
> *From:* TLS  *On Behalf Of * David Benjamin
> *Sent:* Thursday, March 7, 2024 3:41 PM
> *To:* Deirdre Connolly 
> *Cc:* TLS@ietf.org
> *Subject:* RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte
> 
> *CAUTION*: This email originated from outside of the organization. Do 
> not click links or open attachments unless you can confirm the sender 
> and know the content is safe.
>
> 
> This is good work, but we need to be wary of getting too excited about 
> TTLB, and then declaring performance solved. Ultimately, TTLB simply 
> dampens the impact of postquantum by mixing in the 
> (handshake-independent) time to do the bulk transfer. The question is 
> whether that reflects our goals. 
> 
> Ultimately, the thing that matters is overall application performance, 
> which can be complex to measure because you actually have to try that 
> application. Metrics like TTLB, TTFB, etc., are isolated to one 
> connection and thus easier to measure, and without checking each 
> application one by one. But they're only as valuable as they are 
> predictors of overall application performance. For TTLB, both the 
> magnitude and desirability of dampening effect are application-specific:
> 
> If your goal is transferring a large file on the backend, such that you 
> really only care when the operation is complete, then yes, TTLB is a 
> good proxy for application system performance. You just care about 
> throughput in that case. Moreover, in such applications, if you are 
> transferring a lot of data, the dampening effect not only reflects 
> reality but is larger.
> 
> However, interactive, user-facing applications are different. There, 
> TTLB is a poor proxy for application performance. For example, on the 
> web, performance is determined more by how long it takes to display a 
> meaningful webpage to the user. (We often call this the time 

Re: [TLS] Time to first byte vs time to last byte

2024-03-07 Thread Kampanakis, Panos
Thx Deirdre for bringing it up.

David,

ACK. I think the overall point of our paper is that application performance is 
more closely related to PQ TTLB than PQ TTFB/handshake.

Snippet from the paper

> Google’s PageSpeed Insights [12] uses a set of metrics to measure the user 
> experience and webpage performance. The First Contentful Paint (FCP), Largest 
> Contentful Paint (LCP), First Input Delay (FID), Interaction to Next Paint 
> (INP), Total Blocking Time (TBT), and Cumulative Layout Shift (CLS) metrics 
> include this work’s TTLB along with other client-side, browser 
> application-specific execution delays. The PageSpeed Insights TTFB metric 
> measures the total time up to the point the first byte of data makes it to 
> the client. So, PageSpeed Insights TTFB is like this work’s TTFB/TLS 
> handshake time with additional network delays like DNS lookup, redirect, 
> service worker startup, and request time.

Specifically about the Web, TTLB (as defined in the paper) is directly related 
to FCP, LCP, FID, INP, TBT, CLS, which are 6 of the 7 metrics in Google’s 
PageSpeed Insights. We don’t want to declare that TTLB is the ultimate metric, 
but intuitively, I think it is a better indicator when it comes to application 
performance than TTFB.

That does not intend to underestimate the importance of the studies on 
handshake performance which was crucial to identify the best performing new 
KEMs and signatures. It also does not intend to underestimate the importance of 
slimming down PQ TLS 1.3 handshakes as much as possible.

Side note about Rob’s point:
We have not collected QUIC TTLB data yet, but I want to say that the paper’s 
TTLB experimental results could more or less be extended to QUIC be subtracting 
one RTT. OK, I don’t have experimental measurements to prove it yet. So I will 
only make this claim and stop until I have more data.



From: TLS  On Behalf Of David Benjamin
Sent: Thursday, March 7, 2024 3:41 PM
To: Deirdre Connolly 
Cc: TLS@ietf.org
Subject: RE: [EXTERNAL] [TLS] Time to first byte vs time to last byte


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.


This is good work, but we need to be wary of getting too excited about TTLB, 
and then declaring performance solved. Ultimately, TTLB simply dampens the 
impact of postquantum by mixing in the (handshake-independent) time to do the 
bulk transfer. The question is whether that reflects our goals.

Ultimately, the thing that matters is overall application performance, which 
can be complex to measure because you actually have to try that application. 
Metrics like TTLB, TTFB, etc., are isolated to one connection and thus easier 
to measure, and without checking each application one by one. But they're only 
as valuable as they are predictors of overall application performance. For 
TTLB, both the magnitude and desirability of dampening effect are 
application-specific:

If your goal is transferring a large file on the backend, such that you really 
only care when the operation is complete, then yes, TTLB is a good proxy for 
application system performance. You just care about throughput in that case. 
Moreover, in such applications, if you are transferring a lot of data, the 
dampening effect not only reflects reality but is larger.

However, interactive, user-facing applications are different. There, TTLB is a 
poor proxy for application performance. For example, on the web, performance is 
determined more by how long it takes to display a meaningful webpage to the 
user. (We often call this the time to "first contentful paint".) Now, that is a 
very high-level metric that is impacted by all sorts of things, such as whether 
this is a repeat visit, page structure, etc. So it is hard to immediately 
translate that back down to TLS. But it is frequently much closer to the TTFB 
side of the spectrum than the TTLB side. And indeed, we have been seeing 
impacts from PQ to our high-level metrics on mobile.

There's also a pretty natural intuition for this: since there is much more 
focus on latency than throughput, optimizing an interactive application often 
involves trying to reduce the amount of traffic on the critical path. The more 
the application does so, the less accurate TTLB's dampening effect is, and the 
closer we trend towards TTFB. (Of course, some optimizations in this space 
involve making fewer connections, etc. But the point here was to give a rough 
intuition.)

On Thu, Mar 7, 2024 at 2:58 PM Deirdre Connolly 
mailto:durumcrustu...@gmail.com>> wrote:
"At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web 
(MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a metric 
for assessing the total impact of data-heavy, quantum-resistant algorithms such 
as ML-KEM and ML-DSA on real-world TLS 1.3 conn

Re: [TLS] Time to first byte vs time to last byte

2024-03-07 Thread David Benjamin
This is good work, but we need to be wary of getting too excited about
TTLB, and then declaring performance solved. Ultimately, TTLB simply
dampens the impact of postquantum by mixing in the (handshake-independent)
time to do the bulk transfer. The question is whether that reflects our
goals.

Ultimately, the thing that matters is overall application
performance, which can be complex to measure because you actually have to
try that application. Metrics like TTLB, TTFB, etc., are isolated to one
connection and thus easier to measure, and without checking each
application one by one. But they're only as valuable as they are predictors
of overall application performance. For TTLB, both the magnitude and
desirability of dampening effect are application-specific:

If your goal is transferring a large file on the backend, such that you
really only care when the operation is complete, then yes, TTLB is a good
proxy for application system performance. You just care about throughput in
that case. Moreover, in such applications, if you are transferring a lot of
data, the dampening effect not only reflects reality but is larger.

However, interactive, user-facing applications are different. There, TTLB
is a poor proxy for application performance. For example, on the web,
performance is determined more by how long it takes to display a meaningful
webpage to the user. (We often call this the time to "first contentful
paint".) Now, that is a very high-level metric that is impacted by all
sorts of things, such as whether this is a repeat visit, page structure,
etc. So it is hard to immediately translate that back down to TLS. But it
is frequently much closer to the TTFB side of the spectrum than the TTLB
side. And indeed, we have been seeing impacts from PQ to our high-level
metrics on mobile.

There's also a pretty natural intuition for this: since there is much more
focus on latency than throughput, optimizing an interactive application
often involves trying to reduce the amount of traffic on the critical path.
The more the application does so, the less accurate TTLB's dampening effect
is, and the closer we trend towards TTFB. (Of course, some optimizations in
this space involve making fewer connections, etc. But the point here was to
give a rough intuition.)

On Thu, Mar 7, 2024 at 2:58 PM Deirdre Connolly 
wrote:

> "At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web
> (MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a
> metric for assessing the total impact of data-heavy, quantum-resistant
> algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our
> paper shows that the new algorithms will have a much lower net effect on
> connections that transfer sizable amounts of data than they do on the TLS
> 1.3 handshake itself."
>
>
> https://www.amazon.science/blog/delays-from-post-quantum-cryptography-may-not-be-so-bad
>
> ¹
> https://www.amazon.science/publications/the-impact-of-data-heavy-post-quantum-tls-1-3-on-the-time-to-last-byte-of-real-world-connections/
>
>
> ___
> TLS mailing list
> TLS@ietf.org
> https://www.ietf.org/mailman/listinfo/tls
>
___
TLS mailing list
TLS@ietf.org
https://www.ietf.org/mailman/listinfo/tls


Re: [TLS] Time to first byte vs time to last byte

2024-03-07 Thread Rob Sayre
I agree the efficiency concerns are generally overstated, but this study
should have measured QUIC etc, since the web pages will have all sorts of
awful performance problems. But the thing you have to watch out for is when
someone in the datacenter steps on the power cord or something (or DNS is
wrong, etc). Then, you get a stampede of clients reconnecting, and there
you really do care about how expensive the handshake is.

thanks,
Rob

On Thu, Mar 7, 2024 at 11:58 AM Deirdre Connolly 
wrote:

> "At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web
> (MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a
> metric for assessing the total impact of data-heavy, quantum-resistant
> algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our
> paper shows that the new algorithms will have a much lower net effect on
> connections that transfer sizable amounts of data than they do on the TLS
> 1.3 handshake itself."
>
>
> https://www.amazon.science/blog/delays-from-post-quantum-cryptography-may-not-be-so-bad
>
> ¹
> https://www.amazon.science/publications/the-impact-of-data-heavy-post-quantum-tls-1-3-on-the-time-to-last-byte-of-real-world-connections/
>
>
> ___
> TLS mailing list
> TLS@ietf.org
> https://www.ietf.org/mailman/listinfo/tls
>
___
TLS mailing list
TLS@ietf.org
https://www.ietf.org/mailman/listinfo/tls


[TLS] Time to first byte vs time to last byte

2024-03-07 Thread Deirdre Connolly
"At the 2024 Workshop on Measurements, Attacks, and Defenses for the Web
(MADweb), we presented a paper¹ advocating time to last byte (TTLB) as a
metric for assessing the total impact of data-heavy, quantum-resistant
algorithms such as ML-KEM and ML-DSA on real-world TLS 1.3 connections. Our
paper shows that the new algorithms will have a much lower net effect on
connections that transfer sizable amounts of data than they do on the TLS
1.3 handshake itself."

https://www.amazon.science/blog/delays-from-post-quantum-cryptography-may-not-be-so-bad

¹
https://www.amazon.science/publications/the-impact-of-data-heavy-post-quantum-tls-1-3-on-the-time-to-last-byte-of-real-world-connections/
___
TLS mailing list
TLS@ietf.org
https://www.ietf.org/mailman/listinfo/tls