Speculative Request Control

[eligrey]

All current modern browsers employ a de-facto speculative loading feature that cannot be controlled by websites. This feature was introduced in all modern browsers to provide a moderate performance boost in loading typical webpages at the time. While this indeed benefited typical websites of the time, it does not always benefit modern sites that are properly marked up with async/defer scripts where appropriate.

Websites should be able to opt-out of eager speculative requests and be able to accept responsibility for their own site performance.

Giving site owners control over eager speculative requests improves the security implications of generating dynamic <meta> CSPs at runtime based on private locally-stored tracking consent data. Currently, client-side-generated <meta> CSPs are effectively unenforced until DOMContentLoaded due to eager speculative requests. With eager speculative requests disabled, these CSPs can be effectively applied and enforced immediately.

What is an eager speculative request?

An eager speculative request is a speculative request that is sent out before preceding synchronous scripts finish executing.

Motivating Use Cases

The motivating use case for this feature is to increase the ease at which sites could adopt a CSP based on locally-stored consent provided by a third party JS library. In this use case, we can assume that the library vendor and site owner have taken the time to explicitly preload resources asynchronously where appropriate, as they must knowingly disable eager speculative requests.

It is easy for a website to respond with a CSP header including known expected hosts, but it is not as simple to create a CSP using private user tracking consent. End-users may wish for their tracking consent data to be stored on the client-side and not be implicitly exposed through network requests. It is possible to create a client-side JavaScript library (e.g. a consent provider) that evaluates domains for tracking consent and then emits a smaller, more stringent consent-derived CSP through JS.

Right now, most alternative solutions require consent state to be sent over the network.

More in my explainer draft.

[annevk]

Related: whatwg/html#5624. cc @zcorpan @hsivonen

[hsivonen]

It seems to me that this hinges a lot on whether one considers CSP via runtime meta as a misfeature or not. To me, it seems questionable not to provide CSP via HTTP headers.

async/defer etc. are rather beside the point. Markup that's visible to the parser can be deal with on a per-resource basis without having to turn the whole preloader off.

It's possible that the exist super-competent Web devs who can do better than the browser. However, it's all too likely that turning the preloader off becomes a cargo cult applied by less competent developers who end up sabotaging the overall perf.

[zcorpan]

First, from what I can tell, meta CSP is applied and enforced immediately already. It's specified in HTML like so:

When a meta element is inserted into the document, if its http-equiv attribute is present and represents one of the above states, then the user agent must run the algorithm appropriate for that state, as described in the following list:
...
Content security policy state (http-equiv="content-security-policy")
This pragma enforces a Content Security Policy on a Document. [CSP]

1. If the meta element is not a child of a head element, return.

2. If the meta element has no content attribute, or if that attribute's value is the empty string, then return.

3. Let policy be the result of executing Content Security Policy's parse a serialized Content Security Policy algorithm on the meta element's content attribute's value, with a source of "meta", and a disposition of "enforce".

4. Remove all occurrences of the report-uri, frame-ancestors, and sandbox directives from policy.

5. Enforce the policy policy.

https://html.spec.whatwg.org/multipage/semantics.html#attr-meta-http-equiv-content-security-policy

This demo shows that it is enforced immediately in webkit, chromium, gecko: http://software.hixie.ch/utilities/js/live-dom-viewer/saved/8456

Second, I don't see how this relates to speculative parsing. In whatwg/html#5624 (comment) I noted that chromium stops the speculative parser (or at least doesn't speculatively fetch anything) if the speculative parser finds a CSP meta, while webkit and gecko ignore it in the speculative parser. That is, webkit and gecko can speculatively fetch resources after a CSP meta without enforcing the CSP policy to those fetches. I hope those fetches are invalidated when the real parser finds the CSP meta and later the fetching element, but I haven't tested that. Still, this issue talks about script-generated meta, which is not something the speculative parser could find.

Also, I was under the impression that speculative parsing only starts if there's a non-defer non-async non-module script src element. Is this not the case?

[hsivonen]

Also, I was under the impression that speculative parsing only starts if there's a non-defer non-async non-module script src element. Is this not the case?

That applies for document.write-inserted content in Gecko. However, for content arriving from the network, the prefetches start before the corresponding DOM insertions even when the parsing is not speculative. The time difference between those fetches starting and the corresponding DOM insertions happening is just very short.

[hsivonen]

Notably, the DOM insertions look at the wall clock to decide to let the event loop to spin before a script is seen. That in particular would allow non-speculative prefetches to end up a longer time earlier from the corresponding DOM insertions.

[zcorpan]

@hsivonen what you describe is then not relevant for speculative parsing, but for speculative fetches that happen from the normal HTML parser, between the time the tree builder processes a token and the time the parser decides to insert the element to the DOM. Yes?

So, it's possible for a script to insert a CSP meta in between?

[hsivonen]

@hsivonen what you describe is then not relevant for speculative parsing, but for speculative fetches that happen from the normal HTML parser, between the time the tree builder processes a token and the time the parser decides to insert the element to the DOM. Yes?

Yes.

So, it's possible for a script to insert a CSP meta in between?

Yes, e.g. from setTimeout if the DOM insertion batch is so long relative to CPU speed that the event loop gets to spin.

In general, trying to use script to undo fetches that, absent the script, would be caused by what's in the HTML source from the network results in a bad time, and I think we should not change the Web Platform to facilitate such attempts.

[zcorpan]

@hsivonen thanks. So a control that disables speculative parsing, or avoiding script src, is not enough to avoid all speculative fetches, since normal parsing also does speculative fetches.

[eligrey]

I'm renaming this issue to "Speculative Request Control" to better fit with the use-case goals. What I want in my use case is a way to disable speculative fetches alone. Speculative parsing without making requests is safe and doesn't need a control in my opinion.

[eligrey]

I've tweaked the proposed API naming to reflect my focus on eager speculative requests:

API

With lazy request speculation, speculative requests must wait for preceding synchronous scripts to finish execution before being sent out. Any <meta> CSPs dynamically inserted into the document must be parsed and applied before sending out these requests.

Request-Speculation HTTP header

Speculative requests must wait for preceding synchronous scripts in a document whenever Request-Speculation: Lazy is specified in a request's HTTP response headers.

request-speculation attribute on document element

If there is a root document element with a request-speculation attribute and the attribute has a value that case-insensitively equals lazy, then speculative requests must wait for preceding synchronous scripts.

Read-only Document.prototype.requestSpeculation getter

document.requestSpeculation reflects the document's current request speculation setting as either eager or lazy. This is a read-only getter.

Example usage

<html request-speculation="lazy">
  <head>
    <script src="/consent-provider-utils.js"></script>
    <script>
    // Create meta CSP
    const meta = document.createElement('meta');
    meta.httpEquiv = 'Content-Security-Policy';

    // Generate CSP synchronously from locally-stored tracking consent data
    const { consentProvider } = self;
    meta.content = consentProvider
      ? consentProvider.generateCSPFromConsent(localStorage.trackingConsent)
      : 'default-src […];'; // default CSP

    // Enforce CSP on document
    document.head.appendChild(meta).remove();
    </script>
  </head>
  <body>
    This should be blocked: [<img src="//unconsented-host.example"/>]
  </body>
</html>

[othermaciej]

I don't think this feature is a good idea. Enabling websites to disable specific performance optimizations is likely to do more harm than good on net. even if there may be rare cases where there's a good reason to do so. I endorse @hsivonen 's explanation of the dynamic in #19 (comment)

At the extreme, if a site dynamically inserts <meta>, it can also dynamically insert all of its content after the <meta> is added. This may be bad for perf, but disabling speculative loading is also likely to be bad for perf.

Also note that there could be forms of speculative loading besides preload scanning, such as predictive speculative loading (or predictive speculative revalidation), that could occur before a DOM call, an attribute on the document element, or even an HTTP response header would be seen by the browser. So the proposed feature cannot meet its promise of totally preventing speculative loading without potentially preventing some speculative loading for pages that don't even use the feature.

[eligrey]

I've reduced the scope of this proposal down to the control of 'eager speculative requests', which I am defining as speculative requests that are sent out before preceding synchronous scripts finish executing.

This feature can no longer be used to completely disable speculative requests, and describes the minimum features needed to empower the dynamic <meta> CSP use-case without causing as much potential harm to performance.

This update is now reflected in https://github.com/eligrey/speculative-request-control and in this issue