Feature: use Origin-Private Filesystem (OPFS) directory

[ansemjo]

Hey, I'm not sure how much effort this would be, so: would it be viable to support the Origin-Private Filesystem (OPFS) in the fds: [] argument to the WASI constructor? I suppose that you'd need to write a new Directory subclass, which wraps a FileSystemDirectoryHandle or something like that?

// open directory in OPFS
const opfs = await navigator.storage.getDirectory();
const dir = await opfs.getDirectoryHandle("wasi-context", { create: true });

// initialize WASI shim
let wasi = new WASI(args, envs, [
  new OpenFile(new File([])), // stdin
  new OpenFile(new File([])), // stdout
  new OpenFile(new File([])), // stderr
  new WrapDirHandle(dir),     // <----- ?!
]);

...

As far as I can tell, supporting this API would also allow you to use window.showDirectoryPicker() and use an actual directory on the client machine for the execution context.

[bjorn3]

This requires async code which wasm doesn't allow. So it would be necessary to move running the wasm module in a web worker with atomics for communication with the main thread. (might be useful anyway to eg allow downloading parts of the vfs on the fly) In addition firefox doesn't support this api, so I can't easily test any changes. A persistent VFS based on eg IndexedDB should be possible to implement after async support is implemented.

[ansemjo]

Oh, I see. I wasn't aware that there's such a hard limitation. Makes sense though.

According to this article, it is possible to get synchronous handles to individual files – albeit only inside a Worker context.

In addition firefox doesn't support this api, [...]

Not sure what you mean? I'm playing with the OPFS on Firefox 114.0 right now .. 🙂
edit: Oh, you probably mean window.showDirectoryPicker(). Yeah, that one isn't supported, unfortunately. 😬

[bjorn3]

Firefox supports OPFS itself already? If so I don't have objections to implementing it. If you need to use it, it shouldn't be to hard to leave it up to user code to call showDirectoryPicker() I think.

[ansemjo]

Firefox supports OPFS itself already?

Yes! Since Version 111, apparently. My current experiment downloads a binary to OPFS in one Worker and then executes it in another, somewhat like this:

// OPFSWorker: download a WASM binary to OPFS
async function download(url: string) {

    // start fetching
    const download = fetch(url);

    // create the file and open a writable stream
    const stream: WritableStream = await navigator.storage.getDirectory()
    	.then(opfs => opfs.getFileHandle("binary.wasm", { create: true }))
    	.then(handle => (<any>handle).createWritable());

    // pipe the response body to file
    await (await download).body?.pipeTo(stream);

}

// WASMRunner: execute the downloaded WASM from OPFS
async function run() {

    // get arraybuffer of file
    const buf = await navigator.storage.getDirectory()
    	.then(opfs => opfs.getFileHandle("binary.wasm"))
    	.then(handle => handle.getFile())
    	.then(file => file.arrayBuffer());

    // initialize WASI shim
    [...]

     // compile the module
    const { instance } = await WebAssembly.instantiate(buf, imports);

    // ...
    
}

Only having a synchronous access inside a Worker wouldn't be a problem for me, since I am planning to run all WebAssembly modules in Workers anyway.

[ansemjo]

Having thought about this some more after I experimented with Workers a little and checking the SQLite example backed by OPFS it seems to me that supporting OPFS would require:

• The thread executing the WebAssembly must be able to block using Atomics.wait(). That means that the WASM itself – and by extension the browser_wasi_shim – needs to run in a Worker.
• If you block a thread with Atomics.wait(), you can't do anything else in that thread. So the OPFS IO needs to happen in another thread / Worker anyway. You need to request an operation either through postMessage() or a SharedArrayBuffer in some way.
  • Posting a message and then immediately blocking could fail to deliver the message though, as StackBlitz found out.
  • I'm looking at installOpfsVfs from the above SQLite example right now and how they're setting up SharedArrayBuffers to communicate which operation to run. (They're using the Atomics.notify() & Atomics.wait() approach to communicate with the OPFS Worker.)
  • The good thing is that when the requesting thread is blocked anyway, you can also use async code as much as you like. Just write the data to the shared buffer and Atomics.notify() at some point.
• The only exception to the above would probably be a single file opened synchronously through await fileHandle.createSyncAccessHandle() before starting the instance. This still requires you to run WASM in a Worker but would avoid the SharedArrayBuffer communication.

So, it is possible but certainly not trivial .. Also, I realize now that you've basically said all that already in your first comment above. But I'm still new to all of this and just starting to wrap my head around this. 😬

[bjorn3]

Only synchronous file access works currently. Asynchronous file access and directories are yet to be implemented.