extension-v1.21.1

[extension-v1.21.1] - 2025-04-03: Bugfixes

Fixed

• A small crash fix for when some reqres become marked as errored on restart (which is a heisenbug I have not tracked down yet).

Changed

• Split config.debugConfig ("Allow unsafe config values") out of config.debugRuntime ("Debug runtime") since the latter is ok to leave enabled while the former should be disabled most of the time.

extension-v1.21.0

[extension-v1.21.0] - 2025-02-11: Re-archival, reworked internals, many small improvements

Added/Changed/Removed (1): Re-archival replaces re-queueing

• Core + Popup UI + the Saved in Local Storage page + the Help page:

  • Implemented new re-archival machinery. Hoardy-Web now follows the following state diagram:

    (start) -> (request sent) -> (nIO) -> (headers received) -> (nIO) --> (body recived)
       |                           |                              |             |
       |                           v                              v             v
       |                     (no_response)                   (incomplete)   (complete)
       |                           |                              |             |
       |                           \                              |             |
       |\---> (canceled) ----\      \                             |             |
       |                      \      \                            \             |
       |\-> (incomplete_fc) ---\      \                            \            v
       |                        >------>---------------------------->-----> (finished)
       |\--> (complete_fc) ----/                                             /  |
       |                      /                                             /   |
       \----> (snapshot) ----/       /- (collected) <--------- (picked) <--/    |
                                    /       ^                      |            |
     (stashToLS?) <----------------/        |                      v            v
          |                                 \- (in_limbo) <- (stashToLS?) <- (dropped)
          v                                           |                         |
      (queued) <---- (unarchived) <- (stashToLS?)     |                         |
          |                               ^           \-----> (discarded) <-----/
       /-\|          /---> {U} -----------/                        ^
       ^  |          |      \                                      |
       |  |          |       \--> (forgotten) ((-> (saved)))       |
       |  |          |                                             |
     {RE} |          v                  {{!saving && !submitting}} |
          |\----> (saveAs) -> (exported) -------------------------/|
          |                       /                                |
          | {U} <-\    /---------/                                 |
          |        v   v                        {{!saving}}        |
          |\----> (srvIO) -> (submitted) -------------------------/|
          |                     /                                  |
          |                    /              {{delete}}           |
          | {U} <-\     /-----/     /-----------------------------/|
          |        \   /           /                               |
          |        v   v          /       {{re-archive}}           |
          |\---> (saveToLS) -> (saved) -------------------> {RE}   |
          |                                                        |
          \--------------------------------------------------------/
    

    That is, "re-queueing" mechanism was dropped, instead, reqres saved in local storage can now be sent through the archival pipeline multiple times, without touching the queue.
    Also, the re-archival machinery now has a separate set of configurable archival methods.
    And the reqres going through re-archival will be sent through a separate instance of the archival pipeline, in a way that will block Hoardy-Web's main thread, while recording archival errors separately, reporting them at the end, and then forgetting about them (after all, everything is saved in local storage anyway).

    This might not sound that impressive, but it is, because its both more convenient and has much better performance:

    • as before, collection can still work in parallel with this;

    • but normal archival now stops completely while re-archival is working, which locks local storage from modification, which allows re-archival machinery to be much more efficient than the re-queueing machinery was, re-archiving everything requested and applying all updates in a single pass over local storage, and using the smallest required number of disk writes;

    • also, internally, the re-archival machinery is now a simple function, not a collection of distinct schedule-able tasks, which means it can easily be called explicitly, which allows common re-archival workflows to be bound to separate UI buttons and/or keyboard shortcuts,

      which Hoardy-Web now does, making things much more convenient.

    The latter feature solves a common complaint of the Saved in Local Storage page UI being hard to use.
    From now on, its use is completely optional.
    Popup UI now has two simple Re-archive all and Re-archive adjunct buttons that re-archive everything and reqres that were "not previously (re-)archived like this", respectively.

    This can be useful in following use cases:

    • You want to collect and accumulate lots of reqres before re-archiving them by the means of Export via 'saveAs' to make the UI less annoying and compress the resulting WRR bundles better.

    • You run Hoardy-Web under a Fenix-based mobile browser or under Tor Browser, where the use of re-archival is quite common and needing to manually re-queue stuff all the time was really annoying.

    All of this can be done with a single button-press and using a smallest required number of disk writes now.

• Core + Default config:

  • Set the default config.exportAsMaxSize to 64 MiB again and increased the limit to 512 MiB both on Chromium and on Firefox.

    Bigger values are better for WRR bundle compression ratios, and my tests had shown that this actually works fine even on Firefox, even though Mozilla's docs claim it would not, so.

Changed (2): Internals

• Core:

  • Performed a huge internal reworking, splitting everything into a bunch of logically consistent mostly-independent modules, making many functions mostly side-effect pure, etc.

  • Reworked broadcast machinery quite a bit.

    It's significantly faster now, and internal messages no longer get broadcasted to internal pages that do not request them.

    This improved performance when running with many open browser tabs containing internal pages by quite a bit.

  • Simplified internal command handling quite a bit.

    So, the external API is going to be implemented soon-ish.

    This also, essentially, auto-generates implementations of many internal actions.

• Core + manifest.json:

  • Added a bunch more useful, but unbound by default, keyboard shortcuts.

    Since their actual internal implementations are now free.

Added/Changed (3): Capture and logging

• Core + Capture:

  • Renamed fake reqres/WRR flag to request_buggy, because the old name was not at all descriptive.

  • The webRequest::capture::RESPONSE::BROKEN and the old webRequest::pWebArc::RESPONSE::BROKEN errors were remade into a new response_buggy reqres/WRR flag.

    The migration is automatic.

  • Renamed sent reqres/WRR flag to submitted, since it does not guarantee that the request was actually sent, only that it was submitted to the networking pipeline.

    After all, such a request can then be answered from cache, for instance.

  • The logs will display all of these flags now.

  • Also, the final networking state in the logs in prepended with a "$" symbol, to make it distinguishable from flags.

  • Added config options and popup UI for controlling problematic marking and picking of reqres with these flags.

  • From now on config.markProblematicPartialRequest and config.markProblematicBuggy will only get applied to request that were actually submitted and not from_cache, since !submitted || from_cache implies it was not actually sent.

  • On Chromium, request_buggy reqres will no longer get origin_url WRR field set from Referer.

    This adheres to the "capture as raw as possible" philosophy better.

• Documentation:

  • Documented all these flags in doc/data-on-disk.md, which now comes bundled with the extension too.

Added/Changed (4): UI

• Core + Toolbar button + Icons:

  • Added unsnapshottable and unreplayable icons.

    From now on, they are used for displaying the states of the corresponding per-tab config options.

  • Added vertical bar and dot icons.

    They are now used for separating per-tab and per-tab-children toolbar icon animation frames, when they differ.

    This makes things much less confusing.

• Core + Default config:

  • Increased the default config.animateIcon value a bit, to make those icons easier to visually parse.

• Popup UI:

  • Renamed Run tab/tag to RT, Class to CLS, moved UI tab/tag to right after CLS and put spacers at better points.

    Personally, I misclick less with this layout.

  • Moved all snapshottable and replayable toggles to the end of their sections.

  • When viewed from the Help page, per-tab options are now available and settable again.

    This makes no technical sense, but this is less UI-surprising.

  • On Fenix, improved the layout slightly.

• Core + Popup UI:

  • Split config.debugging into a bunch of more fine-grained configuration options.

Changed (5): Documentation

• Documentation:

  • Improved the Help page.

  • Improved keyboard shortcut descriptions.

  • Improved popup UI help strings.

Fixed

• Core:

  • F...

extension-v1.20.0

[extension-v1.20.0] - 2025-01-24: Annoyance and bug fixes

Added

• Core + Popup UI:

  • Implemented config.limboNotifyInterval option and its popup UI.

    It prevents "Too much stuff in limbo" notifications from being generated more frequently than given number of seconds, which makes them much less annoying.

Changed

• Core:

  • From now on, retryUnarchived scheduled action can be canceled (until a new reqres gets captured) with the cancelActions (red square) button.

  • Unavailable and non-archiving servers produce different notification error messages and are handled differently now.

  • Archivals to broken servers and localStorage produce nicer notification errors now.

  • Unrecoverable archival failures now get a hint about using the retryFailed popup UI button in their notification messages.

  • From now on, pressing keyboard shortcuts and popup UI buttons will force immediate updates to displayed notifications.

    I.e. no longer relevant notifications will disappear immediately.

  (Thanks to @douglasg14b for pointing out three out of five of the above.)

• The Help page:

  • Added "Table of Contents" section.

  • Moved and adapted a bunch move stuff from top-level README.md.

  • Improved some pieces of it.

• Other documentation pages:

  • Improved rendering of various references, added more of them.

  • Improved in many random places.

Fixed

• All documentation pages:

  • Clicking on internal links will now replace history states that point to other links.

    This makes the use of "Back" browser button more intuitive.

• Core:

  • On Firefox, fixed an issue when, with a popup UI open, opening "Internal State" page would make the popup show the wrong tab stats until a tab switch.

  • On Firefox, fixed popup UI's tab stats flickering between inconsistent states when, with a popup UI open, a new tab gets opened, starts loading, and then gets redirected, with all of it happening really fast.

tool-v0.23.0

[tool-v0.23.0] - 2025-01-22: Incremental improvements

Fixed

• *:
  • Fixed .part files not being properly filtered out from the inputs.
    (#10 at GitHub. Thanks to @douglasg14b on GitHub for reporting it!)
  • Fixed it fsyncing things out of proper order sometimes.

Changed

• *:

  • From now on, in most cases, newly created .part files should be automatically deleted if hoardy-web encounters an FS error.
  • Improved error messages.

• Improved documentation.

Added

• *:
  • Improved Windows support, added more tests under wine.
  • Added tests for .part filtering and archival with serve.

tool-v0.22.0+simple_server-v1.9.0

[tool-v0.22.0] - 2025-01-17: Incremental improvements

Changed

• *:

  • Formatted code using black.
  • Fixed minor issues found by pylint.
  • Simplified a bunch of code.
  • Moved a bunch of code to kisstdlib.
  • Changed the rest to work with the new version of kisstdlib.
  • Improved error handling and error messages.

• serve:

  • From now on, when archiving/dumping, by default, it will both return errors to the client and also print them to stderr.
    The latter can be disabled by --quiet.

• Improved documentation.

Fixed

• *:

  • Fixed extension detection for .css and .mjs files.
  • Fixed a potential crash when parsing WRR files.

• serve and, likely, organize:

  • Fixed it not working on Windows. (Thanks to @douglasg14b on GitHub!)

[simple_server-v1.9.0] - 2025-01-17: Incremental improvements

Changed

• Formatted code using black.
• Fixed minor issues found by pylint.
• Improved metadata.
• Improved documentation.

Fixed

• Fixed version detection, i.e. --version option works again.

tool-v0.21.0

See CHANGELOG.md.

extension-v1.19.0

See CHANGELOG.md.

extension-v1.18.0+tool-v0.20.0+simple_server-v1.8.0

See CHANGELOG.md.

tool-v0.19.0

See CHANGELOG.md.

tool-v0.18.1

See CHANGELOG.md.