[.NET] More robust ndjson handling for Spec/UnitTests

[obligaron]

🤔 What's changed?

This PR only modifies the Spec/UnitTests, not any shipped artifacts (e.g., NuGet assemblies).

• Used System.Text.Json 9's streaming multiple JSON documents feature for ndjson
• Fixed a file lock in TokensGenerator.GenerateTokens
• Allows tests to be run in parallel
• Removes the System.Text.Json version dependency validation tests because, since [.NET] Removed dependency on System.Text.Json for improved startup time #338, the Gherkin assemblies no longer depend on System.Text.Json.

⚡️ What's your motivation?

While working on #405, it was discovered that the .NET tests are not as robust as other implementations when handling ndjson files. See comment in #402 (comment).

This also improves the test runtime (on my machine) from 3.5 seconds to 2.5 seconds and avoids file lock errors when running the tests on Windows.

🏷️ What kind of change is this?

• 🏦 Refactoring/debt/DX (improvement to code design, tooling, etc. without changing behaviour)

📋 Checklist:

• I agree to respect and uphold the Cucumber Community Code of Conduct
• I've changed the behaviour of the code
  • I have added/updated tests to cover my changes.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• Users should know about my change
  • I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.