Skip to content

v13.0.0

Latest
Latest
Compare
Choose a tag to compare
@github-actions github-actions released this 27 Apr 19:01
· 104 commits to dev since this release
v13.0.0
fab116e

Highlights

This is quite a large and impactful release. We've added a lot of improvements to make your tests rock-solid, but there are also a couple of breaking changes. See the details below, but here's a summary:

  • Browser log management now uses the cross-browser Selenium BiDirectional API, allowing accessing the log of every browser, not just Chrome.
  • Upgraded xUnit to v3, which is not only necessary for maintenance, but also brings proper cancellation of tests. No more dangling browser or driver processes after a test cancellation!
  • You can test not just sending e-mails with a local SMTP server, but also receiving e-mails.
  • Fixing reliability issues that Chrome v134 brought. This Chrome version broke a huge number of tests in intricate ways, most possibly due to its more aggressive resource management.
  • To avoid random bugs due to browsers updating on their own, all browsers are now pinned to a specific version, and automatically managed behind the scenes, in all environments (every OS, locally, in a CI environment...).
  • Tests using Elasticsearch can be run in parallel without any special configuration (this isn't trivial, since they mustn't use the same Elasticsearch index) and they'll automatically wait for the initial indexing to complete, not to try to search too early.
  • And a lot of smaller improvements and bug fixes!

Breaking changes in browser log management

The browser log, i.e. what you also see in the developer console, is useful to pinpoint unhandled JavaScript exceptions, network errors, and other client-side issues. This is what in Selenium can be accessed via IWebDriver.Manage().Logs.GetLog(LogType.Browser). In the UI Testing Toolbox it was accessible via various other ways, and was also automatically asserted after every page load.

In this release, how you can access the browser log, and what objects represent the log, changed. This was necessary because the previous version actually only supported Chrome and Edge, it just silently failed under other browsers (notably Firefox). With a new Selenium version, it now fails with an exception (that's why we actually noticed it).

To overcome this, all browsers now use the new Selenium BiDirectional API to access the logs.

What changed

  • UITestContext.HistoricBrowserLog is now called CumulativeBrowserLog to better reflect that it stores all browser log entries since the start of the test. Instead of containing OpenQA.Selenium.LogEntry objects, it contains OpenQA.Selenium.BiDi.Modules.Log.Entry ones. These have a slightly different interface, but work much the same: Most possibly, you'll only need to change references to LogEntry.Message to Entry.Text, and LogEntry.Level to Entry.Level, which uses a different, but very similar, enum. For the same reason, ClearHistoricBrowserLog() is now called ClearCumulativeBrowserLog() but behaves the same.
  • Similar to the response log, see below, the browser log now has a filter, UITestContext.BrowserLogFilter, to determine what gets logged in the first place. If you need to ignore expected log entries (since a non-empty log fails the test by default), we recommend adjusting the filter instead of the assertion. The samples contain examples of this.
  • OrchardCoreUITestExecutorConfiguration.AssertBrowserLog now works with OpenQA.Selenium.BiDi.Modules.Log.Entry objects too, as do all related helpers.
  • UITestContext.UpdateHistoricBrowserLogAsync() was removed. It's not necessary anymore since the logs are now updated not explicitly, but with events, whenever a new entry is added.
  • UITestContext.AssertCurrentBrowserLogAsync() and the IWebDriver extension GetAndEmptyBrowserLog() were also removed for the same reason.
  • Unexpected response codes, like 404 responses, aren't logged to the browser log anymore. Thus, LogEntry.IsNotFoundLogEntry() was removed since it can't apply to Entry objects. Instead, check out UITestContext.CumulativeResponseLog and UITestContext.ResponseLogFilter. This new behavior is demonstrated in the samples.
  • BiDi keeps a tab open for itself, what you mustn't close (if you do, the browser session will also close). This shouldn't cause an issue in most cases, but check if your tests assume that only their tabs are open (like if you use context.SwitchToLastWindow() or context.SwitchToLastWindow()).

How to adapt

  • The most important issues will show up as build errors. Fix them following the guidelines above.
  • Check if you have assertion code for the browser log containing 404 or other network errors. These won't show up in the browser log anymore, so you'll have to adjust your assertions to use the response log, see above.
  • Running your test suite will make any further issues apparent.
  • We also encourage you to explore the BiDi APIs, since they provide functionality not seen in Selenium before:
var biDi = await context.Scope.Driver.AsBiDiAsync();
// Check out biDi.Network, for example.

Upgraded xUnit to v3 with breaking changes

This version of the UI Testing Toolbox uses v3 of xUnit, which brings many updates. However, it's also a breaking version, requiring you to adapt your test projects, see the official guide.

Migrating UI test projects consuming the UI Testing Toolbox should be simpler, though, with you requiring to do roughly the following steps:

  1. Update the xunit.runner.visualstudio package reference to latest (currently 3.0.2).
  2. Convert test projects to executables, see docs (this will most possibly only need you to add <OutputType>Exe</OutputType> to the first PropertyGroup in the test project's csproj). These are all the projects with xunit.runner.visualstudio references.
  3. In all test projects, change xunit package references to xunit.v3 with the latest version (currently 2.0.0). If a test project lacks such a reference then add it (currently <PackageReference Include="xunit.v3" Version="2.0.0" />). Only test projects should reference xunit, but they must reference it, otherwise tests won't be discovered.
  4. Also update the Microsoft.NET.Test.Sdk references in the test projects to latest (currently 17.13.0) too while we're at it, or add it if it's missing (currently <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />).
  5. ITestOutputHelper is now in the Xunit namespace instead of Xunit.Abstractions. Fix the namespace imports in UITestBase classes first and then anywhere else; they'll show up as build errors.
  6. Clean up unused namespace imports. In Visual Studio, you can do this by right-clicking on the solution -> Analyze and Code Cleanup -> Run Code Cleanup (Profile 1).
  7. Run the tests to check if everything is still OK. If tests don't show up in the Visual Studio Test Explorer, or show up but don't start if you try to run them, check out the Tests pane of the Output Window for clues. Confirm that the Web project doesn't contain any xUnit references (even transitively, like by accidentally referencing UI test projects) but all test projects reference xunit.v3. You might need to do a recursive git clean to be sure to start with a clean slate.
  8. Verify in CI that the tests are actually discovered and all run there too (the CI build being green is not enough, it might miss no tests being discovered).

Complete changelog

New Contributors

Full Changelog: v12.0.0...v13.0.0

Contributors

renovate, Piedone, and 6 other contributors
Assets 8
Loading