Skip to content

meta: update CONTRIBUTING.md to point to test plans for skipping #5140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

meta: update CONTRIBUTING.md to point to test plans for skipping #5140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,13 @@ make test

### Flaky tests

If you see a test being flaky, you should ideally fix it immediately. If that's not feasible, you can disable the test in the test scheme by unchecking it in the Test action:
If you see a test being flaky, you should ideally fix it immediately. If that's not feasible, you can disable the test in the test scheme by unchecking it in the associated test plan:

![Disabling test cases via the Xcode scheme](./develop-docs/disabling_tests_xcode_scheme.png)

or by right-clicking it in the Tests Navigator (⌘6):

![Disabling test cases via the Xcode Tests navigator](./develop-docs/disabling_tests_xcode_tests_navigator.png)
![Disabling test cases via the Xcode Tests navigator](./develop-docs/disabling_tests_xcode_test_plan.png)

Then create a GH issue with the [flaky test issue template](https://github.com/getsentry/sentry-cocoa/issues/new?assignees=&labels=Platform%3A+Cocoa%2CType%3A+Flaky+Test&template=flaky-test.yml).

Disabling the test in the scheme has the advantage that the test report will state "X tests passed, Y tests failed, Z tests skipped", as well as maintaining a centralized list of skipped tests (look in Sentry.xcscheme) and they will be grayed out when viewing in the Xcode Tests Navigator (⌘6):
Disabling the test in the test plan has the advantage that the test report will state "X tests passed, Y tests failed, Z tests skipped", as well as maintaining a centralized list of skipped tests (look in the associated .xctestplan file source in //Plans/) and they will be grayed out when viewing in the Xcode Tests Navigator (⌘6):

![How Xcode displays skipped tests in the Tests Navigator](./develop-docs/xcode_tests_navigator_with_skipped_test.png)

Expand Down
8 changes: 0 additions & 8 deletions Samples/iOS-Swift/iOS-Swift.xcodeproj/xcshareddata/xcschemes/iOS-Swift.xcscheme
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,6 @@
BlueprintName = "iOS-Swift-UITests"
ReferencedContainer = "container:iOS-Swift.xcodeproj">
</BuildableReference>
<SkippedTests>
<Test
Identifier = "SentryFileIOTrackingIntegrationTests/test_DataConsistency_readPath_disabled()">
</Test>
<Test
Identifier = "SentryFileIOTrackingIntegrationTests/test_DataConsistency_readUrl_disabled()">
</Test>
</SkippedTests>
</TestableReference>
</Testables>
</TestAction>
Expand Down
73 changes: 0 additions & 73 deletions Sentry.xcodeproj/xcshareddata/xcschemes/Sentry.xcscheme
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,74 +60,6 @@
BlueprintName = "SentryTests"
ReferencedContainer = "container:Sentry.xcodeproj">
</BuildableReference>
<SkippedTests>
<Test
Identifier = "SentryANRTrackerTests/testANRButAppInBackground_NoANR()">
</Test>
<Test
Identifier = "SentryANRTrackerTests/testMultipleListeners()">
</Test>
<Test
Identifier = "SentryCoreDataTrackerTests/testFetchRequestBackgroundThread()">
</Test>
<Test
Identifier = "SentryCoreDataTrackerTests/testSaveBackgroundThread()">
</Test>
<Test
Identifier = "SentryCrashIntegrationTests/testStartUpCrash_CallsFlush()">
</Test>
<Test
Identifier = "SentryCrashReportStore_Tests/testCrashReportCount1">
</Test>
<Test
Identifier = "SentryCrashReportStore_Tests/testDeleteAllReports">
</Test>
<Test
Identifier = "SentryCrashReportStore_Tests/testStoresLoadsMultipleReports">
</Test>
<Test
Identifier = "SentryHttpTransportTests/testFlush_WhenNoInternet_BlocksAndFinishes()">
</Test>
<Test
Identifier = "SentryNetworkTrackerIntegrationTests/testGetCaptureFailedRequestsEnabled()">
</Test>
<Test
Identifier = "SentryNetworkTrackerIntegrationTests/testGetRequest_CompareSentryTraceHeader()">
</Test>
<Test
Identifier = "SentryOnDemandReplayTests/testAddFrameIsThreadSafe()">
</Test>
<Test
Identifier = "SentryProfilerSwiftTests/testConcurrentSpansWithTimeout()">
</Test>
<Test
Identifier = "SentryReachabilityTest/testMultipleReachabilityObservers">
</Test>
<Test
Identifier = "SentrySDKIntegrationTestsBase">
</Test>
<Test
Identifier = "SentrySDKTests/testStartOnTheMainThread()">
</Test>
<Test
Identifier = "SentrySessionGeneratorTests/testSendSessions()">
</Test>
<Test
Identifier = "SentrySubClassFinderTests/testActOnSubclassesOfViewController()">
</Test>
<Test
Identifier = "SentryThreadInspectorTests/testGetCurrentThreadsWithStacktrace_WithSymbolication()">
</Test>
<Test
Identifier = "SentryThreadInspectorTests/testStacktraceHasFrames_forEveryThread()">
</Test>
<Test
Identifier = "SentryThreadWrapperTests/testOnMainThreadFromNonMainContext()">
</Test>
<Test
Identifier = "SentryTracerTests/testDeadlineTimer_StartedAndCancelledOnMainThread()">
</Test>
</SkippedTests>
</TestableReference>
<TestableReference
skipped = "NO">
Expand All @@ -138,11 +70,6 @@
BlueprintName = "SentryProfilerTests"
ReferencedContainer = "container:Sentry.xcodeproj">
</BuildableReference>
<SkippedTests>
<Test
Identifier = "SentryProfilerTests/testProfilerMutationDuringSlicing">
</Test>
</SkippedTests>
</TestableReference>
</Testables>
</TestAction>
Expand Down
Binary file removed develop-docs/disabling_tests_xcode_scheme.png
View file
Open in desktop
Binary file not shown.
Binary file added develop-docs/disabling_tests_xcode_test_plan.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed develop-docs/disabling_tests_xcode_tests_navigator.png
View file
Open in desktop
Binary file not shown.
Loading