Readable XCUITest with robots, healing lookups and tested rules: iOS-Testing-Automation-Framework 2.0

Table of Contents

• Problem
• Design decisions and alternatives
• What went wrong: a masked password that still reached the log
• Usage
• Limits

Problem

A UI test written against XCUITest's raw API tends to become a list of app.buttons["signIn.submit"].tap() calls. Such a test taps before an element can be tapped, breaks when someone renames an accessibility identifier, and fails with a message that does not say which step of the flow went wrong. Screenshots are compared by eye, accessibility is checked by hand, and the CI log is the only report.

iOS-Testing-Automation-Framework is a Swift package that goes into test targets and fills those gaps. In UI tests it adds robots (one class per screen), lookups that still work after an identifier changes, a five-rule accessibility audit, screenshot comparison with a difference image, and junit.xml plus a Markdown job summary. In any test target it adds SwiftUI snapshots, polling waits, retries, spies and seeded test data. It has no dependencies.

Version 2.0.0, released on 15 October 2026, is a rebuild. The repository was opened in August 2025 and tagged 1.0.0 in February 2026 with a feature list that lived mostly in the README. In June 2026 the sources were replaced with placeholder code: a visual regression engine that compared nothing, an accessibility auditor that only printed, and a test target that no longer compiled. The CHANGELOG says this in the first paragraph of the 2.0.0 entry. The rebuild also merged the useful parts of a sibling repository, iOSTestingTools, which is now archived and points here.

Design decisions and alternatives

Rules in plain Swift, XCUITest as a thin layer. Healing, the audit, image comparison, the reports and secret redaction work on plain values: ElementDescriptor (kind, identifier, label, value, placeholder, frame, depth) and Bitmap. The XCUI folder only turns one snapshot of the screen into descriptors and reports failures through XCTest. That is why swift test can run 118 tests on macOS without a simulator. The alternative, logic written directly in XCUIElement extensions, is less code, but it can only be checked by end-to-end UI tests, which are slow. The June placeholder code is what that path looked like when nothing tested it: results went to print.

No import Testing in the library module. Robots report failures with XCTFail, because XCUITest runs inside XCTestCase anyway. The assertion meant for unit tests, assertSnapshot(of:), throws a SnapshotError instead of recording an issue, and a thrown error fails a test in Swift Testing and in XCTest alike. The first version of the UI layer did the opposite, and the next section explains why it had to go.

Healing that refuses to guess. LocatorHealer tries, in order: the exact identifier, the one element with the locator's label, the one text field with its placeholder, and the most similar identifier (case and punctuation ignored, minimum similarity 0.8 by default). If a strategy finds two equally good candidates it returns .ambiguous, and the test fails with XCTest's usual message. A test that taps the wrong element is worse than one that fails. Every healed lookup is written to HealingLog and to the Markdown summary as a concrete fix, such as Update the locator "loginButton" to "signIn.submit". The alternatives were to always take the best candidate, which risks false passes, or not to heal at all.

Its own pixel comparator and baselines on disk. ImageComparator works on 8-bit RGBA bitmaps with a per-channel tolerance, an allowed share of changed pixels, ignored regions (a clock, a date) and a difference image that paints each changed pixel blue. The first run records the baseline and fails, so a missing baseline never passes silently. A library such as swift-snapshot-testing is more mature, but would have broken the zero-dependency goal; a perceptual tolerance is on the roadmap instead. Inputs from the repository are bounded: snapshot names become safe file names, and a PNG's size is read from its header and checked against a 50-million-pixel limit before decoding.

Reports rewritten after every test. TestResultCollector observes XCTest and rewrites junit.xml and summary.md after each test, not once at the end. When Xcode ends a runner early and starts a new one, a report written at the end can be overwritten by the second process with only its own results, and a run with failures then reads "Passed". The rebuild fixed this in commit 17394ce, which also added perProcessFileNames for parallel runs.

What went wrong: a masked password that still reached the log

Two real bugs from the rebuild show why the package is tested against real output rather than against its own abstractions.

The first one was about building at all. The first XCUITest layer (39d945c) checked at run time whether Swift Testing was active and called Issue.record or XCTFail. Added to a real UI test bundle, it failed to build with cannot load module 'Testing' as 'Testing'. Xcode compiles UI testing bundles with -module-alias Testing=_Testing_Unavailable, so any module that imports Swift Testing, even internally, cannot be used there. Commit bed5f99 removed the import and moved the unit-test assertion to throwing. The package's own unit tests could never have shown this; only a consumer target did.

The second one was a security bug. Robot.typeSecret(_:into:) registered the password with the redactor, so it was masked as •••• in the package's step names, failure messages and reports, and the documentation said so. But the password was still entered with XCUIElement.typeText, and XCTest names that call's activity after the typed text: Type 'correct-horse-batt...' into "signIn.password" SecureTextField. That title is written to the xcodebuild log and to the result bundle, where no redactor reaches. The existing test only checked that the shared redactor masked the password; nothing looked at the log itself.

Commit d7f7c1d changed how the secret is entered. On iOS and macOS, typeSecret puts it on a pasteboard item that stays on the device and presses ⌘V, which XCTest logs as a key press with a modifier. The pasteboard's presence is checked with hasStrings/types, which read no text and show no paste prompt, and the pasteboard is emptied once the field's value changes. On the iOS simulator the first ⌘V after a launch was often dropped, so the paste is repeated up to three times while the field is still unchanged after 1.5 seconds, and a field that never changes fails the step with a message that does not contain the secret.

Commit dc57725 made the guarantee checkable. The example app's workflow now writes a result bundle, and scripts/check-secrets-in-results.sh reads every test's activities and failure details with xcresulttool, together with junit.xml and summary.md, and fails the job if a test password appears in full or by its first 12 characters (XCTest shortens long typed text). Run against a result bundle recorded before the fix, the script reported the leak in three tests; against three runs with pasted secrets, it found nothing. The lesson: test a guarantee by searching the artifact it promises to protect.

The same pass found smaller leaks of the same kind. A secret with a non-ASCII letter was not masked once URL-encoded (904d159), a quoted credential value was masked only up to its first space (43ffcca), and healing suggestions reached summary.md without redaction (db54ad3).

Usage

Add the package to your UI test target, not the app target, because it links XCTest. With an app whose ContentView has a Text with the identifier counter.label and a button with counter.button, the README's quick start test reads:

1import XCTest
2import iOSTestingAutomationFramework
3 
4final class CounterRobot: Robot {
5    var label: XCUIElement { app.staticTexts["counter.label"] }
6    var button: XCUIElement { app.buttons["counter.button"] }
7}
8 
9final class CounterUITests: XCTestCase {
10    @MainActor
11    func testTappingCounts() {
12        let app = XCUIApplication()
13        app.launch()
14 
15        let counter = app.on(CounterRobot.self)
16        counter
17            .tap(counter.button)
18            .tap(counter.button)
19            .assertLabel(of: counter.label, equals: "Tapped 2 times")
20    }
21}

Each tap and the label check show up as their own step in the test report. The same test runs on CI as QuickStartTests in the AutomationShowcase example app, with launch arguments that open the quick-start screen.

A Locator keeps a test working across a redesign:

1// "loginButton" was the identifier before the redesign; the app now uses "signIn.submit".
2let button = app.element(Locator("loginButton", label: "Sign In", kind: .button), timeout: 1)
3XCTAssertTrue(button.exists)
4XCTAssertEqual(button.identifier, "signIn.submit")

To get junit.xml and summary.md, install the collector once with TestResultCollector.install(reportName:) and set TEST_RUNNER_TESTING_AUTOMATION_REPORT_DIR when you run xcodebuild test.

Limits

• The XCUITest layer is verified by 9 UI tests in one example app on CI's iPhone simulator. It has not been tried on physical devices, other iOS versions or other apps.
• iOS 17 and macOS 14 are the minimum, with Swift 6.2 tools. tvOS, watchOS and visionOS are not supported; 1.0.0 had declared iOS 15, watchOS 8 and tvOS 15.
• Image comparison uses a per-channel tolerance only. Baselines recorded on one machine can differ on another; a perceptual tolerance and text and JSON snapshot strategies are on the roadmap.
• Healing is a heuristic. A healed lookup lets the test pass and does not fail CI; someone has to read the suggestions.
• TestResultCollector is an XCTestObservation, so tests written with Swift Testing do not appear in junit.xml.
• typeSecret empties the pasteboard but does not restore its previous contents, since reading them would show a paste prompt.
• The hardening is based on the package's own tests; there has been no external review. The package is not on the Swift Package Index yet, although .spi.yml is already in the repository.

Source code: github.com/muhittincamdali/iOS-Testing-Automation-Framework