Automate XCTest with GitLab’s CI

GitLab has a Continuous Integration (CI) feature that allows you to perform continuous integration without having to deploy a separate solution dedicated to CI.

So when I pushed the code to GitLab, I decided to run XCTest to run unit tests for macOS and iOS apps.

This article show you how I did it and how I set it up.

TOC

Register the runners

In GitLab CI/CD, a Runner runs the process, and there are two types of Runners: the Shared Runner, which is shared between projects, and the Specific Runner, which is dedicated to a particular project.

This time, I registered “Shared Runner” because I want to use it from multiple projects (repositories).

Install a Runner

Since this is the Runner used to run XCTest, it should be installed on the Mac that will run XCTest. Here, I used a Mac Mini.

Instructions for installing GitLab Runner on macOS can be found on the official website.

STEP
Download a Runner for macOS.

The binary to download is different for the Intel Mac and the Apple Silicon. Download in the terminal with curl.

For Intel Mac:

sudo curl --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64"

For Apple Silicon:

sudo curl --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-arm64"

The Runner will be downloaded to /usr/local/bin/gitlab-runner.

STEP
Set the permission to add the right to execute.
sudo chmod +x /usr/local/bin/gitlab-runner
STEP
Install the GitLab Runner as a service in the home directory.
cd ~
gitlab-runner install
STEP
Configure the locale of the gitlab-runner.

Open the ~/Library/LaunchAgents/gitlab-runner.plist file. This file will be created by gitlab-runner install.

STEP
Add the EnvironmentVariables as following.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>

<!-- 省略 -->

	<key>EnvironmentVariables</key>
	<dict>
		<key>LC_ALL</key>
		<string>en_US.UTF-8</string>
	</dict>
</dict>
STEP
Change log files path.

The normal user doesn’t have the permission to read and write to the default log file paths, so the gitlab-runner fail to run. Make the log files can be readable and writable, change the value of the StandardOutPath and StandardErrorPath to change the log files path.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>

<!-- 省略 -->

	<key>StandardOutPath</key>
	<string>/Users/USER_NAME/.gitlab-runner/gitlab-runner.out.log</string>
  <key>StandardErrorPath</key>
	<string>/Users/USER_NAME/.gitlab-runner/gitlab-runner.err.log</string>
STEP
Start the service.
gitlab-runner start

Register a Runner

Register the Runner installed on your Mac to the GitLab.

STEP
Open the “Runners” in the “Admin Area”.
STEP
Click the “Register an instance runner”, and click the “Show runner installation and registration instructions”.
Click the "Show runner installation and registration instructions"
Click the “Show runner installation and registration instructions”
STEP
The detailed instructions are displayed, select macOS from Environment, click the Copy button next to the field labeled Command to register runner, copy the command, and click Close to close.
Copy the command.
Copy the command
STEP
In a terminal on the Mac where Runner is installed, execute the command copied in STEP 3 without sudo.

When running XCTest on a Mac, it must be run in user mode, so it must be run without sudo.

Once executed, you will be asked to enter settings interactively, and enter them in order.

STEP
Enter the GitLab URL.
Runtime platform                                    arch=amd64 os=darwin pid=10297 revision=76984217 version=15.1.0
Running in system-mode.                            
                                                   
Enter the GitLab instance URL (for example, https://gitlab.com/):
[http://code.rkdev.corp/]: 
STEP
You will be asked for your registration token. Since you have already specified a token when executing the command, simply press the Enter key.
Enter the registration token:
[-ABCDEFG4567890YZABC]: 
STEP
Enter the description of the Runner. I entered the machine name.
Enter a description for the runner:
[MacMini2020.local]: MacMini2020
STEP
Enter the tags to be assigned to the Runner.

In my case:

  • The machine is a Mac.
  • It is an Intel Mac.
  • Its OS will be always latest.

Based on these factors, the following three tags were set.

  • mac
  • mac-intel
  • mac-intel-latest
Enter tags for the runner (comma-separated):
mac,mac-intel,mac-intel-latest
STEP
Enter the memo for the maintenance. Nothing in particular, so I left it blank.
Enter optional maintenance note for the runner:
STEP
Enter the Executer.

The official web site says to use shell for macOS and iOS apps, so I specified shell.

Registering runner... succeeded                     runner=-ABCDEFG
Enter an executor: custom, parallels, shell, ssh, kubernetes, docker, docker-ssh, virtualbox, docker+machine, docker-ssh+machine:
shell
STEP
If the registration is successful, the output is as follows.
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
STEP
Reload the Runners page in the Admin Area to see the registered Runners.
Registered Runners are displayed
Registered Runners are displayed

Configure the CI/CD

Create a repository and code for testing and set up the CI/CD.

Test Code

Here, I just want to make sure that XCTest works through GitLab CI, so I made the following unit test with test code that always fails and test code that always succeeds. I hesitate to call it a test code since we are not testing anything, just making XCTest work.

import XCTest
@testable import CITest

class CITestTests: XCTestCase {

    override func setUpWithError() throws {
    }

    override func tearDownWithError() throws {
    }

    func testExample() throws {
        XCTFail("TEST FAILURE")
    }
    
    func testSuccess() throws {
        XCTAssertTrue(true);
    }
}

Activate the Shared Runner

Shared Runner must be enabled on a per-repository basic.

STEP
Open the CI/CD in the project settings.
STEP
Open the Expand of Runners.
STEP
Turn on the “Enable shared runners for this project”.
Activate the shared runner
Activate the shared runner

Add the CI/CD configuration file

Add the CI/CD configuration file by doing the following.

STEP
Open the repository page with the web browser and click the “Set up CI/CD” link.
Click the "Setup CI/CD"
Click the “Set up CI/CD”
STEP
Click the “Create new CI/CD pipeline” button.
Create the CI/CD pipeline
Create the CI/CD pipeline
STEP
Click the “Browse templates’ link, open the “Swift.gitlab-ci.yml” and copy its contents.
STEP
Back to the Pipeline Editor of your repository and paste the configuration which was copied.
STEP
Change the Xcode project file name, a scheme name and so on.

For example, in my test project file, I changed the following values.

Configuration Value
-project CITest.xcodeproj
-scheme CITest
-destination iOS Simulator,name=iPhone 13,OS=15.5
tags mac-intel-latest
-archivePath build/CITest
-archivePath build/CITest.xcarchive
-exportPath build/CITest.ipa
-exportProvisioningProfile “”
paths: build/CITest.ipa
Changed configurations
stages:
  - build
  - test
  - archive
  - deploy

build_project:
  stage: build
  script:
    - xcodebuild clean -project CITest.xcodeproj -scheme CITest | xcpretty
    - xcodebuild test -project CITest.xcodeproj -scheme CITest -destination 'platform=iOS Simulator,name=iPhone 13,OS=15.5' | xcpretty -s
  tags:
    - mac-intel-latest

archive_project:
  stage: archive
  script:
    - xcodebuild clean archive -archivePath build/CITest -scheme CITest
    - xcodebuild -exportArchive -exportFormat ipa -archivePath "build/CITest.xcarchive" -exportPath "build/CITest.ipa" -exportProvisioningProfile ""
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - build/CITest.ipa
  tags:
    - mac-intel-latest

If you only need testing by CI and do not need archiving, remove the archive_project job and make the following configuration file. In my case, this is what I needed.

stages:
  - build
  - test
  - archive
  - deploy

build_project:
  stage: build
  script:
    - xcodebuild clean -project CITest.xcodeproj -scheme CITest | xcpretty
    - xcodebuild test -project CITest.xcodeproj -scheme CITest -destination 'platform=iOS Simulator,name=iPhone 13,OS=15.5' | xcpretty -s
  tags:
    - mac-intel-latest

See the following article for details on specifying the -destination option.

STEP
Click the “Commit changes”.

Test

Push the created test code to the repository. Since this is a must-fail test, the job will fail after a while and you will receive an error notification email.

Open the Pipelines of the CI/CD of the project. Failed jobs will be displayed, click on FAILED.

Displayed the failed jobs
Displayed the failed jobs

Open the Failed Jobs tab. If the error is due to test failure as planned, the test case (method) name of the failed test is output as follows.

2022-06-27 18:59:30.903 xcodebuild[18902:109049] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2022-06-27 18:59:30.903 xcodebuild[18902:109049] [MT] IDETestOperationsObserverDebug: 104.154 sec, +104.154 sec -- end
Failing tests:
	CITestTests:
		CITestTests.testExample()

** TEST FAILED **

ERROR: Job failed: exit status 1

If you are getting other errors, please review the logs and review your settings.

About the UI testing

Here we have only unit testing with XCTest, but it is no different for UI testing.

If the UI Testing test package is included in the Xcode project and registered in the scheme’s test plan, UI testing will be performed automatically, just as it is when testing manually from Xcode.

Trouble shooting

I will post the errors I encountered when I set it up and how to handle them.

Only works when the user is logged in

XCTest must be run by the current user who is logged in. Therefore, the Mac running the Runner must be logged in.

When the machine is rebooted

If you follow the official installation procedure, it will be registered as a service, so when you reboot your machine, all you have to do is log in and Runner will start.

Status remains pending

The status is still pending and the pipeline build_project shows an error that the Runner corresponding to the tag does not exist.

In this case, the following are possible causes.

  1. The tag is incorrect.
  2. Trying to use Shared Runner, but Shared Runner is not enabled in your project.
  3. Runner on Mac is not running in user mode.

2 see “Activate Shared Runner” on this page.

I’m stuck on 3. When registering Runner, the command line output by GitLab has sudo on it, and if I run it as is, Runner won’t work in user mode, so XCTest won’t work either.

Simulator name or OS version is incorrect.

In this case, the following error is output in the build_project log, which lists the available simulator and OS combinations.

xcodebuild: error: Unable to find a destination matching the provided destination specifier:

Correct the values you specify in the .gitlab-ci.yml file to the appropriate values.

Open the Editor of your project’s CI/CD to edit and commit the .gitlab-ci.yml file from within your web browser.

For information on how to specify the simulator name and OS version, see the following article.

Let's share this post !

Author of this article

Akira Hayashiのアバター Akira Hayashi Representative, Software Engineer

I am an application developer loves programming. This blog is a tech blog, its articles are learning notes. In my work, I mainly focus on desktop and mobile application development, but I also write technical books and teach seminars. The websites of my work and books are here -> RK Kaihatsu.

TOC