Running Tests

Currently, we primarily test Zarf through a series of end-to-end tests. These tests are called in the test-*.yml workflows and undergo automatic execution against several K8s distros whenever a pull request is created or updated.

In addition, Zarf implements unit tests for specific functions where edge cases prove difficult to cover through end-to-end testing alone. Unit tests follow standard Go convention and are *_test.go files.

Dependencies # TODO: update any more dependencies

To run the end-to-end tests locally, you must meet the same prerequisites as those required for building and running Zarf, which include:

1. GoLang version >= the version specified in the go.mod
2. Make
3. Docker
4. Docker Buildx
5. Any clean K8s cluster (local or remote) or Linux with root if you want to use the Zarf-installed K3s cluster

Unit Tests # TODO: DONE these have been tested on mac and linux so this section is GTG

There are several ways to run tests depending on your specific situation, such as:

# The default way, from the root directory of the repo. Will run all of the unit tests that are currently defined.
make test-unit

# To get a single-line readable summary of all the unit tests currently defined:
make test-unit 2>&1 | grep -E "^(ok|FAIL)\s"

# If you already have everything built, you can run this inside this folder. This lets you customize the test run.
go test ./src/pkg/... -v

# To run a single test:
go test ./src/pkg/... -v -run TestFooBarBaz

All unit tests should pass locally before submitting a PR.

Adding New Unit Tests # TODO: make new unit test and test instructions

To create a unit test, search for or create a file that ends with _test.go in the package of the file that requires testing, such as auth.go -> auth_test.go. Import the testing library and create test functions as necessary. Generic unit test helper functions can be found in src/test/testutil.

There are two types of unit tests within Zarf:

1. Standard unit tests on isolated functions or files.
2. Tests in the src/cmd package, which test an entire command. These have similar properties to e2e tests, but faster iteration loops.

Unit tests should always take less than one second. Unit tests should run using t.Parallel() unless a specific constraint blocks this.

CLI End-to-End Tests

There are several ways to run tests depending on your specific situation, such as:

# Note: You can prepend CI=true to these commands to force the --no-progress flag like CI does

# The default way, from the root directory of the repo. Will run all of the tests against your chosen k8s distro. Will automatically build any binary dependencies that don't already exist.
make test-e2e ARCH="[amd64|arm64]"

# To get a single-line readable summary of all the e2e tests currently defined:
cd src/test/e2e && go test ./main_test.go ./[01]* -failfast -v -timeout 35m 2>&1 | grep -E "^(--- FAIL|--- PASS|FAIL|ok)"

# To test against a Zarf-created cluster (Linux only with sudo, not compatible with macOS)
APPLIANCE_MODE=true make test-e2e ARCH="amd64"

# If you already have everything built, you can run this inside this folder. This lets you customize the test run.
go test ./src/test/... -v -failfast -count=1

# Let's say you only want to run one test. You would run:
go test ./src/test/... -v -failfast -run TestFooBarBaz -count=1

Note

The -count=1 flag is the idiomatic way to disable test caching explicitly.

Note

The Zarf binary and built packages are required to be stored in the ./build directory. However, if you intend to run tests locally using ‘go test ./…’, the zarf-init package must also be present in this directory.

The Upgrade Test

Caution

This test is meant to be run on a fresh cluster.

Additionally, the documentation for this test is still in progress. Please refer to the code for more information.

Depends on:

• A locally built Zarf binary.
• A fresh cluster.

# The default way, from the root directory of the repo. This will automatically build any Zarf related resources if they don't already exist (i.e. binary, init-package, example packages):
make test-upgrade

# or

# If you are in the root folder of the repository and already have everything built (i.e., the binary, the init-package and the flux-test example package):
go test ./src/test/upgrade/...

The External Test

Caution

This test is meant to be run on a fresh cluster.

Additionally, the documentation for this test is still in progress. Please refer to the code for more information.

Depends on:

• A locally built Zarf binary.
• A fresh cluster.

Note: For this test case, we deploy an ‘external’ Git server and container registry as pods running within the k8s cluster. These are still considered ‘external’ servers since they already existed inside the k8s cluster before zarf init command is executed

# The default way, from the root directory of the repo. This will automatically build any Zarf related resources if they don't already exist (i.e. binary, init-package, example packages):
make test-external

# or

# If you are in the root folder of the repository and already have everything built (i.e., the binary, the init-package and the flux-test example package):
go test ./src/test/external/... -v

Adding New CLI End-to-End Tests

When adding new tests, there are several requirements that must be followed, including:

1. Tests cannot be run in parallel as they utilize the same K8s cluster.
2. Each test should begin with the entries below for standardization and test setup/teardown:

func TestFooBarBaz(t *testing.T) {
    t.Log("E2E: Enter useful description here")

    ...
}

CLI End-to-End Test Naming Conventions

The end-to-end tests are run sequentially and the naming convention is set intentionally:

• 00-19 tests run prior to zarf init (cluster not initialized).
• 20 is reserved for zarf init.
• 22 is reserved for tests required the git-server, which is removed at the end of the test.
• 23-98 are for the remaining tests that only require a basic Zarf cluster without the git-server.
• 99 is reserved for the zarf destroy and YOLO Mode test.

Troubleshooting

Potential e2e testing errors and causes: Docker/Buildx:

unknown flag: --load → Buildx not installed or broken symlink

Cluster:

dial tcp [::1]:8080: connect: connection refused → No cluster configured or kubeconfig not set error loading config file: duplicate name in list → Duplicate entries in ~/.kube/config

Git/1Password:

error: 1Password: Could not connect to socket → 1Password closed during rebase commit signing

Architecture:

no compatible component named k3s found → Either wrong ARCH value or trying to run Linux-only components on macOS