13 KiB
Fleet UI testing
This document contains the testing strategy and plan for the frontend codebase. The testing strategy is a high-level overview of the who, what, when and why when it comes to testing. The testing plan primarily outlines the how of testing and covers the different practices and toolings used in testing.
For instructions on using our testings tools, check out our testing docs.
Table of contents
Testing strategy
Testing philosophy
When we create tests, we keep in mind how an end user will be using this software. This idea influences all other decisions when it comes to testing. Sometimes the end user is a company admin using the Fleet UI, or a DevOps engineer using fleetctl in the terminal, or a developer using a reusable UI component or utility or class. In any case, we should first think about the end user when building our testing plan. Testing software from this perspective has many advantages, including:
- A focus on functionality and behavior over implementation details. This leads to better maintainability of the testing suite which does not have to change as the implementation changes.
- A clear idea of what type of tests are useful and should be prioritized.
- A higher level of confidence that the software behaves as intended in real-world scenarios.
Who tests
The developer is responsible for writing and maintaining tests as part of their work. They can get help from QA when trying to decide what tests are sufficient for the feature.
What we test
We break down our application into different types of systems that we can test in ways that best fit their use cases.
Shared utilities
Shared utilities are generally simple JS functions used throughout the app. This includes code in the
utilities directory such as utilites/url and utilities/string.
UI building blocks (reusable components and hooks)
UI building blocks is reusable code that's primarily driven by props or arguments. This
includes components in the components directory like Radio, FlashMessage, and Button or
reusable hooks like useDeepEffect.
App widgets
App widgets are larger chunks of the application that are more specific (therefore not
reused) and are made up of the UI building blocks. They tend to have more context and dependencies
within them, which means they're tested differently than the building blocks. Examples include forms
like LoginForm and ResetPasswordForm or even simpler pages like Registration Page or LoginPage
User journeys
User journeys are the flows users take through the application to accomplish their goals. These are typically the widest and can include navigating through multiple pages or working with multiple app widgets on a page. This would include goals like creating a new user or team or filtering hosts by software vulnerabilities or policy results.
Testing plan
This section answers how we are testing our code by covering our testing tools and best practices. We like to test the software at different layers (unit, integration, E2E), all of which have their own usefulness and testing practices.
NOTE: Architecture plays a huge role in testing practices and tools. In our current landscape (as of 08/31/2022), we do not have the best separation of concerns between our systems. As a result, the most reliable way to test our software is as a whole with E2E tests. While this is ok at some level, we are still missing a large chunk of important testing that would be better written and maintained at the integration/unit level. We'd like to utilize separation of concerns more between our systems as this will allow us to test more in the integration and unit layers, which are quicker to run and generally easier to work with.
Types of tests
We use a variety of testing to ensure that our software is working as intended. This includes:
- End-to-end (E2E)
- Integration
- Unit
- Static
- Manual
Testing cheat sheet
| Systems | Type of Test | Who is the User | Example | Notes |
|---|---|---|---|---|
| Reusable utilities | Unit with Jest | Devs | String util, url util | Little to no dependencies. Function argument-based. |
| Reusable hooks | Unit with Jest & react-hooks Library | Devs | useToggleDisplayed Hook | Little to no dependencies. Function argument-based. |
| Reusable UI components | Unit with Jest & testing-library | Devs | Radio, button, and input components | Little to no dependencies. Props-based. |
| App widgets | Integration or E2E with testing-library or QA Wolf | End users | Create user form. Reset password form. | Less reusable code with more complex environment setup and dependencies. Depending on the case, can be done with integration or E2E. For integration, mock at the backend level; don't mock other UI systems. For E2E, don't mock other systems except for common network error states. |
| User journeys | E2E with QA Wolf | End users | Filtering a host by software. Creating a team as admin. | Full business flows. Little to no mocking of systems, except for common network error states. |
| N/A | Manual | N/A | // TODO | Manual testing can be used for all types of code. Examples would be for one-offs or states that would require extremely difficult testing setups. |
Manual testing
There will always be a space for manual testing. We use it for testing states that do not occur very often in the application or are not worth the effort to test for unlikely edge cases.
Static analysis
This includes typing and linting to quickly ensure proper typings of data flowing through the application, and that we are following coding conventions and styling rules. This gives us a first line of defense against writing buggy code.
Unit testing
We unit test smaller reusable components that have little to no dependencies within them. These components are primarily parametric-based and require no or minimal mocking to be tested effectively. They tend to be small building blocks of our application (e.g., reusable UI components, common utilities, reusable hooks). With unit testing, the end users tend to be other developers, so we ensure these components work as expected when used as building blocks.
Shared utilities testing
We can test utility functions purely with Jest and don't have to worry about rendering components with react-testing-library. Only Jest is needed with minimal mocking.
View the full url utility testing source.
import { buildQueryStringFromParams } from ".";
describe("url utilities", () => {
it("creates a query string from a params object", () => {
const params = {
query: "test",
page: 1,
order: "asc",
isNew: true,
};
expect(buildQueryStringFromParams(params)).toBe(
"query=test&page=1&order=asc&isNew=true"
);
});
it("filters out undefined values", () => {
const params = {
query: undefined,
page: 1,
order: "asc",
};
expect(buildQueryStringFromParams(params)).toBe("page=1&order=asc");
});
});
UI building block testing
We test the component to ensure it works how a developer would want to use it. There is minimal mocking and we utilize react-testing-library and Jest spies to test.
View the full Radio component testing source.
import React from "react";
import { noop } from "lodash";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import Radio from "./Radio";
describe("Radio - component", () => {
it("renders the radio label text from the label prop", () => {
render(
<Radio
checked
label={"Radio Label"}
value={"radioValue"}
id={"test-radio"}
onChange={noop}
/>
);
const labelText = screen.getByText("Radio Label");
expect(labelText).toBeInTheDocument();
});
it("passes the radio input value when checked", async () => {
const user = userEvent.setup();
const changeHandlerSpy = jest.fn();
render(
<Radio
label={"Radio Label"}
value={"radioValue"}
id={"test-radio"}
onChange={changeHandlerSpy}
/>
);
const radio = screen.getByRole("radio", { name: "Radio Label" });
await user.click(radio);
expect(changeHandlerSpy).toHaveBeenCalled();
expect(changeHandlerSpy).toHaveBeenCalledWith("radioValue");
});
});
Reusable hooks testing
// TODO
Integration testing
We use integration testing to strike a balance between speed and expense to write our tests. We use them to test multiple reusable components that come together into less reusable app widgets. We also try to use minimal mocking, but we can mock at the backend level if required.
App widget testing
This layer of tests is great for testing difficult to obtain states and edge cases as the test setup is simpler than E2E tests. We highly utilize react-testing-library to interface with these components.
View the full ResetPasswordForm app widget testing source.
import React from "react";
import { render, screen } from "@testing-library/react";
import { renderWithSetup } from "test/test-utils";
import ResetPasswordForm from "./ResetPasswordForm";
describe("ResetPasswordForm - component", () => {
const newPassword = "password123!";
const submitSpy = jest.fn();
it("does not submit the form if the password is invalid", async () => {
const invalidPassword = "invalid";
const { user } = renderWithSetup(
<ResetPasswordForm handleSubmit={submitSpy} />
);
await user.type(screen.getByLabelText("New password"), invalidPassword);
await user.type(screen.getByLabelText("Confirm password"), invalidPassword);
await user.click(screen.getByRole("button", { name: "Reset password" }));
const passwordError = screen.getByText(
"Password must meet the criteria below"
);
expect(passwordError).toBeInTheDocument();
expect(submitSpy).not.toHaveBeenCalled();
});
});
E2E testing
Our E2E layer tests all the systems of the software (frontend and backend) together to ensure the application works as intended. We have partnered with QA Wolf to cover these flows, and the E2E tests are written and maintained by them.
Tooling
Here is a quick reference of the current tooling we are using at each layer of testing.
ESLint and TypeScript
We use these for our static analysis testing. These tools have been set up so errors should appear in your editor if they are broken.
Jest
We use Jest as our frontend test runner, assertion library, and spy and mock utilities for unit and integration testing.
Testing library
We rely heavily on the different libraries that are part of the testing-library ecosystem for our unit and integration testing. These including react-testing-library, react-hooks, and user-events. The guiding principles of the testing-library tools align with our own in that we believe tests should resemble real-world usage as closely as possible.
Additional examples
Roles and permissions
// TODO
Mac and Windows hosts
// TODO
Error states
// TODO