Documentation testing

GitLab documentation is stored in projects with code and treated like code. Therefore, we use processes similar to those used for code to maintain standards and quality of documentation.

We have tests:

To lint the words and structure of the documentation.
To check the validity of internal links in the documentation suite.
To check the validity of links from UI elements, such as files in app/views files.

For the specifics of each test run in our CI/CD pipelines, see the configuration for those tests in the relevant projects:

We also run some documentation tests in the:

GitLab Development Kit project: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/.gitlab/ci/test.gitlab-ci.yml.
Gitaly project: https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab-ci.yml.

Run tests locally

Similar to previewing your changes locally, you can also run these tests on your local computer. This has the advantage of:

Speeding up the feedback loop. You can know of any problems with the changes in your branch without waiting for a CI/CD pipeline to run.
Lowering costs. Running tests locally is cheaper than running tests on the cloud infrastructure GitLab uses.

To run tests locally, it's important to:

Install the tools, and keep them up to date.
Run linters, documentation link tests, and UI link tests the same way they are run in CI/CD pipelines. It's important to use same configuration we use in CI/CD pipelines, which can be different than the default configuration of the tool.

Lint checks

Merge requests containing changes to Markdown (.md) files run a docs-lint markdown job. This job runs markdownlint, and a set of tests from /scripts/lint-doc.sh that look for page content problems that Vale and markdownlint cannot test for. The job fails if any of these tests fail:

Curl (curl) commands must use long-form options (--header) instead of short options, like -h.
Documentation pages must contain front matter indicating ownership of the page.
Non-standard Unicode space characters (NBSP, NNBSP, ZWSP) must not be used in documentation, because they can cause irregularities in search indexing and grepping.
CHANGELOG.md must not contain duplicate versions.
No files in the doc/ directory may be executable.
Use index.md instead of README.md.
Directories and file names must use underscores instead of dashes.
Directories and file names must be in lower case.

Run lint checks locally

Lint checks are performed by the lint-doc.sh script and can be executed with the help of a Rake task as follows:

Go to your gitlab directory.
Run:
```
rake lint:markdown
```

To specify a single file or directory you would like to run lint checks for, run:

MD_DOC_PATH=path/to/my_doc.md rake lint:markdown

The output should be similar to:

=> Linting documents at path /path/to/gitlab as <user>...
=> Checking for cURL short options...
=> Checking for CHANGELOG.md duplicate entries...
=> Checking /path/to/gitlab/doc for executable permissions...
=> Checking for new README.md files...
=> Linting markdown style...
=> Linting prose...
✔ 0 errors, 0 warnings and 0 suggestions in 1 file.
✔ Linting passed

This requires you to either:

Have the required lint tools installed on your computer.
A working Docker or containerd installation, to use an image with these tools pre-installed.

Documentation link tests

Merge requests containing changes to Markdown (.md) files run a docs-lint links job, which runs two types of link checks. In both cases, links with destinations that begin with http or https are considered external links, and skipped:

bundle exec nanoc check internal_links: Tests links to internal pages.
bundle exec nanoc check internal_anchors: Tests links to topic title anchors on internal pages.

Failures from these tests are displayed at the end of the test results in the Issues found! area. For example, failures in the internal_anchors test follow this format:

[ ERROR ] internal_anchors - Broken anchor detected!
  - source file `/tmp/gitlab-docs/public/ee/user/application_security/api_fuzzing/index.html`
  - destination `/tmp/gitlab-docs/public/ee/development/code_review.html`
  - link `../../../development/code_review.html#review-response-slo`
  - anchor `#review-response-slo`

Source file: The full path to the file containing the error. To find the file in the gitlab repository, replace /tmp/gitlab-docs/public/ee with doc, and .html with .md.
Destination: The full path to the file not found by the test. To find the file in the gitlab repository, replace /tmp/gitlab-docs/public/ee with doc, and .html with .md.
Link: The actual link the script attempted to find.
Anchor: If present, the topic title anchor the script attempted to find.

Check for multiple instances of the same broken link on each page reporting an error. Even if a specific broken link appears multiple times on a page, the test reports it only once.

Run document link tests locally

To execute documentation link tests locally:

Go to the gitlab-docs directory.

Run the following commands:

# Check for broken internal links
bundle exec nanoc check internal_links

# Check for broken external links (might take a lot of time to complete).
# This test is set to be allowed to fail and is run only in the gitlab-docs project CI
bundle exec nanoc check internal_anchors

UI link tests

The ui-docs-links lint job uses haml-lint to test that all documentation links from UI elements (app/views files, for example) are linking to valid pages and anchors.

To run the ui-docs-links test locally:

Open the gitlab directory in a terminal window.

Run:

bundle exec haml-lint -i DocumentationLinks

If you receive an error the first time you run this test, run bundle install, which installs the dependencies for GitLab, and try again.

If you don't want to install all of the dependencies to test the links, you can:

Open the gitlab directory in a terminal window.
Install haml-lint:
```
gem install haml_lint
```
Run:
```
haml-lint -i DocumentationLinks
```

If you manually install haml-lint with this process, it does not update automatically and you should make sure your version matches the version used by GitLab.

Update linter configuration

Vale configuration and markdownlint configuration is under source control in each project, so updates must be committed to each project individually.

We consider the configuration in the gitlab project as the source of truth and that's where all updates should first be made.

On a regular basis, the changes made in gitlab project to the Vale and markdownlint configuration should be synchronized to the other projects. In omnibus-gitlab, gitlab-runner, and charts/gitlab:

Create a new branch.
Copy the configuration files from the gitlab project into this branch, overwriting the project's old configuration. Make sure no project-specific changes from the gitlab project are included. For example, RelativeLinks.yml is hard coded for specific projects.
Create a merge request and submit it to a technical writer for review and merge.

Update linting images

Lint tests run in CI/CD pipelines using images from the gitlab-docs container registry.

If a new version of a dependency is released (like a new version of Ruby), we should update the images to use the newer version. Then, we can update the configuration files in each of our documentation projects to point to the new image.

To update the linting images:

In gitlab-docs, open a merge request to update .gitlab-ci.yml to use the new tooling version. (Example MR)
When merged, start a Build docs.gitlab.com every hour scheduled pipeline.
Go the pipeline you started, and manually run the relevant build-images job, for example, image:docs-lint-markdown.
In the job output, get the name of the new image. (Example job output)
Verify that the new image was added to the container registry.
Open merge requests to update each of these configuration files to point to the new image. In each merge request, include a small doc update to trigger the job that uses the image.
In each merge request, check the relevant job output to confirm the updated image was used for the test. (Example job output)
Assign the merge requests to any technical writer to review and merge.

Local linters

To help adhere to the documentation style guidelines, and improve the content added to documentation, install documentation linters and integrate them with your code editor.

At GitLab, we mostly use:

markdownlint
Vale

markdownlint

markdownlint checks that Markdown syntax follows certain rules, and is used by the docs-lint test.

Our Documentation Style Guide and Markdown Guide elaborate on which choices must be made when selecting Markdown syntax for GitLab documentation. This tool helps catch deviations from those guidelines.

markdownlint configuration is found in the following projects:

This configuration is also used in build pipelines.

You can use markdownlint:

On the command line, with either:
- markdownlint-cli.
- markdownlint-cli2.
In a code editor.
In a pre-push hook.

Markdown rule `MD044/proper-names` (capitalization)

A rule that can cause confusion is MD044/proper-names. The failure, or how to correct it, might not be immediately clear. This rule checks a list of known words, listed in the .markdownlint.yml file in each project, to verify proper use of capitalization and backticks. Words in backticks are ignored by markdownlint.

In general, product names should follow the exact capitalization of the official names of the products, protocols, and so on.

Some examples fail if incorrect capitalization is used:

MinIO (needs capital IO)
NGINX (needs all capitals)
runit (needs lowercase r)

Additionally, commands, parameters, values, filenames, and so on must be included in backticks. For example:

"Change the needs keyword in your .gitlab-ci.yml..."
- needs is a parameter, and .gitlab-ci.yml is a file, so both need backticks. Additionally, .gitlab-ci.yml without backticks fails markdownlint because it does not have capital G or L.
"Run git clone to clone a Git repository..."
- git clone is a command, so it must be lowercase, while Git is the product, so it must have a capital G.

Vale

Vale is a grammar, style, and word usage linter for the English language. Vale's configuration is stored in the .vale.ini file located in the root directory of projects.

Vale supports creating custom tests that extend any of several types of checks, which we store in the .linting/vale/styles/gitlab directory in the documentation directory of projects.

You can find Vale configuration in the following projects:

This configuration is also used in build pipelines, where error-level rules are enforced.

You can use Vale:

On the command line.
In a code editor.
In a Git hook. Vale only reports errors in the Git hook (the same configuration as the CI/CD pipelines), and does not report suggestions or warnings.

Vale result types

Vale returns three types of results:

Error - For branding guidelines, trademark guidelines, and anything that causes content on the documentation site to render incorrectly.
Warning - For general style guide rules, tenets, and best practices.
Suggestion - For technical writing style preferences that may require refactoring of documentation or updates to an exceptions list.

The result types have these attributes:

Result type	Displays in CI/CD job output	Displays in MR diff	Causes CI/CD jobs to fail	Vale rule link
`error`	{check-circle} Yes	{check-circle} Yes	{check-circle} Yes	Error-level Vale rules
`warning`	{dotted-circle} No	{check-circle} Yes	{dotted-circle} No	Warning-level Vale rules
`suggestion`	{dotted-circle} No	{dotted-circle} No	{dotted-circle} No	Suggestion-level Vale rules

Vale spelling test

When Vale flags a valid word as a spelling mistake, you can fix it following these guidelines:

Flagged word	Guideline
jargon	Rewrite the sentence to avoid it.
correctly-capitalized name of a product or service	Add the word to the vale spelling exceptions list.
name of a person	Remove the name if it's not needed, or add the vale exception code in-line.
a command, variable, code, or similar	Put it in backticks or a code block. For example: `The git clone command can be used with the CI_COMMIT_BRANCH variable.` -> The `git clone` command can be used with the `CI_COMMIT_BRANCH` variable.
UI text from GitLab	Verify it correctly matches the UI, then: If it does not match the UI, update it. If it matches the UI, but the UI seems incorrect, create an issue to see if the UI needs to be fixed. If it matches the UI and seems correct, add it to the vale spelling exceptions list.
UI text from a third-party product	Rewrite the sentence to avoid it, or add the vale exception code in-line.

Vale uppercase (acronym) test

The Uppercase.yml test checks for incorrect usage of words in all capitals. For example, avoid usage like This is NOT important.

If the word must be in all capitals, follow these guidelines:

Flagged word	Guideline
Acronym (likely known by the average visitor to that page)	Add the acronym to the list of words and acronyms in `Uppercase.yml`.
Acronym (likely not known by the average visitor to that page)	The first time the acronym is used, write it out fully followed by the acronym in parentheses. In later uses, use just the acronym by itself. For example: `This feature uses the File Transfer Protocol (FTP). FTP is...`.
Correctly capitalized name of a product or service	Add the name to the list of words and acronyms in `Uppercase.yml`.
Command, variable, code, or similar	Put it in backticks or a code block. For example: Use `FALSE` as the variable value.
UI text from a third-party product	Rewrite the sentence to avoid it, or add the vale exception code in-line.

Vale readability score

In ReadingLevel.yml, we have implemented the Flesch-Kincaid grade level test to determine the readability of our documentation.

As a general guideline, the lower the score, the more readable the documentation. For example, a page that scores 12 before a set of changes, and 9 after, indicates an iterative improvement to readability. The score is not an exact science, but is meant to help indicate the general complexity level of the page.

The readability score is calculated based on the number of words per sentence, and the number of syllables per word. For more information, see the Vale documentation.

When to add a new Vale rule

It's tempting to add a Vale rule for every style guide rule. However, we should be mindful of the effort to create and enforce a Vale rule, and the noise it creates.

In general, follow these guidelines:

If you add an error-level Vale rule, you must fix the existing occurrences of the issue in the documentation before you can add the rule.

If there are too many issues to fix in a single merge request, add the rule at a warning level. Then, fix the existing issues in follow-up merge requests. When the issues are fixed, promote the rule to an error.
If you add a warning-level or suggestion-level rule, consider:
- How many more warnings or suggestions it creates in the Vale output. If the number of additional warnings is significant, the rule might be too broad.
- How often an author might ignore it because it's acceptable in the context. If the rule is too subjective, it cannot be adequately enforced and creates unnecessary additional warnings.
- Whether it's appropriate to display in the merge request diff in the GitLab UI. If the rule is difficult to implement directly in the merge request (for example, it requires page refactoring), set it to suggestion-level so it displays in local editors only.

Install linters

At a minimum, install markdownlint and Vale to match the checks run in build pipelines.

These tools can be integrated with your code editor.

Install markdownlint

You can install either markdownlint-cli or markdownlint-cli2 to run markdownlint.

To install markdownlint-cli, run:

yarn global add markdownlint-cli

To install markdownlint-cli2, run:

yarn global add markdownlint-cli2

You should install the version of markdownlint-cli or markdownlint-cli2 used (see variables: section) when building the image:docs-lint-markdown.

Install Vale

Install vale using either:

The asdf-vale plugin if using asdf. In a checkout of a GitLab project with a .tool-versions file (example), run:
```
asdf plugin add vale && asdf install vale
```
A package manager:
- macOS using brew, run: brew install vale.
- Linux, use your distribution's package manager or a released binary.

Update linters

It's preferable to use linter versions that are the same as those used in our CI/CD pipelines for maximum compatibility with the linting rules we use.

To match the versions of markdownlint-cli (or markdownlint-cli2) and vale used in the GitLab projects, refer to:

For projects managed with asdf, the .tool-versions file in the project. For example, the .tool-versions file in the gitlab project.
The versions used (see variables: section) when building the image:docs-lint-markdown Docker image containing these tools for CI/CD.

Versions set in these two locations should be the same.

Tool	Version	Command	Additional information
`markdownlint-cli`	Latest	`yarn global add markdownlint-cli`	None.
`markdownlint-cli2`	Latest	`yarn global add markdownlint-cli2`	None.
`markdownlint-cli`	Specific	`yarn global add markdownlint-cli@0.35.0`	The `@` indicates a specific version, and this example updates the tool to version `0.35.0`.
`markdownlint-cli2`	Specific	`yarn global add markdownlint-cli2@0.8.1`	The `@` indicates a specific version, and this example updates the tool to version `0.8.1`.
Vale (using `asdf`)	Specific	`asdf install`	Installs the version of Vale set in `.tool-versions` file in a project.
Vale (other)	Specific	Not applicable.	Binaries can be directly downloaded.
Vale (using `brew`)	Latest	`brew update && brew upgrade vale`	This command is for macOS only.

Configure editors

Using linters in your editor is more convenient than having to run the commands from the command line.

To configure markdownlint in your editor, install one of the following as appropriate:

Sublime Text SublimeLinter-contrib-markdownlint package. This package uses markdownlint-cli by default, but can be configured to use markdownlint-cli2 with this SublimeLinter configuration:
```
"markdownlint": {
  "executable": [ "markdownlint-cli2" ]
}
```
Visual Studio Code DavidAnson.vscode-markdownlint extension.
Vim ALE plugin.

To configure Vale in your editor, install one of the following as appropriate:

Sublime Text SublimeLinter-vale package.
Visual Studio Code ChrisChinchilla.vale-vscode extension. You can configure the plugin to display only a subset of alerts.
Vim ALE plugin.
JetBrains IDEs - No plugin exists, but this issue comment contains tips for configuring an external tool.

Emacs Flycheck extension. This requires some configuration:

Flycheck supports markdownlint-cli out of the box, but you must point it to the .markdownlint.yml at the base of the project directory. A .dir-locals.el file can accomplish this:

;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project.
((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml"))))

A minimal configuration for Flycheck to work with Vale could look like this:

(flycheck-define-checker vale
  "A checker for prose"
  :command ("vale" "--output" "line" "--no-wrap"
            source)
  :standard-input nil
  :error-patterns
    ((error line-start (file-name) ":" line ":" column ":" (id (one-or-more (not (any ":")))) ":" (message)   line-end))
  :modes (markdown-mode org-mode text-mode)
  :next-checkers ((t . markdown-markdownlint-cli))
)

(add-to-list 'flycheck-checkers 'vale)

In this setup the markdownlint checker is set as a "next" checker from the defined vale checker. Enabling this custom Vale checker provides error linting from both Vale and markdownlint.

Configure pre-push hooks

Git pre-push hooks allow Git users to:

Run tests or other processes before pushing a branch.
Avoid pushing a branch if failures occur with these tests.

lefthook is a Git hooks manager, making configuring, installing, and removing Git hooks simpler.

Configuration for lefthook is available in the lefthook.yml file for the gitlab project.

To set up lefthook for documentation linting, see Pre-push static analysis.

Show Vale warnings on push

By default, lefthook shows only Vale errors when pushing changes to a branch. The default branches have no Vale errors, so any errors listed here are introduced by commits to the branch.

To also see the Vale warnings when pushing to a branch, set a local environment variable: VALE_WARNINGS=true.

Enable Vale warnings on push to improve the documentation suite by:

Detecting warnings you might be introducing with your commits.
Identifying warnings that already exist in the page, which you can resolve to reduce technical debt.

These warnings:

Don't stop the push from working.
Don't result in a broken pipeline.
Include all warnings for a file, not just warnings that are introduced by the commits.

To enable Vale warnings on push:

Automatically, add VALE_WARNINGS=true to your shell configuration.
Manually, prepend VALE_WARNINGS=true to invocations of lefthook. For example:
```
VALE_WARNINGS=true bundle exec lefthook run pre-push
```

You can also configure your editor to show Vale warnings.

Show subset of Vale alerts

You can set Visual Studio Code to display only a subset of Vale alerts when viewing files:

Go to Preferences > Settings > Extensions > Vale.
In Vale CLI: Min Alert Level, select the minimum alert level you want displayed in files.

To display only a subset of Vale alerts when running Vale from the command line, use the --minAlertLevel flag, which accepts error, warning, or suggestion. Combine it with --config to point to the configuration file in the project, if needed:

vale --config .vale.ini --minAlertLevel error doc/**/*.md

Omit the flag to display all alerts, including suggestion level alerts.

Disable Vale tests

You can disable a specific Vale linting rule or all Vale linting rules for any portion of a document:

To disable a specific rule, add a  tag before the text, and a  tag after the text, replacing rulename with the filename of a test in the GitLab styles directory.
To disable all Vale linting rules, add a  tag before the text, and a  tag after the text.

Whenever possible, exclude only the problematic rule and lines.

For more information, see Vale's documentation.

Disable markdownlint tests

To disable all markdownlint rules, add a  tag before the text, and a  tag after the text.

To disable only a specific rule, add the rule number to the tag, for example  and .

Whenever possible, exclude only the problematic lines.