Repository checks (FREE SELF)
You can use git fsck
to verify the integrity of all data
committed to a repository. GitLab administrators can:
- Manually trigger this check for a project.
- Schedule this check to run automatically for all projects.
- Run this check from the command line.
- Run a Rake task for checking Git repositories, which can be used to run
git fsck
against all repositories and generate repository checksums, as a way to compare repositories on different servers.
Check a project's repository using GitLab UI
To check a project's repository using GitLab UI:
- On the left sidebar, expand the top-most chevron ({chevron-down}).
- Select Admin Area.
- On the left sidebar, select Overview > Projects.
- Select the project to check.
- In the Repository check section, select Trigger repository check.
The checks run asynchronously so it may take a few minutes before the check result is visible on the project page in the Admin Area. If the checks fail, see what to do.
Enable repository checks for all projects
Instead of checking repositories manually, GitLab can be configured to run the checks periodically:
- On the left sidebar, expand the top-most chevron ({chevron-down}).
- Select Admin Area.
- On the left sidebar, select Settings > Repository (
/admin/application_settings/repository
). - Expand the Repository maintenance section.
- Enable Enable repository checks.
When enabled, GitLab periodically runs a repository check on all project repositories and wiki repositories to detect possible data corruption. A project is checked no more than once per month. Administrators can configure the frequency of repository checks. To edit the frequency:
- For Linux package installations, edit
gitlab_rails['repository_check_worker_cron']
in/etc/gitlab/gitlab.rb
. - For source-based installations, edit
[gitlab.cron_jobs.repository_check_worker]
in/home/git/gitlab/config/gitlab.yml
.
If any projects fail their repository checks, all GitLab administrators receive an email notification of the situation. By default, this notification is sent out once a week at midnight at the start of Sunday.
Repositories with known check failures can be found at
/admin/projects?last_repository_check_failed=1
.
Run a check using the command line
You can run git fsck
using the command line on repositories on
Gitaly servers. To locate the repositories:
-
Go to the storage location for repositories:
- For Linux package installations, repositories are stored in the
/var/opt/gitlab/git-data/repositories
directory by default. - For GitLab Helm chart installations, repositories are stored in the
/home/git/repositories
directory inside the Gitaly pod by default.
- For Linux package installations, repositories are stored in the
-
Identify the subdirectory that contains the repository that you need to check.
-
Run the check. For example:
sudo -u git /opt/gitlab/embedded/bin/git \ -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck --no-dangling
The error
fatal: detected dubious ownership in repository
means you're running the command using the wrong account. For example,root
.
What to do if a check failed
If a repository check fails, locate the error in the repocheck.log
file on disk at:
-
/var/log/gitlab/gitlab-rails
for Linux package installations. -
/home/git/gitlab/log
for self-compiled installations. -
/var/log/gitlab
in the Sidekiq pod for GitLab Helm chart installations.
If periodic repository checks cause false alarms, you can clear all repository check states:
- On the left sidebar, expand the top-most chevron ({chevron-down}).
- Select Admin Area.
- On the left sidebar, select Settings > Repository (
/admin/application_settings/repository
). - Expand the Repository maintenance section.
- Select Clear all repository checks.
Troubleshooting
When working with repository checks, you might encounter the following issues.
failed to parse commit <commit SHA> from object database for commit-graph
Error: You can see a failed to parse commit <commit SHA> from object database for commit-graph
error in repository check logs. This error occurs if your commit-graph
cache is out
of date. The commit-graph
cache is an auxiliary cache and is not required for regular Git operations.
While the message can be safely ignored, see the issue error: Could not read from object database for commit-graph for more details.
Dangling commit, tag, or blob messages
Repository check output often includes tags, blobs, and commits that must be pruned:
dangling tag 5c6886c774b713a43158aae35c4effdb03a3ceca
dangling blob 3e268c23fcd736db92e89b31d9f267dd4a50ac4b
dangling commit 919ff61d8d78c2e3ea9a32701dff70ecbefdd1d7
This is common in Git repositories. They're generated by operations like force pushing to branches, because this generates a commit in the repository that is not longer referred to by a ref or by another commit.
If a repository check fails, the output is likely to include these warnings.
Ignore these messages, and identify the root cause of the repository check failure from the other output.
GitLab 15.8 and later no
longer includes these messages in the check output. Use the --no-dangling
option
to suppress then when run from the command line.