A Much Better Way To Backup GitLab to AWS S3

Balaji D Loganathan

54 sec read

Like many of you, we also use the self-hosted version of GitLab CE and it is one of the BEST open source software written in Ruby on Rails. Though GitLab got its own rake tasks to take backup to AWS S3, it wasn’t working for us and several other users. Hence we developed our own solution and sharing it here so that it can be helpful to others in need.

Once the GitLab repositories and uploads sizes become 6GB+, the probability of GitLab’s backup rake task working is 50/50. Most of the time it will fail while it tries to prepare backup or while uploading to AWS S3. To avoid this issue, we simply split the back up into two segments.

First, we upload the repositories and then we upload the uploads directory. It is done using a simple shell script located at https://gist.github.com/balajidl/44c9389835547fa40e88e881ac1d40ee

What does the shell script do?

  1. Initialize the log file
  2. Backup the repositories using the GitLab provided rake tasks gitlab:backup:create by ignoring the uploads folder gitlab:backup:create SKIP=uploads
  3. Backup the uploads by compressing the GitLab uploads folder
  4. Upload the two .gz files to AWS S3
  5. Periodically remove old backup files to avoid excess billing 🙂
  6. Send an email using SendGrid when the back up is successful.
Backup email alert sent via sendgrid

Voila ^_^!

Amazing! Isn’t it? Please feel free to comment if you have questions or need help. Happy to help 🙂

Related posts:

Leave a Reply

Your email address will not be published. Required fields are marked *