Blogging with Hugo and GitLab (5): Let's Encrypt Auto Renewal

Manual renewal actually doesn’t work. Fortunately, there is the experience shared about how to automatically renew certificates under GitLab (link of the original article). The basic idea is to apply the same operation of the manual certificate generation described in the previous article, but automatically through the GitLab pipeline schedule. Here I recompiled the scripts from the original article to not include any domain specific content, which allows them to be directly applied to any repositories (with domain specific variables defined in the pipeline setting).

Scripts for Let’s Encrypt

Download the following three Shell scripts (or the zip package containing them altogether). Place the scripts under the root directory (the same level as .gitlab-ci.yml) of your GitLab repository.

Below are the details of the scripts.

  • letsencrypt_generate.sh
#!/bin/sh

end_epoch=$(date -d "$(echo | openssl s_client -connect $ROOT_DOMAIN:443 -servername $ROOT_DOMAIN 2>/dev/null | openssl x509 -enddate -noout | cut -d'=' -f2)" "+%s")
current_epoch=$(date "+%s")
days_left=$((($end_epoch - $current_epoch) / 60 / 60 / 24))

echo "=============================="
echo "Renewal threshold: $RENEWAL_THRESHOLD_IN_DAYS_LEFT days"
echo "Root domain: $ROOT_DOMAIN"
echo "=============================="

if [ $days_left -le $RENEWAL_THRESHOLD_IN_DAYS_LEFT ]; then
    echo "=============================="
    echo "Certificate is $days_left days old, renewing now."
    echo "=============================="
    certbot certonly \
    --agree-tos \
    --debug \
    --manual \
    --manual-auth-hook letsencrypt_authenticator.sh \
    --manual-cleanup-hook letsencrypt_cleanup.sh \
    --manual-public-ip-logging-ok \
    --preferred-challenges http \
    -d $ROOT_DOMAIN \
    -d www.$ROOT_DOMAIN \
    -m $GITLAB_USER_EMAIL
    echo "=============================="
    echo "Certbot finished. Updating GitLab Pages domains."
    echo "=============================="
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/$ROOT_DOMAIN
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/www.$ROOT_DOMAIN
else
    echo "=============================="
    echo "Certificate still valid for $days_left days, no renewal required."
    echo "=============================="
fi

  • letsencrypt_authenticator.sh
#!/bin/sh

mkdir -p $CI_PROJECT_DIR/static/.well-known/acme-challenge
echo $CERTBOT_VALIDATION > $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git add $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Added certbot challenge file for certificate renewal"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master

interval_sec=15
max_tries=80 # 20 minutes
n_tries=0
while [ $n_tries -le $max_tries ]
do
  status_code=$(curl -L --write-out "%{http_code}\n" --silent --output /dev/null https://$ROOT_DOMAIN/.well-known/acme-challenge/$CERTBOT_TOKEN)
  if [[ $status_code -eq 200 ]]; then
    exit 0
  fi

  n_tries=$((n_tries+1))
  sleep $interval_sec
done

exit 1

  • letsencrypt_cleanup.sh
#!/bin/sh

git rm $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
git commit -m "GitLab runner - Removed certbot challenge file"
git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master

GitLab pipeline configuration

With the above three scripts, add a new job to .gitlab-ci.yml for the GitLab pipeline schedule. Similarly, there is no domain specific content. Simply copy the exact content should just work.

letsencrypt-renewal:
  image: scottchayaa/alpine-certbot:3.7
  only:
    - schedules
  variables:
    CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN
    RENEWAL_THRESHOLD_IN_DAYS_LEFT: $RENEWAL_THRESHOLD_IN_DAYS_LEFT
    ROOT_DOMAIN: $ROOT_DOMAIN
  script:
    - export PATH=$PATH:$CI_PROJECT_DIR
    - git config --global user.name $GITLAB_USER_LOGIN
    - git config --global user.email $GITLAB_USER_EMAIL
    - chmod +x ./letsencrypt*.sh
    - ./letsencrypt_generate.sh

GitLab personal access token

As mentioned earlier, the above scripts just apply the same operation of the manual certificate generation. Recall that we need to add the certificate to the GitLab repository. A personal access token allows the pipeline schedule executing the script to update the GitLab repository, which enables the automatical certificate generation.

To create a personal access token, click “Settings” of the avatar in the upper-right corner.

Then create a new personal access token.
“User Settings” > “Access Tokens” > “Personal Access Tokens”

  • Name: RENEWAL_THRESHOLD_IN_DAYS_LEFT
  • Expires at: (leave it empty)
  • Scopes: select “api”

Click “Create personal access token”
The page will show the generated token which will be used next. Copy it.

GitLab pipeline schedule

With all the above steps ready, let’s create a new pipeline schedule to run the job of certificate auto renewal.

“CI / CD” > “Schedules” > “New schedule”

  • “Description”: Let’s Encrypt Auto Renewal
  • “Interval Pattern”: select “Every week (Sundays at 4:00am)”
  • “Cron Timezone”: (your preferred timezone)
  • “Target Branch”: master
  • “Variables”:
    1. CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: (paste the token generated in the previous step)
    2. RENEWAL_THRESHOLD_IN_DAYS_LEFT: 20
    3. ROOT_DOMAIN: loadbalancing.xyz

Click “Save pipeline schedule”

The three variables defined are used by the scripts for the specified domain. A certificate from Let’s Encrypt is valid for 90 days and the notification starts being sent when the certificate is going to expire in 20 days. Therefore, I set the renewal threshold at 20 days to avoid email noises (i.e., the certificate will be automatically renewed when there are less than 21 days left before the original certificate expires).

Contents