Blogging with Hugo and GitLab (5): Let's Encrypt Auto Renewal

Manual renewal doesn’t work. We need auto renewal.

Fortunately, there is the experience sharing of how to automatically renew certificates with GitLab ( link to the original article). The basic idea is to apply the same operation of the manual certificate generation described in the previous article, but automatically through the GitLab pipeline schedule.

GitLab repository change

Create and check in the following three Shell scripts to your GitLab repository. Put them under the root directory (the same level as .gitlab-ci.yml) and add the corresponding job to .gitlab-ci.yml for the GitLab pipeline schedule. A glance of files updated:

modified:   .gitlab-ci.yml
new file:   letsencrypt_authenticator.sh
new file:   letsencrypt_cleanup.sh
new file:   letsencrypt_generate.sh

I revised the original scripts a bit by replacing the domain specific content with variables (which will be defined later when setting the pipeline schedule). Simply copy the exact content should just work.

  • letsencrypt_generate.sh

    #!/bin/sh
    
    end_epoch=$(date -d "$(echo | openssl s_client -connect $ROOT_DOMAIN:443 -servername $ROOT_DOMAIN 2>/dev/null | openssl x509 -enddate -noout | cut -d'=' -f2)" "+%s")
    current_epoch=$(date "+%s")
    days_left=$((($end_epoch - $current_epoch) / 60 / 60 / 24))
    
    echo "=============================="
    echo "Renewal threshold: $RENEWAL_THRESHOLD_IN_DAYS_LEFT days"
    echo "Root domain: $ROOT_DOMAIN"
    echo "=============================="
    
    if [ $days_left -le $RENEWAL_THRESHOLD_IN_DAYS_LEFT ]; then
    echo "=============================="
    echo "Certificate is $days_left days old, renewing now."
    echo "=============================="
    certbot certonly \
    --agree-tos \
    --debug \
    --manual \
    --manual-auth-hook letsencrypt_authenticator.sh \
    --manual-cleanup-hook letsencrypt_cleanup.sh \
    --manual-public-ip-logging-ok \
    --preferred-challenges http \
    -d $ROOT_DOMAIN \
    -d www.$ROOT_DOMAIN \
    -m $GITLAB_USER_EMAIL
    echo "=============================="
    echo "Certbot finished. Updating GitLab Pages domains."
    echo "=============================="
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/$ROOT_DOMAIN
    curl --request PUT --header "PRIVATE-TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN" --form "certificate=@/etc/letsencrypt/live/$ROOT_DOMAIN/fullchain.pem" --form "key=@/etc/letsencrypt/live/$ROOT_DOMAIN/privkey.pem" https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pages/domains/www.$ROOT_DOMAIN
    else
    echo "=============================="
    echo "Certificate still valid for $days_left days, no renewal required."
    echo "=============================="
    fi
    
  • letsencrypt_authenticator.sh

    #!/bin/sh
    
    mkdir -p $CI_PROJECT_DIR/static/.well-known/acme-challenge
    echo $CERTBOT_VALIDATION > $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
    git add $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
    git commit -m "GitLab runner - Added certbot challenge file for certificate renewal"
    git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master
    
    interval_sec=15
    max_tries=80 # 20 minutes
    n_tries=0
    while [ $n_tries -le $max_tries ]
    do
    status_code=$(curl -L --write-out "%{http_code}\n" --silent --output /dev/null https://$ROOT_DOMAIN/.well-known/acme-challenge/$CERTBOT_TOKEN)
    if [[ $status_code -eq 200 ]]; then
    exit 0
    fi
    
    n_tries=$((n_tries+1))
    sleep $interval_sec
    done
    
    exit 1
    
  • letsencrypt_cleanup.sh

    #!/bin/sh
    
    git rm $CI_PROJECT_DIR/static/.well-known/acme-challenge/$CERTBOT_TOKEN
    git commit -m "GitLab runner - Removed certbot challenge file"
    git push https://$GITLAB_USER_LOGIN:$CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN@gitlab.com/$CI_PROJECT_PATH.git HEAD:master
    

Add a new job to .gitlab-ci.yml. Again, no domain specific content. Simply copy the exact content should just work.

letsencrypt-renewal:
  image: scottchayaa/alpine-certbot:3.7
  only:
    - schedules
  variables:
    CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: $CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN
    RENEWAL_THRESHOLD_IN_DAYS_LEFT: $RENEWAL_THRESHOLD_IN_DAYS_LEFT
    ROOT_DOMAIN: $ROOT_DOMAIN
  script:
    - export PATH=$PATH:$CI_PROJECT_DIR
    - git config --global user.name $GITLAB_USER_LOGIN
    - git config --global user.email $GITLAB_USER_EMAIL
    - chmod +x ./letsencrypt*.sh
    - ./letsencrypt_generate.sh

GitLab personal access token

As mentioned earlier, the above scripts just apply the same operation of the manual certificate generation. Recall that we need to add the certificate to the GitLab repository. A personal access token allows the pipeline schedule executing the script to update the GitLab repository, which enables the automatical certificate creation.

To create a personal access token, click “Settings” of the avatar in the upper-right corner.

Then create a new personal access token.
“User Settings” > “Access Tokens” > “Personal Access Tokens”

  • Name: RENEWAL_THRESHOLD_IN_DAYS_LEFT
  • Expires at: (leave it empty)
  • Scopes: select “api”

Click “Create personal access token”
The page will show the generated token which will be used next. Copy it.

GitLab pipeline schedule

With all the above steps ready, let’s create a new pipeline schedule to run the job of certificate auto renewal.

“CI / CD” > “Schedules” > “New schedule”

  • “Description”: Let’s Encrypt Auto Renewal
  • “Interval Pattern”: select “Every week (Sundays at 4:00am)”
  • “Cron Timezone”: (your preferred timezone)
  • “Target Branch”: master
  • “Variables”:
    1. CERTBOT_RENEWAL_PIPELINE_GIT_TOKEN: (paste the token generated in the previous step)
    2. RENEWAL_THRESHOLD_IN_DAYS_LEFT: 20
    3. ROOT_DOMAIN: loadbalancing.xyz

Click “Save pipeline schedule”

The three variables define are used by the scripts for the specified domain. A certificate from Let’s Encrypt is valid for 90 days and notification starts being sent when the certificate will expire in 20 days. Therefore, I set the renewal threshold at 20 days to avoid email noises (i.e., the certificate will be automatically renewed when there are less than 21 days left before the original certificate expires).

Contents