TL;DR

I automated the procedure of obtaining and installing HTTPS certificates for all my domains and subdomains using the APIs provided by Namesilo (my registrar), GitLab (my host), and Certbot (certificate issuing package).

Long version

Not being on https is seriously bad for your website. Leaving out the actual security issues of not using https, the policy changes in Google Chrome also make a non-https website look bad to an end user. I shifted this blog to https more than an year ago (May 2016) and since then I have been using certificates generated by Letsencrypt for this blog.

If you do not know about Letsencrypt, I suggest you go ahead and read about it here. In short, it provides free trusted SSL certificates for people to enable SSL on their websites. A certificate is valid for 3 months, after which the owner must renew the certificate by providing proof of domain ownership again.

Proving domain ownership

To prove that you have control over a domain name, you need to solve one of multiple challenges:

1. Posting a specified file in a specified location on a web site (the HTTP-01 challenge)
2. Offering a specified temporary certificate on a web site (the TLS-SNI-01 challenge)
3. Posting a specified DNS record in the domain name system (the DNS-01 challenge)

Traditionally, I have always used either (1) or (3) manually to obtain my certificates. However, as the number of domains/sub-domains for which I need to generate these certificates increases, doing this manually for each one starts to suck. A couple of days ago, I decided to automate the whole process so that I don't have to think twice before using a sub-domain (no thinking - oh no, not another one) or getting a new domain.

Choosing the correct challenge

HTTP-01

HTTP-01 challenge would have been the easiest to automate. However, it restricts the locations for which I can generate the certificate. Specifically, since the HTTP-01 challenge requires showing a particular file at a specified URL on that domain, it needs to be a server that's hosting a website. But what if it isn't? For instance alexa.pallav.xyz only hosts an instance of the GeeMusic server. Since there are no HTTP pages served, it can't participate in HTTP-01 challenge.

Also, I tend to change the way my website is generated and/or hosted. Any change would require corresponding changes to the automation script which is unacceptable.

TLS-SNI-01

This one has the same problem as the former challenge. Every server has a different way of serving certificates. The script would need to be tuned for each server. Any domain/subdomain currently not in use would not be included for certificate generation.

DNS-01

DNS-01 was the clear winner. DNS changes don't require any interaction with the domain host, but rather with the registrar or the DNS provider. Since the probability of me changing registrars is quite low, the automation script would be able to generate certificates of all domains registered with Namesilo.

This automatically also takes care of any subdomains I create of the existing domain names, since they are also under the same registrar.

Automation!

So here's the steps:

1. Call certbot for all domains. Indicate preference for DNS-01 challenge.
2. For each domain, use the Namesilo API to update the DNS record as indicated by certbot.
3. Stall the script for 15 minutes to let the DNS record update be published.
4. Let certbot run the verification.
5. Use the GitLab API to install the newly generated certificate into the pages hosted by GitLab.

Step 1: Calling certbot

If you have used certbot before, you'd know that by default it runs in an interactive mode. Clearly, if we want to run it inside a script, interactive mode would be undesirable. Similarly, since the certificates need to be installed on GitLab and not on local server, we need the --manual and certonly modes. After some experimentation, I was able to come up with a launch command that runs the certbot non-interactively, and without any complications.

certbot     --config-dir ${DIR}/gen/conf      \
            --work-dir ${DIR}/gen/work        \
            --logs-dir ${DIR}/gen/logs        \
            --agree-tos                       \
            -m pallavagarwal07@gmail.com      \
            -d pallav.xyz                     \
            -d www.pallav.xyz                 \
            -d alexa.pallav.xyz               \
            -d varstack.com                   \
            -d www.varstack.com               \
            --manual                          \
            --manual-public-ip-logging-ok     \
            --preferred-challenges dns        \
            --noninteractive                  \
            --manual-auth-hook ${DIR}/hook.sh \
            certonly

The first 3 lines just indicate the preference of where I'd like the certificate and other artifcats to be generated. Most of the other flags are rather self explanatory. Flags like --agree-tos and --manual-public-ip-logging-ok just prevent the script from asking the user for permissions. The use of most flags can be understood by running

certbot --help

The flag of interest is --manual-auth-hook. This flag specifies the executable to be called for the completion of the challenge for each domain name (in this case, the challenge is DNS-01).

The challenge executable in my case can be found here. The executable gets in its environment, two important variables:

CERTBOT_DOMAIN:
domain for which challenge is running (eg. alexa.pallav.xyz)

CERTBOT_VALIDATION:
the value corresponding to the _acme-challenge TXT entry

TXT is a type of DNS entry. Thus, if I have the environmental variables set as:

CERTBOT_DOMAIN=abc.pallav.xyz
CERTBOT_VALIDATION=somerandomstring

I need to add a TXT entry for _acme-challenge.abc.pallav.xyz with the value somerandomstring.

Step 2: Calling Namesilo API

Namesilo thankfully has a very complete RESTful API. It took me a short while to write a Go client that can interact with Namesilo via the API and add the required DNS entries. Here is the relevant code (along with helper files in that directory).

The executable called by Certbot is a wrapper around this client, and calls this with the required arguments.

Step 3: Stall 15 minutes

This is a bit hacky. The wrapper script checks if it is being called for the last domain in the list. If so, it goes into a sleep for 15 minutes. This is required because Namesilo publishes DNS changes every 15 (or 5 maybe?) minutes. After 15 minutes, the script returns control to the certbot process.

Step 4: Let certbot run the verification.

Almost done!

Step 5: Installation using GitLab API

At this point we have generated our certificates. The only thing needed to be done is the installation of those certificates into existing websites, in this case, the ones hosted by GitLab. Now, GitLab too has a very comprehensive set of RESTful APIs. I specifically use the Pages Domains API.

This was rather straightforward, except for one bug on which I was stuck was quite some time before finally posting it on stackoverflow. Like always, I figured out a workaround before anybody could post a solution, but the question is still a good record of what the problem was, so I'll just link to it here in case anybody has a similar issue: GitLab Pages API 404 error for certain projects

The installation to GitLab code can be viewed here.