Deployment

The easiest way to deploy the Cargoship models on your own server is using Docker and Traefik. Traefik is a reverse proxy that can be used to route traffic to your models. It can also be used to automatically generate SSL certificates using Let's Encrypt. You can also use your own domain name and configure Traefik to use it.

Install Docker

We are starting with a clean Ubuntu 22.04 server. First, we need to install Docker. You can find the official installation instructions here: https://docs.docker.com/engine/install/ubuntu/

First make sure you have no other Docker installation on your system and remove it:

$ sudo apt-get remove docker docker-engine docker.io containerd runc

Then install Docker. First update your package index:

sudo apt-get update

Then install the required packages:

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Add Docker's official GPG key:

sudo mkdir -m 0755 -p /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Use then use the following command to set up the repository:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update your package index again:

sudo apt-get update

Finally, install Docker:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Test your Docker installation:

sudo docker run hello-world

If everything went well, you should see an output downloading the hello-world image and then running it.

To make interacting with docker easier, you also can add your current user to the docker group (that way you can avoid using sudo with docker commands). First create the group:

sudo groupadd docker

Then add your user to the group:

sudo usermod -aG docker $USER

Run this command to activate the changes to groups:

newgrp docker

Install Traefik

We use Traefik as an easy to use webserver and to make all the models available to the outside under different domains. Traefik also takes care of automatically generating SSL certificates using Let's Encrypt.

First choose in which directory you want to install traefik and all the models we are hosting. We will use the root users home directory (/root) in this example.

Switch to home directory:

cd ~

Create a directory for traefik and switch to it:

mkdir traefik && cd traefik

Create a docker-compose.yml file:

touch docker-compose.yml

Open the file with your favorite editor (e.g. nano docker-compose.yml) and add the following content:

version: "3.5"

services:
  traefik:
    image: "traefik:v2.7"
    restart: always
    container_name: "traefik"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /root/traefik/traefik.yaml:/traefik.yaml
      - /root/traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - web
networks:
  web:
    external: true

If you use a different directory, make sure to change the paths in the volumes section.

Create a traefik.yaml file:

touch traefik.yaml

Open the file with your favorite editor (e.g. nano traefik.yaml) and add the following content:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: default
providers:
  docker:
    watch: true
    exposedByDefault: false
    network: web
certificatesResolvers:
  default:
    acme:
      email: YOUR-EMAIL-ADDRESS # TODO: replace with your email address
      storage: acme.json
      caServer: "https://acme-v01.api.letsencrypt.org/directory"
      tlsChallenge: {}

Make sure to replace YOUR-EMAIL-ADDRESS with your email address. This is required by Let's Encrypt to send you notifications about your certificates.

To store your SSL certificates locally, create a file called acme.json:

touch acme.json

And set the correct permissions:

chmod 600 acme.json

Now you can start traefik:

docker-compose up -d

You can check the status of traefik with:

docker-compose ps

You can also check the logs with:

docker-compose logs -f

Now that traefik is running, you can start other docker containers on your system and make them available to the outside world using traefik. Please make sure, that the ports 443 and 80 are accessible to the outside world and not blocked by your firewall or the firewall of your hoster.

Run your first model

Now that traefik is running, we can start our first model. We will use the Cargoship model cargoshipsh/language-detection as an example.

First, move back to your base directory (in our case the roots home directory /root or ~):

cd ~

Create a directory for the model and switch to it:

mkdir language-detection && cd language-detection

Create a docker-compose.yml file:

touch docker-compose.yml

Open the file with your favorite editor (e.g. nano docker-compose.yml) and add the following content:

version: "3.3"
services:
  language-detection-api:
    restart: unless-stopped
    image: cargoshipsh/language-detection:latest
    environment:
      - API_KEY=CHANGE_ME # TODO: Change to your API key
    networks:
      - web
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.language-detection.rule=Host(`api.example.com`)" # TODO: Change with your own domain
      - "traefik.http.routers.language-detection.tls.certresolver=default"
      - "traefik.http.routers.language-detection.entrypoints=websecure"
      - "traefik.http.services.language-detection.loadbalancer.server.port=80"
networks:
  web:
    external: true

Make sure to replace CHANGE_ME with your API key you want to use to access the API and api.example.com with your own domain. (Note: if you want to deploy more than one model, make sure the router names, e.g. traefik.http.routers.language-detection, and the domain names are unique.)

Make sure your domain is pointing to the IP address of your server. You can check your IP address with:

curl ifconfig.me

If you want to host different models, you can also use subdomains, e.g. language-detection.api.example.com and only set a wildcard DNS record for *.api.example.com once.

Now you can start the model:

docker-compose up -d

You can check the status of the containers with:

docker-compose ps

You can also check the logs with:

docker-compose logs -f

Your model should now be available under your domain. You can test it with:

curl -X POST \
  https://api.example.com \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: CHANGE_ME' \
  -d '{
    "text": "This is a test"
}'

Make sure to replace the API key and also address with the domain you chose. You should get a response like this:

{
  "language": "en"
}

If you get an error, make sure that your domain is pointing to the IP address of your server and that the ports 80 and 443 are accessible to the outside world. You can also check the logs of the traefik container with docker logs -f traefik.

Have a lot of fun with your new machine learning API! 🎉🎉🎉