Certificate Authentication

This guide requires an Anka Enterprise (or higher) license.

There are several different ways you can enable Certificate authentication:

1. With the combined (controller + registry) native macOS package: You’ll edit the /usr/local/bin/anka-controllerd, enable TLS/HTTPS (required), and then enable certificate authentication.
2. With the docker package: You’ll edit the docker-compose.yml, enable TLS/HTTPS, and then enable certificate authentication.
3. With either the controller or registry standalone packages: You’ll edit the proper config files, enable TLS/HTTPS, and then enable certification authentication.

Requirements

1. A Root CA certificate. For more information about CAs, see https://en.wikipedia.org/wiki/Certificate_authority. Usually provided by your organization or where you obtain your certificate signing. We will generate a self-signed one in this guide and refer to this as anka-ca-crt.pem and anka-ca-key.pem.
2. Certificate[s] (signed with the Root CA) for the Anka Build Cloud Controller & Registry.
3. Certificate[s] (signed with the Root CA) for your Anka Build Nodes so they can connect/authenticate with the Anka Build Cloud Controller & Registry.

If bringing your own certs, not password protected (“encrypted”) (use openssl rsa -in <encrypted_private.key> -out <decrypted_private.key> to decrypt).

1. Create a self-signed Root CA certificate

If you don’t have a Root CA yet, you can create it with openssl:

cd ~
openssl req -new -nodes -x509 -days 365 -keyout anka-ca-key.pem -out anka-ca-crt.pem \
  -subj "/O=MyGroup/OU=MyOrgUnit/CN=MyUser"

You can add the Root CA to the System keychain so the Root CA is trusted and you can avoid warnings when you go to access the Controller UI.

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain anka-ca-crt.pem

2. Configuring TLS for Controller & Registry

• TLS/HTTPS is required for Certificate Authentication.
• Your HTTPS/TLS cert can be from any root CA and does not need to be from the same CA as your authentication certs.

Create a self-signed cert for the services (optional)

Certificates should be in PEM (PKCS #8) format.

Ensure your certs are decrypted! They cannot have passwords.

For this guide, we’re running the Controller & Registry locally, so we use 127.0.0.1. Update this depending on where you have things hosted

If you do not have TLS certificates for your Controller & Registry from a signed source, you can create them using your own CA:

export CONTROLLER_ADDRESS="127.0.0.1"
export REGISTRY_ADDRESS=$CONTROLLER_ADDRESS
openssl genrsa -out anka-controller-key.pem 4096
openssl req -new -nodes -sha256 -key anka-controller-key.pem -out anka-controller-csr.pem -subj "/O=MyGroup/OU=MyOrgUnit/CN=MyUser" \
  -reqexts SAN -extensions SAN \
  -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nextendedKeyUsage = serverAuth\nsubjectAltName=IP:$CONTROLLER_ADDRESS"))
openssl x509 -req -days 365 -sha256 -in anka-controller-csr.pem -CA anka-ca-crt.pem -CAkey anka-ca-key.pem -CAcreateserial \
  -out anka-controller-crt.pem -extfile <(echo subjectAltName = IP:$CONTROLLER_ADDRESS)

You can use the same certificate for both the Controller and Registry.

Beginning in Controller version 1.12.0, you can control the allowed TLS Cipher Suites and minimum/maximum TLS versions.

Next, ensure that the certificate has Signature Algorithm: sha256WithRSAEncryption using openssl x509 -text -noout -in ~/anka-controller-crt.pem | grep Signature (https://support.apple.com/en-us/HT210176)

Configure the services to use the TLS cert

MacOS combined Controller & Registry package

Edit /usr/local/bin/anka-controllerd:

1. Change the listen address to 443: export ANKA_LISTEN_ADDR=":443"

SSL will actually work on any port you want.

2. Add the following ENVs to enable HTTPS:

   # SSL + Cert Auth
   export ANKA_USE_HTTPS="true"
   export ANKA_SERVER_CERT="/Users/MyUser/anka-controller-crt.pem"
   export ANKA_SERVER_KEY="/Users/MyUser/anka-controller-key.pem"
   

3. Ensure https is in the registry URL:

   export ANKA_ANKA_REGISTRY="https://anka.registry:8089"
   

The Controller & Registry runs as root. This is why you need to specify the absolute path to the location where you generated or are storing your certs.

Linux/Docker Controller & Registry package

Within the docker-compose.yml:

1. Change the anka-controller ports from 80:80 to 443:80. You can keep the anka-registry ports the same (default: 8089).
2. Under the anka-controller, modify or set ANKA_ANKA_REGISTRY to use https://.
3. Ensure there is a volumes item that points the local cert location inside of the container at /mnt/cert.

Now let’s configure the Controller & Registry containers/services to use those certificates:

version: '2'
services:
  anka-controller:
    container_name: anka-controller
    build:
      context: controller
    ports:
      - "443:80"
    depends_on:
      - etcd
      - anka-registry
    restart: always
    volumes:
      - "/opt/secure/certs:/mnt/certs"
    environment:
      ANKA_ANKA_REGISTRY: "https://anka.registry:8089"
      ANKA_USE_HTTPS: "true"
      ANKA_SKIP_TLS_VERIFICATION: "true" # Only needed if registry cert is self-signed
      ANKA_SERVER_CERT: "/mnt/certs/anka-controller-crt.pem"
      ANKA_SERVER_KEY: "/mnt/certs/anka-controller-key.pem"
  anka-registry:
    container_name: anka-registry
    build:
      context: registry
    ports:
      - "8089:8089"
    restart: always
    volumes:
      - "/opt/anka-storage:/mnt/vol"
      - "/opt/secure/certs:/mnt/certs"
    environment:
      ANKA_USE_HTTPS: "true"
      ANKA_SERVER_CERT: "/mnt/certs/anka-controller-crt.pem"
      ANKA_SERVER_KEY: "/mnt/certs/anka-controller-key.pem"

For the standalone package (separate docker containers for the controller and registry): If the SERVER_CERT and KEY is self-signed, you will need to set ANKA_SKIP_TLS_VERIFICATION to true in the controller config so it can connect to the registry.

Test the Configuration

Start or restart your Controller and/or Registry and test the new TLS configuration using https://. You can also try using curl -v https://$CONTROLLER_OR_REGISTRY_URL/api/v1/status.

If that doesn’t work, try to repeat the above steps and validate that the file names and paths are correct. If you are still having trouble, debug the system as explained in the Debugging Controller section.

3. Creating self-signed Node Certificates

The Controller’s authentication module uses the Root CA (anka-ca-crt.pem) to authenticate any incoming requests. When the Node sends the requests to the Controller, it will present its certificates. Those certificates must have been generated from the Root CA and also, if using Enterprise Plus, have the necessary permissions.

You can use the following openssl commands to create Node certificates using the Root CA:

openssl genrsa -out node-$NODE_NAME-key.pem 4096
openssl req -new -sha256 -key node-$NODE_NAME-key.pem -out node-$NODE_NAME-csr.pem \
  -subj "/O=$ORGANIZATION/OU=$ORG_UNIT/CN=$NODE_NAME"
openssl x509 -req -days 365 -sha256 -in node-$NODE_NAME-csr.pem -CA anka-ca-crt.pem -CAkey anka-ca-key.pem \
  -CAcreateserial -out node-$NODE_NAME-crt.pem

4. Configuring the Controller & Registry to enable authentication

In addition to the node certificates, the controller itself makes API calls to the Registry (you’ve enabled registry auth, right?) to get templates, etc, and will need a client cert to communicate with it. This is where ANKA_CA_CERT comes in as it’s used to validate the incoming requests.

MacOS combined Controller & Registry package

Edit the /usr/local/bin/anka-controllerd and ensure the following ENVs exist:

export ANKA_ENABLE_AUTH="true"
export ANKA_CA_CERT="/Users/MyUser/anka-ca-crt.pem"
export ANKA_CLIENT_CERT="/Users/MyUser/anka-controller-crt.pem"
export ANKA_CLIENT_CERT_KEY="/Users/MyUser/anka-controller-key.pem"

Linux/Docker Controller & Registry package

Within the docker-compose.yml, add the following ENVs:

version: '2'
services:
  anka-controller:
    container_name: anka-controller
    . . .
    environment:
      . . .
      ANKA_ENABLE_AUTH: "true"
      ANKA_CA_CERT: "/mnt/certs/anka-ca-crt.pem"
      ANKA_CLIENT_CERT: "/mnt/certs/anka-controller-crt.pem"
      ANKA_CLIENT_CERT_KEY: "/mnt/certs/anka-controller-key.pem"
  anka-registry:
    container_name: anka-registry
    . . .
    environment:
      . . .
      ANKA_ENABLE_AUTH: "true"
      ANKA_CA_CERT: "/mnt/certs/anka-ca-crt.pem"

The ANKA_CA_CERT is the authority that is used to validate the Anka Node Agent (ankacluster join) certs.

Until you have an Enterprise licensed Node joined to the Controller, it won’t enable authentication for the Controller.

If you’re connecting the Anka CLI with the HTTPS Registry URL, you can use the Node certificates: anka registry --cert /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-crt.pem --key /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-key.pem add $REGISTRY_NAME https://$REGISTRY_ADDRESS:8089

5. Testing & Joining your Node to the Controller with the Node certificate

Copy both the Node certificates (node-$NODE_NAME-crt.pem, node-$NODE_NAME-key.pem) and the anka-ca-crt.pem to the host/node you wish to join.

Testing

Restart your Controller & Registry and then test the status endpoint with curl:

curl --insecure -v https://$HOST/api/v1/status 

The response you should get is a 401 Authentication Required similar to below:

> GET /api/v1/status HTTP/2
> Host: localhost:80
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 401 
< content-type: application/json
< content-length: 54
< date: Thu, 28 Nov 2019 16:58:23 GMT
< 
{"status":"FAIL","message":"Authentication Required"}

If this is the response you get, it means the authentication module is working.

Let’s try to get a response using the Node certificate we created. Execute the same command, but now pass Node certificate and key:

curl --insecure -v https://$CONTROLLER_ADDRESS/api/v1/status --cert /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-crt.pem --key /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-key.pem

If everything is configured correctly, you should see something like this (I used 127.0.0.1 to setup this example):

*   Trying 127.0.0.1...

. . .

> GET /api/v1/status HTTP/2
> Host: 127.0.0.1:80
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< cache-control: no-store
< content-type: application/json
< content-length: 184
< date: Sun, 12 Apr 2020 04:26:13 GMT
< 
{"status":"OK","message":"","body":{"status":"Running","version":"1.7.0-4e6617d3","registry_address":"https://127.0.0.1:8089","registry_status":"Running","license":"enterprise plus"}}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0

Joining

If you previously joined your Nodes to the Controller, you’ll want to sudo ankacluster disjoin on each before proceeding (if it hangs, use ps aux | grep anka_agent | awk '{print $2}' | xargs kill -9 and try disjoin again).

Note: Certificates are cached, so if you update/renew them, you need to either:

1. disjoin and re-join them to the controller, issue sudo pkill -9 anka_agent on each node to restart the agent
2. or, issue a <controller>/v1/node/update PUT to the controller API to forcefully update all nodes.

If you’re using a signed certificate for the controller dashboard, but self-signed certificates for your nodes and CI tools, you’ll need to specify the --cacert for ankacluster join and anka registry add commands and point it to the signed CA certificate. You’ll usually see SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",) if the wrong CA is being used.

Then, use the ankacluster command to connect it to the Controller with:

sudo ankacluster join https://$CONTROLLER_ADDRESS --skip-tls-verification \
  --cert /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-crt.pem --cert-key /Users/$USER_WHERE_CERTS_ARE/node-$NODE_NAME-key.pem
Testing connection to Controller...: OK
Testing connection to registry….: OK
Ok
Cluster join success

The --skip-tls-verification is only necessary if using a self-signed cert. Please avoid using --skip-tls-verification AND the --cacert.

Accessing the Controller UI

Once Cert Auth as been enabled, loading your Controller UI will show Controller Not Connected. This is because the Controller is fully protected. In order to access the UI, you can set up your browser to use client certificates to access the page. Alternatively, you can enable root token auth with ANKA_ROOT_TOKEN which must be set to a minimum of 10 characters. You can read more about it here.

Managing User/Group Permissions (Authorization)

Certificate Authentication users:

When creating certificates, you’ll want to specify CSR values using openssl’s -subj option. For example, if we’re going to generate a certificate so our Jenkins instance can access the Controller & Registry, you’ll want to use something like this:

-subj "/O=MyOrgName/OU=$ORG_UNIT/CN=Jenkins"

• Both O= and CN= are required.
• Within the Controller’s Permission administration panel, we use O= as the Group Name and CN= as the Username.
• Spaces are supported in O= and Anka Build Cloud Controller version >= 1.10.

By default, ANKA_ENABLE_AUTH will not use authorization/permissions and allow any certs or users to connect to all API endpoints and pages in the UI. In order to enable Authorization, you will need to include specific ENVs in your config:

• ANKA_ENABLE_CONTROLLER_AUTHORIZATION works for both combined and standalone (docker) packages.
• ANKA_ENABLE_REGISTRY_AUTHORIZATION is for the combined (controller + registry in one binary) package only.
• ANKA_ENABLE_AUTHORIZATION is only for the standalone registry package.

Permission groups are configurable from your Controller’s https://<controller address>/admin/ui page. You can target and add permissions for either the group name or the username (which is different between the various Advanced Security Features we offer).

This feature requires Enterprise Plus. The regular enterprise license automatically adds all permissions to each certificate or token that is used and gives no control over them.

This also requires that you’ve enabled Root Token Authentication, giving you super user access to the controller UI and permissions.

The permission groups here differ from the groups you assign to nodes within the Controller UI.

The Available Permissions list will display all of the permissions we can assign to the group (see below for the full list). These permissions will allow plugins/users (like Jenkins) to communicate with the Controller & Registry:

Minimum Recommended Permissions

• get_groups
• get_registry_disk_info
• list_images
• list_nodes
• list_vms
• save_image
• start_vm
• terminate_vm
• update_vm
• view_logs

Controller Permissions

Registry Permissions

6. Final Notes

• If you enabled AUTH for the registry, you’ll need to ensure that you set the ANKA_CLIENT_CERT and ANKA_CLIENT_CERT_KEY in your controller config or else it won’t be able to communicate with the registry.

  ANKA_CLIENT_CERT	(string)	(Certificate Authentication) The Controller will use this when making http requests, mainly to the Registry	
  ANKA_CLIENT_CERT_KEY	(string)	(Certificate Authentication) The Controller will use this when making http requests, mainly to the Registry
  

• You may notice that the Controller UI doesn’t load or acts strangely. You will need to enable Root Token Authentication to access the controller UI.
• If you get an invalid cert error from the Controller UI, make sure that you add the root CA you generated to your system keychain.