HashiDays One conference. Three cities. Find a city near you
Vault
Enterprise

Use PKI with external policy services

  • 16min
  • |
  • Enterprise
  • Vault

Note

The Certificate Issuance External Policy Service requires a Vault Enterprise license.

Challenge

When internal teams submit PKI certificate signing requests (CSRs), operators can be limited by Vault semantics when validating and controlling the issued certificate from PKI secrets engines.

Solution

Vault Enterprise version 1.15.0 introduces a new Certificate Issuance External Policy Service (CIEPS) feature for the PKI secrets engine. CIEPS enables PKI operators to have fine grained control over issued certificates using custom policies from a service defined and built by your organization that operates outside of Vault.

If a CSR does not conform to the rules defined in your policy service, the request is rejected.

Operators can also specify any subject attributes or extensions for issued certificates in the policy service.

Personas

The hands-on lab scenario described in this tutorial involves one persona:

  • alice a security operator with privileged permissions to administer Vault secrets engines.

Prerequisites

To perform the tasks described in the tutorial hands-on lab, you need the following.

  • Vault Enterprise version 1.15.0 or greater binary installed and in your system PATH with valid license. Learn how to request a trial license.

  • Familiarity with TLS concepts like certificate fields and extended key usage.

  • Experience with creating and managing certificate authorities with the Vault PKI secrets engine. If you are unfamiliar with these concepts, complete the Build your own certificate authority (CA), and then return to this tutorial.

  • A text editor that you prefer to work with for creating a script.

  • Git installed and in your system PATH.

  • Go installed and in your system PATH for building the example policy service.

    • Familiarity with building Go programs is helpful but not required.

  • A development environment that includes make and openssl for building the example policy service binary.

  • curl installed and in your system PATH for API steps.

  • jq installed and in your system PATH for API output examples.

Versions used for this tutorial

This tutorial was last tested 23 Sep 2023 on macOS using the following software versions.

$ sw_vers --productVersion
13.5.2

go version  
go version go1.21.1 darwin/amd64

$ make --version | head -n 1
GNU Make 3.81

$ openssl version  
OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)

$ vault version
Vault v1.15.0-rc1+ent (d7e7b24dbf5f06fe8b3ba29cfcf4d8f7d961ed45), built 2023-09-11T17:44:31Z

$ curl --version | head -n 1 | awk '{print $2}'
8.1.2

$ jq --version                                                          
jq-1.7

Scenario introduction

Diagram for scenario introduction Diagram for scenario introduction

As the Alice persona, you'll set up a lab environment consisting of an example external policy service and a Vault server.

You'll then configure a PKI infrastructure in Vault consisting of a root CA and intermediate CA. You'll then configure the intermediate CA to use the example external policy service.

After establishing the infrastructure, you can request a certificate and key through the external policy service and then examine the certificate to learn about the extension added by the policy service.

Lab setup

Your goal for this section is to deploy and prepare your external policy service and Vault environments for the steps which follow. This involves cloning the example service repository, building the example service from source, and starting the service.

You can a run Vault dev mode server from a terminal session for the purposes of this hands-on lab.

Create hands-on lab home

You can create a temporary directory to hold all the content needed for this hands-on lab and then assign its path to an environment variable for later reference.

  1. Open a terminal, and create the directory /tmp/learn-vault-pgp.

    $ mkdir /tmp/learn-vault-pki

  2. Export the hands-on directory path as the value to the HC_LEARN_LAB environment variable.

    $ export HC_LEARN_LAB=/tmp/learn-vault-pki

External policy service

The CIEPS protocol is a REST-based, optionally mTLS protected webhook. You can learn more about the protocol along with its request and response formats for interacting with Vault in the Certificate Issuance External Policy (CIEPS) documentation.

Development of an external policy service is beyond the scope of this tutorial, but you'll have access to source code for a public example that your engineering team can review as a starting point.

The example code shows how to add an extension to a certificate and how to return warnings to the end user. You can extend this example with validation logic as necessary to meet your use case requirements.

Build and run example policy service

Your goal for this section is to build and run an example external policy service in a terminal session.

The example external service is open source, written in Go, and its source code is publicly available from the vault-pki-cieps-example repository.

When you use the example policy service, the code from the Evaluate() function in internal/business/policy.go performs evaluation on the request.

In the example version, it also adds a custom extension to certificates so that you know it was processed with the service:

extMsg := "CIEPS Demo Server Certificate"

You should spot the string CIEPS Demo Server Certificate as an extension when decoding the certificate later.

Note

The example policy server processes the certificate request, but does not actually validate the request and instead returns a warning that no validation occurred. You must implement your own validation logic in your policy server code.

  1. Clone the repository.

    $ git clone https://github.com/hashicorp/vault-pki-cieps-example \
    $HC_LEARN_LAB/vault-pki-cieps-example

  2. Change into the vault-pki-cieps-example project directory.

    $ cd $HC_LEARN_LAB/vault-pki-cieps-example

  3. Build the example external service binary.

    $ make build

    Example output:

    go build github.com/hashicorp/vault-pki-cieps-example/cli/cieps-server
go: downloading github.com/hashicorp/vault/sdk v0.10.0

    The Makefile controls building of the binary and downloads the Vault client SDK for Go if necessary.

    The binary named cieps-server, is built and located in the present working directory.

  4. Run cieps-server with the --help flag to learn about the configuration.

    $ ./cieps-server --help

    Example output:

    Usage of ./cieps-server:
-listen string
      TCP listen address in host:port format (default "0.0.0.0:443")
-server-cert string
      Path to the server certificate file (default "server.crt")
-server-key string
      Path to the server key file corresponding to the given certificate file (default "server.key")

  5. The service requires TLS certificate and key material to run; build the TLS certificate and key.

    $ make certs

    Example output:

    openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -sha256 -days 3650 -nodes -subj "/CN=localhost" -addext "subjectAltName = DNS:localhost"
.+...+......+.+........+++++++++++++++++++++++++++++++++++++++*.............+...+...+.....+...+.+.........+++++++++++++++++++++++++++++++++++++++*....+..+...+....+..+...+............+.+..+.+.....+.......+..+..................+.........+.............+...............+...+...+..+...+.........+.......+......+........+.+...........+.........+....+.........+........+....+.......................+..........++++++
.........+...+..+......+.+.....+.+...............+.....+....+......+...+.....+....+++++++++++++++++++++++++++++++++++++++*......+++++++++++++++++++++++++++++++++++++++*.....+..................+.........................+..+...+.+...+.....+......+....+...+........++++++
-----

    The service is now ready for use.

  6. Start the example policy service, instructing it to listen on all interfaces on TCP port 8443, and use the certificate and key files created from the last step. Redirect the standard output and standard error to the file policy-server.log, and background the process.

    $ ./cieps-server \
    -listen=localhost:8443 \
    -server-cert=./server.crt \
    -server-key=./server.key > policy-server.log 2>&1 &

The example policy server is ready for use.

Vault dev mode server

Run a Vault dev mode server as a background process from your terminal session, and prepare it for use.

  1. Change back into the hands-on lab home directory.

    $ cd $HC_LEARN_LAB

  2. Export an environment variable with a valid Vault Enterprise license.

    $ export VAULT_LICENSE='02MC0FFEEBK5...'

    Update example value

    Use your actual Vault Enterprise license string in place of the example value shown.

  3. Open a terminal and start a Vault dev server using the Vault Enterprise binary with root as the initial root token value.

    $ vault server -dev -dev-root-token-id root > "$PWD"/vault-server.log 2>&1 &

    The Vault dev server defaults to running at 127.0.0.1:8200. The server logs to the file vault-server.log in the present working directory, and gets automatically initialized and unsealed.

    Dev mode is not for production

    Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and all contents are lost when the Vault server process is stopped.

  4. Export the environment variable for the vault CLI to address the Vault server.

    $ export VAULT_ADDR='http://127.0.0.1:8200'

  5. Check Vault status.

    $ vault status

    Example output:

    Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.15.0+ent
Build Date      2023-09-11T17:44:31Z
Storage Type    inmem
Cluster Name    vault-cluster-040c159e
Cluster ID      78d63616-f8ac-ef95-bcea-05387a693771
HA Enabled      false

The Vault server is ready for you to proceed with the hands-on lab.

Create a certificate authority

Your goal for this section is to enable PKI infrastructure consisting of 2 PKI secrets engine instances representing a root CA (pki) and intermediate CA (pki_int) for the example.com domain.

This configuration is based on steps just like those in the Build Your Own Certificate Authority (CA) tutorial. You are encouraged to complete the hands-on lab in that tutorial if you are unfamiliar with the PKI secrets engine.

  1. To save time and typing, you can use this example shell script to enable and configure the secrets engines for the hands-on lab.

    Use a text editor to write and save this content to the file /tmp/learn-vault-pki/enable_engines.sh.

    enable_engines.sh

    #!/bin/bash   

set -euxo pipefail   

vault secrets enable pki && \
vault secrets tune -max-lease-ttl=87600h pki && \
vault write -field=certificate pki/root/generate/internal \
     common_name="example.com" \
     issuer_name="root-2023" \
     ttl=87600h > root_2023_ca.crt && \
vault write pki/roles/2023-servers allow_any_name=true && \
vault write pki/config/urls \
     issuing_certificates="$VAULT_ADDR/v1/pki/ca" \
     crl_distribution_points="$VAULT_ADDR/v1/pki/crl" && \
vault secrets enable -path=pki_int pki && \
vault secrets tune -max-lease-ttl=43800h pki_int && \
vault pki issue \
      --issuer_name=example-dot-com-intermediate \
      /pki/issuer/root-2023 \
      /pki_int/ \
      common_name="example.com Intermediate Authority" \
      o="example" \
      ou="education" \
      key_type="rsa" \
      key_bits="4096" \
      max_depth_len=1 \
      permitted_dns_domains="test.example.com" \
      ttl="43800h"
vault write pki_int/roles/example-dot-com \
     issuer_ref="$(vault read -field=default pki_int/config/issuers)" \
     allowed_domains="example.com" \
     allow_subdomains=true \
     max_ttl="12000h"

    Note

    The values used here are just for the hands-on lab. You should change the values of common_name and issuer_name to match your own values when using these commands outside of the context of the hands-on lab.

  2. Make the script executable.

    $ chmod +x $HC_LEARN_LAB/pki/enable_engines.sh

  3. Execute the script to enable and configure the secrets engines.

    $ ./enable_engines.sh

    Abbreviated expected output:

    + vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
+ vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/
   + vault write -field=certificate pki/root/generate/internal common_name=example.com issuer_name=root-2023 ttl=87600h
+ vault write pki/roles/2023-servers allow_any_name=true
...snip...
street_address                        []
ttl                                   0s
use_csr_common_name                   true
use_csr_sans                          true
use_pss                               false

  4. List the enabled PKI secrets engines.

    $ vault secrets list

    Example expected output:

    Path          Type         Accessor              Description
----          ----         --------              -----------
   cubbyhole/    cubbyhole    cubbyhole_5030c17b    per-token private secret storage
identity/     identity     identity_21924a85     identity store
pki/          pki          pki_fd06124b          n/a
pki_int/      pki          pki_05cfa4c0          n/a
secret/       kv           kv_83eeb931           key/value secret storage
   sys/          system       system_c0e34cd1       system endpoints used for control, policy and debugging

    You have enabled the PKI secrets engines with a basic configuration.

Enable and configure CIEPS

Now that you've created your CAs in Vault, you need to enable and configure CIEPS on the intermediate CA so that you can generate a certificate and key with the external policy service.

  1. Configure CIEPS using the Root 2023 CA and the PEM bundle created earlier.

    $ vault write /pki_int/config/external-policy \
    enabled=true \
    external_service_url='https://localhost:8443/evaluate' \
    trusted_leaf_certificate_bundle=@vault-pki-cieps-example/server.crt

    Example output:

    Key                                 Value
---                                 -----
enabled                             true
entity_jmespath                     n/a
external_service_last_updated       2023-09-25T15:16:23-04:00
external_service_url                https://localhost:8443/evaluate
external_service_validated          false
group_jmespath                      n/a
last_successful_request             n/a
timeout                             15000000000
trusted_ca                          n/a
trusted_leaf_certificate_bundle     -----BEGIN CERTIFICATE-----
MIIDHzCCAgegAwIBAgIUBueTj/kjYKpD+CNZl7RBQ/pNo28wDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIzMDkyNTE5MTEyMFoXDTMzMDky
MjE5MTEyMFowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAvnc3P/Qvek2BaUbAs6rWqFboFYy8T+arRKmZSBnqyLt2
z/xK4AsvFPVqMLt/VLKPpEGBIE78h1sxXuTCVNilG42I++xm5hD1V1M2D67nL3/o
MgHRStjSn5c9dMYRcqrLu6+i+jhr53aomaw79EtImlPJdDQmO9rGvvP0287/CjU8
5OwKpzeUlvuJt9BsxwadWlzkrkSgwnHn+SqsQotWy2tY36H47almIzNDcQUh8ZVQ
uBmK6wzOazUHhQzh0zbR9UYztMw4Jsx/UgPeWlIIfy9o0V+rj4zAhwiEKEqOj4b4
P0K3/2S5F/Oa7g0RmYZcRX6NPfNc9YEiVH6oLyEVkwIDAQABo2kwZzAdBgNVHQ4E
FgQUjB4p4AZ9DVoanVbRsN1OrVLtkB4wHwYDVR0jBBgwFoAUjB4p4AZ9DVoanVbR
sN1OrVLtkB4wDwYDVR0TAQH/BAUwAwEB/zAUBgNVHREEDTALgglsb2NhbGhvc3Qw
DQYJKoZIhvcNAQELBQADggEBAFapM4qWqhVpd3wC84Yd12nJfKoBpTham8m6AXKI
49x6oK6c6PwoPxqG7WyaN6GAXjLfZOShlSq5FTl8daqLxoHsh7Ym1uzCkqO+/1Pp
GP+AcJUc+wJTw6uOML6Ck9kcePqZ8ufYVwNn3ZfLkwEjgEFRnRUNVKJGt4maYDJM
D+lWKkfLx9UpkGiWtKrQ1xb1in6qPRGzW2yra0ZC91bd9NiecGkAo8PVCWxdNZcZ
OEnSGqESdiUkWlVuO/b1xDLnAWkMbDePHavJx6GqnxX5s5em2CVGAA3+2aO4l62S
20G1VkelJA8KhgK3Z2pOzyeCXJDvrgGIpoHGuAoxs7I3C0I=
-----END CERTIFICATE-----
vault_client_cert_bundle_no_keys    n/a

Generate certificate and key with external policy

Your goal in this section is to use the Vault CLI or API to request a certificate and key from Vault that uses the example external policy service.

Tip

It is suggested to limit access to the path-overridden issue API endpoint (on /pki/issuer/:issuer_ref/external-policy/issue/:policy) and let the CIEPS engine override the issuer as necessary.

Generate a certificate and key with the external policy service.

$ vault write \
    /pki_int/issuer/example-dot-com-intermediate/external-policy/issue \
    key_type=rsa \
    key_bits=2048

Example abbreviated output:

WARNING! The following warnings were returned from Vault:

  * result from demo server; no validation occurred

Key                 Value
---                 -----
ca_chain            [-----BEGIN CERTIFICATE-----
MIIE3jCCA8agAwIBAgIUE4wpL+2L0VxF5lKxX6SR1V5xBzUwDQYJKoZIhvcNAQEL
BQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjMwOTI1MTkxNTA2WhcNMjgw
...snip...
9po1Ulp0ctcjIL78+IeERNzz/wEFWkylTIzCBnOubkKqhg2I/FXYmviTt8TyvRLo
lz1QaWMcRc2nDCQy7NzriRbtFaywP8l5bPsisU6BvMVpQnLebj9Weg==
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       61:8e:17:6f:15:fb:e6:1f:3b:1f:d1:40:f1:cb:66:96:d4:7c:96:60

Vault returns the certificate and key as expected. This behavior is the same for a client whether using CIEPS or not. The difference in this case though, is that the external policy service handled this request and emitted a warning message.

WARNING! The following warnings were returned from Vault:

  * result from demo server; no validation occurred

The service also added a custom extension to the certificate which Vault cannot natively add. You can parse the certificate with a tool to learn about this extension.

Capture a certificate and export its value in the CIEPS_CERTIFICATE environment variable.

$ export CIEPS_CERTIFICATE="$(vault write -field=certificate \
    /pki_int/issuer/example-dot-com-intermediate/external-policy/issue \
    key_type=rsa \
    key_bits=2048)"

Examine a certificate

You can use a tool to examine the certificate and learn about the fields included. You can use a CLI tool like openssl or another application to examine the certificate.

Here, you'll use a web based tool to visualize the certificate content.

  1. From your terminal session, copy the contents of the certificate file to your system clipboard.

    $ echo $CIEPS_CERTIFICATE | pbcopy

  2. Open your browser and navigate to http://lapo.it/asn1js/.

  3. Paste the certificate content from your clipboard into the text area.

  4. Click decode.

    Web application decoding Vault certificate and showing the custom extension

    You can view all the certificate fields, including the custom extension and its message CIEPS Demo Server Certificate.

Next steps

You've built and run an example CIEPS server, deployed a Vault dev mode server and used the CIEPS server with the Vault PKI secrets engine. You also learned about the example CIEPS server code, user warnings, and how to a custom extension is added to a certificate by the CIEPS server.

The Vault PKI secrets engine is a popular solution, and there is much for you to learn about it. Some tutorials you can explore next to expand your PKI knowledge include Enable ACME with PKI secrets engine, PKI Unified CRL and OCSP with cross cluster revocation, and PKI secrets engine with managed keys.

Cleanup

  1. Unset the environment variables.

    $ unset VAULT_ADDR HC_LEARN_LAB

  2. Stop the external policy service by executing the following command in your terminal session.

    $ pgrep -f cieps-server | xargs kill

  3. Stop the Vault dev server by executing the following command in your terminal session.

    $ pgrep -f vault | xargs kill

  4. Change into your home directory.

    $ cd

  5. Remove the hands-on lab home directory, /tmp/learn-vault-pki.

    $ rm -rf "$HC_LEARN_LAB"

Help and reference

Previous

SAML authentication

 Next

Synchronize cloud native secrets

This tutorial also appears in:

  • 18 tutorials
    Secrets management
    Centrally store, access, and deploy secrets across applications, systems, and infrastructure.
    • Vault