Using Barbican as a Secret storage



Secrets management

You can refer to the official client documentation for more examples and options about how to use CLI

How to store a generic secret

It is possible to store any secret data with Barbican. Command below will create a secret of type passphrase encrypted with AES algorithm with key length of 256 bits:

(openstack) $ openstack secret store --secret-type passphrase -p passphrase -n mysecret
+---------------+--------------------------------------------------------------------------------+
| Field         | Value                                                                          |
+---------------+--------------------------------------------------------------------------------+
| Secret href   | https://dx1.citycloud.com:9311/v1/secrets/33ef0985-f89e-4bf0-b318-887ebcac0cba |
| Name          | mysecret                                                                       |
| Created       | None                                                                           |
| Status        | None                                                                           |
| Content types | None                                                                           |
| Algorithm     | aes                                                                            |
| Bit length    | 256                                                                            |
| Secret type   | passphrase                                                                     |
| Mode          | cbc                                                                            |
| Expiration    | None                                                                           |
+---------------+--------------------------------------------------------------------------------+
(openstack) $ 

How to retrieve secrets

Secrets (EK) are stored in barbican in an encrypted format. You can see a list of secrets created for your user with the following command:

(openstack) $ openstack secret list
+--------------------------------------------------------------------------------+----------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                                    | Name     | Created                   | Status | Content types                           | Algorithm | Bit length | Secret type | Mode | Expiration |
+--------------------------------------------------------------------------------+----------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| https://dx1.citycloud.com:9311/v1/secrets/33ef0985-f89e-4bf0-b318-887ebcac0cba | mysecret | 2021-04-29T10:33:18+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes       |        256 | passphrase  | cbc  | None       |
| https://dx1.citycloud.com:9311/v1/secrets/ad628532-53b8-4d2f-91e5-b0097b51da4e | None     | 2021-04-27T13:52:10+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
+--------------------------------------------------------------------------------+----------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
(openstack) dmitriy@dmitriy-XPS-13-9380:~/Documents/ansible$ 

In order to retrieve decrypted content of the secret you should ask for payload:

(openstack) $ openstack secret get https://dx1.citycloud.com:9311/v1/secrets/33ef0985-f89e-4bf0-b318-887ebcac0cba -p
+---------+------------+
| Field   | Value      |
+---------+------------+
| Payload | passphrase |
+---------+------------+
(openstack) $ 

Block storage encryption

How to create encrypted volume for cinder

Note that instances with encrypted volumes attached can not be live migrated or booted after evacuation.

  • This means that such instances may have to be stopped during maintenance windows and at other occasions when live migrations are normally performed. 
  • Detach the encrypted volume from the instance and perform live-migration.
  • In case of compute node failure you would need to boot such instances after evacuation manually

If you still want CityNetwork to perform live migration and evacuation, you will need to share secret to the volume through ACL. Instructions on how to do that can be found on the page: Maintenance Window Instructions for instances with encrypted volumes attached#SharingsecretwithACL


Creation of the LUKS encrypted volumes does not differ from creation any other one, except you need to provide specific volume type. You can retrieve list of available volume types with command:

openstack volume type list

(openstack) $ openstack volume type lis
+--------------------------------------+-----------------------+-----------+
| ID                                   | Name                  | Is Public |
+--------------------------------------+-----------------------+-----------+
| a479a6b0-b283-41a5-b38b-5b08e7f902ca | volumes_hdd_encrypted | True      |
| d9dfa98a-238d-4ca0-9abf-701fceb05623 | __DEFAULT__           | True      |
| 86796611-fb12-4628-b6b1-e09469e301d7 | volumes_hdd           | True      |
+--------------------------------------+-----------------------+-----------+
(openstack) $


Volume types, that will be LUKS encrypted, have suffix _encrypted for example, volumes_hdd_encrypted. LUKS key is encrypted with symmetric AES 256 bit key and stored in Barbican.

To create volume with this volume type you need to explicitly specify it:

(openstack) $ openstack volume create --type volumes_hdd_encrypted --size 10 enc_drive
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2021-04-27T13:52:10.000000           |
| description         | None                                 |
| encrypted           | True                                 |
| id                  | 33211b21-8d4f-48e9-b76f-ec73ffd19def |
| multiattach         | False                                |
| name                | enc_drive                            |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 10                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | volumes_hdd_encrypted                |
| updated_at          | None                                 |
| user_id             | 966ad341f4e14920b5f589f900246ccc     |
+---------------------+--------------------------------------+
(openstack) $


You can define volume type while creating volumes in ansible and terraform as well.

UI integration is not ready yet.

Usual gotchas

While creating encrypted volumes a secret is created in barbican with user token. During volume deletion Cinder will also try to remove previously created EK. However as secrets are also related to the user, only user who created secret can delete it afterwards. So in case you created encrypted volume for project with user A, and try to delete with user B, this request will fail. You must issue volume delete request as user, that has created that volume. Otherwise you need to create an ACL rule, to allow other user to manage specific secret:

(openstack) $ openstack acl user add --user c1d20e4b7e7d4917aee6f0832152269b https://dx1.citycloud.com:9311/v1/secrets/33ef0985-f89e-4bf0-b318-887ebcac0cba
+----------------+----------------+--------------------------------------+---------------------------+---------------------------+------------------------------------------------------------------------------------+
| Operation Type | Project Access | Users                                | Created                   | Updated                   | Secret ACL Ref                                                                     |
+----------------+----------------+--------------------------------------+---------------------------+---------------------------+------------------------------------------------------------------------------------+
| read           | True           | ['c1d20e4b7e7d4917aee6f0832152269b'] | 2021-04-29T10:58:23+00:00 | 2021-04-29T10:58:23+00:00 | https://dx1.citycloud.com:9311/v1/secrets/33ef0985-f89e-4bf0-b318-887ebcac0cba/acl |
+----------------+----------------+--------------------------------------+---------------------------+---------------------------+------------------------------------------------------------------------------------+
(openstack) $


Object storage encryption

Client-side encryption with S3 API (SSE-C)

You can use objects encryption via S3 API according to the Amazon SSE-C specification. This means, that you need to provide encryption key explicitly with each request to the object. Encryption key can be stored in barbican and provided to the S3 client during the runtime. Below we will provide more detailed explanation regarding how to use this in your workloads.

At the moment s3cmd does not support SSE-C encryption. This feature should be implemented in 2.3.0.

Generate S3 login details

In order to interact with S3 API you would need to create application credentials. For that you can run following commands:

  1. Generate credentials in keystone
    ec2=$(openstack ec2 credentials create -f value -c access -c secret)
  2. Define variables
    access_key_id=$(echo ${ec2} | cut -d ' ' -f 1)
    secret_access_key=$(echo ${ec2} | cut -d ' ' -f 2)

Create encryption details

According to SSE-C specification, in order to use server-side encryption, we need to provide 3 additional headers:

  1. Encryption algorithm - the only valid option here is AES256
  2. Encryption key - generated random key, that we will store in Barbican. It should be valid AES key, which means that key length must be 32 characters.
  3. Encryption key checksum - MD5 checksum of the encryption key. It's used for integrity check.

Required tools for that:

  1. openstackclient
  2. pwgen
  3. openssl
  4. rclone>=1.54

In order to generate encryption key and store it in Barbican please follow next commands:

  1. Generate secret
    secret_raw=$(pwgen 32 1)
  2. Store secret in Barbican
    babrican_secret_url=$(openstack secret store --name objectSecret --algorithm aes --bit-length 256 --payload ${secret_raw} -f value -c 'Secret href')
  3. Retireve secret from Barbican
    secret=$(openstack secret get ${babrican_secret_url} -p -c Payload -f value)

Store encrypted object in S3

The trickiest part for interacting S3 API with cURL is to create valid signature. It is dynamic and will vary from request to request and should contain headers passed in the request itself, current date and requested URL. This data should be hashed with sha HMAC by ec2_secret token. You may read more about the format and requirements provided for the Signature below: https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#ConstructingTheAuthenticationHeader

For simplicity we will show example of how to use client-side encryption with rclone.

SSE-C encryption has been implemented/fixed with version 1.54. Rclone of prior versions won't work here.

  1. Download and install latest rclone for your distribution: https://rclone.org/downloads/
  2. Export OS_REGION_NAME or source .openrc file
  3. Create config file ~/.rclone.conf :

    $ cat <<EOF > ~/.rclone.conf
    [citycloud] 
    type = s3 
    provider = Ceph 
    env_auth = false 
    access_key_id = ${access_key_id}
    secret_access_key = ${secret_access_key} 
    endpoint = ${OS_REGION_NAME}.citycloud.com:8080 
    acl = private
    sse_customer_algorithm = AES256
    EOF
    
  4. Create S3 bucket
    rclone mkdir citycloud:encrypted
  5. Sync directory to the S3 where files will be encrypted with key:
    rclone sync ~/media/ citycloud:encrypted --s3-sse-customer-key=${secret}
  6. Retrieve file from S3 and decrypt it
    rclone copy citycloud:encrypted/file.png ~/ --s3-sse-customer-key=${secret}

For more examples on how to use rclone please reference documentation: https://rclone.org/docs/#subcommands


How to store a secret for Octavia

There are several ways to store a SSL certificate in Barbican that can be consumed by Octavia. It depends on the format of the certificate you have - PEM/PKCS#12. Of course you can convert between these types in advance, but we will discuss both options here.

Official documentation: https://docs.openstack.org/octavia/train/user/guides/basic-cookbook.html#deploy-a-tls-terminated-https-load-balancer

PKCS#12 Certificate

PKCS#12 certificate format include both certificate, chain and private key in the same file. So it's stored as a regular secret inside barbican.

In case you have certificate in PEM fromat you can easily convert it to PKCS#12: openssl pkcs12 -export -inkey server.key -in server.cert -out server.p12


To create secret with stored PKCS#12 certificate, you need to feed p12 file in base64 format as payload to the secret:

~$ openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
+---------------+---------------------------------------------------------------------------------+
| Field         | Value                                                                           |
+---------------+---------------------------------------------------------------------------------+
| Secret href   | https://kna1.citycloud.com:9311/v1/secrets/69bd82f5-60c9-4764-99ec-7a3dff05d2aa |
| Name          | tls_secret1                                                                     |
| Created       | None                                                                            |
| Status        | None                                                                            |
| Content types | {'default': 'application/octet-stream'}                                         |
| Algorithm     | aes                                                                             |
| Bit length    | 256                                                                             |
| Secret type   | opaque                                                                          |
| Mode          | cbc                                                                             |
| Expiration    | None                                                                            |
+---------------+---------------------------------------------------------------------------------+
~$

After that you can easily create a listener that will use certificate:

~$ openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=https://kna1.citycloud.com:9311/v1/secrets/69bd82f5-60c9-4764-99ec-7a3dff05d2aa lb2
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                       | Value                                                                                                                                                                                                                                                                              |
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up              | True                                                                                                                                                                                                                                                                               |
| connection_limit            | -1                                                                                                                                                                                                                                                                                 |
| created_at                  | 2021-01-20T11:51:46                                                                                                                                                                                                                                                                |
| default_pool_id             | None                                                                                                                                                                                                                                                                               |
| default_tls_container_ref   | https://kna1.citycloud.com:9311/v1/secrets/69bd82f5-60c9-4764-99ec-7a3dff05d2aa                                                                                                                                                                                                    |
| description                 |                                                                                                                                                                                                                                                                                    |
| id                          | 4ec6b23d-d08a-4de0-9e12-54ac690ee1ec                                                                                                                                                                                                                                               |
| insert_headers              | None                                                                                                                                                                                                                                                                               |
| l7policies                  |                                                                                                                                                                                                                                                                                    |
| loadbalancers               | 2c2a0760-c3a8-48d2-bdd0-288c3d33a43f                                                                                                                                                                                                                                               |
| name                        | listener1                                                                                                                                                                                                                                                                          |
| operating_status            | OFFLINE                                                                                                                                                                                                                                                                            |
| project_id                  | 4a9484063d4c40d29301ad745c0e2c69                                                                                                                                                                                                                                                   |
| protocol                    | TERMINATED_HTTPS                                                                                                                                                                                                                                                                   |
| protocol_port               | 443                                                                                                                                                                                                                                                                                |
| provisioning_status         | PENDING_CREATE                                                                                                                                                                                                                                                                     |
| sni_container_refs          | []                                                                                                                                                                                                                                                                                 |
| timeout_client_data         | 50000                                                                                                                                                                                                                                                                              |
| timeout_member_connect      | 5000                                                                                                                                                                                                                                                                               |
| timeout_member_data         | 50000                                                                                                                                                                                                                                                                              |
| timeout_tcp_inspect         | 0                                                                                                                                                                                                                                                                                  |
| updated_at                  | None                                                                                                                                                                                                                                                                               |
| client_ca_tls_container_ref | None                                                                                                                                                                                                                                                                               |
| client_authentication       | NONE                                                                                                                                                                                                                                                                               |
| client_crl_container_ref    | None                                                                                                                                                                                                                                                                               |
| allowed_cidrs               | None                                                                                                                                                                                                                                                                               |
| tls_ciphers                 | TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256 |
| tls_versions                |                                                                                                                                                                                                                                                                                    |
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
~$


PEM (X.509) Certificate

PEM format is pretty much default format in which you usually see certificate on Linux systems. While it's convenient, certificate, chain and private key are all different files, which mean that you need to store several different secrets, which are linked together with container of certificate type.

This is old method, that is being used by Octavia as a fallback one. That means when Octavia fails to decode certificate as PKCS#12, it catches an exception and tries to retrieve certificate as PEM.

On the contrary to the PKCS#12, PEM certificates are stored in Barbican in plain, without base64 hashing.

So we're creating secrets for both certificate and private key:

~$ openstack secret store --name crt-key -s private -t text/plain --payload="$(cat server.key)"
+---------------+---------------------------------------------------------------------------------+
| Field         | Value                                                                           |
+---------------+---------------------------------------------------------------------------------+
| Secret href   | https://kna1.citycloud.com:9311/v1/secrets/c6a9117d-94af-4b41-bf3d-67efbe51dcbe |
| Name          | crt-key                                                                         |
| Created       | None                                                                            |
| Status        | None                                                                            |
| Content types | {'default': 'text/plain'}                                                       |
| Algorithm     | aes                                                                             |
| Bit length    | 256                                                                             |
| Secret type   | private                                                                         |
| Mode          | cbc                                                                             |
| Expiration    | None                                                                            |
+---------------+---------------------------------------------------------------------------------+
~$ openstack secret store --name crt-cert -s certificate -t text/plain --payload="$(cat server.cert)"                                                                                                                                        +---------------+---------------------------------------------------------------------------------+
| Field         | Value                                                                           |
+---------------+---------------------------------------------------------------------------------+
| Secret href   | https://kna1.citycloud.com:9311/v1/secrets/91091ff2-640a-461a-920e-df983fbe5d68 |
| Name          | crt-key                                                                         |
| Created       | None                                                                            |
| Status        | None                                                                            |
| Content types | {'default': 'text/plain'}                                                       |
| Algorithm     | aes                                                                             |
| Bit length    | 256                                                                             |
| Secret type   | private                                                                         |
| Mode          | cbc                                                                             |
| Expiration    | None                                                                            |
+---------------+---------------------------------------------------------------------------------+

And after that we can create a container, that will be passed to the octavia listener

~$ openstack secret container create --name crt --type certificate --secret "certificate=https://kna1.citycloud.com:9311/v1/secrets/91091ff2-640a-461a-920e-df983fbe5d68" --secret "private_key=https://kna1.citycloud.com:9311/v1/secrets/c6a9117d-94af-4b41-bf3d-67efbe51dcbe" 
+----------------+------------------------------------------------------------------------------------+
| Field          | Value                                                                              |
+----------------+------------------------------------------------------------------------------------+
| Container href | https://kna1.citycloud.com:9311/v1/containers/35ae87af-47df-4184-b3d5-21f60cf01a9d |
| Name           | crt                                                                                |
| Created        | None                                                                               |
| Status         | ACTIVE                                                                             |
| Type           | certificate                                                                        |
| Certificate    | https://kna1.citycloud.com:9311/v1/secrets/91091ff2-640a-461a-920e-df983fbe5d68    |
| Intermediates  | None                                                                               |
| Private Key    | https://kna1.citycloud.com:9311/v1/secrets/c6a9117d-94af-4b41-bf3d-67efbe51dcbe    |
| PK Passphrase  | None                                                                               |
| Consumers      | None                                                                               |
+----------------+------------------------------------------------------------------------------------+
~$ openstack loadbalancer listener create --protocol-port 8443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=https://kna1.citycloud.com:9311/v1/containers/35ae87af-47df-4184-b3d5-21f60cf01a9d lb2
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                       | Value                                                                                                                                                                                                                                                                              |
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up              | True                                                                                                                                                                                                                                                                               |
| connection_limit            | -1                                                                                                                                                                                                                                                                                 |
| created_at                  | 2021-01-20T11:53:47                                                                                                                                                                                                                                                                |
| default_pool_id             | None                                                                                                                                                                                                                                                                               |
| default_tls_container_ref   | https://kna1.citycloud.com:9311/v1/containers/35ae87af-47df-4184-b3d5-21f60cf01a9d                                                                                                                                                                                                 |
| description                 |                                                                                                                                                                                                                                                                                    |
| id                          | c5029e06-2788-47d3-bc92-98dd9f383a6d                                                                                                                                                                                                                                               |
| insert_headers              | None                                                                                                                                                                                                                                                                               |
| l7policies                  |                                                                                                                                                                                                                                                                                    |
| loadbalancers               | 2c2a0760-c3a8-48d2-bdd0-288c3d33a43f                                                                                                                                                                                                                                               |
| name                        | listener1                                                                                                                                                                                                                                                                          |
| operating_status            | OFFLINE                                                                                                                                                                                                                                                                            |
| project_id                  | 4a9484063d4c40d29301ad745c0e2c69                                                                                                                                                                                                                                                   |
| protocol                    | TERMINATED_HTTPS                                                                                                                                                                                                                                                                   |
| protocol_port               | 8443                                                                                                                                                                                                                                                                               |
| provisioning_status         | PENDING_CREATE                                                                                                                                                                                                                                                                     |
| sni_container_refs          | []                                                                                                                                                                                                                                                                                 |
| timeout_client_data         | 50000                                                                                                                                                                                                                                                                              |
| timeout_member_connect      | 5000                                                                                                                                                                                                                                                                               |
| timeout_member_data         | 50000                                                                                                                                                                                                                                                                              |
| timeout_tcp_inspect         | 0                                                                                                                                                                                                                                                                                  |
| updated_at                  | None                                                                                                                                                                                                                                                                               |
| client_ca_tls_container_ref | None                                                                                                                                                                                                                                                                               |
| client_authentication       | NONE                                                                                                                                                                                                                                                                               |
| client_crl_container_ref    | None                                                                                                                                                                                                                                                                               |
| allowed_cidrs               | None                                                                                                                                                                                                                                                                               |
| tls_ciphers                 | TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256 |
| tls_versions                |                                                                                                                                                                                                                                                                                    |
+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
~$