SSH Access Management - Control SSH Access With Vault OTP Engine (PART 1)

Written by Ananth Kamath on April 28, 2020; tagged under vault, bastion, devops, aws, ssh, hashicorp

In this blog series you will go through the problem involved with SSH Access Management and how to fix the same without compromising your organisations security and compliance using Hashicorp’s Vault and Bastion server.

SSH access management is a known problem in DevOps world today. It is a tedious job to create access for your team on demand and then ensure that the access is revoked post their task. To simplify this task you end up creating your application workloads(servers) in the public subnets, sharing pem files or copying users public key into the workloads themselves. Thereby risking exposure of your critical application or user data to the world.

Let Us Address The Elephant In The Room

Generally, you would provide your users access to the application workloads via Bastion/jump server and keep all our workloads in a private subnet hidden from the rest of the world. By doing this you can restrict access for the external users and the attackers to your workloads, but you would still need to take care of access and audit-ability of your organisation’s internal users.

Controlling access to the bastion server and logging the access requests for auditing are critical for your organisation’s security and compliance goals. There are multiple authorities with packages to help you achieve this, one such authority is Hashicorp’s Vault. Vault helps you control access to the bastion server and also provides audit logs for the access requests.

Implementation

Vault SSH Engine Image

We will be using two EC2 instances, one for setting up the Vault server and the other as the bastion server in the public subnet. Users will interact with the Vault server via Vault CLI to fetch the OTP(One Time Password) and then SSH to the bastion server using the OTP as the password.

The implementation includes 4 steps

Setup Vault Server

Let us start by setting up the Vault server on an EC2 instance. We will be using the Ubuntu AMI to spin up the EC2 instance.

  1. Download & Install Vault Server

    # Download Vault binary based on the version, replace ${VAULT_VERSION} with required version below
    $ wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip -O vault.zip
    
    # Unzip the Vault zip file
    $ unzip vault.zip
    
    # Make Vault executable and accessible
    $ chmod +x vault
    $ sudo mv vault /usr/local/bin/vault
    
    # Set Linux capability flag on the binary, this will let the binary do memory locking without granting unnecessary #privileges
    $ sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault
    
    # To check if vault is installed
    $ vault --version
    vault v1.3.1
    
    # To install command line completion
    $ vault -autocomplete-install
    
    # Create a vault system user
    sudo useradd --system --home /etc/vault.d --shell /bin/false vault
    
    # Create vault service file
    sudo touch /etc/systemd/system/vault.service
    
    # Update /etc/systemd/system/vault.service with below content
    [Unit]
    Description="HashiCorp Vault - A tool for managing secrets"
    Documentation=https://www.vaultproject.io/docs/
    Requires=network-online.target
    After=network-online.target
    ConditionFileNotEmpty=/etc/vault.d/vault.hcl
    StartLimitIntervalSec=60
    StartLimitBurst=3
    
    [Service]
    User=vault
    Group=vault
    ProtectSystem=full
    ProtectHome=read-only
    PrivateTmp=yes
    PrivateDevices=yes
    SecureBits=keep-caps
    AmbientCapabilities=CAP_IPC_LOCK
    Capabilities=CAP_IPC_LOCK+ep
    CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
    NoNewPrivileges=yes
    ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
    ExecReload=/bin/kill --signal HUP $MAINPID
    KillMode=process
    KillSignal=SIGINT
    Restart=on-failure
    RestartSec=5
    TimeoutStopSec=30
    StartLimitInterval=60
    StartLimitIntervalSec=60
    StartLimitBurst=3
    LimitNOFILE=65536
    LimitMEMLOCK=infinity
    
    [Install]
    WantedBy=multi-user.target
    
  2. Setup Vault Server

    Provide vault user access to /etc/vault.d

    sudo mkdir /etc/vault.d
    sudo touch /etc/vault.d/vault.hcl
    sudo chown --recursive vault:vault /etc/vault.d
    sudo chmod 640 /etc/vault.d/vault.hcl
    

    Vault stores the data in its storage in an encrypted format. There are multiple types of storage supported by Vault. We will be using the Filesystem as storage for the demonstration. On a production server though, it is recommended to use a highly available key value store like etcd, Consul, etc. Let’s update the file /etc/vault.d/vault.hcl with the values below:

    storage "file" {
      path = "/mnt/vault/data"
    }
    
    listener "tcp" {
     address     = "0.0.0.0:8200"
     tls_disable = 1
    }
    
    ui = true
    log_level = "Info"
    

    Provide vault user permission to /mnt/vault directory

    # Create /mnt/vault
    $ sudo mkdir /mnt/vault
    
    # Give vault permission to the directory
    $ sudo chown -R vault:vault /mnt/vault
    
  3. Initialise Vault Server

    Vault server always boots up in sealed state, data in its storage is stored in an encrypted format. Vault is configured to know how to access the physical storage and its location, but doesn’t know how to decrypt any of it. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data.

    When you initialise the Vault for the first time you will be given a set of 5 keys. To unseal Vault, you will need at least 3 of these. You will also be provided a root token which will be used later to login to the vault and create required users, groups and policies. The number of keys to create master key and threshold of unseal keys are configurable, they are set to 5 and 3 by default.

    Every time the vault is restarted or if the underlying EC2 instance gets restarted the vault goes back to sealed state and will have to be unlocked by using at least 3 of the 5 keys provided during the initialisation step. To avoid this manual effort you can use AWS KMS to Auto Unseal Vault. (We will be writing a detailed blog on this soon)

    Let’s initialise the vault server now, these keys and token are displayed only once on initialisation so keep them somewhere safe. You would not be able to unseal vault later if you lose the keys.

    # Now start the vault server as a service
    $ sudo systemctl enable vault # Enable vault to start automatically when the system boots up
    $ sudo systemctl start vault
    $ sudo systemctl status vault
    
    # Initialise the Vault server this will return root token and 5 keys used to unseal
    $ export VAULT_ADDR="http://127.0.0.1:8200"
    $ vault operator init >> ~/vault.keys
    
    Initial Root Token: s.######
    
    Unseal Key 1: key1###
    Unseal Key 2: key2###
    Unseal Key 3: key3###
    Unseal Key 4: key4###
    Unseal Key 5: key5###
    
    

    Now the vault server setup is complete and is ready for use.

Enable SSH OTP Engine On Vault

Now that the vault server is initialised, we can access it over http://{{VaultEC2PublicIP}}:8200. Let’s go ahead and enable the SSH OTP engine on Vault. Follow below steps to accomplish the same.

Vault SSH Engine Image

Vault SSH Engine Image

Vault SSH Engine Image

Vault SSH Engine Image

Once this step is complete, you now have the Vault server enabled with SSH OTP engine ready to use.

What Next?

As discussed earlier the implementation includes 4 steps, we have covered just two of those four steps in this blog. Part 2 of this blog series will cover:

Appendix

If you have any questions or feedback, feel free to drop us a mail at team@codemancers.com.