This article describe the upcoming release of the Vault with Apache integration, and show the details of configuration.

With the rise of microservice-based architecture, keeping things secure has become much more challenging than earlier. We are far beyond the point where our 100 instances of backend servers are accessing our database server with a single static secret credential because if in case of a credential leakage the whole system is compromised and revocation of that credential causes a massive service outage (now no one can access anything unless the instances are reconfigured). We can't eliminate the possibility of a security breach because sometimes unexpected does happen. Instead, it's totally up to us to control the blast radius in these situations. To tackle scenarios like this, a popular solution like HashiCorp Vault comes into the picture in a production environment to act as an identity-based secrets and encryption management system. In this article, I have demonstrated how to integrate Vault with Apache (a cloud-native API Gateway) jwt-auth plugin to effectively use excellence from both worlds.

What is Vault​

HashiCorp Vault is designed to help organizations manage access to secrets and transmit them safely within an organization. Secrets are defined as any form of sensitive credentials that need to be tightly controlled and monitored and can be used to unlock sensitive information. Secrets could be in the form of passwords, API keys, SSH keys, RSA tokens, or OTP. In the real world where it is very common to have a secret sprawl where secrets get stored into the config file or as a variable in actual program code which as a consequence sometimes even end up in a version control system like GitHub, BitBucket or GitLab, possess a major threat in security. Vault solves this problem by centralizing secrets. It provides encrypted storage for static secrets, generation of dynamic secrets with a TTL lease, authentication of users (machines or humans) to make sure they’re authorized to access a particular secret and many more. So that even in case of a security breach the blast radius is much small and contained.

Vault makes it very easy to control and manage access by providing us with a unilateral interface to manage every secret in your infrastructure. Not only that, it also provides the flexibility to create detailed audit logs and keep track of who accessed what.

About jwt-auth Plugin​

It is an authentication plugin that can be attached to any route to perform JWT (JSON web token, read more) authentication before the request gets forwarded to the upstream URI. In short, it is a secure authentication mechanism that leads to authorization to critical resources. Typically, a private key, or a text secret, is used by the issuer to sign the JWT. The receiver of the JWT will verify the signature to ensure that the token hasn’t been altered after it was signed by the issuer. The total integrity of the whole jwt mechanism depends on the signing secret (may it be a text secret of RSA keypairs). That makes it difficult for unauthenticated sources to guess the signing key and attempt to change the claims within the JWT.

So the storage of these keys in a secure environment is extremely crucial. Falling into wrong hands may jeopardize the security of the whole infrastructure. Though we from the side take all the means to follow standard SecOps practices, it's quite natural in the production environment to have a centralized key management solution like HashiCorp vault to have elaborate audit trails, periodic key rotation, key revocation etc. And it would be quite a troublesome issue if each time you have to update Apache configuration whenever a key rotation occurs throughout the infrastructure.

Steps to Use Vault with Apache ​

For integration with Vault, Apache needs to be loaded with vault configuration at config.yaml.

Internally, communicates with vault server KV secret engine v1 HTTP APIs. As most enterprise solution prefers to stick with KV Secrets Engine - Version 1 in their production environment, during the initial phase of -Vault support we have gone with version 1 only. In later releases, we will add the support of K/V version 2.

The main idea of using vault, instead of the etcd backend is the security concern in a low trust environment. We, the developers, understand your priorities seriously. That's why we recommend using vault access tokens that are short scoped and can grant server limited access.

Configure Vault​

If you have already a Vault instance running with the necessary privileges, feel free to skip this section. This section shares the best practices to use Vault inside the Apache ecosystem. Please follow the steps mentioned below.

Step 1: Spin Up a Vault Server​

Here you have multiple options, feel free to choose between docker, precompiled binary or building from source. As to communicate with the vault server, you need a vault CLI client, I would prefer going with precompiled binary instead of the Docker approach. Anyway, it's totally up to you (feel free to consult Vault's official installation docs). To spin up a development server, please run the following command.

$ vault server -dev -dev-root-token-id=root…WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memoryand starts unsealed with a single unseal key. The root token is alreadyauthenticated to the CLI, so you can immediately begin using Vault.You may need to set the following environment variable:export VAULT_ADDR='http://127.0.0.1:8200'The unseal key and root token are displayed below in case you want toseal/unseal the Vault or re-authenticate.Unseal Key: 12hURx2eDPKK1tzK+8TkgH9pPhPNJFpyfc/imCLgJKY=Root Token: rootDevelopment mode should NOT be used in production installations!

Set your current CLI with the correct environment variables.

$ export VAULT_ADDR='http://127.0.0.1:8200'$ export VAULT_TOKEN='root'

Enable vault k/v version 1 secret engine backend with a suitable path prefix. In this demo, we are going to choose the kv path so that we don't have a collision with the vault default secret path for kv version 2.

$ vault secrets enable -path=kv -version=1 kvSuccess! Enabled the kv secrets engine at: kv/# To reconfirm the status, run$ vault secrets listPath          Type         Accessor              Description----          ----         --------              -----------cubbyhole/    cubbyhole    cubbyhole_4eeb394c    per-token private secret storageidentity/     identity     identity_5ca6201e     identity storekv/           kv           kv_92cd6d37           n/asecret/       kv           kv_6dd46a53           key/value secret storagesys/          system       system_2045ddb1       system endpoints used for control, policy and debugging

Step 2: Generate a Vault Access Token for ​

This article is regarding using vault in jwt-auth plugin perspective. So, for an consumer (if you are unfamiliar with consumers in the ecosystem, please read the document about Apache Consumer) with username jack the jwt-auth plugin looks up (if enabled with vault configuration) for secret/s at <vault.prefix inside config.yaml>/consumer/<consumer.username>/jwt-auth into vault kv storage. In this context, if you are assigning kv/ namespace (vault path) as vault.prefix inside config.yaml for all related data retrieval, we suggest you to create a policy for path kv//consumer/*. The extra asterisk (*) at the end ensure the policy allows read for any path that has a kv//consumer prefix.

Create a policy file in HashiCorp Configuration Language (HCL).

$ tee -policy.hcl << EOFpath "kv//consumer/*" {    capabilities = ["read"]}EOF

Applying the policy into vault instance.

$ vault policy write -policy -policy.hclSuccess! Uploaded policy: -policy

Generate a token with the newly defined policy that has been configured with the small access boundary.

$ vault token create -policy="-policy"Key                  Value---                  -----token                s.KUWFVhIXgoRuQbbp3j1eMVGatoken_accessor       nPXT3q0mfZkLmhshfioOyx8Ltoken_duration       768htoken_renewable      truetoken_policies       ["-policy" "default"]identity_policies    []policies             ["-policy" "default"]

In this demonstration s.KUWFVhIXgoRuQbbp3j1eMVGa is your access token.

Adding vault configuration into Apache ​

As discussed earlier, Apache communicates with Vault instance through Vault HTTP APIs. The necessary configuration must be added into config.yaml. Here is the brief information about different fields that you can use:

• host: The host address where the vault server is running.
• timeout: HTTP timeout for each request.
• token: The generated token from vault instance that can grant access to read data from the vault.
• prefix: enabling a prefix allows you to better enforcement of policies, generate limited scoped tokens and tightly control the data that can be accessed from . Valid prefixes are (kv/, secret etc.)

vault:  host: 'http://0.0.0.0:8200'  timeout: 10  token: 's.KUWFVhIXgoRuQbbp3j1eMVGa'  prefix: 'kv/'

Create an Consumer​

has a consumer-level abstraction that goes side by side with authentication scenarios. To enable authentication for any route, a consumer is needed with a suitable configuration for that specific type of authentication service. Then only can forward the request to the upstream URI by successfully performing authentication wrt. the consumer configuration. consumer has two fields - one is username (required) to identify one consumer from the others and another is plugins that holds the consumer specific plugin configurations.

Here, in this article, we will create a consumer with jwt-auth plugin. It performs JWT authentication for the respective route/s or service/s.

To enable jwt-auth with vault configuration, make a request to:

$ curl http://127.0.0.1:9080//admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '{    "username": "jack",    "plugins": {        "jwt-auth": {            "key": "test-key",            "vault": {}        }    }}'

Here the plugin looks up for key secret inside vault path (<vault.prefix from conf.yaml>/consumer/jack/jwt-auth) for consumer jack mentioned in the consumer config and uses it for subsequent signing and jwt verification. If the key is not found in the same path, the plugin logs error and fails to perform jwt authentication.

Set Up a Test Upstream Server​

To test the behaviour, you can create a route for an upstream (a simple ping handler that returns pong). You can set it up with a plain go HTTP-Server.

// simple upstream serverpackage mainimport "net/http"func ping(w http.ResponseWriter, req *http.Request) {    w.Write([]byte("secure/pong\n"))}func main() {    http.HandleFunc("/secure/ping", ping)    http.ListenAndServe(":9999", <span class="token boolean" style="color: rgb(54, 17