We’ve just launched  API Gateway’s support for OpenTelemetry, which means that with ’s native OpenTelemetry integration, developers and API platform teams gain complete visibility into their API traffic, making troubleshooting and issue resolution faster.

In our development process, our team questioned: How do we ensure our instrumentation undergoes thorough testing with every Gateway update? How do we prevent errors from reporting misleading information and impacting our users’ API monitoring?

In this blog post, I want to share how we have implemented Tracetest for effective integration testing of OpenTelemetry in ’s open source Gateway.

The challenge: Accurate data reporting across diverse protocols

’s diverse proxying capabilities—encompassing HTTP, HTTPS, gRPC, GraphQL, and our proprietary internal proxy protocol—introduce complexity to our environment.

This diversity, alongside a dynamic middleware chain that adjusts based on the API configuration, presents a distinctive challenge: guaranteeing that our OpenTelemetry implementation accurately captures and reports data across these varied protocols.

The solution: Trace-based testing with Tracetest

In our search for a solution, we found our answer in Tracetest. Designed specifically for Observability-Driven Development (ODD), Tracetest became an indispensable tool in our arsenal. It allowed us to build and execute comprehensive end-to-end tests encompassing the full breadth of our API scenarios.

Integrating Tracetest into our workflow

A key aspect of our integration strategy was ensuring a seamless and consistent testing experience locally and on our Continuous Integration (CI) platform. This approach was crucial for improving the Developer Experience (DX), allowing our team to test new features efficiently and fix issues under similar conditions, irrespective of the environment.

We achieved this by crafting a comprehensive Taskfile, which included a series of commands for setting up, executing, and tearing down our e2e OpenTelemetry++Tracetest testing environment. These commands – detailed in our Taskfile – are integral to our testing workflow and include:

• opentelemetry:setup: This command is responsible for initialising the Tracetest environment, installing the necessary CLI tools, and building a local image of the gateway. It ensures that all prerequisites for testing are met, both in local development setups and in our CI environment.
• opentelemetry:test: This command triggers the execution of all our end-to-end test scenarios using Tracetest CLI.
• opentelemetry:teardown: Post-testing, this command is used to dismantle the testing environment cleanly, ensuring that no residual configurations or data impact subsequent tests.

Our GitHub Actions pipeline, which can be viewed in detail here, is configured to trigger each pull request to the master branch, ensuring that every change proposed is thoroughly vetted for compatibility and performance before integration.

Thanks to the verbose option of Tracetest CLI, we can have a clear understanding of what’s tested and what scenarios fail on each PR, for example:

This uniformity between local and CI testing environments not only streamlines the development process but also instils confidence in our team. Developers can replicate the exact testing scenarios locally executed in CI, ensuring that any issues are caught early and reliably.

Diving into our test scenarios

Our approach to testing goes far beyond basic checks; it’s a comprehensive exploration of the varied landscape that an API gateway presents:

• Diverse API protocols: From HTTP to gRPC and GraphQL, each protocol is thoroughly tested to ensure accurate trace collection.
• Internal protocols: Special attention is given to our internal protocols, ensuring they integrate seamlessly with OpenTelemetry.
• Middleware execution: Each middleware’s execution is traced and analysed, ensuring the integrity of our API management.
• attributes: We ensure that our attributes follow the expected semantic convention and are present on each request.

This is an example of a basic test scenario for a valid API request:

Copy to clipboard
type: Test
spec:
 id: 4pnmVurVg
 name: HTTP Test API - ok request
 trigger:
   type: http
   httpRequest:
     method: GET
     url: ':8080/test/ip'
     headers:
       - key: Content-Type
         value: application/json
 specs:
   - selector: 'span[tracetest.span.type="http" name="GET /test/ip" http.method="GET"]'
     name: Test main span attributes
     assertions:
       - 'attr:http.method         =         "GET"'
       - 'attr:http.status_code         =         200'
       - 'attr:http.user_agent         =         "Go-http-client/1.1"'
       - 'attr:http.wrote_bytes         !=         0'
       - 'attr:tracetest.span.type         =         "http"'
       - 'attr:.api.id       =       "3"'
       - 'attr:.api.name       =       "TestAPI"'
       - 'attr:.api.orgid       =       "default"'
       - 'attr:.api.tags       not-contains      "test"'
       - 'attr:.api.path       =       "/test/"'
   - selector: >-
       span[tracetest.span.type="general" name="Tracetest trigger"]
       span[tracetest.span.type="general"]
     name: Check if it contains Version and Rate middlewares
     assertions:
       - 'attr:tracetest.selected_spans.count  =  2'
       - 'attr:tracetest.span.parent_id != ""'
   - selector: 'span[tracetest.span.type="general" name="RateCheckMW"]'
     name: Check for RateCheckMiddleware
     assertions:
       - 'attr:name  =  "RateCheckMW"'
   - selector: 'span[tracetest.span.type="general" name="VersionCheck"]'
     name: VersionCheck MW attributes
     assertions:
       - 'attr:.api.version = "Non Versioned"'

You can find the whole list of tested scenarios here.

This endeavour goes beyond a one-time integration; it’s about continuous improvement and adaptation. As we add more scenarios to our tests, we refine our approach, ensuring that every step in our API management is accurately reflected in our traces.

Transparency and reliability at

In embracing OpenTelemetry for API Gateway, we needed to ensure robust integration testing. Tracetest is our solution to accurate data reporting across diverse protocols. Now seamlessly woven into our workflow, Tracetest ensures consistent, reliable testing across local and CI environments and has a permanent spot in the toolkit to make sure our future releases are rock solid.

Want to connect with the team about  API Gateway’s support for OpenTelemetry? You can ask us anything here.