Grafana Agent AWS Request Signing: Setup & Best Practices

The modern cloud infrastructure, dominated by services like Amazon Web Services (AWS), thrives on interconnected components exchanging vast amounts of data. For organizations striving for comprehensive observability, Grafana Agent stands out as a lightweight, powerful data collector, adept at gathering metrics, logs, and traces from diverse sources. Integrating Grafana Agent with AWS services, however, is not merely a matter of configuring endpoints; it demands a robust security framework to ensure data integrity, authenticity, and authorization. This is precisely where AWS Request Signing, specifically Signature Version 4 (SigV4), becomes an indispensable component. Without proper request signing, Grafana Agent's attempts to interact with protected AWS APIs would be met with immediate rejection, rendering it incapable of fulfilling its critical monitoring role.

This comprehensive guide delves deep into the intricacies of setting up Grafana Agent to securely communicate with AWS services using SigV4. We will unravel the complexities of AWS authentication, explore the various configuration options within Grafana Agent, and present a detailed array of best practices to ensure not only secure but also efficient and scalable integrations. From understanding the fundamental principles of IAM roles to navigating common troubleshooting scenarios, this article aims to equip engineers and architects with the knowledge to build resilient observability pipelines that leverage the full power of both Grafana Agent and AWS. In the landscape of distributed systems, where every api call matters, securing these interactions is paramount, and understanding the role of an api gateway or any gateway in general for managing such interactions can further enhance the overall system's robustness and security.

---

Unveiling Grafana Agent: Your Observability Sidekick

Grafana Agent is a versatile, purpose-built agent designed to simplify the collection of telemetry data – metrics, logs, and traces – for consumption by Grafana Cloud or compatible open-source observability stacks. Unlike traditional, monolithic agents, Grafana Agent is highly modular and resource-efficient, making it ideal for deployment across diverse environments, from bare metal servers to Kubernetes clusters. Its design philosophy centers around being a universal collector, capable of scraping Prometheus-compatible metrics, tailing logs for Loki, and capturing traces for Tempo.

At its core, Grafana Agent operates in two primary modes: Static Mode and Flow Mode. * Static Mode is the more traditional configuration approach, where a single YAML file defines all the data sources, processing pipelines, and remote write endpoints. It’s well-suited for simpler deployments and users familiar with standard Prometheus or Loki configuration patterns. In static mode, you define a set of 'instances' for different types of telemetry, such as metrics_instance or logs_instance, each with its own set of scrape configurations and remote write targets. When interacting with AWS services in this mode, the AWS authentication details are often configured directly within the respective remote_write or scrape_config blocks, or globally via an aws_sdk_client configuration, which we will explore in detail later. * Flow Mode, introduced more recently, offers a more dynamic and declarative approach inspired by dataflow programming. It treats each component (e.g., a Prometheus scraper, a Loki log target, an AWS credentials provider) as a building block that can be connected to other components using expressions. This allows for highly flexible and composable pipelines, making it particularly powerful for complex, multi-source, or multi-destination scenarios. Flow Mode leverages a declarative language akin to CUE, providing granular control over data processing and routing. For AWS integrations in Flow Mode, dedicated components like aws.credentials provide a robust and reusable way to manage authentication, which can then be passed to other components that need to interact with AWS APIs.

Regardless of the mode, the fundamental role of Grafana Agent remains consistent: to efficiently gather observability data and securely transmit it to a central storage and analysis platform. This transmission, especially when targeting AWS services like CloudWatch, S3, Kinesis, or even custom api endpoints backed by an api gateway, necessitates stringent security measures. The data collected often contains sensitive operational insights, and unauthorized access or tampering could lead to severe consequences. Therefore, understanding how Grafana Agent establishes and maintains secure communication channels with AWS through request signing is not just a best practice, but a foundational requirement for any robust cloud observability strategy.

The agent's lightweight nature means it often runs closer to the data source, be it an EC2 instance, an ECS task, or a Kubernetes pod. This proximity reduces network latency for data collection but also places a higher emphasis on its ability to securely authenticate with remote services without compromising local security posture. The efficiency of its data collection and forwarding capabilities directly impacts the cost and performance of the overall monitoring system, making optimized and secure configurations a critical factor in cloud operations.

---

Deconstructing AWS Request Signing: Signature Version 4

At the heart of secure programmatic interaction with almost all modern AWS services lies Signature Version 4 (SigV4). SigV4 is the protocol AWS uses to authenticate requests made to its APIs, ensuring that only authorized entities can interact with their services and that the requests themselves have not been tampered with in transit. It's a cryptographic process designed to provide both authentication (proving who you are) and integrity (proving the request hasn't been altered). Without a valid SigV4 signature, most AWS API calls are immediately rejected with an "Access Denied" error, regardless of the permissions associated with the calling identity.

The complexity of SigV4 arises from its multi-step process, which involves hashing, canonicalization, and signing various parts of an HTTP request. Let's break down the core components and the dance involved in generating a valid signature:

1. Canonical Request: The first step is to transform the original HTTP request into a "canonical" format. This involves:
   • HTTP Method: The request method (e.g., GET, POST, PUT).
   • Canonical URI: The URI component of the request, stripped of query string parameters, then URL-encoded.
   • Canonical Query String: All query string parameters, sorted alphabetically by parameter name, then URL-encoded.
   • Canonical Headers: All required headers (Host, Content-Type, X-Amz-Date, and any X-Amz- prefixed headers), converted to lowercase, sorted alphabetically, and concatenated.
   • Signed Headers: A list of the headers included in the Canonical Headers, also lowercase and sorted alphabetically.
   • Payload Hash: A SHA256 hash of the request body. Even for an empty body, a specific hash is used. This canonical request ensures that even minor, semantically irrelevant differences in how a request is constructed (e.g., header order) do not lead to different signatures, maintaining consistency.
2. String to Sign: This string combines crucial metadata with the hash of the canonical request:
   • Algorithm: Always AWS4-HMAC-SHA256.
   • Request Date/Time: The timestamp of the request in ISO 8601 format (e.g., 20230101T120000Z). This must be precise and match the X-Amz-Date header.
   • Credential Scope: Defines the region and service for which the request is valid. It follows the format: YYYYMMDD/region/service/aws4_request (e.g., 20230101/us-east-1/s3/aws4_request).
   • Hashed Canonical Request: The SHA256 hash of the canonical request generated in step 1. The string to sign is critical because it binds the signature to a specific time, region, and service, preventing replay attacks and ensuring context.
3. Signing Key: Before the final signature can be generated, a "signing key" must be derived. This is a multi-step HMAC-SHA256 process using your AWS secret access key, progressively hashing it with the date, region, and service information. This creates a unique key for each request's specific context, further enhancing security by limiting the scope of any potential key compromise.
4. Signature: Finally, the signing key is used with HMAC-SHA256 to sign the "string to sign." The resulting hexadecimal string is the SigV4 signature. This signature is then typically included in the Authorization header of the HTTP request, alongside other SigV4-related information.

AWS services rely heavily on SigV4 for all programmatic interactions. Whether Grafana Agent is pushing metrics to CloudWatch, fetching logs from S3, streaming data to Kinesis Data Firehose, or interacting with a custom api gateway endpoint that requires AWS authentication, SigV4 is the underlying security mechanism. The precise time synchronization between the client (Grafana Agent) and AWS servers is crucial, as even a small clock drift can invalidate the signature, leading to authentication failures. This entire process, while complex under the hood, is abstracted away by AWS SDKs and well-designed clients like Grafana Agent, allowing users to focus on configuration rather than cryptographic details. However, understanding the fundamentals of SigV4 is vital for effective troubleshooting and for designing robust, secure api integrations.

---

The Nexus: Grafana Agent and AWS Request Signing

When Grafana Agent needs to interact with AWS services, it doesn't just send plain HTTP requests. Instead, it leverages the AWS SDK or implements a compatible SigV4 signing mechanism to securely communicate with the target AWS api endpoints. This is a critical aspect of its functionality, enabling it to collect data from and send data to a variety of AWS resources. The need for SigV4 arises in several common scenarios:

• Collecting Metrics from CloudWatch: Grafana Agent can be configured to scrape metrics directly from AWS CloudWatch. For instance, to monitor EC2 CPU utilization, RDS database connections, or Lambda invocation counts, Grafana Agent makes GetMetricData or GetMetricStatistics api calls to CloudWatch. These calls must be signed with SigV4.
• Fetching Logs from S3 or CloudWatch Logs: While Grafana Agent primarily uses Loki for log aggregation, it can be configured to read logs from AWS S3 buckets (e.g., application logs stored by an EC2 instance) or stream logs from CloudWatch Logs. Accessing S3 objects or CloudWatch Logs streams requires appropriate permissions and SigV4-signed requests.
• Sending Data to Kinesis Data Firehose/Kinesis Streams: For high-throughput streaming data, Grafana Agent can be configured to send collected metrics or logs to Kinesis Data Firehose or directly to Kinesis Streams. Pushing data to these services involves PutRecord or PutRecordBatch api calls, which, again, must be SigV4-signed.
• Interacting with other AWS Services: In more advanced scenarios, Grafana Agent might need to query other AWS services for dynamic configuration, service discovery, or to enrich telemetry data. Each such interaction with an AWS api endpoint necessitates SigV4 authentication.

Grafana Agent handles AWS authentication through various mechanisms, prioritizing security and operational convenience:

1. IAM Roles for EC2/EKS/ECS: This is the recommended and most secure method. When Grafana Agent runs on an EC2 instance, within an EKS pod (using IAM roles for Service Accounts, IRSA), or as an ECS task (using task roles), it automatically inherits credentials from the underlying instance or task role. The AWS SDKs, which Grafana Agent utilizes internally, are designed to automatically fetch temporary credentials from the instance metadata service (IMDS) or the EKS/ECS credential provider. These temporary credentials are then used to sign SigV4 requests. This method eliminates the need to store static, long-lived credentials on the host, significantly reducing the risk of credential compromise.
2. AWS Shared Credential Files (~/.aws/credentials): Grafana Agent can be configured to use credentials from a standard AWS shared credential file. This file typically stores aws_access_key_id and aws_secret_access_key pairs under named profiles. While less secure than IAM roles, it's a common method for local development or non-EC2 deployments where IAM roles are not readily available.
3. Environment Variables: AWS credentials can also be provided via environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). This method offers similar security risks to shared credential files but is often used in CI/CD pipelines or containerized environments for specific, short-lived tasks.
4. Direct Configuration (Least Recommended): In some Grafana Agent configuration blocks, you might find options to directly specify access_key_id and secret_access_key. This is strongly discouraged for production environments due to the inherent security risks of embedding sensitive credentials directly in configuration files.

The importance of IAM policies cannot be overstated. When Grafana Agent assumes an IAM role or uses an IAM user's credentials, the permissions associated with that role or user dictate exactly which AWS api calls the agent is authorized to make. For example, if Grafana Agent is meant to collect CloudWatch metrics, its IAM policy must include permissions like cloudwatch:GetMetricData and cloudwatch:ListMetrics. Similarly, for S3 access, s3:GetObject on specific buckets might be required. Adhering to the principle of least privilege – granting only the minimum necessary permissions – is paramount for enhancing the security posture of your AWS environment. A well-defined IAM policy, combined with SigV4, forms the bedrock of secure and authorized interactions between Grafana Agent and AWS services.

---

Setting Up Grafana Agent for AWS Request Signing: Step-by-Step Guide

Successfully configuring Grafana Agent to interact with AWS services using SigV4 requires a meticulous approach, encompassing both AWS Identity and Access Management (IAM) setup and Grafana Agent configuration. This section provides a detailed, step-by-step guide for both Static Mode and Flow Mode.

Prerequisites:

Before you begin, ensure you have: * An active AWS account. * An EC2 instance or a Kubernetes cluster (EKS) where Grafana Agent will be deployed. * Grafana Agent installed and accessible (downloaded binary or Docker image). * Basic understanding of IAM roles and policies.

Step 1: IAM Configuration – The Foundation of Security

The most secure way for Grafana Agent to access AWS services is by leveraging IAM roles. This eliminates the need for hardcoded credentials.

1.1 Create an IAM Policy with Least Privilege Permissions:

Define a custom IAM policy that grants only the necessary permissions for Grafana Agent to perform its tasks. For instance, if you intend to collect CloudWatch metrics and read logs from S3, your policy might look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricStatistics",
                "logs:DescribeLogGroups",
                "logs:FilterLogEvents",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:cloudwatch:*:*:metric/*",
                "arn:aws:logs:*:*:log-group:*",
                "arn:aws:s3:::your-log-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-log-bucket"
            ]
        }
    ]
}

• Explanation:
  • cloudwatch:GetMetricData, ListMetrics, GetMetricStatistics: Required for querying CloudWatch metrics. The resource arn:aws:cloudwatch:*:*:metric/* allows access to all metrics across all regions (you might want to scope this down further).
  • logs:DescribeLogGroups, logs:FilterLogEvents: Necessary for tailing CloudWatch Logs.
  • s3:GetObject on arn:aws:s3:::your-log-bucket/*: Allows reading objects from a specific S3 bucket. Replace your-log-bucket with your actual bucket name.
  • s3:ListBucket on arn:aws:s3:::your-log-bucket: Allows listing the contents of the bucket, often a prerequisite for GetObject.

Save this policy in the IAM console (under "Policies" -> "Create policy") or via AWS CLI/CloudFormation. Give it a descriptive name, e.g., GrafanaAgentCloudWatchS3ReaderPolicy.

1.2 Create an IAM Role and Attach the Policy:

Next, create an IAM role that Grafana Agent can assume.

• For EC2 Instances:
  1. Go to "IAM" -> "Roles" -> "Create role".
  2. Select "AWS service" -> "EC2" as the trusted entity. This allows EC2 instances to assume this role.
  3. Attach the GrafanaAgentCloudWatchS3ReaderPolicy you created.
  4. Give the role a name, e.g., GrafanaAgentEC2Role, and create it.
  5. When launching an EC2 instance, or for an existing instance, attach this IAM role to it. EC2 instances automatically get temporary credentials from their attached role via the instance metadata service.
• For EKS (IAM Roles for Service Accounts - IRSA):
  1. Ensure your EKS cluster has an OIDC provider.
  2. When creating the IAM role, select "Web identity" as the trusted entity.
  3. Configure the trust policy to allow your Kubernetes service account to assume this role. A typical trust policy for EKS looks like this: json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<YOUR_ACCOUNT_ID>:oidc-provider/oidc.eks.<YOUR_REGION>.amazonaws.com/id/<YOUR_OIDC_ID>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.<YOUR_REGION>.amazonaws.com/id/<YOUR_OIDC_ID>:sub": "system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT_NAME" } } } ] } Replace placeholders with your account ID, region, OIDC ID, Kubernetes namespace, and service account name.
  4. Attach the GrafanaAgentCloudWatchS3ReaderPolicy.
  5. Annotate your Kubernetes service account (or the pod's service account) with the ARN of this IAM role: eks.amazonaws.com/role-arn: arn:aws:iam::<YOUR_ACCOUNT_ID>:role/GrafanaAgentEKSRole

Step 2: Grafana Agent Configuration

Now, configure Grafana Agent to use these permissions.

2.1 Static Mode Configuration Example:

In Static Mode, you typically define metrics_instance or logs_instance blocks. For AWS services, you can configure an aws_sdk_client or specify aws credentials directly within the remote write or scrape configurations. Using aws_sdk_client is generally preferred as it centralizes credential management.

# agent-config-static.yaml
server:
  http_listen_port: 12345

# Global AWS SDK client configuration (optional but good for consistency)
aws_sdk_client:
  # If running on EC2/EKS/ECS with an IAM role, you don't need access_key_id/secret_access_key.
  # The SDK will automatically use the role credentials.
  # region: us-east-1 # Specify your desired AWS region

metrics:
  configs:
    - name: default
      remote_write:
        - url: https://prometheus-us-east-1.grafana.net/api/prom/push
          basic_auth:
            username: <YOUR_GRAFANA_CLOUD_PROMETHEUS_USER>
            password: <YOUR_GRAFANA_CLOUD_PROMETHEUS_API_KEY>
      scrape_configs:
        # Example: Scrape EC2 instance metrics (requires CloudWatch permissions on the IAM role)
        - job_name: 'aws-ec2-cloudwatch'
          cloudwatch_exporter:
            # The region here will use the credentials from the attached IAM role
            # or the global aws_sdk_client config
            region: us-east-1
            # dimensions:
            #  - Name: "InstanceId"
            #    Value: "<YOUR_EC2_INSTANCE_ID>"
            metrics:
              - aws_namespace: AWS/EC2
                aws_metric_name: CPUUtilization
                aws_statistics: [Average]
                aws_period: 60
                aws_dimensions:
                  - InstanceId
              - aws_namespace: AWS/EC2
                aws_metric_name: NetworkIn
                aws_statistics: [Sum]
                aws_period: 300
                aws_dimensions:
                  - InstanceId
            # If explicit credentials were needed (NOT RECOMMENDED for roles):
            # access_key_id: "YOUR_ACCESS_KEY_ID"
            # secret_access_key: "YOUR_SECRET_ACCESS_KEY"

logs:
  configs:
    - name: default
      clients:
        - url: https://logs-us-east-1.grafana.net/loki/api/v1/push
          basic_auth:
            username: <YOUR_GRAFANA_CLOUD_LOKI_USER>
            password: <YOUR_GRAFANA_CLOUD_LOKI_API_KEY>
      scrape_configs:
        # Example: Scrape logs from an S3 bucket (requires S3 GetObject permissions)
        - job_name: 'aws-s3-logs'
          s3:
            s3_path:
              - "s3://your-log-bucket/path/to/logs/*.log" # Adjust path as needed
            # The region will use credentials from the attached IAM role
            # or the global aws_sdk_client config
            region: us-east-1
            # poll_interval: 1m
            # sns:
            #   sqs_queue_url: "https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/your-sqs-queue"
            #   sqs_region: us-east-1
            # If explicit credentials were needed (NOT RECOMMENDED for roles):
            # access_key_id: "YOUR_ACCESS_KEY_ID"
            # secret_access_key: "YOUR_SECRET_ACCESS_KEY"

• Key Points for Static Mode:
  • IAM Role Autodetection: If Grafana Agent is running on an EC2 instance with GrafanaAgentEC2Role or an EKS pod with GrafanaAgentEKSRole (via IRSA), the AWS SDK (used internally by Grafana Agent) will automatically discover and use these temporary credentials. You generally do not need to specify access_key_id or secret_access_key in the configuration.
  • Region: Always specify the correct AWS region for the resources you are interacting with.
  • cloudwatch_exporter and s3 scrape configs: These are examples of Grafana Agent components that leverage AWS SDK and thus SigV4 for interaction. Ensure your IAM policy covers the necessary actions for these components.

2.2 Flow Mode Configuration Example:

Flow Mode offers more modularity. You define components and connect them.

// agent-config-flow.river
// Define an AWS credentials component. This will automatically use IAM roles
// if running on EC2/EKS/ECS. Otherwise, it falls back to environment variables
// or shared credential files.
aws.credentials "default" {
  // If running on an instance with an IAM role, this component will automatically
  // fetch temporary credentials. No explicit configuration needed for roles.
  // profile = "default" // Optionally specify a profile from ~/.aws/credentials
}

// Scrape CloudWatch metrics
prometheus.exporter.cloudwatch "ec2_cpu" {
  aws_client {
    credentials = aws.credentials.default.output
    region      = "us-east-1"
  }
  metrics_configs {
    aws_namespace  = "AWS/EC2"
    aws_metric_name = "CPUUtilization"
    aws_statistics = ["Average"]
    aws_dimensions = ["InstanceId"]
    # You might want to filter instances, e.g., by Tag or specific InstanceId
    # aws_dimension_filters {
    #   "InstanceId" = ["i-1234567890abcdef0"]
    # }
  }
  # Additional configurations like `period`, `delay_interval`, etc.
}

// Scrape logs from an S3 bucket
loki.source.s3 "app_logs" {
  aws_client {
    credentials = aws.credentials.default.output
    region      = "us-east-1"
  }
  bucket_name = "your-log-bucket"
  key_prefix  = "path/to/logs/"
  # Example: Only process files ending with .log
  # path_regexp = "^.*\\.log$"
  # poll_interval = "1m"
  # sns {
  #   sqs_queue_url = "https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/your-sqs-queue"
  #   sqs_region    = "us-east-1"
  # }
}

// Remote write collected metrics to Grafana Cloud Prometheus
prometheus.remote_write "grafana_cloud_metrics" {
  endpoint {
    url = "https://prometheus-us-east-1.grafana.net/api/prom/push"
    basic_auth {
      username = "<YOUR_GRAFANA_CLOUD_PROMETHEUS_USER>"
      password = "<YOUR_GRAFANA_CLOUD_PROMETHEUS_API_KEY>"
    }
  }
  # Connect the CloudWatch exporter's output to this remote write
  forward_to = [prometheus.exporter.cloudwatch.ec2_cpu.output]
}

// Remote write collected logs to Grafana Cloud Loki
loki.write "grafana_cloud_logs" {
  endpoint {
    url = "https://logs-us-east-1.grafana.net/loki/api/v1/push"
    basic_auth {
      username = "<YOUR_GRAFANA_CLOUD_LOKI_USER>"
      password = "<YOUR_GRAFANA_CLOUD_LOKI_API_KEY>"
    }
  }
  # Connect the S3 log source's output to this write
  forward_to = [loki.source.s3.app_logs.output]
}

• Key Points for Flow Mode:
  • aws.credentials component: This centralizes the logic for obtaining AWS credentials. It automatically tries instance profiles, environment variables, and shared credential files, making it highly adaptable.
  • aws_client block: Components that need to interact with AWS (like prometheus.exporter.cloudwatch or loki.source.s3) accept an aws_client block where you pass the output of the aws.credentials component and specify the region. This is how the SigV4 signing context is established for that specific component.
  • forward_to: This attribute is how components are chained together in Flow Mode, sending data from a source component to a destination component.

Step 3: Run Grafana Agent

Save your configuration file (e.g., agent-config-static.yaml or agent-config-flow.river) and run Grafana Agent:

# For Static Mode
./grafana-agent -config.file=agent-config-static.yaml

# For Flow Mode
./grafana-agent -config.file=agent-config-flow.river -enable-feature=agent-mode-flow

After starting, monitor Grafana Agent's logs for any errors related to AWS authentication or API calls. You should see it successfully making requests to AWS services and forwarding data to your remote write endpoints.

By following these steps, Grafana Agent will be configured to securely access AWS services, leveraging the power of IAM roles and SigV4 without ever exposing sensitive, long-lived credentials directly within its configuration files. This approach forms the cornerstone of a secure and robust observability pipeline in AWS.

---

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Best Practices for Secure and Efficient Integration

Implementing Grafana Agent with AWS Request Signing effectively goes beyond mere configuration; it necessitates adhering to a set of best practices that enhance security, improve efficiency, and ensure the long-term reliability of your observability infrastructure.

1. Embrace the Principle of Least Privilege (PoLP)

This is perhaps the most fundamental security principle in cloud environments. When creating IAM policies for Grafana Agent, ensure that they grant only the absolute minimum permissions required for its operations. * Granular Actions: Instead of cloudwatch:*, specify cloudwatch:GetMetricData, cloudwatch:ListMetrics, etc. * Resource Scoping: Instead of Resource: "*", define specific ARNs for S3 buckets, CloudWatch log groups, or Kinesis streams. For example, limit S3 access to arn:aws:s3:::your-specific-bucket/* instead of all buckets. * Conditional Access: Utilize IAM policy conditions to restrict access based on IP address, time of day, or other contextual factors, if applicable, though this adds complexity.

Adhering to PoLP minimizes the blast radius in case of a credential compromise or a misconfigured agent. If Grafana Agent's role is compromised, an attacker would only gain access to the very specific AWS services and resources needed for observability, not the entire AWS account.

2. Prioritize IAM Roles over Static Credentials

As demonstrated in the setup guide, using IAM roles (for EC2 instances, EKS service accounts, or ECS tasks) is the gold standard for AWS authentication. * Dynamic, Short-Lived Credentials: IAM roles provide temporary credentials that are automatically rotated by AWS. This eliminates the risk of long-lived static keys being stolen or accidentally exposed. * No Hardcoding: Credentials are never stored directly in configuration files, environment variables, or version control, significantly reducing exposure. * Simplified Management: You manage permissions at the role level, not individual key pairs, simplifying policy updates and audits.

Avoid using access_key_id and secret_access_key directly in Grafana Agent configurations or environment variables in production. Reserve static credentials only for development, testing, or highly controlled CI/CD scenarios where temporary credentials might be impractical, and even then, use them with extreme caution and short lifespans.

3. Regular Credential Rotation (Even for IAM Roles, Understand the Nuances)

While IAM roles provide automatic rotation of temporary credentials, it's essential to understand the underlying mechanics. The temporary credentials issued to an EC2 instance or an EKS pod have a configurable duration (up to 12 hours). The AWS SDK, and thus Grafana Agent, will automatically refresh these credentials before they expire. * Monitor IAM Best Practices: Regularly review your IAM setup using AWS IAM Access Analyzer or other security tools to ensure no static credentials are inadvertently created or have excessive permissions. * Audit Trail: Use AWS CloudTrail to monitor AssumeRole events and API calls made by Grafana Agent's IAM role. This provides an invaluable audit trail for security investigations.

4. Ensure Correct Region Specificity

AWS services are region-specific. Always configure Grafana Agent with the correct AWS region for the resources it needs to interact with. * Configuration Consistency: Specify the region consistently in your Grafana Agent configuration (e.g., region: us-east-1 in Static Mode, or region = "us-east-1" in Flow Mode). * Credential Scope: Remember that SigV4 signatures are scoped to a specific region. A mismatch will result in SignatureDoesNotMatch errors.

Incorrect region configuration is a common cause of authentication failures and can lead to data loss or delayed troubleshooting.

5. Implement Robust Error Handling and Retries

AWS services, like any distributed system, can experience transient errors, rate limiting, or network issues. Grafana Agent's underlying AWS SDK generally includes built-in retry mechanisms with exponential backoff, which helps in handling these transient failures gracefully. * Monitor Agent Logs: Keep a close eye on Grafana Agent's logs for repeated errors or warnings related to AWS API calls. This can indicate persistent issues that require intervention. * Alerting: Set up alerts based on Grafana Agent's internal metrics (e.g., agent_metrics_remote_write_queue_lengths_sum) to detect if data is not being sent successfully to AWS or remote endpoints.

Understanding how Grafana Agent handles these failures is crucial for maintaining data reliability and ensuring continuous observability.

6. Monitor Grafana Agent's Health and Performance

Just as Grafana Agent monitors your other systems, you should monitor Grafana Agent itself. * Agent Metrics: Grafana Agent exposes its own internal metrics (e.g., Prometheus scrape counts, remote write queue lengths, memory usage, CPU usage) via its /metrics endpoint. Scrape these metrics with another Grafana Agent or a dedicated Prometheus instance. * System Metrics: Monitor the host system where Grafana Agent runs for resource saturation (CPU, memory, disk I/O, network). * AWS Service Quotas: Be aware of AWS service quotas (e.g., CloudWatch API request limits, Kinesis throughput limits). A high volume of data collection by Grafana Agent can inadvertently hit these limits.

Proactive monitoring allows you to identify performance bottlenecks, configuration issues, or resource constraints before they impact your observability pipeline.

7. Fortify Network Security

Layered network security is vital for any component interacting with cloud services. * Security Groups/NACLs: Configure security groups and Network Access Control Lists (NACLs) to restrict network access to and from the Grafana Agent hosts. Allow outbound traffic only to necessary AWS service endpoints (e.g., CloudWatch, S3, Grafana Cloud endpoints) and inbound traffic only from trusted management sources. * VPC Endpoints (AWS PrivateLink): For enhanced security and reduced data transfer costs, consider using AWS PrivateLink to connect Grafana Agent directly to AWS service endpoints within your VPC. This bypasses the public internet, reducing exposure to threats. * Firewalls: Ensure any host-level firewalls (e.g., iptables, firewalld) are correctly configured to allow Grafana Agent's network traffic.

8. Consider AWS API Gateway for Custom Integrations

While Grafana Agent directly interacts with many core AWS services, in a larger microservices architecture, you might have custom api endpoints that Grafana Agent needs to interact with, or you might want to proxy existing AWS services. An api gateway like AWS API Gateway can play a crucial role here. * Unified Access Point: AWS API Gateway can act as a single entry point for all api calls, routing requests to various backend services (Lambda, EC2, other AWS services). * Enhanced Security: API Gateway provides features like throttling, caching, request validation, and built-in authorization mechanisms (IAM, Cognito, custom authorizers). If Grafana Agent needs to push data to a custom api that your organization manages, routing it through API Gateway allows you to apply additional security layers and enforce access policies. * Abstraction and Transformation: API Gateway can transform requests and responses, allowing Grafana Agent to send data in a specific format that is then adapted for the backend service. This provides an extra layer of abstraction, decoupling the client (Grafana Agent) from backend implementation details.

For instance, if Grafana Agent needs to send custom application metrics to a non-standard endpoint, and that endpoint is exposed via AWS API Gateway, the agent would then make requests to the api gateway URL, which might itself be secured with IAM authentication (requiring SigV4 from the agent) or other methods. In such a setup, the api gateway acts as a crucial gateway for managing and securing the flow of data.

By meticulously applying these best practices, you can build a highly secure, resilient, and efficient observability pipeline leveraging Grafana Agent and AWS, ensuring that your critical telemetry data is collected and transmitted with integrity and confidentiality.

---

Troubleshooting Common Issues

Even with the most careful setup, issues can arise when integrating Grafana Agent with AWS. Understanding common pitfalls and effective troubleshooting strategies is key to quickly resolving problems and minimizing downtime.

1. AccessDenied Errors

This is perhaps the most frequent issue. It indicates that the IAM principal (the role or user) Grafana Agent is using does not have the necessary permissions to perform the requested AWS API action on the specified resource.

• Symptom: Grafana Agent logs show errors like AccessDeniedException, NotAuthorized, or User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/GrafanaAgentEC2Role/i-xxxx is not authorized to perform: <action> on resource: <resource_arn>.
• Troubleshooting Steps:
  1. Check Grafana Agent Logs: The logs usually provide the exact AWS api action and resource that was denied. This is your primary clue.
  2. Verify IAM Policy:
     • Navigate to the IAM role (or user) associated with Grafana Agent in the AWS console.
     • Review the attached policies. Ensure the required Action (e.g., cloudwatch:GetMetricData, s3:GetObject) is explicitly allowed.
     • Check the Resource specified in the policy. Is it * (less secure but rules out resource restriction), or a specific ARN? If it's an ARN, does it exactly match the resource Grafana Agent is trying to access? Remember ARN formats can be tricky (e.g., log-group:* vs. log-group:/aws/lambda/*).
     • Examine any Condition blocks in the policy that might restrict access unexpectedly.
  3. Use AWS IAM Policy Simulator: This powerful tool allows you to test whether a specific IAM principal has permission to perform certain actions on specific resources. It's invaluable for validating policy changes before deployment.
  4. Check Service Quotas: Less common for AccessDenied but possible if the denial is related to reaching a service limit which implicitly denies further access.

2. SignatureDoesNotMatch Errors

This error specifically points to a problem with the SigV4 signature generated by Grafana Agent not matching the one AWS expects. This is a cryptographic failure.

• Symptom: Grafana Agent logs show errors like SignatureDoesNotMatch, The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details., or Malformed or invalid request signature.
• Troubleshooting Steps:
  1. Time Synchronization (NTP): The most common cause. SigV4 is highly sensitive to clock skew. If the system clock of the host running Grafana Agent is out of sync with AWS's servers (even by a few minutes), the timestamp in the signature will be invalid.
     • Ensure Network Time Protocol (NTP) is correctly configured and working on the Grafana Agent host. For Linux, ntpstat or timedatectl status can help. For EC2 instances, this is usually handled automatically, but manual intervention might be needed for custom VMs.
  2. Incorrect Credentials:
     • If using static access_key_id and secret_access_key (again, NOT RECOMMENDED), double-check for typos.
     • If using shared credential files (~/.aws/credentials), ensure the correct profile is being referenced and the credentials are valid.
     • For IAM roles, while direct credential errors are rare, an issue with the underlying STS (Security Token Service) or IMDS (Instance Metadata Service) could prevent proper credential acquisition. Check IMDS access from the instance.
  3. Incorrect Region/Service: The SigV4 signature includes a credential scope for the region and service. A mismatch here will invalidate the signature.
     • Verify that the region configured in Grafana Agent precisely matches the region of the AWS service endpoint it's trying to reach.
     • Ensure the service endpoint is correct (e.g., not accidentally trying to hit S3 in us-east-1 when your bucket is in eu-west-1).

3. Network Connectivity Issues

Grafana Agent needs outbound network access to AWS service endpoints and its remote write targets (e.g., Grafana Cloud).

• Symptom: Grafana Agent logs show errors like Connection timed out, Network unreachable, No route to host, or TLS handshake failed.
• Troubleshooting Steps:
  1. Security Groups/NACLs: Check the security groups attached to the Grafana Agent host and the network ACLs of its subnet. Ensure outbound traffic is allowed on port 443 (HTTPS) to the necessary AWS service IP ranges or specific endpoints.
  2. Firewalls: Verify any host-level firewalls (e.g., iptables, firewalld) are not blocking outbound connections on port 443.
  3. Proxy Configuration: If Grafana Agent operates behind an HTTP proxy, ensure its proxy settings are correctly configured (e.g., HTTP_PROXY, HTTPS_PROXY environment variables or specific agent configurations).
  4. DNS Resolution: Ensure the host can resolve AWS service endpoints' DNS names. Use dig or nslookup to test.
  5. VPC Endpoints (PrivateLink): If using VPC endpoints, confirm they are correctly configured, the routing tables are updated, and the endpoint policies allow access.

4. Agent Not Sending Data (Silent Failures)

Sometimes Grafana Agent appears to be running, but no data reaches the intended destination, and logs aren't immediately revealing.

• Symptom: No metrics/logs/traces appear in Grafana Cloud or your observability backend, but Grafana Agent logs show no explicit errors.
• Troubleshooting Steps:
  1. Check Grafana Agent's Internal Metrics: Access Grafana Agent's /metrics endpoint (e.g., http://localhost:12345/metrics) and look for:
     • agent_metrics_remote_write_queue_lengths_sum: If this is high and growing, data is queued but not sent.
     • agent_metrics_remote_write_sent_batches_total: Should be increasing.
     • agent_metrics_target_scrape_duration_seconds: Indicates if scraping itself is working.
     • agent_metrics_target_scrape_errors_total: Check for scrape-specific errors.
  2. Configuration Review: Carefully re-read the entire Grafana Agent configuration file. Even a small typo (e.g., incorrect endpoint URL, wrong label selector, missing forward_to in Flow Mode) can prevent data flow.
  3. Target Service Health: Check the health and availability of the target AWS service (CloudWatch, S3, Kinesis) or your remote write endpoint (Grafana Cloud, Loki, Prometheus).
  4. Resource Saturation: High CPU, memory, or disk I/O on the Grafana Agent host can cause it to lag or drop data. Monitor system resources.
  5. Data Volume: If the volume of data being scraped is too high, it might overwhelm the agent or hit remote write limits, leading to backpressure and potential data drops.

Debugging with AWS CloudTrail and Grafana Agent Logs

• AWS CloudTrail: For any AWS API-related issues, AWS CloudTrail is your best friend. It records all API calls made to your AWS account. Filter CloudTrail events by the IAM role ARN used by Grafana Agent, the AWS service name, and the time range of the issue. This often reveals the exact API call that failed and the reason for the failure, especially for AccessDenied issues.
• Grafana Agent's Debug Logging: Increase Grafana Agent's log level to debug (e.g., -log.level=debug or log_level: debug in config) to get more verbose output. This can often provide granular details about the internal processes, including AWS SDK interactions, which might shed light on the problem.

By systematically applying these troubleshooting techniques, you can efficiently diagnose and resolve most issues encountered when setting up and operating Grafana Agent with AWS Request Signing, ensuring the reliability of your observability data.

---

Advanced Scenarios and Considerations

Beyond the basic setup, several advanced scenarios and considerations can further enhance the security, efficiency, and scalability of Grafana Agent's AWS integrations. These often involve more complex network configurations, cross-account strategies, or deployment models that demand a deeper understanding of both AWS and Grafana Agent capabilities.

Cross-Account Access for Centralized Observability

In large organizations, it's common to have multiple AWS accounts (e.g., development, staging, production, shared services). For centralized observability, you might want a Grafana Agent deployed in a "monitoring" account to collect metrics, logs, and traces from resources in other "source" accounts.

• IAM Roles for Cross-Account Access: This is achieved by using IAM roles with a trust policy that allows the monitoring account to assume a role in the source account.
  1. Source Account Role: Create an IAM role in each "source" account (e.g., GrafanaAgentSourceAccountRole). This role's trust policy allows the monitoring account's Grafana Agent IAM role (e.g., arn:aws:iam::<MONITORING_ACCOUNT_ID>:role/GrafanaAgentMonitorRole) to assume it.
  2. Source Account Policy: Attach a policy to GrafanaAgentSourceAccountRole that grants read permissions to the specific AWS resources (CloudWatch metrics, S3 buckets, etc.) in that source account.
  3. Monitoring Account Role: The Grafana Agent in the "monitoring" account runs with its own IAM role (GrafanaAgentMonitorRole). This role needs sts:AssumeRole permission to assume GrafanaAgentSourceAccountRole in the source accounts.
• Grafana Agent Configuration:
  • Static Mode: For each scrape configuration targeting a source account, you might need to specify the external_id (if used in the AssumeRole policy) and the role_arn of the role in the source account. The aws_sdk_client block can sometimes support this for certain scrape configurations.
  • Flow Mode: The aws.credentials component can be configured to assume a role from another account. You'd typically create separate aws.credentials components for each target account, each specifying the role_arn to assume. ```river aws.credentials "source_account_1_creds" { role_arn = "arn:aws:iam:::role/GrafanaAgentSourceAccountRole" # Optional: external_id = "your-external-id" }prometheus.exporter.cloudwatch "source_account_1_ec2" { aws_client { credentials = aws.credentials.source_account_1_creds.output region = "us-east-1" } // ... scrape configs for source account 1 } `` This setup centralizes observability while maintaining a strict security boundary between accounts, using SigV4 for allAssumeRole` and subsequent service API calls.

PrivateLink and VPC Endpoints: Enhanced Security and Cost Efficiency

Directly accessing AWS service endpoints over the public internet, even with SigV4, might not meet the stringent security or compliance requirements of some organizations. AWS PrivateLink allows you to connect your VPC directly to supported AWS services using private endpoints, bypassing the public internet entirely.

• Security Benefits: Traffic flows within the AWS network, never traversing the public internet. This reduces exposure to external threats and simplifies network security controls.
• Cost Benefits: For high-volume data transfer, using VPC endpoints can sometimes reduce data egress costs compared to public internet routes.
• Configuration:
  1. Create VPC Endpoints (Interface Endpoints) for the specific AWS services Grafana Agent needs to access (e.g., CloudWatch, S3, Kinesis).
  2. Ensure the security group attached to the VPC endpoint allows inbound traffic on port 443 from the Grafana Agent hosts.
  3. Verify that your Grafana Agent hosts are in subnets that can route traffic to these VPC endpoints.
  4. Grafana Agent automatically uses VPC endpoints if they are configured in the VPC where it runs, as the AWS SDK prefers private endpoints when available. No special agent configuration is typically needed beyond specifying the region.

Grafana Agent in Containerized Environments (EKS, ECS): IAM Roles for Service Accounts (IRSA) and Task Roles

Containerization adds another layer of abstraction, but the principle of IAM roles remains paramount.

• Amazon EKS (Kubernetes): For pods running Grafana Agent in EKS, IAM Roles for Service Accounts (IRSA) is the recommended method.
  • You associate an IAM role with a Kubernetes Service Account.
  • Pods that use that Service Account automatically receive temporary AWS credentials from the associated IAM role.
  • This is far more secure than kube2iam or kiam as it provides credentials directly to the pod via the OIDC provider, leveraging the sts:AssumeRoleWithWebIdentity action.
  • The setup for IRSA was covered briefly in the IAM configuration section for EKS.
• Amazon ECS (Fargate/EC2): For tasks running Grafana Agent in ECS, Task Roles are used.
  • When defining an ECS task definition, you specify a "Task IAM role."
  • Any container within that task can then assume the permissions granted to that task role.
  • This works seamlessly with Grafana Agent, as the AWS SDK automatically picks up these credentials.

Both IRSA and Task Roles ensure that each containerized Grafana Agent instance operates with its own specific, least-privileged IAM role, further enhancing the security and isolation of your observability data collection.

Integrating with a Broader API Ecosystem and API Gateway Relevance

While Grafana Agent focuses on collecting and sending observability data, its interactions with AWS services (which are essentially api calls) highlight a larger theme: secure api management. In a modern distributed system, countless apis are consumed and produced, not just AWS's own. This is where dedicated api gateway solutions become indispensable for managing the entire api lifecycle.

Consider a scenario where Grafana Agent might need to: * Push custom metrics to a proprietary api endpoint that your organization developed. * Fetch configuration data from an internal configuration api. * Trigger a Lambda function via an api gateway for on-demand debugging or data processing.

In such cases, your organization's custom apis would likely be managed by an api gateway (like AWS API Gateway, or a third-party gateway solution). This gateway would handle: * Authentication and Authorization: Ensuring Grafana Agent (or any client) is authorized to call the custom api. This could involve AWS SigV4, OAuth, JWTs, or API keys. * Rate Limiting and Throttling: Protecting your backend services from being overwhelmed. * Traffic Management: Routing, load balancing, and versioning of apis. * Monitoring and Logging: Centralized tracking of api calls.

The principles of secure api interaction, including robust authentication (like SigV4), least privilege, and network security, apply universally whether the api is an AWS service or a custom api managed by an api gateway. Grafana Agent's successful integration with AWS underscores the importance of these concepts in any api-driven architecture.

In this broader context of api management, robust tools are essential. For organizations dealing with a multitude of APIs, especially those leveraging AI models, platforms like APIPark offer a comprehensive solution. APIPark is an open-source AI gateway and API developer portal that provides end-to-end API lifecycle management. While Grafana Agent excels at securely collecting observability data from AWS, APIPark focuses on managing, integrating, and deploying a variety of AI and REST services, acting as a powerful gateway for all your custom and AI-driven APIs. It standardizes API formats, encapsulates prompts into REST APIs, and offers features like independent API and access permissions for each tenant, ensuring secure and efficient API sharing within teams. Its performance, rivaling Nginx, and detailed API call logging capabilities make it a strong contender for organizations seeking a powerful api management platform. You can learn more about this robust solution at ApiPark. By understanding how Grafana Agent secures its interactions with AWS, we gain valuable insights into the fundamental requirements for securing any api in a complex cloud ecosystem, a challenge that platforms like APIPark are specifically designed to address for a diverse set of apis.

---

Conclusion

The secure and efficient integration of Grafana Agent with AWS services through Request Signing (SigV4) is not merely a technical configuration task; it is a fundamental pillar of building a robust and reliable cloud observability infrastructure. Throughout this extensive guide, we have traversed the critical landscape of understanding Grafana Agent's capabilities, deconstructing the cryptographic intricacies of AWS Signature Version 4, and meticulously detailing the step-by-step process of configuring Grafana Agent to leverage IAM roles for secure interactions.

We have emphasized the paramount importance of the Principle of Least Privilege, advocating for IAM roles over static credentials, and meticulously outlining best practices that range from diligent region specificity and robust error handling to the often-overlooked yet critical aspect of network security. Troubleshooting common issues like AccessDenied and SignatureDoesNotMatch errors was covered to equip practitioners with the knowledge to swiftly diagnose and rectify problems. Furthermore, we explored advanced scenarios such as cross-account access and the utilization of PrivateLink, showcasing how these advanced configurations contribute to heightened security and operational efficiency in complex, multi-account AWS environments. The discussion extended to how Grafana Agent operates within containerized environments like EKS and ECS, highlighting the seamless integration with IRSA and Task Roles.

Ultimately, the successful deployment of Grafana Agent with proper AWS Request Signing ensures that your valuable telemetry data—metrics, logs, and traces—are collected and transmitted to your observability platform with integrity, confidentiality, and authenticity. This foundational security, coupled with efficient data collection, empowers organizations to gain deep insights into their cloud applications and infrastructure, enabling proactive problem-solving, performance optimization, and informed decision-making. The robust and scalable nature of a properly configured Grafana Agent system, securely communicating with AWS, forms the bedrock upon which modern, resilient cloud operations are built, ready to tackle the dynamic challenges of today's digital landscape, where every api interaction demands unwavering security and reliability.

---

Frequently Asked Questions (FAQs)

1. What is AWS Signature Version 4 (SigV4) and why is it important for Grafana Agent? AWS Signature Version 4 (SigV4) is the cryptographic protocol used by AWS to authenticate requests made to almost all its API services. It ensures that only authorized entities can interact with AWS and that the requests have not been tampered with. For Grafana Agent, SigV4 is crucial because it allows the agent to securely communicate with AWS services like CloudWatch, S3, or Kinesis to collect metrics, logs, and traces, proving its identity and the integrity of its requests. Without a valid SigV4 signature, Grafana Agent's API calls to AWS would be rejected.

2. What is the most secure way to configure Grafana Agent for AWS authentication? The most secure way to configure Grafana Agent for AWS authentication is by using IAM Roles. If Grafana Agent runs on an EC2 instance, an EKS pod (using IAM Roles for Service Accounts - IRSA), or an ECS task (using task roles), it can automatically obtain temporary, short-lived credentials from the instance metadata service. This method eliminates the need to hardcode sensitive access_key_id and secret_access_key in configuration files or environment variables, significantly reducing the risk of credential compromise.

3. My Grafana Agent is getting AccessDenied errors when trying to read CloudWatch metrics. How do I troubleshoot this? An AccessDenied error indicates that the IAM role or user Grafana Agent is using lacks the necessary permissions. To troubleshoot: 1. Check Grafana Agent logs: Identify the exact AWS API action (e.g., cloudwatch:GetMetricData) and resource ARN that was denied. 2. Verify IAM Policy: Go to the IAM role/user in the AWS console and review its attached policies. Ensure the required Action and Resource are explicitly allowed. Pay close attention to resource ARNs and any Condition blocks. 3. Use IAM Policy Simulator: This AWS tool can help you test if your IAM principal has the necessary permissions.

4. Why is my Grafana Agent receiving SignatureDoesNotMatch errors when interacting with AWS? SignatureDoesNotMatch errors typically point to a cryptographic failure in the SigV4 signature generation. The most common reasons are: 1. Clock Skew: The system clock of the host running Grafana Agent is significantly out of sync with AWS's servers. Ensure NTP is correctly configured on the host. 2. Incorrect Credentials: If static credentials are used (not recommended), there might be a typo in the access_key_id or secret_access_key. 3. Incorrect Region/Service: The AWS region specified in Grafana Agent's configuration (e.g., us-east-1) does not match the region of the AWS service endpoint being called.

5. Can Grafana Agent interact with custom API endpoints secured by an AWS API Gateway? Yes, Grafana Agent can interact with custom API endpoints secured by an AWS API Gateway. If the API Gateway endpoint requires AWS authentication (e.g., IAM authorization), Grafana Agent will need to sign its requests using SigV4, similar to how it interacts with other core AWS services. This allows the API Gateway to authenticate and authorize Grafana Agent's calls before routing them to the backend service. This scenario highlights the broader importance of secure api gateway solutions in managing api interactions within a complex cloud environment.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.