Unlock AI Potential with Kong AI Gateway

The landscape of technology is undergoing an unprecedented transformation, fueled by the relentless march of Artificial Intelligence. From automating mundane tasks to powering sophisticated decision-making systems, AI is no longer a futuristic concept but a tangible, indispensable component of modern enterprises. At the heart of this revolution lies a critical operational challenge: how to effectively manage, secure, scale, and observe the myriad of AI models and services that organizations increasingly rely upon. This is where the concept of an AI Gateway emerges as a foundational pillar, acting as the intelligent intermediary that orchestrates interactions between applications and the complex world of AI. Among the various solutions available, Kong Gateway stands out as a particularly powerful and versatile platform, capable of evolving from a robust API Gateway into a sophisticated LLM Gateway and comprehensive AI Gateway, unlocking the true potential of AI within any organization.

The journey to harness AI's full capabilities is often fraught with complexities. Developers integrating AI models, data scientists deploying new algorithms, and business leaders seeking tangible value all face a common set of hurdles. These include disparate APIs from various AI providers, the imperative for stringent security and compliance, the need for cost optimization in a usage-based billing model, and the inherent demands for high performance and reliability. Without a centralized, intelligent management layer, these challenges can quickly spiral into an unmanageable tangle, hindering innovation and eroding the very benefits AI promises. This article will delve into the profound impact an AI Gateway can have, exploring its essential features, demonstrating how Kong Gateway is uniquely positioned to fulfill this role, and providing practical insights into leveraging its capabilities to transform your AI operations, making them more secure, efficient, and scalable.

The AI Revolution and its Operational Challenges

The current era is unequivocally defined by the AI revolution. Generative AI, spearheaded by Large Language Models (LLMs) like GPT, Claude, and Llama, has not merely captured the public imagination but has fundamentally altered how businesses conceive of content creation, customer service, data analysis, and even software development itself. Beyond LLMs, specialized AI models for vision, speech, anomaly detection, and predictive analytics are woven into the fabric of countless applications, driving efficiencies and enabling entirely new functionalities. This pervasive integration of AI, while immensely beneficial, introduces a significant operational overhead that traditional IT infrastructure was not designed to handle. Organizations are finding themselves grappling with a new set of challenges that demand specialized solutions, far beyond what a conventional API Gateway might offer.

Firstly, the proliferation and diversity of AI models present a formidable integration challenge. Companies rarely commit to a single AI provider or model. Instead, they leverage a mosaic of services: some from major cloud providers (AWS Bedrock, Azure OpenAI), others from specialized vendors (Anthropic, Cohere), and an increasing number of open-source or internally developed models. Each of these models typically exposes its own unique API, with varying authentication mechanisms, request/response formats, rate limits, and even data schema. This lack of standardization forces developers to write bespoke integration logic for every AI service, leading to increased development time, brittle codebases, and a high maintenance burden. Imagine building an application that needs to summarize text using one LLM, translate it using another, and generate images using a third. Without a unified interface, each interaction becomes a distinct, complex engineering task.

Secondly, security concerns are amplified in the context of AI. Feeding sensitive proprietary data or Personally Identifiable Information (PII) into third-party AI models raises significant data governance and privacy issues. Organizations must ensure that data shared with AI models is appropriately masked, encrypted, or anonymized, and that responses from AI models do not inadvertently expose confidential information. Moreover, AI endpoints themselves are prime targets for abuse, including denial-of-service attacks, unauthorized data access, and prompt injection vulnerabilities, where malicious inputs can trick an LLM into performing unintended actions or divulging sensitive data. Traditional access control mechanisms, while necessary, are often insufficient to address these nuanced AI-specific threats. The need for robust authentication, authorization, and data redaction at the gateway level becomes paramount.

Thirdly, cost management and optimization are critical, especially with the prevalent usage-based billing models for many commercial AI services. LLMs, in particular, incur costs based on token usage for both input prompts and output completions. Without granular visibility and control, costs can quickly skyrocket, making it difficult to allocate budgets, identify inefficiencies, or even predict expenditures. Enterprises need mechanisms to track token usage per application, per team, or per user, and intelligently route requests to the most cost-effective model for a given task, without compromising performance or quality. This might involve directing simpler queries to cheaper, smaller models, or leveraging cached responses for frequently asked questions.

Fourthly, performance and reliability are non-negotiable for production AI systems. Applications relying on AI services cannot afford significant latency, as it directly impacts user experience and business operations. Managing rate limits imposed by AI providers, implementing effective caching strategies, ensuring high availability through load balancing and failover mechanisms, and monitoring the health of diverse AI endpoints are all complex tasks. A single point of failure or an overloaded AI service can bring down critical business functions, underscoring the need for a resilient and performant intermediary. The sheer volume of concurrent requests that AI applications can generate also necessitates a highly scalable infrastructure that can handle fluctuating loads without degradation.

Finally, observability in AI interactions is challenging. Unlike traditional microservices where request/response payloads are often structured and predictable, AI interactions can involve complex, unstructured text or multi-modal data. Logging and monitoring these interactions effectively, tracing the journey of a request through multiple AI models, and capturing AI-specific metrics like token count, model version, and confidence scores are crucial for debugging, auditing, and performance analysis. Without comprehensive visibility, diagnosing issues, understanding AI behavior, or proving compliance becomes exceedingly difficult. The lack of standardized tools and approaches for AI observability further complicates matters, pushing organizations to seek integrated solutions.

These multifaceted challenges highlight a fundamental truth: while traditional API Gateways are essential for managing RESTful APIs, the unique characteristics and demands of AI services necessitate a more intelligent, specialized solution. An ordinary api gateway might handle basic routing, authentication, and rate limiting for an AI endpoint, but it lacks the deeper, AI-aware capabilities required for advanced security, cost optimization, prompt management, and intelligent routing across diverse AI models. This evolution gives rise to the specialized function of an AI Gateway.

What is an AI Gateway? Beyond the Traditional API Gateway

To truly unlock the potential of AI, organizations must move beyond generic API management and embrace a dedicated AI Gateway. An AI Gateway serves as an intelligent, specialized proxy layer that sits between your applications and various AI models, providing a unified control plane for all AI interactions. While it shares some foundational principles with a traditional API Gateway, its capabilities are significantly extended and tailored to the unique demands of Artificial Intelligence services, especially LLM Gateways.

A standard API Gateway acts as a single entry point for all API requests, providing functionalities like request routing, load balancing, authentication, authorization, and rate limiting. It's an indispensable component for managing microservices architecture and external API exposure. However, an AI Gateway builds upon this foundation by adding a suite of AI-specific features designed to address the challenges outlined in the previous section.

Let's delve into the core functionalities that define an AI Gateway:

1. Unified Interface for Diverse AI Models: This is perhaps the most fundamental feature. An AI Gateway abstracts away the idiosyncrasies of different AI model APIs. Whether you're interacting with OpenAI, Anthropic, Google Gemini, a locally deployed open-source LLM, or a custom vision model, the gateway presents a consistent, standardized API to your consuming applications. This means developers write integration code once against the gateway, regardless of the underlying AI provider. This dramatically reduces integration complexity, accelerates development cycles, and makes it trivial to switch or swap out AI models without impacting downstream applications. For example, instead of sending requests in OpenAI's JSON format and then in Anthropic's distinct JSON format, the gateway can normalize requests and responses to a single schema, translating between your application's request and the specific AI model's requirement.
2. Advanced Security and Compliance: An AI Gateway offers robust security mechanisms tailored for AI interactions. This includes:
   • Authentication & Authorization: Granular access control to specific AI models or endpoints, ensuring that only authorized applications or users can invoke certain AI capabilities. This extends beyond basic API key management to support OAuth 2.0, OpenID Connect, JWTs, and other enterprise-grade authentication protocols.
   • Data Masking & Redaction: Automatically identifying and obscuring sensitive information (like PII, credit card numbers, or medical data) in both incoming prompts and outgoing AI responses before they leave the organization's control or are stored in logs. This is critical for data privacy and compliance regulations like GDPR, HIPAA, and CCPA.
   • Prompt Injection Prevention: Implementing techniques to detect and mitigate prompt injection attacks, where malicious users attempt to manipulate LLMs into misbehaving or revealing confidential information through cleverly crafted inputs. This might involve input validation, content filtering, or re-prompting strategies.
   • Content Moderation: Filtering out harmful, inappropriate, or biased content from AI-generated responses before it reaches end-users, protecting brand reputation and ensuring responsible AI usage.
   • Auditing & Logging: Comprehensive, immutable logs of all AI interactions, including requests, responses, model versions, and user details, which are essential for compliance audits, security investigations, and debugging.
3. Observability Tailored for AI: Beyond standard API metrics, an AI Gateway provides AI-specific observability:
   • Token Usage Tracking: Crucial for cost management, the gateway can accurately track input and output token counts for each LLM interaction, allowing for detailed cost allocation and analysis.
   • Latency & Error Rates (per model): Monitoring performance metrics specific to each underlying AI model, enabling quick identification of underperforming services or providers.
   • Model Versioning: Tracking which version of an AI model was used for a particular request, vital for debugging and reproducibility.
   • Semantic Logging: Enriching logs with AI-specific context, such as the intent of the prompt, the confidence score of a classification, or the specific AI features invoked.
   • Integration with Monitoring Stacks: Seamlessly exporting AI metrics and logs to popular monitoring and logging platforms (e.g., Prometheus, Grafana, ELK Stack, Splunk).
4. Intelligent Traffic Management and Optimization:
   • Dynamic Routing: Routing requests to the most appropriate AI model based on predefined rules. This could be based on cost (route to the cheapest available model), performance (route to the fastest model), model capability (route to an image generation model for image requests), geographical location, or even specific user groups. This functionality is particularly vital for an LLM Gateway that needs to balance requests across multiple LLM providers or instances.
   • Load Balancing & Failover: Distributing requests across multiple instances of an AI model or across different providers to ensure high availability and prevent any single AI service from becoming a bottleneck. If one AI provider experiences an outage, the gateway can automatically failover to an alternative.
   • Rate Limiting & Quotas (AI-aware): Applying sophisticated rate limits based on tokens per minute, requests per second, or even dollar cost per consumer/application, preventing abuse and managing consumption within provider limits.
   • Caching AI Responses: Storing and serving responses for frequently asked or deterministic AI queries (e.g., common translations, fixed knowledge base lookups) to reduce latency and save costs on repeated API calls to expensive models.
5. Prompt Engineering & Management: This is a rapidly evolving and critical area, especially for LLM Gateways:
   • Prompt Templating & Versioning: Storing, managing, and versioning prompts centrally within the gateway, allowing for easy updates and A/B testing of different prompt strategies without changing application code.
   • Dynamic Prompt Augmentation: Automatically injecting system instructions, context, or examples into user prompts before forwarding them to the LLM, ensuring consistent behavior and higher quality outputs.
   • Prompt Chaining: Orchestrating sequences of prompts to different AI models or even the same model, enabling complex multi-step AI workflows (e.g., extracting entities, then summarizing, then translating).
6. Cost Optimization:
   • Cost Visibility: Providing real-time dashboards and reports on AI expenditure broken down by model, application, user, and token type.
   • Budget Alerts: Notifying administrators when consumption thresholds are approaching or exceeded.
   • Strategic Routing for Cost Savings: Actively using dynamic routing rules to send requests to the cheapest model that meets performance and quality requirements.

In essence, an AI Gateway elevates the traditional api gateway by introducing deep awareness of AI's unique characteristics and requirements. It transforms a basic connectivity layer into an intelligent control plane that not only manages traffic but also enhances security, optimizes costs, and streamlines the development and deployment of AI-powered applications, making the dream of scalable, secure, and efficient AI a reality for enterprises.

Kong as the Premier AI Gateway Solution

When considering a platform capable of evolving into a sophisticated AI Gateway and LLM Gateway, Kong Gateway emerges as a standout choice. Rooted in its open-source heritage, Kong has established itself as a leading API Gateway solution, renowned for its performance, flexibility, and extensive plugin ecosystem. These very attributes position Kong uniquely to address the complex requirements of modern AI operations, making it an ideal candidate for unlocking AI's full potential. Kong's architecture is inherently designed for extensibility, allowing organizations to tailor its capabilities precisely to the nuanced demands of AI models and workflows, going far beyond the basic functionalities of a traditional api gateway.

Kong Gateway operates on a powerful, lightweight core, typically deployed in front of microservices or external APIs. Its architecture revolves around several key concepts:

• Services: Represent upstream APIs or microservices, pointing to their network location. In an AI context, a Service could represent an OpenAI API endpoint, an Anthropic Claude endpoint, or a custom-trained LLM running on a private server.
• Routes: Define how client requests are matched and routed to a specific Service. Routes can be configured based on paths, hostnames, HTTP methods, headers, and more. This is crucial for directing specific AI requests to the correct AI Service.
• Consumers: Represent the end-users or client applications that interact with your APIs. For AI, Consumers could be internal teams, specific applications, or external partners, each requiring potentially different access policies or rate limits.
• Plugins: This is where Kong's true power and flexibility reside. Plugins are modular components that execute specific logic on requests and responses as they pass through Kong. They can be applied globally, to specific Services, Routes, or even Consumers, allowing for highly granular control.

How Kong Extends its Capabilities for AI: The Plugin Ecosystem

Kong's plugin architecture is the cornerstone of its transformation into an AI Gateway. It allows for the injection of AI-specific logic at various stages of the request/response lifecycle, enabling advanced features that would be cumbersome or impossible with a generic api gateway.

1. Authentication and Authorization for AI Endpoints: Kong offers a rich set of authentication plugins, including key-auth (API Key), jwt (JSON Web Token), openid-connect, and oauth2. For AI services, these are vital. Imagine having different internal teams or external partners needing access to specific LLMs. Kong allows you to assign unique credentials (e.g., API keys, JWTs) to each Consumer and enforce authentication before any request reaches the AI model. For instance, you could configure a jwt plugin on an LLM Gateway Service, ensuring that only requests with a valid JWT, issued by your internal identity provider, can invoke your text generation AI. This secures access at the perimeter, preventing unauthorized use and potential data breaches.
2. Sophisticated Rate Limiting and Quotas: While traditional rate limiting (requests per second/minute) is available, Kong can be extended to implement AI-aware rate limits. With custom plugins or thoughtful configuration, you can enforce limits based on:
   • Tokens per minute: Crucial for LLMs where billing is token-based. A custom Lua plugin could inspect the request payload, estimate or parse the token count, and apply a token-based rate limit.
   • Cost per period: Limiting the total expenditure a consumer can incur on AI services within a given timeframe.
   • Concurrency limits: Preventing a single consumer from overwhelming an AI model with too many simultaneous requests. These advanced controls are essential for managing costs and ensuring fair usage across multiple consumers sharing expensive AI resources.
3. Enhanced Security Plugins for AI: Beyond authentication, Kong's security plugins can be leveraged and enhanced for AI-specific threats:
   • Input/Output Validation: Custom plugins written in Lua, Go, or WebAssembly can inspect the content of prompts and AI-generated responses. This can involve:
     • Prompt Injection Detection: Analyzing input text for patterns indicative of prompt injection attacks, blocking or sanitizing them before they reach the LLM.
     • PII Detection and Redaction: Automatically identifying sensitive data (e.g., names, addresses, credit card numbers, health information) in both incoming prompts and outgoing AI responses and replacing them with placeholders or masking them to prevent leakage.
     • Content Moderation: Scanning AI-generated text for harmful, hate speech, or inappropriate content, and blocking or modifying responses to ensure ethical AI usage.
   • Web Application Firewall (WAF) Integration: While Kong itself isn't a full WAF, it can integrate with external WAF solutions or leverage plugins for basic IP restriction and bot detection, adding another layer of defense against malicious actors targeting AI endpoints.
4. Custom Logic for AI-Specific Transformations (Prompt Engineering & Response Handling): This is where Kong's extensibility truly shines for an LLM Gateway. Custom plugins allow you to inject powerful AI-specific logic:
   • Prompt Templating and Augmentation: A plugin can dynamically modify incoming user prompts. For example, it can prepend a standard system instruction (e.g., "You are a helpful assistant...") to every user query, or inject contextual data retrieved from a database, ensuring consistent and effective interaction with the LLM. It can also manage different versions of prompts, allowing A/B testing or gradual rollout of new prompting strategies.
   • Response Parsing and Transformation: After receiving a response from an AI model, a plugin can parse, validate, or transform the output. This might involve extracting specific fields from a JSON response, converting markdown to HTML, or even invoking another AI model for a secondary task (e.g., summarization followed by sentiment analysis).
   • Error Handling and Fallback: If an AI model returns an error or takes too long to respond, a custom plugin can implement retry logic, fall back to a different AI provider, or serve a cached response, enhancing resilience.
5. Intelligent Traffic Management for AI Workloads: Kong's traffic management capabilities are critical for optimizing AI performance and cost:
   • Load Balancing Across AI Providers/Instances: Distribute requests across multiple instances of the same AI model or even different AI providers (e.g., round-robin between OpenAI and Anthropic) to improve response times and ensure high availability. If one provider experiences throttling or downtime, Kong can automatically shift traffic to another.
   • Circuit Breaking: Protect AI services from cascading failures. If an AI endpoint starts exhibiting high error rates or prolonged latency, Kong can temporarily stop sending requests to it, giving it time to recover, and preventing further degradation.
   • Blue/Green & Canary Deployments for AI Models/Prompts: Use Kong to route a small percentage of traffic to a new version of an AI model or a new prompt template, allowing you to test its performance and impact in a production environment before a full rollout. This is invaluable for iteratively improving AI capabilities without disrupting users.
   • Dynamic Routing Based on AI-Specific Logic: Beyond simple path-based routing, Kong can use custom plugins to implement intelligent routing decisions. For example:
     • Route text summarization requests to the cheapest LLM.
     • Route image generation requests to a specialized image AI.
     • Route complex customer service queries to a premium, more accurate LLM, while simpler FAQs are routed to a less expensive, faster one.
     • Route requests containing specific keywords or PII to a locally hosted, privacy-preserving model.
6. Comprehensive Observability for AI Interactions: Kong excels at providing detailed visibility, which is further enhanced for AI:
   • Integration with Monitoring & Logging Stacks: Kong seamlessly integrates with industry-standard tools like Prometheus (for metrics), Grafana (for dashboards), and popular logging aggregators such as Splunk, Elastic Stack (ELK), and Datadog. This allows organizations to collect and analyze AI-specific metrics and logs alongside their existing infrastructure data.
   • Detailed Logging of AI Requests/Responses: Kong can log every aspect of an AI interaction – the original prompt, the model invoked, the AI's response (with PII redacted), latency, token usage, and any errors. This level of detail is critical for debugging, auditing compliance, and understanding AI model behavior.
   • Tracing AI Calls End-to-End: With distributed tracing plugins, Kong can propagate trace headers, allowing you to follow a single AI request through multiple internal services and external AI models, providing an end-to-end view of its journey and identifying performance bottlenecks.
   • AI-Specific Metrics: Beyond standard HTTP metrics, Kong can be configured or extended to capture metrics like:
     • ai_model_latency_seconds_bucket (latency per specific AI model)
     • ai_token_usage_total (total tokens consumed per model, per consumer)
     • ai_error_rate_total (error rate specific to AI service responses) These metrics provide invaluable insights into the health, performance, and cost of your AI infrastructure.

By leveraging its powerful plugin architecture and robust traffic management capabilities, Kong Gateway transcends the role of a traditional api gateway. It transforms into a versatile AI Gateway and LLM Gateway, providing the essential control plane for securing, optimizing, and scaling AI interactions across the enterprise. Its open-source nature, coupled with enterprise-grade features through Konnect, ensures that organizations have the flexibility and power to meet their evolving AI operational needs.

Implementing Kong AI Gateway: A Practical Approach

Implementing Kong as an AI Gateway involves strategic deployment and configuration, moving beyond basic API proxying to leverage its advanced capabilities for AI workloads. The journey typically starts with choosing the right deployment model, followed by carefully designing services, routes, and applying a tailored set of plugins to manage AI traffic effectively. This section will explore practical deployment options, conceptual configuration examples, and compelling use case scenarios that demonstrate Kong's prowess in managing AI services.

Deployment Options for Kong AI Gateway

Kong's flexibility extends to its deployment models, allowing organizations to choose the environment that best suits their infrastructure and operational preferences:

1. On-Premises Deployment: For organizations with stringent data sovereignty requirements or existing on-premises data centers, Kong can be deployed directly on bare metal servers or virtual machines. This gives complete control over the infrastructure, but also requires managing the underlying operating system, database (PostgreSQL or Cassandra), and Kong instances. This approach is suitable for companies that prefer to keep all AI interactions within their own network, especially when dealing with highly sensitive data or proprietary AI models.
2. Containerized Deployment (Kubernetes): This is arguably the most popular and scalable deployment model for Kong. Deploying Kong as a set of Docker containers orchestrated by Kubernetes offers immense benefits for an AI Gateway:
   • Scalability: Kubernetes can automatically scale Kong instances horizontally based on traffic load, ensuring that your LLM Gateway can handle fluctuating AI request volumes.
   • Resilience: Kubernetes' self-healing capabilities ensure that if a Kong instance fails, a new one is automatically started, maintaining high availability for your AI services.
   • DevOps Synergy: Aligns perfectly with modern DevOps practices, enabling infrastructure-as-code and automated deployments.
   • Kong Ingress Controller: For Kubernetes environments, the Kong Ingress Controller allows you to manage Kong Gateway directly through Kubernetes Ingress resources, simplifying configuration and integrating seamlessly with your cluster's networking.
3. Cloud Deployment (AWS, Azure, GCP): Kong can be deployed on any major cloud provider, leveraging cloud-native services for compute (EC2, Azure VMs, GCE), databases (RDS, Azure Database for PostgreSQL, Cloud SQL), and load balancing. This offers scalability, high availability, and the ability to integrate with cloud-specific AI services directly. Cloud deployment often allows for easier geographical distribution of your AI Gateway, reducing latency for global users.
4. Managed Services (Kong Konnect): For organizations seeking to offload operational burden, Kong offers Konnect, a cloud-native, managed service that provides a global control plane for managing Kong Gateway deployments across various environments (cloud, on-premises, Kubernetes). Konnect simplifies the management of an AI Gateway by handling infrastructure, upgrades, and scaling, allowing teams to focus purely on configuring AI services and policies. It also provides advanced analytics, developer portals, and centralized policy enforcement, which are highly beneficial for managing a diverse AI landscape.

Conceptual Configuration Examples

Let's illustrate how Kong can be configured as an AI Gateway with a few conceptual examples. These are simplified to highlight the core concepts but reflect real-world possibilities.

Scenario 1: Protecting an OpenAI LLM Service with Rate Limiting and Authentication

First, define a Kong Service pointing to the OpenAI API:

# Kong Service for OpenAI's Chat Completions
apiVersion: configuration.konghq.com/v1
kind: KongService
metadata:
  name: openai-chat-service
spec:
  host: api.openai.com
  port: 443
  protocol: https
  path: /v1/chat/completions # Specific path for chat completions
  tls_verify: true
---
# Kong Route to expose the OpenAI service
apiVersion: configuration.konghq.com/v1
kind: KongRoute
metadata:
  name: openai-chat-route
spec:
  paths:
    - /ai/chat
  strip_path: true # Remove /ai/chat before forwarding
  service: openai-chat-service

Now, apply plugins for authentication and rate limiting:

# Apply Key Authentication (API Key) to the route
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: openai-key-auth
config: {}
plugin: key-auth
---
apiVersion: configuration.konghq.com/v1
kind: KongRoute
metadata:
  name: openai-chat-route
spec:
  paths:
    - /ai/chat
  strip_path: true
  service: openai-chat-service
  plugins:
    - name: key-auth # Link the key-auth plugin
      config: {}
---
# Apply Rate Limiting (e.g., 50 requests per minute per Consumer)
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: openai-rate-limit
config:
  minute: 50
  policy: local # Or 'redis' for cluster-wide limits
  limit_by: consumer
plugin: rate-limiting
---
apiVersion: configuration.konghq.com/v1
kind: KongRoute
metadata:
  name: openai-chat-route
spec:
  paths:
    - /ai/chat
  strip_path: true
  service: openai-chat-service
  plugins:
    - name: key-auth
      config: {}
    - name: rate-limiting # Link the rate-limiting plugin
      config:
        minute: 50
        policy: local
        limit_by: consumer

To use this, you would create a Kong Consumer and assign it an API key. Requests to /ai/chat would then need to include that API key in the header (e.g., apikey: YOUR_KEY) and would be limited to 50 requests per minute.

Scenario 2: Implementing a Custom Plugin for Prompt Augmentation (Conceptual Lua Plugin)

Imagine you want to ensure every prompt sent to an LLM includes a specific system instruction. You could write a custom Lua plugin:

-- custom-prompt-plugin.lua
local BasePlugin = require "kong.plugins.base_plugin"
local custom_prompt_plugin = BasePlugin:extend("custom-prompt-plugin")

function custom_prompt_plugin:new()
  custom_prompt_plugin.super.new(self, "custom-prompt-plugin")
end

function custom_prompt_plugin:access(conf)
  custom_prompt_plugin.super.access(self)

  local request_method = kong.request.get_method()
  if request_method ~= "POST" then
    return
  end

  local body_bytes, err = kong.request.get_raw_body()
  if err then
    kong.log.err("Failed to get raw body: ", err)
    return kong.response.error(500, "Internal Server Error")
  end

  local body_json = kong.json.decode(body_bytes)
  if not body_json or not body_json.messages then
    kong.log.err("Invalid request body for prompt augmentation")
    return kong.response.error(400, "Invalid AI request format")
  end

  -- Prepend a system message
  local system_message = {
    role = "system",
    content = "You are a helpful and concise AI assistant. Always strive to provide accurate and brief answers."
  }
  table.insert(body_json.messages, 1, system_message)

  -- Re-encode the body and set it for the upstream request
  kong.service.request.set_raw_body(kong.json.encode(body_json))
end

return custom_prompt_plugin

This plugin, once loaded into Kong, could be applied to your openai-chat-route. Every incoming request would have the system message prepended, ensuring consistent LLM behavior without modifying client applications.

Use Case Scenarios for Kong AI Gateway

The strategic implementation of Kong as an AI Gateway unlocks a multitude of powerful use cases:

1. Developer Empowerment and Standardized AI Access: Instead of developers needing to understand and integrate with N different AI APIs, they interact with a single, unified LLM Gateway exposed by Kong. This dramatically simplifies AI integration, reduces cognitive load, and speeds up development cycles. Kong provides a consistent interface, allowing developers to focus on building features rather than managing API variations. With Kong Konnect's developer portal, API consumers can self-serve, discovering available AI services, reading documentation, and obtaining credentials without direct intervention from platform teams.
2. Strategic Cost Optimization for LLM Usage: With Kong as an AI Gateway, organizations can implement intelligent routing rules to direct AI requests to the most cost-effective model or provider. For instance, a plugin could analyze the complexity or length of a prompt: simple, short queries might be routed to a cheaper, smaller LLM, while complex analytical tasks requiring higher accuracy are directed to a premium, more expensive model. Additionally, by caching responses for frequently asked questions, Kong can reduce redundant calls to expensive external AI services, leading to significant cost savings. The ability to track token usage per consumer provides granular billing and budgeting capabilities.
3. Enhanced Security and Data Governance for AI Interactions: Kong's robust security features are paramount for AI. By placing the AI Gateway at the edge, organizations can:
   • Prevent Unauthorized Access: Only authenticated and authorized applications can interact with AI models.
   • Protect Sensitive Data: Implement data masking plugins to redact PII or confidential information from prompts and responses, preventing data leakage to external AI providers or insecure logs.
   • Mitigate AI-Specific Attacks: Custom plugins can detect and block prompt injection attempts, ensuring LLMs behave as intended and don't reveal sensitive internal information.
   • Ensure Compliance: Comprehensive logging and auditing capabilities provide a clear trail of all AI interactions, essential for demonstrating compliance with privacy regulations (GDPR, HIPAA, CCPA).
4. Superior Performance and Reliability of AI Services: Kong enhances the operational resilience of AI-powered applications. Load balancing across multiple AI model instances or providers ensures high availability and distributes traffic efficiently, preventing any single point of failure. Circuit breakers protect downstream AI services from being overwhelmed by aberrant client behavior. Caching mechanisms dramatically reduce latency for repeated requests, providing a snappier user experience. For example, if your primary LLM provider experiences an outage, Kong can automatically failover to a secondary provider or a locally hosted model, maintaining service continuity.
5. Seamless A/B Testing and Iterative Improvement of AI Models/Prompts: The AI Gateway becomes a powerful tool for experimentation. Kong allows organizations to conduct A/B tests on different AI models, model versions, or even different prompt strategies. You can route a small percentage of user traffic to a new LLM or a refined prompt template, collect metrics on performance, quality, and user satisfaction, and then gradually roll out the best-performing option. This iterative approach is crucial for continuous improvement of AI capabilities without impacting all users. For instance, a traffic-split plugin could direct 10% of requests to a new prompt version, comparing its output quality and latency against the control.
6. Building a Multi-Cloud/Hybrid AI Strategy: Many enterprises adopt multi-cloud strategies to avoid vendor lock-in and leverage specialized services. An AI Gateway like Kong abstracts away the underlying cloud-specific AI APIs (e.g., Azure OpenAI, AWS Bedrock, Google Cloud AI). This allows organizations to switch between providers, deploy AI models on-premises, or combine them in a hybrid fashion without re-architecting their applications. The gateway acts as a flexible adapter, enabling true vendor agnosticism and maximizing flexibility in AI model selection and deployment.

By strategically deploying and configuring Kong Gateway, organizations transform a traditional api gateway into a dynamic, intelligent AI Gateway. This move is not merely an architectural upgrade; it's a fundamental shift in how enterprises manage, secure, and leverage their AI investments, driving efficiency, innovation, and competitive advantage.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Advanced AI Gateway Capabilities with Kong

Beyond the foundational aspects of security, traffic management, and observability, Kong as an AI Gateway can be extended to implement highly sophisticated functionalities that are critical for modern, enterprise-grade AI operations. These advanced capabilities transform Kong into a truly intelligent orchestration layer, deeply integrated with the AI development lifecycle and focused on optimizing every facet of AI interaction.

1. Advanced Prompt Management and Orchestration

For Large Language Models (LLMs), prompts are the new code. Managing them effectively is paramount. Kong, as an LLM Gateway, can provide robust prompt management:

• Prompt Versioning and Rollback: Store and manage different versions of prompts centrally within Kong. This allows teams to iterate on prompts, experiment with different phrasing or instructions, and easily roll back to a previous version if a new prompt degrades performance or output quality. Instead of hardcoding prompts in application logic, they become configurable assets managed at the gateway level.
• Dynamic Prompt Templating: Instead of static prompts, Kong can dynamically inject variables or context into prompts based on incoming request parameters, user profiles, or data retrieved from external systems. For example, a plugin could fetch user preferences or historical interaction data from a database and inject it into the LLM prompt, making the AI's response more personalized and relevant.
• Prompt Chaining and Flow Orchestration: For complex tasks, a single LLM call might not suffice. Kong can orchestrate a sequence of AI interactions. For instance, an initial prompt could go to one LLM for entity extraction, the output of which is then used to construct a second prompt for a different LLM (or a specialized model) for summarization, and finally, the summary might be sent to a translation model. This multi-stage AI workflow is seamlessly managed by the gateway, abstracting the complexity from the consuming application. This significantly streamlines complex AI pipelines, which might otherwise require bespoke microservices or intricate client-side logic.
• Prompt Guards and Sanitization: Implement proactive measures to filter out malicious or unintentionally harmful content from user prompts before they even reach the LLM. This could involve using a smaller, specialized AI model at the gateway level to check for prompt injection patterns, hate speech, or sensitive data, enhancing the security posture even further.

2. Intelligent Response Handling and Transformation

The output from AI models often needs further processing before being consumed by applications or end-users. Kong can manage this crucial step:

• Response Content Filtering and Moderation: Beyond filtering prompts, the AI Gateway can inspect AI-generated responses for undesirable content (e.g., toxicity, bias, non-compliance with brand guidelines) and automatically modify, redact, or block the response. This acts as a critical safety net, ensuring that only appropriate content is delivered to users.
• AI Response Caching with TTL: For queries that frequently recur and produce deterministic or nearly deterministic AI outputs (e.g., common customer support FAQs, fixed data lookups), Kong can cache the AI responses for a configured Time-To-Live (TTL). This drastically reduces latency for subsequent identical requests and significantly cuts down on API call costs to expensive external AI services. The caching strategy can be fine-tuned based on the prompt's characteristics or the expected variability of the AI's response.
• Response Transformation and Normalization: AI models can return responses in varying formats (e.g., plain text, JSON, XML, custom data structures). Kong can transform these outputs into a standardized format expected by the consuming application, reducing the integration burden on developers. This might involve parsing an LLM's natural language response into structured JSON or extracting specific data points for further processing.
• Error Enrichment and Fallback Responses: If an AI model returns an error, Kong can enrich the error message with additional context, provide a more user-friendly error explanation, or serve a predefined fallback response (e.g., "AI service is temporarily unavailable, please try again later") instead of simply propagating the raw AI service error. This improves the robustness and user experience of AI-powered applications.

3. Granular Cost Tracking and Financial Optimization

Managing the financial aspects of AI usage is a major concern. Kong, as an AI Gateway, provides powerful tools for this:

• Detailed Token Usage Accounting: Track input and output tokens for every LLM interaction, broken down by individual Consumer, Service, Route, or custom metadata. This provides unparalleled visibility into AI consumption patterns.
• Real-time Cost Aggregation and Reporting: Aggregate token usage data and convert it into estimated dollar costs based on current AI provider pricing. Integrate with monitoring dashboards (e.g., Grafana) to display real-time and historical cost trends, allowing finance and operations teams to monitor expenditures.
• Budget Enforcement and Alerts: Configure budget thresholds for specific teams, applications, or projects. Kong can then trigger alerts (e.g., via Slack, email, PagerDuty) when these budgets are approaching or exceeded, preventing unexpected cost overruns.
• Dynamic Cost-Based Routing: Implement sophisticated routing rules that prioritize cost-efficiency. For instance, during peak hours, route to a faster but slightly more expensive model, but switch to a slower, cheaper model during off-peak hours. Or, for non-critical tasks, always default to the most economical AI provider, falling back to a premium one only if the cheaper option fails or doesn't meet quality requirements.

4. Comprehensive Data Governance and Compliance Audit Trails

Ensuring responsible AI usage and adherence to regulatory mandates is critical. Kong strengthens this aspect:

• Immutable Audit Logs: Capture every detail of AI interactions, including the full request and (redacted) response payloads, timestamps, originating IP addresses, consumer identity, model used, and any plugins applied. These logs are invaluable for compliance audits, security investigations, and debugging.
• Data Residency Enforcement: For multinational corporations, data residency is a key concern. Kong can enforce rules to ensure that AI requests from users in a specific geographical region are only routed to AI models hosted within that same region, complying with local data privacy laws.
• Consent Management Integration: Integrate with consent management platforms. Kong can ensure that AI services only process user data for which explicit consent has been obtained, or apply specific data handling policies based on user consent preferences.
• Transparency and Explainability (Limited): While Kong doesn't provide intrinsic AI explainability, its detailed logging of prompts, model choices, and transformations contributes to greater transparency about how an AI response was generated and which inputs contributed to it, aiding in potential debugging or compliance inquiries.

By implementing these advanced capabilities, Kong transcends the role of a basic proxy and becomes a strategic asset in an organization's AI ecosystem. It provides the centralized intelligence and control necessary to operationalize AI responsibly, cost-effectively, and at scale, transforming fragmented AI services into a cohesive, secure, and highly performant AI platform.

The Broader Ecosystem and Future of AI Gateways

The rapid evolution of AI necessitates a robust and adaptable infrastructure, and the AI Gateway sits at the nexus of this transformation. While Kong offers a powerful and extensible platform for building an AI Gateway, it's important to recognize that it operates within a broader ecosystem of tools and platforms, each contributing to a complete AI operational framework. Understanding this ecosystem, and where other innovative solutions like APIPark fit in, provides a comprehensive view of the future of AI management.

The journey of an AI model, from data ingestion and training to deployment and consumption, involves multiple stages and specialized tools. Kong, as an AI Gateway, primarily focuses on the consumption and management of deployed AI models. It acts as the intelligent proxy that secures, optimizes, and orchestrates access to these models for various applications and users. However, AI operations also involve MLOps platforms for model training and versioning, data governance tools for managing training data, and observability platforms for deeper AI model performance analysis. The ideal scenario involves seamless integration between these components, with the AI Gateway serving as the critical enforcement point at the interaction layer.

This holistic view underscores the need for comprehensive solutions in the AI space. One such compelling offering is APIPark, which provides an open-source AI Gateway and API Management Platform. While Kong (especially with its AI-specific plugins) focuses intensely on the gateway functionality, APIPark positions itself as an all-in-one solution that combines robust AI gateway capabilities with a full-fledged API developer portal, all under the Apache 2.0 license. This makes APIPark a particularly attractive option for developers and enterprises looking for a streamlined approach to manage, integrate, and deploy both AI and REST services with exceptional ease.

APIPark's feature set directly addresses many of the challenges discussed earlier, often providing out-of-the-box solutions that complement or enhance the customizability offered by Kong's plugin system:

• Quick Integration of 100+ AI Models: Similar to a core AI Gateway function, APIPark emphasizes rapid integration with a diverse range of AI models. It offers a unified management system for authentication and cost tracking across these models, significantly reducing the integration overhead for developers dealing with various AI providers. This feature directly aligns with the need for abstraction and standardization in AI consumption.
• Unified API Format for AI Invocation: A cornerstone of any effective LLM Gateway, APIPark standardizes the request data format across all AI models. This means applications can send requests in a consistent manner, regardless of the underlying AI model's specific API. The benefit is profound: changes in AI models or prompts do not necessitate changes in the application or microservices, drastically simplifying AI usage and reducing long-term maintenance costs.
• Prompt Encapsulation into REST API: This is a particularly innovative feature. APIPark allows users to quickly combine AI models with custom prompts to create new, specialized APIs. For instance, a user can define a prompt for sentiment analysis and expose it as a dedicated REST API endpoint. This democratizes prompt engineering, enabling domain experts to create AI-powered capabilities that can be easily consumed by other applications without deep AI expertise.
• End-to-End API Lifecycle Management: Going beyond just AI, APIPark provides comprehensive tools for managing the entire lifecycle of APIs, encompassing design, publication, invocation, and decommission. This helps organizations regulate API management processes, manage traffic forwarding, load balancing, and versioning of published APIs—critical aspects for both AI and traditional REST services.
• API Service Sharing within Teams: The platform facilitates centralized display of all API services, making it effortless for different departments and teams to discover and utilize required API services. This fosters internal collaboration and accelerates innovation by making AI capabilities readily accessible across the enterprise.
• Independent API and Access Permissions for Each Tenant: APIPark supports multi-tenancy, allowing for the creation of multiple teams (tenants), each with independent applications, data, user configurations, and security policies. This is vital for large organizations or SaaS providers, allowing them to share underlying infrastructure while maintaining strict separation and customization for each tenant, improving resource utilization and reducing operational costs.
• API Resource Access Requires Approval: To enhance security and governance, APIPark allows for the activation of subscription approval features. Callers must subscribe to an API and await administrator approval before they can invoke it, preventing unauthorized API calls and potential data breaches—a critical safeguard for sensitive AI resources.
• Performance Rivaling Nginx: Performance is non-negotiable for an AI Gateway. APIPark boasts impressive performance, capable of achieving over 20,000 TPS with an 8-core CPU and 8GB of memory, and supports cluster deployment for large-scale traffic handling. This demonstrates its readiness for demanding AI workloads.
• Detailed API Call Logging and Powerful Data Analysis: Comprehensive logging capabilities, recording every detail of each API call, are crucial for debugging, auditing, and troubleshooting. APIPark complements this with powerful data analysis features that analyze historical call data, displaying long-term trends and performance changes. This predictive analytics helps businesses with preventive maintenance, identifying potential issues before they impact operations—a significant advantage for proactive AI management.

APIPark's ability to be quickly deployed in just 5 minutes with a single command line ( curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh ) underscores its commitment to developer experience and rapid adoption. While the open-source product caters to startups' basic API resource needs, a commercial version with advanced features and professional technical support is available for leading enterprises, demonstrating a commitment to supporting diverse organizational requirements. Backed by Eolink, a leader in API lifecycle governance, APIPark brings a wealth of experience in API management to the burgeoning field of AI operations.

The future of AI Gateways will likely see even deeper integration with MLOps pipelines, more sophisticated AI-driven routing and optimization, and increased emphasis on ethical AI and compliance features built directly into the gateway. Self-optimizing AI routing, where the gateway itself uses AI to determine the best model based on real-time performance, cost, and contextual data, is an exciting frontier. The evolution will continue towards gateways that are not just proxies but intelligent orchestrators, actively participating in the decision-making process of AI interactions. Choosing the right AI Gateway – whether a highly customizable platform like Kong or a comprehensive, open-source solution like APIPark – will be paramount for organizations aiming to build scalable, secure, and intelligent AI ecosystems that truly unlock their potential. The journey to fully harness AI is ongoing, and the AI Gateway will remain a central, indispensable component of this transformative expedition.

Conclusion

The era of Artificial Intelligence is defined not just by technological breakthroughs but by the operational paradigms that enable their secure, efficient, and scalable deployment. As organizations increasingly integrate diverse AI models, from powerful LLM Gateways to specialized machine learning services, the complexities of managing these interactions amplify significantly. The traditional API Gateway, while fundamental for RESTful services, proves insufficient for the unique demands of AI—demands rooted in security, cost optimization, performance, and nuanced data governance. This is precisely where the AI Gateway emerges as an indispensable architectural component, providing an intelligent control plane that orchestrates every aspect of AI interaction.

Kong Gateway, with its robust open-source foundation, extensible plugin architecture, and enterprise-grade offerings, is uniquely positioned to serve as a premier AI Gateway. It transcends the role of a simple proxy by offering a comprehensive suite of functionalities specifically tailored for AI workloads. From sophisticated authentication and authorization mechanisms that secure access to sensitive AI models, to intelligent rate limiting and dynamic routing that optimize costs and performance across a heterogeneous mix of AI providers, Kong empowers organizations to exert granular control over their AI consumption. Its powerful plugin ecosystem facilitates the implementation of advanced features like prompt engineering, data masking, content moderation, and comprehensive observability, ensuring that AI deployments are not only efficient but also compliant and responsible.

By transforming Kong into an AI Gateway, enterprises can unlock myriad benefits. Developers are empowered with a unified, simplified interface to consume AI services, drastically accelerating application development. Business leaders gain unprecedented visibility into AI costs and performance, enabling strategic optimization and informed decision-making. Security and operations teams are equipped with powerful tools to protect sensitive data, mitigate AI-specific threats like prompt injection, and ensure continuous availability of critical AI-powered applications. Furthermore, the ability to conduct A/B testing of AI models and prompts, alongside flexible multi-cloud and hybrid AI strategies, fosters continuous innovation and resilience.

In a landscape where solutions like APIPark further highlight the innovation in open-source AI Gateway and API management platforms, the choice of an AI Gateway becomes a strategic imperative. Whether leveraging Kong's deep extensibility or APIPark's comprehensive, out-of-the-box feature set for rapid integration and full API lifecycle management, the goal remains the same: to provide a robust, secure, and scalable foundation for all AI initiatives. The future success of AI-driven enterprises hinges on their ability to manage, secure, and optimize their AI interactions effectively. An intelligently implemented AI Gateway is not just an architectural component; it is the central nervous system that empowers organizations to safely, efficiently, and innovatively harness the full, transformative potential of Artificial Intelligence.

Comparison of Core AI Gateway Features

Feature Category	Traditional API Gateway (e.g., Basic Kong)	AI Gateway (Kong with AI plugins, APIPark)	Benefits for AI Operations
Core Functionality	Routing, Authentication, Rate Limiting, Load Balancing	All traditional + AI-specific routing, prompt management, cost tracking, data masking	Simplified integration of diverse AI models, comprehensive control over AI interactions.
AI Model Support	Treats AI endpoints as any REST API	Unified interface for 100+ diverse AI models (LLMs, vision, etc.)	Reduces development complexity, enables easy switching between AI providers, fosters vendor agnosticism.
Security	API Key, JWT, OAuth; basic WAF	Advanced AI-aware PII masking/redaction, prompt injection prevention, content moderation, granular AI access control	Protects sensitive data, prevents AI abuse, ensures compliance with privacy regulations (GDPR, HIPAA).
Cost Management	General request limits	Token usage tracking, cost-based routing, budget alerts, intelligent caching for AI responses	Significant cost savings on expensive AI models, granular cost allocation, prevents unexpected overruns.
Performance	Request/response latency, general load balancing	AI-specific latency/error rates, dynamic load balancing for AI, intelligent AI response caching, circuit breaking	Ensures high availability and low latency for AI-powered applications, improves user experience.
Observability	HTTP access logs, basic metrics	AI-specific logging (prompts, responses, tokens), model version tracking, AI metrics (token usage, AI error rates)	Faster debugging of AI issues, better understanding of AI behavior, compliance auditing, proactive issue detection.
Prompt Management	N/A (prompts embedded in client code)	Centralized prompt templating, versioning, A/B testing, dynamic prompt augmentation, prompt chaining	Consistent AI interactions, faster iteration on AI capabilities, reduced client-side prompt logic, enabling complex AI workflows.
Data Governance	Basic logging, access control	PII redaction, data residency enforcement, immutable audit trails for AI interactions, consent management integration	Ensures legal and ethical use of AI, simplifies compliance reporting, builds trust in AI systems.
Deployment Ease	Standard API deployment	Often simplified for AI (e.g., APIPark's quick-start script), integrates with MLOps	Accelerates deployment of AI services, reduces operational overhead for AI infrastructure.
Ecosystem & Value	Foundation for microservices	Core component of enterprise AI strategy, integrates with broader MLOps, offers developer portal for AI services (e.g., APIPark)	Boosts developer productivity, provides a scalable and secure foundation for AI innovation, enables strategic AI investment.

5 FAQs

Q1: What is the primary difference between a traditional API Gateway and an AI Gateway? A1: A traditional API Gateway primarily handles routing, authentication, and rate limiting for general RESTful APIs. An AI Gateway, while built on these fundamentals, extends its capabilities with AI-specific features such as token-based rate limiting, AI-aware data masking (PII redaction in prompts/responses), intelligent routing based on AI model cost or performance, prompt management (templating, versioning), and advanced observability tailored for AI interactions. It's designed to manage the unique complexities and security demands of AI services, especially Large Language Models (LLMs).

Q2: How does an AI Gateway help with cost optimization for LLMs? A2: An AI Gateway significantly aids cost optimization for LLMs by providing granular token usage tracking for both input and output, allowing organizations to monitor and attribute costs precisely. It enables intelligent routing rules that direct requests to the most cost-effective LLM for a given task (e.g., cheaper models for simple queries). Additionally, it can implement caching of AI responses for frequently asked questions, drastically reducing the number of repeated calls to expensive external LLMs, thereby directly saving costs.

Q3: Can an AI Gateway protect against prompt injection attacks? A3: Yes, a robust AI Gateway can implement various mechanisms to protect against prompt injection attacks. This includes using custom plugins or integrated tools to analyze incoming prompts for malicious patterns or keywords before they reach the LLM. It can perform input validation, content filtering, or even use smaller, specialized AI models at the gateway level to detect and mitigate such attempts, ensuring the LLM behaves as intended and doesn't reveal sensitive information.

Q4: How does Kong Gateway become an effective AI Gateway? A4: Kong Gateway leverages its highly extensible plugin architecture to become an effective AI Gateway. It can utilize existing plugins for authentication, rate limiting, and traffic management, and further extend these with AI-specific plugins (often custom-developed in Lua, Go, or WebAssembly) for functionalities like PII redaction, prompt manipulation (templating, augmentation), AI-aware load balancing, cost tracking, and detailed AI interaction logging. Its routing capabilities allow for intelligent traffic direction to various AI models based on custom logic, performance, or cost.

Q5: What role does APIPark play in the AI Gateway landscape? A5: APIPark is an open-source AI Gateway and API management platform that offers an all-in-one solution for managing both AI and REST services. It provides quick integration with 100+ AI models, a unified API format for AI invocation, prompt encapsulation into REST APIs, and end-to-end API lifecycle management. APIPark stands out for its ease of deployment, robust performance, comprehensive logging, and powerful data analysis, offering a streamlined, open-source approach to unlocking AI potential, particularly appealing for organizations seeking a complete API and AI management solution.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.