Mastering eBPF Packet Inspection in User Space

Modern computing infrastructures, characterized by highly distributed microservices, containerization, and rapidly escalating data volumes, demand an unprecedented level of visibility and control over network traffic. The traditional approaches to network monitoring and security, often relying on user-space tools that interact with the kernel through high-latency interfaces or static kernel modules, are increasingly proving inadequate for the dynamic, high-performance environments of today. The inherent overheads associated with moving vast amounts of packet data from the kernel to user space for analysis can bottleneck even the most robust systems, obscuring critical insights and delaying incident response. This persistent challenge has spurred a relentless pursuit of more efficient, powerful, and flexible mechanisms for observing and manipulating network packets at their source.

Enter eBPF (extended Berkeley Packet Filter), a revolutionary technology that has fundamentally transformed how we interact with the Linux kernel. No longer merely a simple packet filtering mechanism, eBPF has evolved into a powerful, in-kernel virtual machine capable of executing custom programs safely and efficiently across various kernel event points. Its profound impact on networking, security, and observability stems from its ability to extend the kernel's functionality without requiring module compilation or kernel modifications, offering a programmable data plane that was once the exclusive domain of specialized hardware. For the discerning network engineer, security analyst, or system administrator, eBPF presents an unparalleled opportunity to gain deep, granular insights into network traffic with minimal overhead, directly at the source. However, the true power of eBPF packet inspection isn't just in what it can do inside the kernel, but how effectively it can communicate those vital observations and processed data back to user-space applications for comprehensive analysis, alerting, and action.

This article embarks on an extensive journey to master eBPF packet inspection, with a particular emphasis on the critical techniques for efficiently transferring processed network data from the kernel to user space. We will dissect the architectural underpinnings of eBPF, explore its diverse attachment points within the network stack, and meticulously examine the sophisticated mechanisms—such as BPF maps, perf buffers, ring buffers, and AF_XDP sockets—that bridge the kernel-user space divide. By understanding and strategically applying these concepts, readers will be equipped to build robust, high-performance network analysis tools, bolster system security, and unlock new frontiers in performance optimization, ultimately achieving unparalleled network observability and control in complex, modern environments.

The Foundational Pillars of eBPF: A Paradigm Shift in Kernel Programmability

To truly master eBPF packet inspection, one must first grasp its fundamental principles and architectural marvels. eBPF is not simply a tool; it is a paradigm shift in how we extend and interact with the Linux kernel. At its core, eBPF is a highly efficient, in-kernel virtual machine that allows users to run sandboxed programs within the kernel without altering kernel source code or loading potentially unstable kernel modules. This capability unlocks a new era of flexibility, allowing custom logic to be injected into various kernel subsystems, from networking to tracing, security, and more.

The lineage of eBPF traces back to the classic Berkeley Packet Filter (cBPF), originally designed in the early 1990s to efficiently filter packets for tools like tcpdump. cBPF provided a simple bytecode interpreter in the kernel, allowing network captures to be offloaded and processed more efficiently. However, cBPF was limited in its expressiveness and capabilities. eBPF represents a significant evolution, expanding the instruction set, registers, and memory model, thereby transforming a simple filter into a general-purpose, programmable engine. This transformation has empowered developers to craft sophisticated programs that can dynamically react to kernel events, process data, and share information with user-space applications with unprecedented agility and performance.

Core Principles and Architectural Components

The efficacy and robustness of eBPF are underpinned by several critical architectural components:

1. eBPF Programs: These are the heart of the eBPF ecosystem. Written typically in a subset of C, these programs are compiled into BPF bytecode using specialized compilers like LLVM/Clang. Unlike traditional user-space programs, eBPF programs are event-driven and relatively small, designed to execute quickly at specific points within the kernel. They are loaded into the kernel, where they are subject to rigorous verification before execution. The versatility of eBPF programs allows them to perform a wide array of tasks, from simple packet dropping and counting to complex stateful flow analysis and custom load balancing.
2. eBPF Maps: Maps are essential data structures that facilitate communication and state sharing within the eBPF ecosystem. They serve as efficient key-value stores, residing in kernel memory, which can be accessed by eBPF programs, other eBPF programs, and user-space applications. This duality is crucial for packet inspection: eBPF programs can store aggregated statistics, connection states, or filtered data in maps, while user-space applications can read from or write to these same maps to retrieve insights, configure program behavior, or inject policy rules. A diverse range of map types exists, each optimized for different use cases, including hash maps, array maps, Lru hash maps, and specialized maps like perf_event_array and ringbuf for asynchronous data streaming. The ability to share data between kernel and user space via maps is a cornerstone of effective eBPF-based network monitoring and observability.
3. eBPF Helpers: To interact with the kernel context and perform operations that go beyond simple data manipulation, eBPF programs rely on a set of kernel-provided helper functions. These helpers offer a safe and controlled interface for eBPF programs to perform tasks like looking up or updating map elements, getting current time, generating random numbers, interacting with network packets (e.g., bpf_skb_load_bytes, bpf_xdp_adjust_head), and printing debug messages (bpf_printk). The specific set of available helpers varies depending on the program type and kernel version, but they are instrumental in unlocking the full potential of eBPF programs.
4. The eBPF Verifier: Safety is paramount when executing custom code within the kernel. The eBPF verifier is a critical security component that statically analyzes every eBPF program before it is loaded and executed. Its role is to ensure that the program is safe, will terminate, does not contain infinite loops, does not access invalid memory locations, and does not crash the kernel. The verifier checks against maximum instruction limits, verifies pointer arithmetic, and ensures that all access to kernel memory is legitimate. This stringent verification process is what makes eBPF a secure and trusted mechanism for extending kernel functionality, a key differentiator from traditional kernel modules that can arbitrarily access memory and potentially destabilize the system.
5. The JIT Compiler (Just-In-Time Compiler): For optimal performance, once an eBPF program passes the verifier, it is often translated by a Just-In-Time (JIT) compiler into native machine code. This eliminates the overhead of interpreting bytecode, allowing eBPF programs to execute at speeds comparable to natively compiled kernel code. Most modern Linux architectures (x86, ARM, PowerPC, etc.) have JIT compilers for eBPF, ensuring that performance is not compromised by the flexibility of dynamic program loading. This combination of safety (verifier) and speed (JIT) is what makes eBPF so compelling for high-performance applications like network packet processing.

Why eBPF is a Game-Changer for Packet Inspection

The unique architecture of eBPF makes it exceptionally well-suited for network packet inspection and manipulation:

• Unprecedented Visibility: eBPF programs can attach to virtually any point in the network stack, from the earliest possible ingress point in the network driver (XDP) to later stages in the traffic control layer (TC BPF) or even directly to sockets. This allows for an extremely fine-grained view of packet flow, enabling developers to observe, filter, and modify packets at precise locations within the kernel.
• Minimal Overhead: By processing packets directly in the kernel and only transferring relevant metadata or summarized statistics to user space, eBPF significantly reduces the overhead associated with traditional packet capture methods. Operations like filtering, dropping, or redirecting packets can occur before they consume significant kernel resources, leading to higher throughput and lower latency.
• Flexibility and Programmability: Unlike fixed kernel functionalities, eBPF allows for dynamic, custom logic. Security policies, load balancing algorithms, and telemetry collection can be programmed and updated on the fly without rebooting the system or recompiling the kernel. This flexibility is invaluable in rapidly evolving network environments.
• Enhanced Security: By enabling deep inspection and filtering within the kernel, eBPF can power advanced security solutions, such as dynamic firewalls, intrusion detection, and DDoS mitigation, with unmatched performance and granularity. The ability to drop malicious traffic at the earliest possible stage, even before it hits the main network stack, is a powerful defensive capability.

In essence, eBPF provides a robust, secure, and high-performance framework for extending kernel capabilities, making it an indispensable tool for anyone looking to gain mastery over network packet inspection and the broader domain of system observability. The foundation laid by these core principles is what enables the sophisticated data transfer mechanisms and user-space applications we will explore in subsequent sections.

eBPF Attachment Points for Granular Network Data Interception

The effectiveness of eBPF packet inspection hinges on its ability to attach programs at strategic points within the Linux network stack. Each attachment point offers distinct advantages, trade-offs, and use cases, allowing developers to precisely target their packet processing logic where it will have the most impact and provide the most relevant data. Understanding these various hooks is crucial for designing efficient and performant eBPF-based network solutions.

The Linux network stack is a complex beast, comprising multiple layers and functions that a packet traverses from its arrival at the network interface card (NIC) to its delivery to a user-space application, and vice versa. eBPF provides hooks across this entire journey, allowing for truly fine-grained control and observation.

XDP (eXpress Data Path): The Front Line of Packet Processing

XDP is arguably the most revolutionary eBPF attachment point for high-performance network packet processing. It allows eBPF programs to run directly within the network driver, at the earliest possible point after a packet arrives at the NIC, even before the kernel's full network stack begins processing it. This "zero-copy" architecture means packets can be inspected, dropped, or redirected without ever being copied into a kernel sk_buff structure or traversing the entire network stack. This minimizes CPU cycles, reduces latency, and maximizes throughput, making XDP ideal for highly demanding scenarios.

• Deep Dive into XDP: When a network interface card receives a frame, it typically stores it in a DMA ring buffer. An XDP program attaches to this ring, receiving a pointer to the raw packet data. At this stage, the packet is as close to hardware as possible within the kernel context. The eBPF program then processes this raw data.
• Use Cases: XDP excels in scenarios where extreme performance and low latency are paramount.
  • DDoS Mitigation: Malicious traffic can be identified and dropped at line rate, preventing it from consuming valuable kernel resources or impacting legitimate services.
  • High-Performance Load Balancing: XDP can implement sophisticated load balancing algorithms, redirecting incoming packets to appropriate backend servers (e.g., using XDP_REDIRECT to a different NIC or to a AF_XDP socket in user space) before they even enter the main IP stack.
  • Stateless Firewalling: Dropping packets based on simple rules (e.g., source IP, destination port) with minimal overhead.
  • Packet Forwarding: Implementing custom routing logic or even building a fast path for specific types of traffic.
  • Network Telemetry: Extracting headers or specific packet fields for analysis, sending metadata to user space without copying entire packets.
• Zero-Copy Packet Processing: The key differentiator of XDP is its ability to operate on packets in place in the NIC's receive ring. This avoids costly memory allocations and data copying that are typical in later stages of the network stack.
• XDP Actions: After an XDP program processes a packet, it returns an action code instructing the kernel what to do next:
  • XDP_PASS: Allow the packet to continue its journey up the normal network stack.
  • XDP_DROP: Discard the packet immediately.
  • XDP_TX: Transmit the packet back out the same NIC, useful for reflection attacks or specialized loopback.
  • XDP_REDIRECT: Redirect the packet to another network interface or to a user-space socket via AF_XDP.
  • XDP_ABORTED: An error occurred in the XDP program, and the packet is dropped. The choice of action directly influences the packet's fate and the efficiency of the network operation.

TC BPF (Traffic Control BPF): Integrated Network Stack Management

While XDP offers raw speed at the driver level, TC BPF provides more traditional network stack integration and context. Traffic Control (TC) is a long-standing Linux kernel subsystem responsible for managing network queues, shaping traffic, and applying various rules. eBPF programs can be attached to the ingress (incoming) or egress (outgoing) queues of network interfaces as part of the TC classification process.

• Attachment to Ingress/Egress Queues: TC BPF programs operate on sk_buff structures, which encapsulate the packet along with rich metadata that the kernel has already attached (e.g., destination cache, routing information, timestamps). This provides a more comprehensive view of the packet within the kernel's network processing context compared to XDP's raw packet view.
• Use Cases: TC BPF is suitable for more sophisticated, context-aware network operations.
  • Fine-Grained Traffic Shaping and QoS: Prioritizing certain types of traffic, limiting bandwidth for others, or ensuring minimum bandwidth guarantees.
  • Advanced Firewalling: Implementing stateful firewall rules, inspecting transport layer headers, or even application-layer information, leveraging the sk_buff context.
  • Detailed Packet Manipulation: Modifying packet headers, encapsulating/decapsulating tunnels, or rewriting addresses based on complex logic.
  • Network Service Mesh Integration: Providing fine-grained policy enforcement and observability within containerized environments.
• Comparison with XDP:
  • Placement: XDP is pre-network stack (driver level); TC BPF is within the network stack (traffic control layer).
  • Context: XDP has minimal context (raw packet data); TC BPF has rich sk_buff context.
  • Performance: XDP generally offers higher raw throughput due to zero-copy; TC BPF might have slightly higher overhead but provides more context.
  • Flexibility: Both are highly flexible, but TC BPF allows deeper interaction with existing kernel network functionalities. Choosing between XDP and TC BPF often depends on the specific performance requirements and the depth of kernel network stack interaction needed for the packet inspection or manipulation task.

Socket Filters (SO_ATTACH_BPF): Application-Specific Packet Control

eBPF programs can also be attached directly to sockets, influencing how a specific application's network traffic is processed. This is achieved using the SO_ATTACH_BPF socket option, allowing a user-space application to apply a custom eBPF filter to its own socket. This mechanism has been around since the early days of classic BPF, famously used by tcpdump and wireshark to efficiently filter packets before they are copied to user space.

• Attaching to Sockets: When an eBPF program is attached to a socket, it acts as a filter for packets attempting to be delivered to that socket. Only packets that pass the filter will be buffered for the application to read.
• Use Cases:
  • Application-Specific Packet Filtering: A web server could attach a filter to its listening socket to drop malformed requests or specific types of attack traffic before they are processed by the application logic.
  • Custom Network Stack Behavior: An application could customize how it receives packets, allowing only certain protocols or payload structures.
  • Security for Specific Processes: Isolating a sensitive application by giving it its own eBPF-based firewall directly on its sockets.
  • High-Performance Packet Capture for Specific Applications: Tools needing to monitor a particular application's traffic can use socket filters to get only relevant packets.

sockmap and sockhash: Advanced Socket Steering and Optimization

Beyond simple filtering, eBPF introduces powerful mechanisms like sockmap and sockhash for highly optimized socket management and inter-process communication. These map types allow eBPF programs to steer connections between sockets or enable efficient lookup of sockets based on connection properties, drastically reducing context switches and improving latency.

• sockmap: Allows an eBPF program to create a "map" of sockets. An eBPF program can then directly redirect data from one socket in the map to another, or from a network interface to a socket in the map. This is particularly useful for building high-performance proxies or message queues, where data needs to move between different processing stages efficiently without unnecessary round trips through the full network stack.
• sockhash: A hash map for storing sockets, keyed by connection tuples (e.g., source IP, destination IP, ports). An eBPF program can quickly look up an existing socket for a given connection and redirect packets to it, enabling highly efficient connection steering, load balancing, and connection reuse.
• Use Cases:
  • High-Performance Proxies: Building transparent proxies that can forward traffic between front-end and back-end services with minimal overhead.
  • Service Mesh Data Plane: Optimizing inter-service communication within a service mesh by steering traffic directly between application proxies.
  • Custom Load Balancers: Implementing kernel-based load balancing for specific services, leveraging eBPF to select the appropriate backend socket. These advanced features push the boundaries of kernel bypass and direct data plane programming, allowing developers to craft highly optimized network paths for critical applications.

Other Relevant Attachment Points (Briefly)

While not strictly for "packet inspection" in the traditional sense, other eBPF attachment points contribute significantly to overall network observability and debugging:

• Tracepoints and Kprobes: eBPF programs can attach to kernel tracepoints (pre-defined instrumentation points) or kprobes (dynamic instrumentation points) to monitor kernel function calls, including those related to the network stack. This allows for deep introspection into kernel behavior, helping to debug network issues or understand performance bottlenecks at a very low level. While they don't directly process packets, the context they provide is invaluable for understanding the network's inner workings.
• Cgroups: eBPF programs can be attached to cgroups, allowing for network policy enforcement and accounting on a per-cgroup basis, enabling fine-grained control over resource utilization and security for groups of processes.

In conclusion, the rich array of eBPF attachment points provides an unparalleled toolkit for network engineers and developers. From the raw speed of XDP to the context-rich environment of TC BPF and the application-specific control of socket filters, eBPF enables precise, high-performance packet inspection and manipulation tailored to virtually any network use case. The next crucial step, which we will explore, is how to efficiently extract the valuable insights gleaned from these in-kernel operations and make them accessible to user-space applications for further analysis and action.

Efficient Data Transfer from Kernel to User Space: Bridging the Divide

The true power of eBPF for packet inspection isn't solely in its ability to process data within the kernel, but crucially, in its sophisticated mechanisms for efficiently transferring relevant insights, metadata, or even raw samples of packets back to user-space applications. Without an effective kernel-to-user space communication channel, the wealth of data processed by eBPF programs would remain trapped within the kernel, rendering it largely unactionable. This section delves into the primary conduits that bridge this critical divide, examining their architectures, use cases, and performance characteristics.

The challenge lies in balancing performance with flexibility. Copying entire packets to user space for every network event, as is common with traditional libpcap-based tools, quickly becomes a bottleneck in high-throughput environments. eBPF tackles this by enabling intelligent filtering and aggregation in the kernel, minimizing the data transferred, and utilizing specialized shared-memory mechanisms to reduce copy overheads.

BPF Maps: The Primary Conduit for Aggregated Data

As discussed earlier, BPF maps are versatile key-value stores. While they facilitate communication between eBPF programs, their role as a bridge between kernel and user space is equally vital, particularly for aggregated statistics, counters, and state information.

• BPF_MAP_TYPE_HASH and BPF_MAP_TYPE_ARRAY: These are general-purpose maps suitable for storing simple, aggregated data.
  • Architecture: Hash maps (flexible key-value stores) and array maps (indexed arrays) reside in kernel memory. eBPF programs in the kernel can atomically update elements (e.g., increment counters, store flow metadata).
  • User-Space Interaction: User-space applications interact with these maps by polling them. They can use the bpf() system call to look up elements, update elements, or iterate through the map to retrieve all stored data. This is a synchronous, on-demand pull mechanism.
  • Use Cases:
    • Packet Counters: Counting packets per IP address, port, protocol, or application flow. An eBPF program increments a counter in a map for each matching packet, and user space periodically reads these counts to build dashboards or alerts.
    • Flow Statistics: Storing bytes/packets transferred for active connections, along with timestamps.
    • Policy Tables: User-space applications can populate a hash map with IP addresses to block, and eBPF programs can quickly consult this map to drop malicious traffic.
  • Performance Characteristics: Efficient for aggregated data. Polling frequency needs to be tuned: too frequent, and it adds CPU overhead; too infrequent, and data is stale. Best for data that changes relatively slowly or where instant real-time updates are not strictly required.

BPF_MAP_TYPE_PERF_EVENT_ARRAY (Perf Buffer): Asynchronous Event Streaming

When real-time event streaming and low-latency transfer of individual events or sampled data are required, the perf_event_array map type, commonly referred to as a "perf buffer," is the mechanism of choice. It leverages the Linux perf_event subsystem, which is typically used for CPU profiling.

• Architecture: A perf buffer is essentially a ring buffer allocated in shared memory (mmap'd) between the kernel and user space. Each CPU has its own dedicated ring buffer.
  • Kernel-Side (bpf_perf_event_output): An eBPF program uses the bpf_perf_event_output helper function to push arbitrary data structures (e.g., sampled packet headers, connection events, security alerts) into the per-CPU ring buffer. The data is written to a designated slot in the buffer.
  • User-Space Interaction: A user-space application opens perf_event file descriptors for each CPU's ring buffer and mmaps them into its address space. It then reads data as it becomes available. Typically, the user-space process blocks (e.g., using poll or epoll) until new data is written to the buffer, indicating an event.
• Mechanism: The kernel writes data directly into the shared memory region. When the buffer fills up, an interrupt can notify the user-space process. User space then reads the data, processes it, and advances its read pointer. This design minimizes kernel-to-user copy operations and context switches.
• Use Cases:
  • Sampled Packet Data: Sending header information for a percentage of packets for deeper analysis without overwhelming user space.
  • Connection Events: Notifying user space of new TCP connections, connection terminations, or unusual network activity.
  • Security Alerts: Triggering an alert in user space when a specific type of malicious packet or network behavior is detected.
  • Detailed Flow Records: Exporting richer flow information than simple counters, including timestamps, flags, and payload snippets.
• Performance Characteristics: Excellent for high-volume, asynchronous event streaming. Very low latency due to shared memory and direct kernel writes. Handles bursts well. Management of multiple per-CPU buffers in user space can add some complexity, requiring careful aggregation if a global view is needed.

BPF_MAP_TYPE_RINGBUF (BPF Ring Buffer): The Modern Alternative

Introduced in Linux kernel 5.8, the BPF_MAP_TYPE_RINGBUF map type offers a simpler, more efficient, and often preferred alternative to perf buffers for many event-streaming scenarios. It is designed specifically for eBPF and addresses some of the complexities and overheads of the generic perf_event subsystem.

• Architecture: Similar to perf buffers, it uses a single, shared-memory ring buffer (can be per-CPU or global). It's designed as a single-producer (kernel), single-consumer (user space) mechanism, although multiple eBPF programs can write to the same buffer.
  • Kernel-Side (bpf_ringbuf_output): eBPF programs use bpf_ringbuf_output to push data. It offers a simpler API than bpf_perf_event_output. Data is written directly to the next available slot.
  • User-Space Interaction: User space mmaps the ring buffer. It constantly checks for new data by comparing read and write pointers in the shared memory. It can also use poll on the BPF map file descriptor to be notified when new data is available.
• Key Advantages over Perf Buffer:
  • Simpler API: Easier for eBPF programs and user-space applications to use.
  • Lower Overhead: Generally incurs less overhead than perf buffers because it avoids the generic perf_event infrastructure.
  • Fixed-Size Records: Optimized for fixed-size data structures, though variable-length data can be handled with some care.
  • Zero-Copy: Data is written directly into the shared user-space buffer, avoiding copies.
  • Less Boilerplate: Requires less setup code in both kernel and user space.
• Use Cases: Effectively replaces perf buffers for most eBPF event streaming needs, including sampled packet data, connection logging, security events, and custom telemetry.
• Performance Characteristics: Extremely performant for event streaming, offering minimal latency and high throughput. It is becoming the go-to mechanism for transferring event-driven data from eBPF programs.

AF_XDP (Address Family eXpress Data Path): Raw Packet Access in User Space

For applications that need direct, zero-copy access to raw network packets in user space at extremely high rates, AF_XDP is the ultimate solution. It's not a BPF map type but a dedicated socket family (AF_XDP) that works in conjunction with XDP programs. AF_XDP allows user-space applications to receive and transmit raw packets directly from/to the NIC's XDP ring buffers, completely bypassing the kernel's network stack for those packets.

• Architecture: AF_XDP creates a shared memory region (UMEM - User-space Memory) between the kernel and a user-space application. This UMEM contains buffers for raw packet data. In addition, there are several ring queues (Rx ring, Tx ring, Completion ring) that facilitate communication and ownership transfer of these packet buffers between the kernel and user space.
  • Kernel-Side (XDP Program): An XDP program running at the NIC driver can explicitly redirect packets (XDP_REDIRECT) to an AF_XDP socket. Instead of copying the packet, the kernel merely updates pointers in the shared ring buffer, effectively transferring ownership of the packet buffer to user space.
  • User-Space Interaction: A user-space application (mmaps the UMEM and ring queues. It constantly monitors the Rx ring for new packet pointers. When a packet arrives, the application reads its data from the UMEM buffer without any kernel copies. It can then process the packet and potentially send it back out via the Tx ring, also using zero-copy.
• Zero-Copy Packet Transfer: The defining feature of AF_XDP is its true zero-copy nature. Packets are never copied by the kernel for user-space consumption; only pointers and metadata are exchanged in the shared ring buffers.
• Use Cases:
  • User-Space Network Stacks: Building custom TCP/IP stacks or specialized network protocols entirely in user space (e.g., for NFV, SDN).
  • High-Performance Packet Processors: Implementing specialized firewalls, load balancers, intrusion prevention systems, or monitoring tools that require line-rate processing without kernel overhead.
  • Research and Development: Prototyping new networking technologies that demand ultimate control over packet handling.
• The Role of libbpf: Interacting with AF_XDP can be complex due to the intricate management of UMEM and ring buffers. The libbpf library provides helpful abstractions and utilities (e.g., xsk.h) to simplify the setup and interaction with AF_XDP sockets, making it more accessible to developers.
• Performance Characteristics: Unmatched performance for raw packet access in user space, rivaling or even exceeding specialized kernel bypass solutions like DPDK (without requiring specialized NIC drivers).

Comparing Kernel-to-User Space Data Transfer Mechanisms

To summarize the various kernel-to-user space data transfer mechanisms, a comparative table can illuminate their strengths and optimal use cases:

Feature/Mechanism	BPF Hash/Array Maps	Perf Buffer (BPF_MAP_TYPE_PERF_EVENT_ARRAY)	BPF Ring Buffer (BPF_MAP_TYPE_RINGBUF)	AF_XDP
Data Type	Aggregated stats, Counters, State	Individual events, Sampled data, Logs	Individual events, Sampled data, Logs	Raw full packets
Transfer Model	Synchronous pull (user space polls)	Asynchronous push (kernel to shared ring)	Asynchronous push (kernel to shared ring)	Zero-copy redirect (XDP to shared socket)
Copy Overhead	Low (only map data copied on read)	Low (data written directly to shared memory)	Very low (data written directly to shared memory)	Zero-copy (packet ownership transferred)
Latency	Moderate (depends on polling frequency)	Very Low	Extremely Low	Extremely Low
Throughput	Moderate (for aggregated reads)	High (for event streaming)	Very High (for event streaming)	Max (line-rate packet processing)
Kernel Hook Req.	Any program type can access maps	Any program type can output to perf buffer	Any program type can output to ring buffer	Requires XDP program redirecting to AF_XDP socket
Complexity	Low	Moderate (perf_event API)	Low-Moderate (simpler API)	High (UMEM, ring queue management, libbpf helps)
Best For	Metrics, policies, lookup tables	General event logging, security alerts, tracing	High-volume event streaming, tracing, logging	User-space network stacks, high-perf packet processing

While eBPF offers granular control for low-level network operations, managing the resulting data streams and integrating them into a broader system can be complex. The insights gained from eBPF might need to be consumed by various applications, some of which interact via APIs. For higher-level API management, AI model integration, and robust API lifecycle governance, platforms like ApiPark provide invaluable capabilities. APIPark abstracts away much of the complexity of API deployment and scaling, ensuring that the critical insights derived from eBPF (e.g., network flow data, security alerts) can be effectively ingested, managed, and exposed as APIs for consumption by other services or analytics platforms. This separation of concerns allows developers to focus on eBPF's low-level power for data generation, while leveraging APIPark for efficient, secure, and scalable API delivery and consumption.

The choice of data transfer mechanism depends heavily on the specific requirements of the application, including the type of data, required latency, and overall throughput. By judiciously selecting and implementing these mechanisms, developers can unlock the full potential of eBPF-driven packet inspection, transforming raw kernel events into actionable intelligence for user-space applications.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Building User-Space Applications for eBPF Packet Inspection: Bringing Insights to Life

Having explored the foundational aspects of eBPF and the crucial mechanisms for transferring data from the kernel to user space, the next logical step is to understand how to build robust user-space applications that harness this power. A user-space component is essential for loading eBPF programs, attaching them to kernel hooks, configuring maps, consuming data, and ultimately presenting insights in a meaningful way. This section focuses on the tools, libraries, and best practices for developing such applications.

The complexity of interacting with eBPF system calls directly can be daunting. Fortunately, a mature ecosystem of libraries and frameworks has emerged, significantly simplifying the development process and abstracting away much of the low-level kernel interaction.

The Indispensable Role of libbpf

libbpf is a C library developed as part of the Linux kernel project, specifically designed to simplify the loading, management, and interaction with eBPF programs and maps from user space. It has become the de facto standard for building production-grade eBPF applications due to its tight integration with the kernel, robustness, and focus on stability and performance.

• Simplifying eBPF Program Loading and Management: libbpf handles the intricate details of opening BPF file descriptors, loading compiled BPF bytecode (often from ELF files), verifying programs, and managing their lifecycle (attaching, detaching). It takes care of program relocation, map definition, and linking programs with maps.
• BTF (BPF Type Format) Integration: One of libbpf's most powerful features is its deep integration with BTF. BTF embeds C type information into the compiled eBPF object files. libbpf uses this information to:
  • Improve Program Portability: libbpf can automatically adjust BPF programs for different kernel versions by re-locating fields in kernel data structures if their offsets change across kernel updates. This significantly reduces the "write once, run anywhere" challenge.
  • Enhance Introspection: User-space tools can use BTF to understand the data structures used by eBPF programs and maps, making it easier to parse and interpret the data exchanged.
• Map Interaction: libbpf provides a straightforward API for creating, looking up, updating, and iterating over BPF maps, regardless of their type. This simplifies the process of sending configuration from user space to kernel programs and retrieving data.
• Perf Buffer and Ring Buffer Consumption: libbpf offers convenient helper functions for setting up and consuming data from perf buffers and ring buffers, abstracting away the complexities of perf_event_open and mmap directly. For instance, ringbuf_poll simplifies event processing.
• AF_XDP Integration: libbpf provides an XDP socket API (xsk.h) that significantly simplifies the creation and management of AF_XDP sockets, UMEM, and ring queues, making high-performance raw packet processing more accessible.

For any serious eBPF development, especially for network packet inspection, leveraging libbpf is highly recommended. It reduces boilerplate, improves robustness, and makes applications more resilient to kernel version changes.

High-Level Language Bindings and Frameworks

While libbpf is a C library, various projects provide bindings for higher-level languages, making eBPF development accessible to a broader audience:

• Go (cilium/ebpf): The cilium/ebpf library offers a native Go API for loading, managing, and interacting with eBPF programs and maps. It's widely used in projects like Cilium itself and provides excellent performance and concurrency features inherent to Go. It's a strong choice for building network and security tools.
• Python (bcc, pybpf):
  • bcc (BPF Compiler Collection): A powerful framework that combines clang and LLVM to compile C code into eBPF bytecode on the fly. It provides Python bindings for loading, attaching, and interacting with eBPF programs, and it comes with a vast collection of ready-to-use eBPF tools for tracing and monitoring. While bcc is incredibly useful for rapid prototyping and interactive exploration, its runtime compilation might not be ideal for all production deployments.
  • pybpf: A newer project aiming to provide lower-level, libbpf-like Python bindings, offering more control and better performance than bcc for certain use cases, often used with pre-compiled eBPF programs.
• Rust (aya, redbpf): The Rust ecosystem is also rapidly embracing eBPF, with projects like aya and redbpf providing safe, high-performance bindings and frameworks for eBPF development in Rust. Rust's memory safety guarantees align well with the critical nature of kernel-level programming.

Choosing the right toolchain depends on project requirements, performance needs, developer familiarity, and the desired level of abstraction. For production-grade network appliances or critical security tools, C/C++ with libbpf or Go with cilium/ebpf are often preferred due to their control and performance characteristics.

Example Walkthrough (Conceptual): Flow Monitoring with eBPF and User Space

Let's conceptualize a practical example: building a simple network flow monitoring application.

Goal: Identify active network flows (e.g., source IP, destination IP, source port, destination port, protocol) and report aggregated statistics (bytes, packets) to user space.

1. eBPF Program (Kernel-Side, C):

• Attachment Point: Likely TC_BPF on the ingress hook for detailed sk_buff context, or XDP for higher throughput if only basic headers are needed. Let's assume TC_BPF for richness.
• Data Structure: Define a struct flow_key (source IP, dest IP, src port, dest port, protocol) and a struct flow_stats (total bytes, total packets, start time, end time).
• BPF Map: Create a BPF_MAP_TYPE_HASH map called flow_map where flow_key is the key and flow_stats is the value.
• Logic:
  1. When a packet arrives:
  2. Extract flow_key from the sk_buff.
  3. Lookup flow_key in flow_map.
  4. If found: Update flow_stats (increment bytes, packets, update end time).
  5. If not found: Initialize flow_stats with current packet data and timestamp, then insert into flow_map.
  6. Optionally, if a flow has been inactive for a while (tracked by end_time and a user-space timer), delete it from flow_map and perhaps send a "flow expired" event to user space via a ring buffer.

2. User-Space Application (e.g., Go with cilium/ebpf):

• Loading and Attaching:
  1. Load the compiled eBPF flow_monitor.o ELF file using cilium/ebpf's LoadCollection or LoadCollectionSpec.
  2. Find the ingress program (e.g., tc_ingress_flow_monitor).
  3. Create a tc filter and attach the eBPF program to the desired network interface's ingress hook.
• Map Interaction:
  1. Get a handle to the flow_map (e.g., collection.LookupMap("flow_map")).
  2. Periodically (e.g., every 5 seconds) iterate through flow_map to retrieve all active flow_key and flow_stats entries.
  3. Process the data:
     • Calculate throughput.
     • Identify top talkers.
     • Detect long-running connections.
     • Detect anomalies (e.g., unusually high packet counts for a single flow).
  4. Optionally, delete flows from the map based on inactivity timeouts determined by user space.
• Data Presentation:
  1. Print flow information to the console.
  2. Send data to an observability platform (Prometheus, Grafana, ELK stack).
  3. Generate alerts based on defined thresholds.

This conceptual example demonstrates how eBPF programs handle the high-performance, in-kernel data collection, while user-space applications manage program lifecycle, policy, data aggregation, and presentation.

Leveraging Existing Tools and Frameworks

Before embarking on building a custom eBPF solution from scratch, it's often beneficial to explore the rich ecosystem of existing eBPF-based tools and frameworks:

• Cilium: A cloud-native networking, security, and observability solution for Kubernetes and other container orchestration platforms. Cilium extensively uses eBPF for fast data path, network policy enforcement, load balancing, and network telemetry. It's a prime example of eBPF's power in a production environment.
• Falco: A runtime security tool that uses eBPF (among other kernel sources) to detect suspicious activity in applications, containers, and hosts. Falco provides rules to detect common attacks and policy violations.
• bpftrace: A high-level tracing language built on eBPF. It allows users to write short, powerful scripts to trace kernel functions, user-space functions, and various events with minimal effort. While primarily a tracing tool, bpftrace can be invaluable for debugging eBPF network programs or quickly understanding kernel network behavior.
• BCC Tools: The BPF Compiler Collection (BCC) ships with hundreds of ready-to-use eBPF tools for various monitoring and tracing tasks, including network-specific tools like tcpretrans, opensnoop, execsnoop, and many others. These tools are excellent for learning and quickly gaining insights without writing any eBPF code.

Integration with Observability Stacks

The ultimate goal of packet inspection is to gather actionable intelligence. User-space eBPF applications play a pivotal role in integrating this intelligence into broader observability stacks.

• Metrics Export: Aggregated data from BPF maps (e.g., packet counts, byte rates, connection numbers) can be exposed as Prometheus metrics via a custom exporter. Grafana dashboards can then visualize these metrics, providing real-time insights into network health and performance.
• Event Logging: Events streamed from perf buffers or ring buffers (e.g., new connections, dropped packets, security alerts) can be formatted into JSON or other structured logs and sent to an ELK (Elasticsearch, Logstash, Kibana) stack or a Splunk instance for centralized logging, searching, and anomaly detection.
• Alerting: Based on the processed data, user-space applications can trigger alerts through various channels (e.g., Slack, PagerDuty) when predefined thresholds are breached or suspicious patterns are detected.

By carefully designing the user-space component, developers can transform raw eBPF insights into a robust, integrated observability solution that enhances network performance, bolsters security, and simplifies troubleshooting in complex, distributed systems. The synergy between kernel-side eBPF processing and sophisticated user-space analysis is where the true mastery of eBPF packet inspection lies.

Advanced Techniques, Challenges, and Best Practices in eBPF Packet Inspection

Mastering eBPF packet inspection goes beyond understanding the basics; it involves delving into advanced techniques, navigating inherent challenges, and adopting best practices to build robust, efficient, and maintainable solutions. As eBPF continues to evolve, pushing the boundaries of kernel programmability, so too do the complexities and opportunities it presents.

Advanced Techniques for Network Control

The flexibility of eBPF allows for incredibly sophisticated network operations that extend far beyond simple packet filtering and counting.

• Packet Re-injection and Modification:
  • Intelligent Routing with bpf_redirect: eBPF programs, particularly at the XDP or TC level, can use bpf_redirect to dynamically route packets. This can be to another network interface, a specific CPU queue, or even a user-space AF_XDP socket. This capability is foundational for programmable load balancing, network overlays, and building custom high-performance routers entirely in eBPF.
  • Direct Egress with XDP_TX: An XDP program can process an incoming packet and then, instead of passing it up the stack, immediately transmit it back out the same network interface using XDP_TX. This is incredibly fast and useful for specific scenarios like responding to certain network probes directly from the NIC, or implementing very low-latency "reflection" services.
  • Packet Header Rewriting: eBPF programs can directly modify packet headers (e.g., MAC, IP, TCP/UDP ports) in place within the kernel (e.g., using bpf_xdp_adjust_head, bpf_xdp_adjust_tail, or by directly writing to sk_buff data in TC BPF). This enables sophisticated network address translation (NAT), custom tunneling protocols, or even on-the-fly protocol conversions.
• Stateful Packet Inspection (SPI) in eBPF: While many eBPF examples focus on stateless operations, eBPF programs can maintain state using maps. For example, a hash_map can store connection tuples (source/dest IP, port, protocol) as keys and connection state (SYN received, ACK received, bytes transferred, timeout) as values.
  • Use Cases: Building advanced stateful firewalls, implementing connection tracking for load balancers (e.g., sticky sessions), or detecting complex multi-packet attack patterns. The challenge here is managing map size, eviction policies, and race conditions, which require careful design.
• Programmable Load Balancing: eBPF allows implementing custom, highly efficient load balancing algorithms. Instead of relying on traditional kernel load balancers or user-space proxies, eBPF programs can analyze incoming packets (e.g., HTTP headers, SSL SNI), select a backend server based on a custom algorithm (e.g., least connections, consistent hashing), rewrite the destination MAC/IP, and redirect the packet to the chosen backend.
  • Advantages: Extreme performance, high scalability, and fine-grained control over balancing logic.
• Tunnel Encapsulation/Decapsulation: eBPF programs can be used to encapsulate packets into various tunneling protocols (e.g., VXLAN, Geneve) or decapsulate them. This is crucial for building high-performance overlay networks in cloud-native environments, allowing services to communicate across different hosts or subnets transparently. This is heavily utilized by projects like Cilium.

Navigating the Challenges of eBPF Development

Despite its power, eBPF development comes with its own set of unique challenges that require careful consideration.

• Debugging eBPF Programs: Debugging kernel-side eBPF programs is notoriously difficult. Unlike user-space applications, you can't easily attach a debugger.
  • Tools: Rely heavily on bpf_printk for basic logging (printed to dmesg). bpftool is invaluable for inspecting loaded programs, maps, and verifier logs. bpftrace can be used to observe kernel functions that your eBPF program interacts with. Understanding the verifier's output is critical for fixing common errors like invalid memory access or infinite loops.
  • Isolation: Since eBPF programs are sandboxed, they cannot call arbitrary kernel functions, allocate dynamic memory, or trigger system calls, which limits debugging options.
• Kernel Version Compatibility: eBPF is a rapidly evolving technology. New features, helper functions, and map types are introduced with almost every major kernel release. This means an eBPF program written for a newer kernel might not work on an older one, and vice versa.
  • Solution: Using libbpf with BTF (BPF Type Format) significantly mitigates this. libbpf can automatically adjust programs to different kernel versions. Conditional compilation (__KERNEL__ macros in C code) can also be used, but it increases complexity. It's often pragmatic to target a minimum kernel version.
• Security Implications: While the verifier ensures memory safety and termination, poorly designed eBPF programs can still have security implications. For instance, an eBPF program that consumes excessive CPU cycles in a tight loop, even if it terminates, can starve other kernel processes. Malicious eBPF programs, if they circumvent verifier checks (highly unlikely with recent kernels), could potentially expose sensitive data or manipulate network traffic in unintended ways.
  • Mitigation: Strict code review, thorough testing, adherence to best practices, and keeping up-to-date with kernel security patches.
• Resource Management: eBPF programs operate within kernel resource constraints. Maps consume kernel memory, and program execution consumes CPU cycles. Large maps, particularly those with frequent updates or complex keys, can impact kernel performance. Programs with high instruction counts or many loops can also consume significant CPU.
  • Optimization: Design maps carefully, prune old entries, optimize program logic to minimize instructions and memory access. Profile programs (bpftool prog profile) to identify hotspots.
• Steep Learning Curve: eBPF development requires a solid understanding of Linux kernel internals, networking concepts, and often C programming. The mental model of a safe, sandboxed VM within the kernel is unique and takes time to grasp.

Best Practices for Robust eBPF Solutions

To overcome challenges and effectively leverage eBPF's power, adopting a set of best practices is crucial.

• Start Simple and Iterate: Begin with basic programs (e.g., packet counters) and gradually add complexity. Test each component in isolation before integrating.
• Leverage libbpf and BTF: For portability, maintainability, and reduced development overhead, use libbpf with BTF for almost all production-grade eBPF applications.
• Comprehensive Testing:
  • Unit Tests: Test the eBPF program logic in isolation using tools like bpf_prog_test_run.
  • Integration Tests: Test the complete eBPF program and user-space interaction in a controlled environment (e.g., containerized tests, virtual machines).
  • Performance Benchmarking: Measure the impact of your eBPF programs on system resources (CPU, memory, network throughput) and optimize hot paths.
• Defensive Programming:
  • Bounds Checking: Always perform bounds checks when accessing packet data or map elements to satisfy the verifier and prevent out-of-bounds access.
  • Error Handling: Implement robust error handling in both the eBPF program and the user-space component.
  • Fallback Mechanisms: Design your application to degrade gracefully or fall back to traditional methods if eBPF program loading or attachment fails (e.g., due to kernel version incompatibility).
• Meaningful Documentation and Comments: eBPF programs can be terse and complex. Clear comments explaining the logic, map structures, and kernel interactions are essential for maintainability. Document kernel version requirements and testing procedures.
• Monitor and Profile: Use bpftool, perf, and other system monitoring tools to observe the behavior and resource consumption of your eBPF programs in production. Set up alerts for unexpected behavior.
• Stay Updated: The eBPF ecosystem evolves rapidly. Regularly follow kernel development, libbpf updates, and community discussions to leverage new features and learn from others' experiences.

By adhering to these advanced techniques and best practices, developers can navigate the complexities of eBPF, build highly performant and secure network packet inspection solutions, and truly master this transformative kernel technology. The journey of mastering eBPF is continuous, reflecting the dynamic nature of both modern networks and the Linux kernel itself.

Conclusion: eBPF as the Future of Network Visibility and Control

The journey through the intricate world of eBPF packet inspection in user space reveals a technology that is nothing short of revolutionary. We have explored how eBPF transcends its origins as a simple packet filter, evolving into a sophisticated, in-kernel virtual machine capable of executing custom, sandboxed programs with unparalleled efficiency and safety. This paradigm shift has fundamentally redefined the possibilities for observing, securing, and optimizing network traffic within the Linux kernel.

Our deep dive into eBPF's foundational architecture highlighted the critical roles of its programs, maps, helper functions, the stringent verifier, and the performance-boosting JIT compiler. These components collectively enable developers to craft bespoke solutions that operate directly at the heart of the operating system, bypassing the traditional overheads that plague conventional network monitoring tools. We then meticulously examined the diverse array of eBPF attachment points within the network stack—from the blazing-fast, zero-copy XDP at the network driver level, through the context-rich TC BPF in the traffic control layer, to application-specific socket filters and advanced sockmap/sockhash structures. Each attachment point offers unique advantages, allowing for surgical precision in intercepting and manipulating network packets precisely where it matters most.

The critical bridge between the kernel's processing power and user-space analytics lies in the sophisticated data transfer mechanisms. We dissected how BPF maps serve as versatile conduits for aggregated statistics, while perf buffers and the newer, more efficient BPF ring buffers provide low-latency, asynchronous event streaming. For applications demanding absolute raw packet access at line rates, AF_XDP stands out as the ultimate solution, enabling true zero-copy transfer of full packets to user space. The strategic selection and implementation of these mechanisms are paramount to ensuring that the rich insights generated within the kernel are efficiently and effectively delivered to user applications for further analysis, visualization, and action.

Furthermore, we detailed the essential components for building robust user-space applications, emphasizing the indispensable role of libbpf for program loading, map interaction, and ensuring kernel version portability through BTF. We briefly touched upon high-level language bindings and the wealth of existing eBPF tools and frameworks that simplify development and accelerate problem-solving. Finally, we explored advanced techniques like packet rewriting and stateful inspection, alongside the inherent challenges of debugging and resource management, concluding with a set of best practices to guide developers in crafting resilient and high-performance eBPF solutions.

In essence, mastering eBPF packet inspection is about more than just understanding individual components; it's about appreciating the synergy between its kernel-side prowess and the intelligent user-space orchestration that brings those insights to life. This mastery empowers engineers to build superior network observability platforms, implement robust, high-performance security policies, and unlock unprecedented levels of control and optimization for complex, cloud-native infrastructures.

The future of eBPF in networking, security, and observability is incredibly bright. Its continued evolution promises even more sophisticated capabilities, further entrenching it as a fundamental technology in the Linux ecosystem. As networks grow in complexity and performance demands escalate, the ability to programmatically peer into and manipulate network traffic directly at the kernel level, then efficiently communicate those findings to user space, will not just be an advantage—it will be a necessity. Investing in mastering eBPF is an investment in understanding and controlling the very fabric of modern distributed systems.

Frequently Asked Questions (FAQs)

1. What is eBPF and why is it important for packet inspection? eBPF (extended Berkeley Packet Filter) is a powerful, in-kernel virtual machine within the Linux kernel that allows users to run custom, sandboxed programs for various purposes, including networking, security, and tracing. For packet inspection, it's critical because it enables programs to inspect, filter, modify, or drop packets directly within the kernel network stack, at extremely high speeds and with minimal overhead, before they are processed by the full kernel or copied to user space. This provides unprecedented visibility, performance, and flexibility compared to traditional methods.

2. How does eBPF ensure safety when running code in the kernel? eBPF ensures safety primarily through the eBPF verifier. Before any eBPF program is loaded into the kernel, the verifier statically analyzes its bytecode. It ensures the program will terminate, does not contain infinite loops, does not access invalid memory locations, adheres to resource limits (like maximum instructions), and cannot crash the kernel. This rigorous verification process prevents malicious or faulty eBPF code from destabilizing the operating system, a key differentiator from traditional kernel modules.

3. What are the main ways to get packet data from the eBPF kernel program to a user-space application? There are several efficient mechanisms to transfer data from eBPF programs in the kernel to user space: * BPF Maps (e.g., Hash/Array Maps): Primarily used for aggregated statistics, counters, and state information, which user space can poll periodically. * Perf Buffer (BPF_MAP_TYPE_PERF_EVENT_ARRAY): An asynchronous, shared-memory ring buffer mechanism suitable for streaming individual events, sampled packet data, or logs with low latency. * BPF Ring Buffer (BPF_MAP_TYPE_RINGBUF): A newer, simpler, and often more efficient shared-memory ring buffer alternative to perf buffers, also for high-volume asynchronous event streaming. * AF_XDP (Address Family eXpress Data Path): A dedicated socket type working with XDP that provides true zero-copy access to raw network packets directly in user space, completely bypassing the kernel's network stack for high-performance applications.

4. What is the difference between XDP and TC BPF for network packet processing? * XDP (eXpress Data Path): Attaches eBPF programs at the earliest possible point in the network driver, before the kernel's main network stack processes the packet. It operates on raw packet data, enabling zero-copy operations and offering the highest throughput and lowest latency. It's ideal for tasks like DDoS mitigation, high-performance load balancing, and fast packet filtering. * TC BPF (Traffic Control BPF): Attaches eBPF programs within the kernel's traffic control (TC) layer, after some initial kernel processing. It operates on sk_buff structures, which contain richer metadata from the kernel's network stack. TC BPF is suitable for more context-aware operations like fine-grained traffic shaping, advanced firewalling, and packet manipulation that benefit from the sk_buff context.

5. What are some common use cases for eBPF packet inspection in user space? eBPF packet inspection, with its user-space integration, enables a wide range of use cases: * Network Observability: Building high-performance network monitors, flow exporters, and telemetry tools that feed data to Prometheus/Grafana or ELK stacks. * Security: Implementing dynamic firewalls, intrusion detection/prevention systems (IDS/IPS), DDoS mitigation, and runtime security enforcement. * Performance Optimization: Custom load balancing, traffic shaping, kernel bypass networking (e.g., user-space network stacks via AF_XDP), and optimizing inter-service communication. * Troubleshooting: Deeply analyzing network issues, identifying bottlenecks, and debugging complex distributed systems by gaining granular visibility into packet paths and behavior.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.