This article documents the process of developing the file-logger plugin by a front-end engineer with no back-end experience.

Over the past few months, community users have added many plugins to Apache , enriching the Apache ecosystem. From the user's point of view, the emergence of more diverse plugins is certainly a good thing, as they fulfill more of the user's expectations for a gateway that is "one-stop" and "multi-functional" processor on top of perfecting the high performance and low latency of Apache .

None of the articles on the Apache blog seem to go into detail about the process of developing plugins. So let's take a look at the process from the perspective of a plugin developer and see how a plugin is born!

This article documents the process of developing the file-logger plugin on API Gateway by a front-end engineer with no back-end experience. Before digging into the details of the implementation process, we will briefly introduce the functionality of file-logger.

Introduction of file-logger plugin​

file-logger supports generating custom log formats using Apache plugin metadata. Users can append request and response data in JSON format to log files via the file-logger plugin, or push the log data stream to a specified location.

Imagine this: when monitoring the access log of a route, we not only care about the value of certain request and response data, but also want to write the log data to a specified file. This is where the file-logger plugin can be used to help achieve these goals.

We can use file-logger to write log data to a specific log file to simplify the process of monitoring and debugging.

How to implement a plugin?​

After introducing the features of file-logger, you will have a better understanding of this plugin. The following is a detailed explanation of how I, a front-end developer with no server-side experience, develop the plugin for Apache and add the corresponding tests for it.

Confirm the name and priority of the plugin​

Open the Apache Plugin Development Guide and in order of priority you need to determine the following two things:

1. Determine the plugin category.
2. prioritize the plugins and update the conf/config-default.yaml file.

Since this development of file-logger is a logging type plugin, I refer to the name and ordering of the existing logging plugins for Apache and place file-logger here.

After consulting with other plugin authors and enthusiastic members of the community, the name file-logger and priority 399 of the plugin were finally confirmed.

Note that the priority of the plugin is related to the order of execution; the higher the value of the priority, the more forward the execution. And the ordering of plugin names is not related to the order of execution.

Create a minimum executable plugin file​

After confirming the plugin name and priority, you can create our plugin code file in /plugins/ directory . There are two points to note here:

• If the plugin code file is created directly in the /plugins/ directory, there is no need to change the Makefile file.
• If your plugin has its own code directory, you need to update the Makefile file, please refer to the Apache Plugin Development Guide for detailed steps.

1. Here we create the file-logger.lua file in the /plugins/ directory.
2. Then we will complete an initialized version based on the example-plugin.

-- Introduce the module we need in the headerlocal log_util     =   require(".utils.log-util")local core         =   require(".core")local plugin       =   require(".plugin")local ngx          =   ngx-- Declare the plugin's namelocal plugin_name = "file-logger"-- Define the plugin schema formatlocal schema = {    type = "object",    properties = {        path = {            type = "string"        },    },    required = {"path"}}-- Plugin metadata schemalocal metadata_schema = {    type = "object",    properties = {        log_format = log_util.metadata_schema_log_format    }}local _M = {    version = 0.1,    priority = 399,    name = plugin_name,    schema = schema,    metadata_schema = metadata_schema}-- Check if the plugin configuration is correctfunction _M.check_schema(conf, schema_type)    if schema_type == core.schema.TYPE_METADATA then        return core.schema.check(metadata_schema, conf)    end    return core.schema.check(schema, conf)end-- Log phasefunction _M.log(conf, ctx)    core.log.warn("conf: ", core.json.encode(conf))    core.log.warn("ctx: ", core.json.encode(ctx, true))endreturn _M

Once the minimal available plugin file is ready, the plugin's configuration data and request-related data can be output to the error.log file via core.log.warn(core.json.encode(conf)) and core.log.warn("ctx: ", core.json.encode(ctx, true)).

Enable and test the plugin​

The following are a couple of steps for testing. In order to test whether the plugin can successfully print the plugin data and request-related data information we configured for it to the error log file, we need to enable the plugin and create a test route.

1. Prepare a test upstream locally (the test upstream used in this article is 127.0.0.1:3030/api/hello, which I created locally).

2. Create a route via curl command and enable our new plugin.

    curl http://127.0.0.1:9080//admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "plugins": {     "file-logger": {     "path": "logs/file.log"     } }, "upstream": {     "type": "roundrobin",     "nodes": {     "127.0.0.1:3030": 1     } }, "uri": "/api/hello" }'

   Copy

   You will then see a status code 200, indicating that the route was successfully created.

3. Run the curl command to send a request to the route to test whether the file-logger plugin has been started.

   curl -i http://127.0.0.1:9080/api/helloHTTP/1.1 200 OK...hello, world

   Copy

4. In the logs/error.log file there will be a record:

   As you can see, the path: logs/file.log that we configured for the plugin in the conf parameter has been successfully saved. At this point we have successfully created a minimally usable plugin that prints the conf and ctx parameters in the logging phase.

   After that, we can write the core functionality for the file-logger.lua plugin directly in its code file. Here we can directly run the reload command to reload the latest plugin code without restarting Apache .

Write core function for the file-logger plugin​

The main function of the file-logger plugin is to write log data. After asking other people from the community and checking the information, I learned about Lua's IO library, and confirmed that the logic of the plugin's function is roughly the following steps.

1. After each accepted request, output the log data to the path configured by the plugin.

   1. First, get the value of path in file-logger through conf in the logging phase.
   2. Then, the Lua IO library is used to create, open, write, refresh the cache, and close the file.

2. Handle errors such as open file failure, create file failure, etc.

    local function write_file_data(conf, log_message)     local msg, err = core.json.encode(log_message)     if err then         return core.log.error("message json serialization failed, error info : ", err)     end     local file, err = io_open(conf.path, 'a+')     if not file then         core.log.error("failed to open file: ", conf.path, ", error info: ", err)     else         local ok, err = file:write(msg, '\n')         if not ok then             core.log.error("failed to write file: ", conf.path, ", error info: ", err)         else             file:flush()         end         file:close()     end end

   Copy

3. Referring to the source code of http-logger plugin, I finished the method of passing the log data to the write log data and some judgment and processing of the metadata.

    function _M.log(conf, ctx)     local metadata = plugin.plugin_metadata(plugin_name)     local entry     if metadata and metadata.value.log_format         and core.table.nkeys(metadata.value.log_format) > 0     then         entry = log_util.get_custom_format_log(ctx, metadata.value.log_format)     else         entry = log_util.get_full_log(ngx, conf)     end     write_file_data(conf, entry) end

   Copy

Validate and add tests​

Validate the log records​

Since the file-logger plugin was enabled when the test route was created and the path was configured as logs/file.log, we can simply send a request to the test route to verify the results of the log collection at this point.

curl -i http://127.0.0.1:9080/api/hello

In the corresponding logs/file.log we can see that each record is saved in JSON format. After formatting one of the data, it looks like this.

{    "server":{        "hostname":"....",        "version":"2.11.0"    },    "client_ip":"127.0.0.1",    "upstream":"127.0.0.1:3030",    "route_id":"1",    "start_time":1641285122961,    "latency":13.999938964844,    "response":{        "status":200,        "size":252,        "headers":{            "server":"\/2.11.0",            "content-type":"application\/json; charset=utf-8",            "date":"Tue, 04 Jan 2022 08:32:02 GMT"<sp