Building custom plugins for Kong API gateway

Learn how Nylas uses Kong API gateway to authenticate, control routing, and transform requests/responses.

Hero Banner

Intro

At Nylas, we use Kong API gateway to authenticate, control routing, transform requests and responses. All of these are done by creating custom Kong plugin. This blog explains:

  • Why build a custom plugin
  • How to implement the custom plugin logic
  • How to test the plugin locally
  • How to deploy the plugin to production

Kong API gateway

At Nylas, APIs are supported by different backends. For example:

  1. Email sending API (/send) are handled by an email sending service.
  2. Calendar event scheduling APIs (/events and /calendars) are handled by a calendar service.
  3. All other APIs are supported by a legacy monolith service.

API requests to different endpoints are routed to different backend services. To better manage the routing, we use Kong as our API gateway provider:

We choose Kong as our API gateway provider. Nylas handles thousands of requests per second. For such a large throughput, Kong can route these requests in the order of a few milliseconds.

We separate our API gateway into control plane and data plane:

  • Control plane establishes routing policies. It also instruments the system, such as pushing configs and fetching logs. It also hosts internal admin APIs and serves a manager dashboard.
  • Data plane processes the actual external requests. API middleware (or as Kong calls it, “plugins”) are running inside data plane.

The separation of control plane and data plane improves the reliability and scalability of our API gateway. Even if control plane crashes, Nylas can still process requests through data plane. Data plane and control plane are also provisioned differently. High memory usage in data plane won’t affect the critical functionalities running in control plane. You can read more about control-plane and data-plane in their documentation: https://docs.konghq.com/gateway/latest/plan-and-deploy/hybrid-mode/.

Another thing Kong provides are a number of built-in plugins. A plugin is a middleware which processes requests inside the API gateway layer. It sit between customers and API backend. Some of the built-in plugins we use are:

  • File log plugin that writes a log line for each request.
  • Correlation id plugin that generates a random request id.
  • StatsD plugin that emits API metrics.

Built-in plugins come with limitations. It does not support complex plugin logic. Our goal is to do the followings in the API gateway layer:

  1. Authenticate request with an internal authentication service.
  2. Control routing destination by API version, user account and feature flags.
  3. Add account info to request and response headers.

No built-in plugins handle such complicated logic, so we built our own API gateway plugin.

A request lifecycle will look like this:

Authenticating in the API gateway

Why authenticate in API gateway layer? Can’t we build an authentication library and use it in every backend service? Let’s consider this alternative architecture:

There are a few reasons:

  1. Request routing is not only depending on endpoint, but by user account information. If a user is in an older version, we need to route them to our legacy service. The version of a user account can only be retrieved after authentication. Therefore, we must do authentication in the request routing phase.
  2. The earlier we do authentication, the earlier we can reject illegitimate requests. The alternative architecture above is very vulnerable to DDoS attacks. Illegitimate requests will spread out to all backend API services. We have to stop such requests as early as possible.
  3. If authentication service makes a backward incompatible change in service contract, it will introduce trouble to all backend services. We will have to upgrade the auth-library one service after another. It is easier to do just one upgrade in the API gateway layer.

Authentication is tightly coupled with API gateway. It makes sense to do authentication using a Kong plugin.

How to build a custom plugin

Kong offers an open sourced Plugin Developer Kit (or “PDK”) in various languages. You can build a Kong plugin with Go, Javascript, Python, and Lua:

Kong PDK languageSource codeGuide
Gohttps://github.com/Kong/go-pdkhttps://docs.konghq.com/gateway/latest/reference/external-plugins/#developing-go-plugins
Javascripthttps://github.com/Kong/kong-js-pdkhttps://docs.konghq.com/gateway/latest/reference/external-plugins/#developing-javascript-plugins
Pythonhttps://github.com/Kong/kong-python-pdkhttps://docs.konghq.com/gateway/latest/reference/external-plugins/#developing-python-plugins
Luahttps://github.com/Kong/kong/tree/master/kong/pdkhttps://docs.konghq.com/gateway/latest/pdk/

We chose Go as our plugin development language because it is fast, light-weighted and has good community support. As a side note, we also use Go for most of our backend services. Here is a simple hello world Go plugin:

package main

import (
	"github.com/Kong/go-pdk"
	"github.com/Kong/go-pdk/server"
)

func main() {
	server.StartServer(New, Version, Priority)
}

var Version = "0.2"
var Priority = 1

type Config struct {
	Message string
}

func New() interface{} {
	return &Config{}
}

func (conf Config) Access(kong *pdk.PDK) {
	message := conf.Message
	if message == "" {
		message = "hello"
	}
	kong.Log.Notice("Message: " + message)
}

The main plugin logic is written in Access(kong *pdk.PDK) function. You can also define plugin version and priority. Priority is the ordering of plugin execution. The higher the priority, the earlier it gets executed.

Kong offers a wide range of built-in functions. You can find the source code in https://github.com/Kong/go-pdk. Here are some examples:

Plugin capabilityFunctions
Request parsingkong.Request.GetScheme()
kong.Request.GetHost()
kong.Request.GetPort()
kong.Request.GetMethod()
kong.Request.GetPath()
kong.Request.GetQueryArg()
kong.Request.GetHeader(headerKey string)
kong.Request.GetRawBody()
Request transformationkong.ServiceRequest.SetScheme(scheme string)
kong.ServiceRequest.SetPath(path string)
kong.ServiceRequest.SetMethod(method string)
kong.ServiceRequest.SetQuery(query map[string][]string)
kong.ServiceRequest.SetHeader(name string, value string)
kong.ServiceRequest.SetRawBody(body string)
Response parsingkong.ServiceResponse.GetStatus()
kong.ServiceResponse.GetHeader(name string)
kong.ServiceResponse.GetRawBody()
Response transformationkong.Response.SetStatus()
kong.Response.SetHeader(k string, v string)
Change routing destinationkong.Service.SetUpstream(host string)
kong.Service.SetTarget(host string, port int)
Loggingkong.Info(args …interface{})
kong.Notice(args …interface{})
kong.Warn(args …interface{})
kong.Err(args …interface{})
kong.Crit(args …interface{})

With these functions, you can do anything in the gateway layer. The built-in functions are the building blocks of custom plugins.

Writing unit test and function test

Kong’s PDK libraries for Go, Javascript, and Python are not native. They are wrapper of Lua functions. For example, this is the source code of kong.Service.SetUpstream(host string) function:

func (s Service) SetUpstream(host string) error {
	return s.Ask(`kong.service.set_upstream`, bridge.WrapString(host), nil)
}

The Go function wraps the Lua command kong.service.set_upstream. It doesn’t actually do anything except executing Lua. For this reason, writing unit test for custom Kong plugin is a little tricky. We will have to mock Lua like this:

service := Service{
	bridge.New(
		bridgetest.Mock(
			t,
			[]bridgetest.MockStep{
				{
					Method: "kong.service.set_upstream",
					Args:   bridge.WrapString("farm_4"),
					Ret:    nil
				},
			},
		),
	),
}

assert.NoError(t, service.SetUpstream("farm_4"))

Unit test won’t be able to test the actual effect of Kong plugin. That is why we don’t solely rely on unit test for code quality.

Then how do we test the plugin functionality? The answer is building a developer environment on your laptop. You can run a minikube and deploy Kong with your plugin. Make a fake backend service and configure routing accordingly.

Once Kong is up, write a function test that calls Kong proxy. See whether the request/response is transformed, and see if the routing is expected.

You can read more about building developer environment in my previous blog: https://www.nylas.com/blog/how-we-test-microservices-locally-at-nylas/

Deploy API gateway with custom plugin

After writing custom logic to the plugin, we can deploy it with Kong API gateway.

First, compile the plugin into an executable. The plugin must be runnable on linux. This is an example compile command:

GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/go-hello plugin/go-hello.go

Second, build a docker image. In your Dockerfile, use kong-gateway as base image and copy the executable into the container:

FROM kong/kong-gateway:2.8

ADD bin/go-hello /usr/local/bin/go-hello

USER kong

# Prove that go-hello works
RUN ["/usr/local/bin/go-hello", "-dump"]

Run docker build command to build the kong-with-plugin-image:

docker build -f Dockerfile -t kong-with-plugin-image:latest .

Third, configure the Helm values.yaml file to register the plugin.

image:
  repository: kong-with-plugin-image
  tag: "latest"

env:
  pluginserver_names: go-hello
  pluginserver_go_hello_socket: /usr/local/kong/go-hello.socket
  pluginserver_go_hello_start_cmd: /usr/local/bin/go-hello
  pluginserver_go_hello_query_cmd: /usr/local/bin/go-hello -dump
  plugins: bundled,go-hello

Run helm upgrade with the new values.yaml:

helm upgrade kong kong/kong --values values.yaml -n kong

If you deploy Kong with control-plane and data-plane, kong image should be changed in both planes. You also need to add the environment variables (such as pluginserver_names and pluginserver_go_hello_socket) to the values.yaml file for both control-plane and data-plane. During deployment, upgrade control plane first, and then data-plane.

Now the plugin should be available in Kong. You will see these logs proving that plugin binary is registered:

[proxy] 2022/07/14 22:46:01 [notice] 5591#0: *28 [kong] process.lua:263 Starting go-hello, context: ngx.timer

At last, you can use [deck command](https://docs.konghq.com/deck/latest/) or Admin API to enable this plugin. If you are a Kong enterprise user, go to Kong manager and enable it in the UI:

Instead of “nylas-proof-of-concept”, you will see “go-hello”.

Troubleshooting a custom plugin

Once deployed to production, you will need to check whether the plugin is behaving properly.

If the plugin crashes, Kong API gateway will “fail-open” where requests pass through Kong as if there is no plugin. After the fail-open, plugin binary will restart immediately. However, if the fundamental issue is not fixed, plugin will keep crashing and restarting.

You won’t see any alert for Kong plugin crashes. The only way to check is looking at logs. Here is an example of a plugin crash logs:

kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [error] 2104#0: *1840 [kong] pb_rpc.lua:404 [nylas-auth] closed, client: 10.177.59.116, server: kong, request: "GET /calendars HTTP/1.1", host: "api-staging.nylas.com"
kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: signal 17 (SIGCHLD) received from 2106
kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: *23 [kong] process.lua:279 external pluginserver 'nylas-auth' terminated: exit 2, context: ngx.timer
kong-data-plane-kong-9b4cd96b6-nq52b proxy 2022/07/08 06:15:39 [notice] 2104#0: *23 [kong] process.lua:263 Starting nylas-auth, context: ngx.timer

Search with keyword signal 17 (SIGCHLD) or external pluginserver '<your-plugin-name>' terminated. If you see many error logs, it is an indication that your custom plugin is crashing.

Here is another tip: use the file-log plugin and log with kong.Notice(args ...interface{}) in your custom code. It can narrow down your search for bugs. Write as many logs as possible locally, and remove those logs before deploying to production.

If you are a Kong Enterprise user, you can always reach out to Kong Support. Although Kong does not support custom plugin development, they can still offer some guidance in their best efforts.

Lastly, we recommend running Kong plugin in a developer environment (minikube). Try to reproduce the bug locally, and write comprehensive function tests. If a bug cannot be reproduced locally, it may not be caused by Kong, but by the upstream service.

Build time!

After reading this blog, be sure to checkout my example plugin: https://github.com/quzhi1/KongPlayground. This repository contains the plugin code, the deployment configuration, and a developer environment to play with.

Conclusion

The process of developing Kong custom plugins can be summarized in this meme:

Special thanks to all Nylanauts who helped building and deploying Kong plugins:

  • Prem Keshari
  • Caleb Geene
  • Mudit Seth
  • John Jung
  • Bill Jo
  • Danton Pimentel

You May Also Like

Why it’s mission-critical to build bi-directional email in your CRM with an email API… now
Why it’s mission-critical to build bi-directional email in your CRM with an email API… now
hero banner
Everything you need to know about Microsoft’s basic authentication deprecation
How we secure APIs at Nylas using JSON Web Tokens