调研对象：

• Kong
• Traefix
• Envoy

Kong CE 版本

官网 和 GitHub Repo。

Kong 版本：

• Community Edition (CE)
• Enterprise Edition (EE)

CE 当前版本 0.14.1。Kong 底层采用 OpenRestry 开发。

Kong is a cloud-native, fast, scalable, and distributed Microservice Abstraction Layer (also known as an API Gateway, API Middleware or in some cases Service Mesh). Made available as an open-source project in 2015, its core values are high performance and extensibility

Kong CE Features

• Cloud-Native: Platform agnostic, Kong can run from bare metal to
  Kubernetes.
• Dynamic Load Balancing: Load balance traffic across multiple upstream
  services.
• Hash-based Load Balancing: Load balance with consistent hashing/sticky
  sessions.
• Circuit-Breaker: Intelligent tracking of unhealthy upstream services.
• Health Checks: Active and passive monitoring of your upstream services.
• Service Discovery: Resolve SRV records in third-party DNS resolvers like
  Consul.
• Serverless: Invoke and secure AWS Lambda or OpenWhisk functions directly
  from Kong.
• WebSockets: Communicate to your upstream services via WebSockets.
• OAuth2.0: Easily add OAuth2.0 authentication to your APIs.
• Logging: Log requests and responses to your system over HTTP, TCP, UDP,
  or to disk.
• Security: ACL, Bot detection, whitelist/blacklist IPs, etc…
• Syslog: Logging to System log.
• SSL: Setup a Specific SSL Certificate for an underlying service or API.
• Monitoring: Live monitoring provides key load and performance server
  metrics.
• Forward Proxy: Make Kong connect to intermediary transparent HTTP proxies.
• Authentications: HMAC, JWT, Basic, and more.
• Rate-limiting: Block and throttle requests based on many variables.
• Transformations: Add, remove, or manipulate HTTP requests and responses.
• Caching: Cache and serve responses at the proxy layer.
• CLI: Control your Kong cluster from the command line.
• REST API: Kong can be operated with its RESTful API for maximum
  flexibility.
• Geo-Replicated: Configs are always up-to-date across different regions.
• Failure Detection & Recovery: Kong is unaffected if one of your Cassandra
  nodes goes down.
• Clustering: All Kong nodes auto-join the cluster keeping their config
  updated across nodes.
• Scalability: Distributed by nature, Kong scales horizontally by simply
  adding nodes.
• Performance: Kong handles load with ease by scaling and using NGINX at
  the core.
• Plugins: Extendable architecture for adding functionality to Kong and
  APIs.

OpenRestry

官网 和 GitHub Repo。
作者章亦春 (agentzh)，底层实现采用 Nginx + Lua 结合的方式进行，非常灵活，易于扩展。
Nginx 在 2018 年 4 月 17 号发布的版本 1.13.10 已经支持增加了对于 gRPC 的支持，
参见：Introducing gRPC Support with NGINX 1.13.10。

OpenRestry 当前支持最新版为 2018 年 5 月 14 号发布的 1.13.6.x ，还不支持 Nginx 1.13.10，其版本列表参见：changes.html。 倒数第二个版本 1.11.2 发布时间为 2017 年 8月 18 号。

Nginx 对于 HTTP 2 支持的相关情况可以参见 http2 , gRPC 支持的情况参见 gRPC。

Traefik

官网地址 和 GitHub Repo。

Træfik is a modern HTTP reverse proxy and load balancer that makes
deploying microservices easy. Træfik integrates with your existing infrastructure
components (Docker, Swarm mode,Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, …)
and configures itself automatically and dynamically. Pointing Træfik at your orchestrator
should be the onlyconfiguration step you need.

Traefik Features

• Continuously updates its configuration (No restarts!)
• Supports multiple load balancing algorithms
• Provides HTTPS to your microservices by leveraging Let’s Encrypt (wildcard certificates support)
• Circuit breakers, retry
• High Availability with cluster mode (beta)
• See the magic through its clean web UI
• Websocket, HTTP/2, GRPC ready
• Provides metrics (Rest, Prometheus, Datadog, Statsd, InfluxDB)
• Keeps access logs (JSON, CLF)
• Fast
• Exposes a Rest API
• Packaged as a single binary file (made with :heart: with go) and available as a tiny official docker image

Ambassador (Envoy Gateway)

官网 GitHub Repo。 Ambassador 底层基于 Envoy。由于 Envoy 天然支持 gRPC， 因此 Ambassador 也完全支持 gRPC 代理。

Ambassador is an open source Kubernetes-native API Gateway built on Envoy, designed for microservices. Ambassador essentially serves as an Envoy ingress controller, but with many more features.

Ambassador Features

• Self-service configuration, via Kubernetes annotations
• First class gRPC and HTTP/2 support
• Support for CORS, timeouts, weighted round robin (canary), rate limiting
• Istio integration
• Authentication
• Robust TLS support, including TLS client-certificate authentication

结论

• Kong CE 对于 gRPC 的支持取决于 OpenRestry 对于 Nginx 版本的支持情况，按照 OpenRestry 的开发进度，预计年底可能会支持 gRPC，可以持续关注。
• Traefix 对于 WebSocket、HTTP 2 和 GRPC 都提供了支持，可以作为重点测试对象。
• Ambassador 基于 Envoy 实现，将 gRPC 和 HTTP/2 作为一等公民来进行对待，对于 gRPC 的支持应该是最好的，但是考虑到其底层强依赖于 Kubernetes，因此可以在使用 Istio 的时候进行集成比较方面，持续关注。

特性\方案	Nginx	Kong（CE）	Traefik	Envoy（Ambassador）
性能	转发效率高	转发效率高	吞吐量 Nginx 85%	底层 C++，转发效率高
配置难易度	简单	简单	简单，与微服务对接方便	简单，ServiceMesh 原生支持
负载均衡机制	一般	可以灵活扩展	会话保持和健康机制	会话保持和健康机制
社区活跃度	高	一般	一般	一般
功能适用性	代理或者web/mail 服务器	代理或者负载均衡器	代理或负载均衡器	负载均衡器
支持平台	各种平台	各种平台	各种平台	主要是 Kubernets
K8S Ingress	ingress-nginx 成熟度高	kubernetes-ingress-controller 成熟度低	天然支持，成熟度高	天然支持，成熟度中
HTTP 2支持情况	支持	支持	支持	支持
gRPC 代理	支持 >=1.13.10	暂时不支持	支持	原生支持
与 ServiceMesh 集成度	良好	良好	良好	优秀，原生支持

Traefik gRPC 代理测试

本文以 traefik 作为测试对象，来作为 gRPC 支持情况。完整代码库参见：examples

特别备注： gRPC 在 Mac 环境下测试过程中，如果将 backend.local 写入到 /etc/hosts 中，使用 gRPC Client 使用 backend.local 地址进行访问的时候，不能够正常解析出来地址，问题待确认。后切换到 Linux 下则无此问题。

gRPC 依赖准备

# gRPC requires Go 1.6 or higher.
$ go version
go version go1.11

# Install gRPC 需要翻墙
$ go get -u google.golang.org/grpc

# install the protoc plugin for Go
$ go get -u github.com/golang/protobuf/protoc-gen-go

gRPC 测试样例参见 go quickstart

$ cd $GOPATH/src/google.golang.org/grpc/examples/helloworld)

# 启动服务端
$ go run greeter_server/main.go

# 启动客户端
$ go run greeter_client/main.go

# 期待能够正常工作

Traefik 配置

traefix grpc 样例配置参考 user-guide

As gRPC needs HTTP2, we need HTTPS certificates on both gRPC Server and Træfik.

# download traefix
$ wget https://github.com/containous/traefik/releases/download/v1.6.6/traefik_linux-amd64

生成服务端证书：

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/backend.key \
-out ./certs/backend.crt \
-subj "/CN=backend.local/O=backend.local"

生成客户端证书

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/frontend.key \
-out ./certs/frontend.crt \
-subj "/CN=frontend.local/O=frontend.local"

修改 grpc server 和 client 添加证书支持，同时修改 client 代码的连接地址，在本地 hosts 添加以下记录：

127.0.0.1 frontend.local
127.0.0.1 backend.local

traefik.toml 配置文件样例：

$ cat traefik.toml
defaultEntryPoints = ["https"]

logLevel = "INFO"

RootCAs = [ "../certs/backend.crt" ]

[entryPoints]
  [entryPoints.https]
  address = ":4443"
    [entryPoints.https.tls]
     [[entryPoints.https.tls.certificates]]
     certFile = "../certs/frontend.crt"
     keyFile  = "../certs/frontend.key"

[api]

[file]

[backends]
  [backends.backend1]
    [backends.backend1.servers.server1]
    url = "https://backend.local:50051"


[frontends]
  [frontends.frontend1]
  backend = "backend1"
    [frontends.frontend1.routes.test_1]
    rule = "Host:frontend.local"

[traefikLog]

[metrics]
  [metrics.prometheus]
    entryPoint = "traefik"

为方便启动，增加 start.sh 脚本：

$ cat start.sh
#!/bin/bash

./traefik --configFile=traefik.toml

traefik server 启动成功后可以，会启动一个 dashboard 地址，可以通过地址访问 http://172.16.71.9:8080/dashboard/，其中 172.16.71.9 为启动的 IP 地址。

greeter_client 必须使用域名访问，如果使用 IP 地址访问，会导致证书不匹配的错误：

$ go run greeter_client/main.go
2018/08/29 10:01:56 could not greet: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for backend.local, not localhost"
exit status 1

gRPC 配置证书

gRPC HelloWorld 样例程序采用非证书模式，为配合 traefik 测试，需要将服务端与客户端代码增加证书支持。

greet_server

// ...

// Read cert and key file
BackendCert, _ := ioutil.ReadFile("./backend.cert")
BackendKey, _ := ioutil.ReadFile("./backend.key")

// Generate Certificate struct
cert, err := tls.X509KeyPair(BackendCert, BackendKey)
if err != nil {
  log.Fatalf("failed to parse certificate: %v", err)
}

// Create credentials
creds := credentials.NewServerTLSFromCert(&cert)

// Use Credentials in gRPC server options
serverOption := grpc.Creds(creds)
var s *grpc.Server = grpc.NewServer(serverOption)
defer s.Stop()

pb.RegisterGreeterServer(s, &server{})
err := s.Serve(lis)

// ...

greet_client

// ...

// Read cert file
FrontendCert, _ := ioutil.ReadFile("./frontend.cert")

// Create CertPool
roots := x509.NewCertPool()
roots.AppendCertsFromPEM(FrontendCert)

// Create credentials
credsClient := credentials.NewClientTLSFromCert(roots, "")

// Dial with specific Transport (with credentials)
conn, err := grpc.Dial("frontend.local:4443", grpc.WithTransportCredentials(credsClient))
if err != nil {
    log.Fatalf("did not connect: %v", err)
}

defer conn.Close()
client := pb.NewGreeterClient(conn)

name := "World"
r, err := client.SayHello(context.Background(), &pb.HelloRequest{Name: name})

// ...

TODO 性能相关的数据收集

mdl -r~MD013 grpc_gateway.md 验证 md 的时候去除长度限制。

参考

1. Pattern: API Gateway / Backend for Front-End
2. Service Mesh 和 API Gateway的关系探讨
3. Building Microservices: Using an API Gateway
4. Using an API Gateway in Your Microservices Architecture
5. Turbo 能生成一个反向代理服务器，把HTTP请求转换为 grpc 或者 Thrift 格式的请求。 grpc-gateway的替代品–Turbo
6. Part 3: Deploying Envoy as an API Gateway for Microservices
7. gRPC through Traefik || Envoy
8. gRPC naming
9. Secure gRPC with TLS/SSL
10. 深入玩转K8S之如何访问业务应用（Traefik-ingress配置https篇）
11. OpenSSL自签发配置有多域名或ip地址的证书