Venil Noronha

Dead Reckoning for Databases

2026-05-09T00:00:00+00:00

One file. One source of truth. Write a .proto message, get a database table and type-safe Go bindings.

The Core Idea

Every Go service that touches a database ends up maintaining the same entity in at least two places: a Protobuf message for the API layer and a SQL table for persistence. Add a Go struct in between and you have three. They drift. You forget to migrate. You fat-finger a column rename.

The fix is to make the .proto file authoritative and derive everything else from it. Concretely:

A custom protoc plugin reads your message definition and emits a CREATE TABLE DDL file.
A reflection-based scanner maps SQLite rows back onto your proto-generated Go structs at runtime — no hand-written Scan(&a, &b, &c) calls.

We’re using SQLite via modernc.org/sqlite (a pure-Go driver, zero CGo) to keep the demo self-contained. The same pattern ports cleanly to Postgres or MySQL.

The Pipeline

The runtime has no knowledge of protoc or code generation — it just sees a regular Go struct and a *sql.DB. The generated DDL is applied once at startup via db.Exec(ddlString). reflectScan() in the diagram refers to the sqlreflect package covered later: it uses Go’s reflect package to map SQL column names back to struct fields automatically, so query results land in your struct without a single hand-written rows.Scan(...) call.

Annotating Your Proto

We use custom field options to embed storage hints directly in the message. This keeps column type overrides, primary key declarations, and index hints co-located with the field they describe.

// proto/options/db.proto
syntax = "proto3";
package options;

import "google/protobuf/descriptor.proto";

option go_package = "github.com/yourorg/app/gen/options";

message DbFieldOptions {
  string column_type = 1; // e.g. "TEXT", "INTEGER", "REAL"
  bool   primary_key = 2;
  bool   unique      = 3;
  bool   not_null    = 4;
  string default_val = 5;
  bool   index       = 6;
}

message DbMessageOptions {
  string table_name = 1;
}

extend google.protobuf.FieldOptions   { DbFieldOptions   db_field = 50001; }
extend google.protobuf.MessageOptions { DbMessageOptions db_msg   = 50002; }

Now the domain message. Notice how everything a DBA would want to know about storage is right here, next to the field definitions:

// proto/user/v1/user.proto
syntax = "proto3";
package user.v1;

import "proto/options/db.proto";

option go_package = "github.com/yourorg/app/gen/user/v1";

message User {
  option (options.db_msg) = { table_name: "users" };

  string id = 1 [(options.db_field) = {
    primary_key: true
    column_type: "TEXT"
    not_null:    true
  }];

  string email = 2 [(options.db_field) = {
    unique:   true
    not_null: true
  }];

  string display_name = 3 [(options.db_field) = {
    not_null: true
  }];

  int64 created_at_unix = 4 [(options.db_field) = {
    column_type: "INTEGER"
    not_null:    true
    default_val: "(unixepoch())"
  }];

  string org_id = 5 [(options.db_field) = {
    not_null: true
    index:    true
  }];
}

Generating DDL with a protoc Plugin

A protoc plugin is just a binary that reads a CodeGeneratorRequest from stdin and writes a CodeGeneratorResponse to stdout. The protogen package wraps this protocol cleanly.

// cmd/protoc-gen-sql/main.go
package main

import (
  "fmt"
  "strings"

  "google.golang.org/protobuf/compiler/protogen"
  "google.golang.org/protobuf/proto"
  "google.golang.org/protobuf/reflect/protoreflect"

  options "github.com/yourorg/app/gen/options"
)

func main() {
  protogen.Options{}.Run(func(gen *protogen.Plugin) error {
    for _, f := range gen.Files {
      if !f.Generate { continue }
      if err := generateSQL(gen, f); err != nil {
        return err
      }
    }
    return nil
  })
}

func generateSQL(gen *protogen.Plugin, f *protogen.File) error {
  var out strings.Builder

  for _, msg := range f.Messages {
    msgOpts, _ := proto.GetExtension(
      msg.Desc.Options(), options.E_DbMsg,
    ).(*options.DbMessageOptions)
    if msgOpts == nil { continue }

    table := msgOpts.GetTableName()
    out.WriteString(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n", table))

    var indexes []string

    for i, field := range msg.Fields {
      fo, _ := proto.GetExtension(
        field.Desc.Options(), options.E_DbField,
      ).(*options.DbFieldOptions)

      col  := toSnake(string(field.Desc.Name()))
      typ  := sqliteType(field.Desc.Kind(), fo)
      line := fmt.Sprintf("  %s %s", col, typ)

      if fo != nil {
        if fo.GetPrimaryKey() { line += " PRIMARY KEY" }
        if fo.GetUnique()     { line += " UNIQUE" }
        if fo.GetNotNull()    { line += " NOT NULL" }
        if d := fo.GetDefaultVal(); d != "" {
          line += " DEFAULT " + d
        }
        if fo.GetIndex() {
          indexes = append(indexes,
            fmt.Sprintf("CREATE INDEX IF NOT EXISTS idx_%s_%s ON %s (%s);",
              table, col, table, col))
        }
      }

      if i < len(msg.Fields)-1 { line += "," }
      out.WriteString(line + "\n")
    }

    out.WriteString(");\n\n")
    for _, idx := range indexes {
      out.WriteString(idx + "\n")
    }
  }

  g := gen.NewGeneratedFile(f.GeneratedFilenamePrefix+"_schema.sql", "")
  g.P(out.String())
  return nil
}

func sqliteType(k protoreflect.Kind, fo *options.DbFieldOptions) string {
  if fo != nil && fo.GetColumnType() != "" { return fo.GetColumnType() }
  switch k {
  case protoreflect.BoolKind,
       protoreflect.Int32Kind, protoreflect.Int64Kind,
       protoreflect.Sint32Kind, protoreflect.Sint64Kind:
    return "INTEGER"
  case protoreflect.FloatKind, protoreflect.DoubleKind:
    return "REAL"
  case protoreflect.BytesKind:
    return "BLOB"
  case protoreflect.MessageKind:
    return "TEXT" // JSON-encode nested messages
  default:
    return "TEXT"
  }
}

The plugin emits this DDL — which you apply at startup via a single db.Exec():

-- generated: proto/user/v1/user_schema.sql
CREATE TABLE IF NOT EXISTS users (
  id               TEXT    PRIMARY KEY NOT NULL,
  email            TEXT    UNIQUE NOT NULL,
  display_name     TEXT    NOT NULL,
  created_at_unix  INTEGER NOT NULL DEFAULT (unixepoch()),
  org_id           TEXT    NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_users_org_id ON users (org_id);

Scanning Rows Back via Reflection

Here’s where it gets interesting. Instead of writing rows.Scan(&u.Id, &u.Email, ...) for every query, we use Go’s reflect package to match column names to struct fields automatically. Because protoc-gen-go names struct fields in PascalCase and our columns are snake_case, a simple converter bridges the gap.

// internal/sqlreflect/scan.go
package sqlreflect

import (
  "database/sql"
  "fmt"
  "reflect"
  "strings"
  "unicode"
)

// ScanRow scans a single *sql.Rows into a proto-generated struct pointer.
// dst must be a non-nil pointer to a struct.
func ScanRow[T any](row *sql.Rows, dst *T) error {
  cols, err := row.Columns()
  if err != nil { return err }

  rv := reflect.ValueOf(dst).Elem() // dereference pointer
  rt := rv.Type()

  // Build col → field index map once per call
  fieldByCol := make(map[string]int, rt.NumField())
  for i := 0; i < rt.NumField(); i++ {
    fieldByCol[toSnake(rt.Field(i).Name)] = i
  }

  ptrs := make([]any, len(cols))
  for i, col := range cols {
    if idx, ok := fieldByCol[col]; ok {
      ptrs[i] = rv.Field(idx).Addr().Interface()
    } else {
      var discard any
      ptrs[i] = &discard // unknown column — silently skip
    }
  }
  return row.Scan(ptrs...)
}

// ScanRows collects all rows into a slice.
func ScanRows[T any](rows *sql.Rows) ([]T, error) {
  var out []T
  for rows.Next() {
    var dst T
    if err := ScanRow(rows, &dst); err != nil { return nil, err }
    out = append(out, dst)
  }
  return out, rows.Err()
}

// toSnake converts "DisplayName" → "display_name"
func toSnake(s string) string {
  var b strings.Builder
  for i, r := range s {
    if unicode.IsUpper(r) && i > 0 { b.WriteByte('_') }
    b.WriteRune(unicode.ToLower(r))
  }
  return b.String()
}

Performance note: The column→field map is rebuilt on every call here for clarity. In a hot path, cache it in a sync.Map keyed on reflect.TypeOf(dst). The allocation cost is negligible for most workloads.

A Zero-Boilerplate Repository

We can go one step further: generate the repository itself. A second plugin, protoc-gen-repo, reads the same field options and emits a complete Go repository — no hand-written code at all. The rules are simple:

primary_key or unique → GetBy{Field}(ctx, val) (*T, error)
index → ListBy{Field}(ctx, val) ([]*T, error)
Fields with a default_val are omitted from INSERT (the DB supplies the default)

The template lives in its own file — a plain text file where backticks in SQL strings cause no issues. The plugin loads it at compile time with //go:embed and only concerns itself with collecting data to pass in.

// cmd/protoc-gen-repo/repo.tmpl
package repo

import (
	"context"
	"database/sql"
	_ "embed"

	_ "modernc.org/sqlite"
	{{.Pkg}} "{{.Imp}}"
	scan "github.com/yourorg/app/internal/sqlreflect"
)

//go:embed {{.SchemaFile}}
var {{.Lower}}Schema string

type {{.GoType}}Repo struct{ db *sql.DB }

func New{{.GoType}}Repo(path string) (*{{.GoType}}Repo, error) {
	db, err := sql.Open("sqlite", path)
	if err != nil { return nil, err }
	if _, err = db.Exec({{.Lower}}Schema); err != nil { return nil, err }
	return &{{.GoType}}Repo{db: db}, nil
}

func (r *{{.GoType}}Repo) Create(ctx context.Context, v *{{.Pkg}}.{{.GoType}}) error {
	_, err := r.db.ExecContext(ctx,
		`INSERT INTO {{.Table}} ({{.InsertCols}}) VALUES ({{.Placeholders}})`,
		{{.InsertVals}},
	)
	return err
}
{{range .Methods}}
func (r *{{.GoType}}Repo) {{.Name}}(ctx context.Context, val {{.GoKind}}) {{.Return}} {
	rows, err := r.db.QueryContext(ctx,
		`SELECT * FROM {{.Table}} WHERE {{.Col}} = ?`, val)
	if err != nil { return nil, err }
	defer rows.Close()
	{{- if .IsList}}
	items, err := scan.ScanRows[{{.Pkg}}.{{.GoType}}](rows)
	if err != nil { return nil, err }
	out := make([]*{{.Pkg}}.{{.GoType}}, len(items))
	for i := range items { out[i] = &items[i] }
	return out, nil
	{{- else}}
	if !rows.Next() { return nil, sql.ErrNoRows }
	var v {{.Pkg}}.{{.GoType}}
	return &v, scan.ScanRow(rows, &v)
	{{- end}}
}
{{end}}

// cmd/protoc-gen-repo/main.go
package main

import (
  _ "embed"
  "strings"
  "text/template"
  "unicode"

  "google.golang.org/protobuf/compiler/protogen"
  "google.golang.org/protobuf/proto"
  "google.golang.org/protobuf/reflect/protoreflect"

  options "github.com/yourorg/app/gen/options"
)

//go:embed repo.tmpl
var repoTmplSrc string

var repoTmpl = template.Must(template.New("repo").Parse(repoTmplSrc))

type repoData struct {
  Pkg, Imp, GoType, Lower, Table, SchemaFile string
  InsertCols, Placeholders, InsertVals       string
  Methods                                    []methodData
}

type methodData struct {
  Name, GoType, Pkg, Table, Col, GoKind, Return string
  IsList                                         bool
}

func main() {
  protogen.Options{}.Run(func(gen *protogen.Plugin) error {
    for _, f := range gen.Files {
      if !f.Generate { continue }
      generateRepo(gen, f)
    }
    return nil
  })
}

func generateRepo(gen *protogen.Plugin, f *protogen.File) {
  for _, msg := range f.Messages {
    msgOpts, _ := proto.GetExtension(
      msg.Desc.Options(), options.E_DbMsg,
    ).(*options.DbMessageOptions)
    if msgOpts == nil { continue }

    goType := msg.GoIdent.GoName
    lower  := strings.ToLower(goType)
    data := repoData{
      Pkg: lower, Imp: string(f.GoImportPath),
      GoType: goType, Lower: lower,
      Table:      msgOpts.GetTableName(),
      SchemaFile: f.GeneratedFilenamePrefix + "_schema.sql",
    }

    // Collect columns for INSERT, skipping fields with a server-side default.
    var cols, vals []string
    for _, field := range msg.Fields {
      fo, _ := proto.GetExtension(
        field.Desc.Options(), options.E_DbField,
      ).(*options.DbFieldOptions)
      if fo != nil && fo.GetDefaultVal() != "" { continue }
      col := toSnake(string(field.Desc.Name()))
      cols = append(cols, col)
      vals = append(vals, "v."+toPascal(col))
    }
    data.InsertCols   = strings.Join(cols, ", ")
    data.Placeholders = strings.TrimSuffix(strings.Repeat("?, ", len(cols)), ", ")
    data.InsertVals   = strings.Join(vals, ", ")

    // One query method per annotated field.
    for _, field := range msg.Fields {
      fo, _ := proto.GetExtension(
        field.Desc.Options(), options.E_DbField,
      ).(*options.DbFieldOptions)
      if fo == nil { continue }
      col    := toSnake(string(field.Desc.Name()))
      goKind := kindToGoType(field.Desc.Kind())

      if fo.GetPrimaryKey() || fo.GetUnique() {
        data.Methods = append(data.Methods, methodData{
          Name: "GetBy" + toPascal(col), GoType: goType, Pkg: lower,
          Table: data.Table, Col: col, GoKind: goKind,
          Return: "*" + lower + "." + goType + ", error",
        })
      } else if fo.GetIndex() {
        data.Methods = append(data.Methods, methodData{
          Name: "ListBy" + toPascal(col), GoType: goType, Pkg: lower,
          Table: data.Table, Col: col, GoKind: goKind,
          Return: "[]*" + lower + "." + goType + ", error",
          IsList: true,
        })
      }
    }

    var buf strings.Builder
    repoTmpl.Execute(&buf, data)
    g := gen.NewGeneratedFile(f.GeneratedFilenamePrefix+"_repo.go", f.GoImportPath)
    g.P(buf.String())
  }
}

func kindToGoType(k protoreflect.Kind) string {
  switch k {
  case protoreflect.BoolKind:                            return "bool"
  case protoreflect.Int32Kind, protoreflect.Sint32Kind:  return "int32"
  case protoreflect.Int64Kind, protoreflect.Sint64Kind:  return "int64"
  case protoreflect.FloatKind:                           return "float32"
  case protoreflect.DoubleKind:                          return "float64"
  case protoreflect.BytesKind:                           return "[]byte"
  default:                                               return "string"
  }
}

func toPascal(s string) string {
  var b strings.Builder
  upper := true
  for _, r := range s {
    if r == '_' { upper = true; continue }
    if upper { b.WriteRune(unicode.ToUpper(r)); upper = false } else { b.WriteRune(r) }
  }
  return b.String()
}

Running this against user.proto produces the following — the entire repository, derived purely from the annotations already in the .proto file:

// generated: proto/user/v1/user_repo.go
package repo

import (
  "context"
  "database/sql"
  _ "embed"

  _ "modernc.org/sqlite"
  userv1 "github.com/yourorg/app/gen/user/v1"
  scan   "github.com/yourorg/app/internal/sqlreflect"
)

//go:embed proto/user/v1/user_schema.sql
var userSchema string

type UserRepo struct{ db *sql.DB }

func NewUserRepo(path string) (*UserRepo, error) {
  db, err := sql.Open("sqlite", path)
  if err != nil { return nil, err }
  if _, err = db.Exec(userSchema); err != nil { return nil, err }
  return &UserRepo{db: db}, nil
}

func (r *UserRepo) Create(ctx context.Context, v *userv1.User) error {
  _, err := r.db.ExecContext(ctx,
    `INSERT INTO users (id, email, display_name, org_id) VALUES (?, ?, ?, ?)`,
    v.Id, v.Email, v.DisplayName, v.OrgId,
  )
  return err
}

func (r *UserRepo) GetById(ctx context.Context, val string) (*userv1.User, error) {
  rows, err := r.db.QueryContext(ctx,
    `SELECT * FROM users WHERE id = ?`, val)
  if err != nil { return nil, err }
  defer rows.Close()
  if !rows.Next() { return nil, sql.ErrNoRows }
  var v userv1.User
  return &v, scan.ScanRow(rows, &v)
}

func (r *UserRepo) GetByEmail(ctx context.Context, val string) (*userv1.User, error) {
  rows, err := r.db.QueryContext(ctx,
    `SELECT * FROM users WHERE email = ?`, val)
  if err != nil { return nil, err }
  defer rows.Close()
  if !rows.Next() { return nil, sql.ErrNoRows }
  var v userv1.User
  return &v, scan.ScanRow(rows, &v)
}

func (r *UserRepo) ListByOrgId(ctx context.Context, val string) ([]*userv1.User, error) {
  rows, err := r.db.QueryContext(ctx,
    `SELECT * FROM users WHERE org_id = ?`, val)
  if err != nil { return nil, err }
  defer rows.Close()
  items, err := scan.ScanRows[userv1.User](rows)
  if err != nil { return nil, err }
  out := make([]*userv1.User, len(items))
  for i := range items { out[i] = &items[i] }
  return out, nil
}

Wiring it all together

A single protoc invocation drives all three plugins, producing .pb.go, _schema.sql, and _repo.go in one pass:

.PHONY: generate
generate:
	protoc \
	  --go_out=gen  --go_opt=paths=source_relative \
	  --sql_out=gen --sql_opt=paths=source_relative \
	  --repo_out=gen --repo_opt=paths=source_relative \
	  proto/user/v1/user.proto

A single make generate now produces all three artifacts from one .proto file.

Using the generated repository

repo, err := NewUserRepo("app.db")
if err != nil { log.Fatal(err) }

// Insert a new user.
err = repo.Create(ctx, &userv1.User{
  Id:          "u_123",
  Email:       "alice@example.com",
  DisplayName: "Alice",
  OrgId:       "org_456",
})

// Look up by primary key.
user, err := repo.GetById(ctx, "u_123")

// Look up by unique field.
user, err = repo.GetByEmail(ctx, "alice@example.com")

// List all users in an org.
users, err := repo.ListByOrgId(ctx, "org_456")

Every method, its signature, and its SQL query were derived entirely from the annotations in user.proto — no repository code was written by hand.

Tradeoffs

What you gain:

One file to rule them all — add a field to the proto, re-run make generate, done.
No ORM magic — plain SQL queries, full visibility into what hits the DB.
Embeddable binary — schema ships inside the binary via //go:embed.
Natural relations — when two proto messages reference each other (e.g. User and Org), annotate both with db_msg and use an index: true field on the foreign key. Each message gets its own table and its own generated repository; joining them is a plain SQL query rather than a nested-message blob.

Watch out for:

Reflection cost — cache the column map for hot paths.
Proto field numbers vs column order — proto field numbers are for wire encoding; column order in SELECT * is defined by DDL order. Keep them aligned or always name columns explicitly.
Schema migrations — CREATE TABLE IF NOT EXISTS is idempotent for creation but won’t add new columns to an existing table. Use a migration tool like golang-migrate or Atlas for production schema evolution.
Complex types — nested proto messages become TEXT (JSON-encoded). Fine for rarely-queried data; a poor fit for deeply queryable nested structures.

Beyond SQLite — Porting to Other Databases

The same proto annotation trick extends beyond SQL. Point a different plugin at your .proto file and emit a MongoDB validator schema, a DynamoDB attribute map, or a Redis hash structure. The plugin ecosystem is the leverage point — the proto file stays the same regardless of where the data lands.

Conclusion

The .proto file has always been the contract between services. This post extends that idea one step further: make it the contract between your code and your database too. A custom protoc plugin turns field annotations into DDL; a reflection-based scanner maps rows back to structs without manual binding; a second plugin generates the entire repository from the same source. The result is a system where adding a field, an index, or a new query method is a single edit in one file followed by make generate — no drift, no duplication, no boilerplate.

Disclaimer: My postings are my own and don’t necessarily represent Stripe’s positions, strategies or opinions.

Introduction to Original Destination in Envoy

2021-06-26T19:15:00+00:00

Envoy is a service mesh proxy with a ton of built-in capabilities. In this post, I’ll be discussing the Original Destination feature in Envoy.

Read it on Medium

Disclaimer: My postings are my own and don’t necessarily represent Stripe’s positions, strategies or opinions.

A Beginner’s Guide to AuthN and AuthZ with Istio

2021-03-02T00:04:00+00:00

Service meshes solve some of the key challenges in the cloud-native world today, and in this post I’ll be discussing about security.

The Istio service mesh provides several security features including identity assignment for workloads, TLS encryption, AuthN (Authentication), AuthZ (Authorization), and more. In this article, I’ll be focusing mainly on the aspect of AuthN and AuthZ policies.

Introduction

In our day-to-day lives we provide some form of identification to avail services such as healthcare, access services like email, etc. Authentication is the process of verifying these identities in order for one to be granted access to a system.

In the example of a healthcare system, providers may have access to a wide array of patient data, while patients would have limited access i.e. typically to their own data. Both entities here have access to the same system, but it’s their role that determines the kind of data available to them. Simply put, Authorization is the process of protecting a system by granting users permissions to access only the resources they’re entitled to.

Sessions

Web applications leverage the concept of sessions for the purpose of AuthN and AuthZ. A session is used to identify and store temporary information related to a user over several requests as the user interacts with the system. Typically, a session is established at the time of login and is torn down when the user logs out.

There are mainly 2 types of sessions:

Server-side sessions
Client-side sessions

Server-Side Sessions

Server-side session implementations typically generate a random ID for the authenticating user and it is stored in the browser in the form of a cookie. The cookie is then presented to the server on each request made by the user, thereby allowing it to identify and store information related to the user.

One of the main challenges with server-side sessions is that it makes scaling difficult. If you need to scale the number of web server instances, you’d either need to replicate session information across all the instances, or ensure that a given user’s requests always land on the same instance.

Client-Side Sessions

Client-side session implementations also make use of cookies; however, instead of storing just a random ID, useful user information is embedded in the cookie.

Client-side sessions allow for better scalability on the server side as the overhead of mapping an ID to a user entity disappears. This also means that we don’t need to replicate user session information across web server instances, and each instance can be stateless i.e. a request could be routed to any instance and it should be able to easily identify the user using the details directly available in the cookie.

In this article I’ll be focusing on one such implementation called JWT. You can read more about sessions in this article.

JWTs (JSON Web Tokens)

JWTs (JSON Web Tokens), pronounced jot, are essentially JSON objects that are encoded and digitally signed to protect their integrity. JWTs are made up of 3 Base64-URL segments (header, payload, and signature) separated by dots. The header specifies the algorithm used for signing. The payload holds claims i.e. information about the user. The signature can be generated using an alogrithm like HMAC with a secret, or RSA using a pair of public/private keys.

For more information on JWTs you can refer to this article.

A Sample JWT

Let’s look at the sample JWT below. There are 3 segments, separated by the 2 white dots. The first segment in green is the header, the second one in pink is the payload, and the third one in blue is the signature. This JWT is signed using the HS256 HMAC algorithm.

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

You can decode this JWT using the following commands.

# Export the JWT string into a variable.
$ export JWT='eyJhbGc....dQssw5c'

# Decode the header.
$ echo $JWT -s | cut -d '.' -f 1 | base64 -d
{"alg":"HS256","typ":"JWT"}

# Decode the payload.
$ echo $JWT -s | cut -d '.' -f 2 | base64 -d
{"sub":"1234567890","name":"John Doe","iat":1516239022}

Generating a JWT

Let’s now look at how you could generate your own JWT and sign it. In this example, I’ll be using a pair of public/private keys and sign the JWT using the RSA256 RSA algorithm.

To generate an RSA256 signature for the JWT, we first need to generate a pair of public/private keys. This can be done like so:

$ openssl genrsa -des3 -out private_encrypted.pem 2048
$ openssl rsa -pubout -in private_encrypted.pem -out public.pem
$ openssl rsa -in private_encrypted.pem -out private.pem -outform PEM

Next, we can generate the JWT using the Python Authlib library.

# Code to generate the JWT.
$ cat <<EOF | python3 -
from authlib.jose import jwt
header = {'alg': 'RS256'}
payload = {'iss': 'venilnoronha.io', 'sub': 'user1', 'exp': 1257894000}
private_key = open('private.pem', 'r').read()
bytes = jwt.encode(header, payload, private_key)
print(bytes.decode('utf-8'))
EOF

# The generated JWT.
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ2ZW5pbG5vcm9uaGEuaW8iLCJzdWIiOiJ1c2VyMSIsImV4cCI6MTI1Nzg5NDAwMH0.fmMlvXcDrDw1_yCVUSxTFP0ptOEr94-Yo2QXGUc19qitYg8zpVWUkEBvty1dlsWw9nRkQUFzFoq6plYly_UbKQIbIFwoRepiicrqLDBFuNEnknks8c9-b5AMTZl6IrY7uWhjHquMR7zTRnLVjHaJYI_dH-wlMZpet5snGbGFl2HxfUm03HrdjVP0ICapPUUt5L0MZzu1CoJ1eoS0fceWf2nORZeIwhl45g4edlrHc3GwoTDa4GilS88W8UEFZxPs_GDYKpnv6Z5eMdccwa2H0FQEWxb717bBQHS2_C6oX3_MLkm86wXetg9_GVWQudHicI0x90BDRdX8GLrrU3rG1A

Note that I have set the following 3 claims in the JWT payload.

iss identifies the issuer of the JWT.
sub refers to the subject of the JWT.
exp is the expiration time after which the JWT shouldn’t be accepted in the system.

To learn more about JWT claims, you can refer this RFC.

Verifying a JWT

We can use the same Authlib library to parse and validate the generated JWT as shown below.

# Export the JWT to a variable.
$ export JWT="eyJhbGc....rU3rG1A"

# Code to verify the JWT.
$ cat <<EOF | python3 -
from authlib.jose import jwt
public_key = open('public.pem', 'r').read()
claims = jwt.decode('$JWT', public_key)
claims.validate()
print(claims)
EOF

# The parsed claims from the JWT.
{'iss': 'venilnoronha.io', 'sub': 'user1', 'exp': 1257894000}

Web applications typically run these validations and process JWT claims for providing users access to resources. Although straightforward, this adds overhead on application developers and we can simplify this to a large extent using AuthN/AuthZ policies in Istio.

AuthN/AuthZ Policies in Istio

Istio supports mainly 2 Kubernetes CRDs (Custom Resource Definitions) in the context of AuthZ/AuthZ.

RequestAuthentication enables authentication of requests based on authentication information in requests, and configured rules.
AuthorizationPolicy enables authorization of requests based on apply/deny rules configured for a workload.

RequestAuthentication in Istio

Let’s now look at a RequestAuthentication policy in action. To do that I’ll first deploy the httpbin and sleep applications into my cluster and make a few HTTP requests to observe the behavior.

# Deploy the httpbin and sleep applications.
$ kubectl apply -f istio-1.9.0/samples/httpbin/httpbin.yaml
$ kubectl apply -f istio-1.9.0/samples/sleep/sleep.yaml

# Create an alias for executing the curl command from within the sleep pod.
$ alias kcurl="kubectl exec sleep-67d8bf54cb-fh5x2 -- curl"

# Send an HTTP GET request from the sleep pod to the httpbin service.
$ kcurl -sI -X GET httpbin:8000/get | head -n1
HTTP/1.1 200 OK

# Send an HTTP POST request from the sleep pod to the httpbin service.
$ kcurl -sI -X POST httpbin:8000/post | head -n1
HTTP/1.1 200 OK

Next, we can apply a RequestAuthentication policy to validate JWTs that are sent in the requests to perform authentication. To do that, we’ll first generate a JWK (JSON Web Key) from the public key we previously generated.

# Code to generate the JWK.
$ cat <<EOF | python3 -
from authlib.jose import jwk
public_key = open('public.pem', 'r').read()
key = jwk.dumps(public_key, kty='RSA')
print(key)
EOF

# The generated JWK.
{'n': 'ueebNpkIO0Jxog3U9NEQbgix8BV0MQZAvjIFnGs4OpL8_W2xJBFORcDJFE58bR0MMfnrD342hQMkOmGKoBoBzu01D5aoPR0TIusLk8JeLwa7x8uC5gSCOAbpkIhRrYGhMs0LtsQP481p5mdZuzIbLpWn7yHf1KV1Tq7wsWlT7y1xA52msa2rN_z7loUaaRJ2OBSIeFzkhD8IS2kRLCjGdove3kUXtTy2KzlyNMVo_2GjK0lrydZwlXlD8CUDMVYSpQUOOeH_LMFf2otu92UAUsEEcxvHaTPJ1GWCp7qhyy5q6V1GA5K-1ZfUUZr_CuChmITeXF9lV2701fwWCfhILQ', 'e': 'AQAB', 'kty': 'RSA'}

At this point, we can create the Istio RequestAuthentication policy to validate JWTs in incoming requests to httpbin. Note that you could also provide the JWK by setting the jwksUri instead of passing in the JSON object in the jwks field.

$ cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: request-auth
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - forwardOriginalToken: true
    issuer: venilnoronha.io
    jwks: |
      {'keys': [{'n': 'ueebNpk....WCfhILQ', 'e': 'AQAB', 'kty': 'RSA'}]}
EOF

To verify that the authentication policy is in effect, we can send 2 requests from the sleep application to the httpbin application, one with an expired JWT and one with a valid JWT. You can easily generate these JWTs using the code snippet from the Generating a JWT section and using the payload as described below.

# Payload for generating an expired JWT.
payload = {'iss': 'venilnoronha.io', 'sub': 'user1', 'exp': 1257894000}

# Payload for generating a JWT without expiry.
payload = {'iss': 'venilnoronha.io', 'sub': 'user1'}

We can now send requests and verify the behavior.

# JWT verification fails.
$ export EXPIRED_JWT=eyJhbGc...rU3rG1A
$ kcurl -sI -H "Authorization: Bearer $EXPIRED_JWT" -X GET httpbin:8000/get | head -n1
HTTP/1.1 401 Unauthorized

# JWT verification succeeds.
$ export VALID_JWT=eyJhbGc...8qLwfAA
$ kcurl -sI -H "Authorization: Bearer $VALID_JWT" -X GET httpbin:8000/get | head -n1
HTTP/1.1 200 OK

Note that by default Istio will validate the JWT only if the Authorization header is specified. If not, it will allow the request unless there’s also a AuthorizationPolicy defined to reject it.

AuthorizationPolicy in Istio

Now that we understand how Istio performs authorization, we can go one step further and define a AuthorizationPolicy to perform access control using JWT claims. We can begin by creating a new valid JWT for another user user2 using the following payload.

payload = {'iss': 'venilnoronha.io', 'sub': 'user2'}

We can now apply a simple AuthorizationPolicy to allow user1 to only make GET calls, and for user2 to make calls only to the /status/* endpoints.

$ cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: auth-policy
spec:
  selector:
    matchLabels:
      app: httpbin
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["venilnoronha.io/user1"]
    to:
    - operation:
        methods: ["GET"]
  - from:
    - source:
        requestPrincipals: ["venilnoronha.io/user2"]
    to:
    - operation:
        paths: ["/status/*"]
EOF

In this example, I’ve set action to ALLOW, but you could specify policies with action set to DENY in which case those rules will be evaluated prior to rules that have it set to ALLOW.

Let’s make a few requests and observe the behavior of Istio with this authorization policy.

# Export the JWT for user2 into a variable.
$ export VALID_JWT2=eyJhbGc....AWO0LZw

# GET /headers returns a valid response for the JWT token for user1.
$ kcurl -sI -H "Authorization: Bearer $VALID_JWT" -X GET httpbin:8000/headers | head -n1
HTTP/1.1 200 OK

# GET /get returns a valid response for the JWT token for user1.
$ kcurl -sI -H "Authorization: Bearer $VALID_JWT" -X GET httpbin:8000/get | head -n1
HTTP/1.1 200 OK

# POST /post returns an error for the JWT token for user1.
$ kcurl -sI -H "Authorization: Bearer $VALID_JWT" -X POST httpbin:8000/post | head -n1
HTTP/1.1 403 Forbidden

# GET /get returns an error for the JWT token for user2.
$ kcurl -sI -H "Authorization: Bearer $VALID_JWT2" -X GET httpbin:8000/get | head -n1
HTTP/1.1 403 Forbidden

# GET /status/418 returns a valid response for the JWT token for user2.
$ kcurl -s -H "Authorization: Bearer $VALID_JWT2" -X GET httpbin:8000/status/418

    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

In my examples above, policies were specifically applied to the httpbin application using the selector attribute, but you could apply such policies to the Istio gateways too.

Conclusion

Istio can easily perform AuthN and AuthZ using simple policies as we observed in the examples above. This can greatly simplify the life of application developers as access control is taken care of by the service mesh itself. I used a simple HTTP application to showcase the power of policies in Istio; however, you could apply similar policies for gRPC traffic too. These policies can be further customized by specifying conditions using other supported attributes defined on this page.

I’d love to hear your feedback!

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

A Simple State Machine Framework in Go

2020-05-24T23:51:00+00:00

The concept of State Machines is not new, and they form the core of a lot of systems that we come across daily, for example, elevators and traffic lights. State machines make it a lot easy to understand and implement multi-faceted systems where inputs can be asynchronous and are triggered by multiple sources.

In this post, I will describe the details of a simple state machine framework that I built to handle a complex scenario to obtain deterministic results.

Introduction

State machines are of several different types and newer models (like Statecharts) have also been designed in order to handle extremely complicated systems. The fundamentals of all of these patterns remain the same i.e. they consist of states and transitions. For the purpose of this post, I’ll be using the term state to describe the status of the system, and event to describe an input to the FSM (Finite State Machine) that can possibly trigger a transition from one state to another.

Light Switch Example

Let’s take a simple example of the light switch state machine that’s shown above. The machine starts in the off state and the user action of switching on i.e. triggering the switch-on event, causes the machine to transition over to the on state. And the reverse is also true, i.e. on triggering the switch-off event, the machine then transitiona back from the on state back to the off state. Note that triggering the switch-off event when in the off state means nothing i.e. the machine doesn’t react to it, and the same is also true when the switch-on event is triggered when the system is in the on state.

The Framework

Here is the framework in Go in under 100 lines of code.

// ErrEventRejected is the error returned when the state machine cannot process
// an event in the state that it is in.
var ErrEventRejected = errors.New("event rejected")

const (
	// Default represents the default state of the system.
	Default StateType = ""

	// NoOp represents a no-op event.
	NoOp EventType = "NoOp"
)

// StateType represents an extensible state type in the state machine.
type StateType string

// EventType represents an extensible event type in the state machine.
type EventType string

// EventContext represents the context to be passed to the action implementation.
type EventContext interface{}

// Action represents the action to be executed in a given state.
type Action interface {
	Execute(eventCtx EventContext) EventType
}

// Events represents a mapping of events and states.
type Events map[EventType]StateType

// State binds a state with an action and a set of events it can handle.
type State struct {
	Action Action
	Events Events
}

// States represents a mapping of states and their implementations.
type States map[StateType]State

// StateMachine represents the state machine.
type StateMachine struct {
	// Previous represents the previous state.
	Previous StateType

	// Current represents the current state.
	Current StateType

	// States holds the configuration of states and events handled by the state machine.
	States States

	// mutex ensures that only 1 event is processed by the state machine at any given time.
	mutex sync.Mutex
}

// getNextState returns the next state for the event given the machine's current
// state, or an error if the event can't be handled in the given state.
func (s *StateMachine) getNextState(event EventType) (StateType, error) {
	if state, ok := s.States[s.Current]; ok {
		if state.Events != nil {
			if next, ok := state.Events[event]; ok {
				return next, nil
			}
		}
	}
	return Default, ErrEventRejected
}

// SendEvent sends an event to the state machine.
func (s *StateMachine) SendEvent(event EventType, eventCtx EventContext) error {
	s.mutex.Lock()
	defer s.mutex.Unlock()

	for {
		// Determine the next state for the event given the machine's current state.
		nextState, err := s.getNextState(event)
		if err != nil {
			return ErrEventRejected
		}

		// Identify the state definition for the next state.
		state, ok := s.States[nextState]
		if !ok || state.Action == nil {
			// configuration error
		}

		// Transition over to the next state.
		s.Previous = s.Current
		s.Current = nextState

		// Execute the next state's action and loop over again if the event returned
		// is not a no-op.
		nextEvent := state.Action.Execute(eventCtx)
		if nextEvent == NoOp {
			return nil
		}
		event = nextEvent
	}
}

For the sake of brevity, I’ve skipped over database handling in this post. It’s also quite easy to wire into this state machine framework.

Light Switch Example in Go

Here is the same light switch example from above, but this time implemented using the Go framework. Let’s first define the states and events for the light switch state machine.

const (
	Off StateType = "Off"
	On  StateType = "On"

	SwitchOff EventType = "SwitchOff"
	SwitchOn  EventType = "SwitchOn"
)

We can now also define the actions corresponding to each state.

// OffAction represents the action executed on entering the Off state.
type OffAction struct{}

func (a *OffAction) Execute(eventCtx EventContext) EventType {
	fmt.Println("The light has been switched off")
	return NoOp
}

// OnAction represents the action executed on entering the On state.
type OnAction struct{}

func (a *OnAction) Execute(eventCtx EventContext) EventType {
	fmt.Println("The light has been switched on")
	return NoOp
}

We can now define our state machine using the previously defined states, events and actions.

func newLightSwitchFSM() *StateMachine {
  return &StateMachine{
		States: States{
			Default: State{
				Events: Events{
					SwitchOff: Off,
				},
			},
			Off: State{
				Action: &OffAction{},
				Events: Events{
					SwitchOn: On,
				},
			},
			On: State{
				Action: &OnAction{},
				Events: Events{
					SwitchOff: Off,
				},
			},
		},
	}
}

Let’s now put our light switch state machine to test.

func TestLightSwitchStateMachine(t *testing.T) {
	// Create a new instance of the light switch state machine.
	lightSwitchFsm := newLightSwitchFSM()

	// Set the initial "off" state in the state machine.
	err := lightSwitchFsm.SendEvent(SwitchOff, nil)
	if err != nil {
		t.Errorf("Couldn't set the initial state of the state machine, err: %v", err)
	}

	// Send the switch-off event again and expect the state machine to return an error.
	err = lightSwitchFsm.SendEvent(SwitchOff, nil)
	if err != ErrEventRejected {
		t.Errorf("Expected the event rejected error, got nil")
	}

	// Send the switch-on event and expect the state machine to transition to the
	// "on" state.
	err = lightSwitchFsm.SendEvent(SwitchOn, nil)
	if err != nil {
		t.Errorf("Couldn't switch the light on, err: %v", err)
	}

	// Send the switch-on event again and expect the state machine to return an error.
	err = lightSwitchFsm.SendEvent(SwitchOn, nil)
	if err != ErrEventRejected {
		t.Errorf("Expected the event rejected error, got nil")
	}

	// Send the switch-off event and expect the state machine to transition back
	// to the "off" state.
	err = lightSwitchFsm.SendEvent(SwitchOff, nil)
	if err != nil {
		t.Errorf("Couldn't switch the light off, err: %v", err)
	}
}

As you see above, the whole state machine becomes very clear just by looking at the lightSwitchFsm definition. The OffAction and OnAction implement the Action interface from the framework and in this simple example they both print a statement to stdout. Events are sent to the state machine using statements like lightSwitchFsm.SendEvent(SwitchOn, nil) where the first parameter is the event itself and the second parameter represents the EventContext. In this example, there is no need for passing any context, so we send nil, but we will also see another example that showcases its usage.

Following is the output from executing the above test.

=== RUN   TestLightSwitchStateMachine
The light has been switched off
The light has been switched on
The light has been switched off
--- PASS: TestLightSwitchStateMachine (0.00s)

As expected, the state machine transitions to the initial off state, then to the on state and finally back to the off state, rejecting any events that it can’t handle according to the definition of the state machine and the state that it is in when the event arrives.

Order Processing Example in Go

Here is a slightly fancier state machine example of order processing. The state machine starts in the order creating state where it validates the order details (passed via EventContext). It then can transition to the order created state if the order is valid, and if not, it transitions over to the order failed state. Then another event is sent to it externally for charging the user’s card and shipping the order. The state machine here transitions to the charging card state, and if the card is valid, transitions over to the order shipped state, and if not, to the transaction failed state.

To implement this in Go, we first define the set of states and events as in the previous example.

const (
	CreatingOrder     StateType = "CreatingOrder"
	OrderFailed       StateType = "OrderFailed"
	OrderPlaced       StateType = "OrderPlaced"
	ChargingCard      StateType = "ChargingCard"
	TransactionFailed StateType = "TransactionFailed"
	OrderShipped      StateType = "OrderShipped"

	CreateOrder     EventType = "CreateOrder"
	FailOrder       EventType = "FailOrder"
	PlaceOrder      EventType = "PlaceOrder"
	ChargeCard      EventType = "ChargeCard"
	FailTransaction EventType = "FailTransaction"
	ShipOrder       EventType = "ShipOrder"
)

This time we also define special context types for the purpose of passing data to event actions.

type OrderCreationContext struct {
	items []string
	err   error
}

func (c *OrderCreationContext) String() string {
	return fmt.Sprintf("OrderCreationContext [ items: %s, err: %v ]",
		strings.Join(c.items, ","), c.err)
}

type OrderShipmentContext struct {
	cardNumber string
	address    string
	err        error
}

func (c *OrderShipmentContext) String() string {
	return fmt.Sprintf("OrderShipmentContext [ cardNumber: %s, address: %s, err: %v ]",
		c.cardNumber, c.address, c.err)
}

We can now define actions for all our states.

type CreatingOrderAction struct{}

func (a *CreatingOrderAction) Execute(eventCtx EventContext) EventType {
	order := eventCtx.(*OrderCreationContext)
	fmt.Println("Validating, order:", order)
	if len(order.items) == 0 {
		order.err = errors.New("Insufficient number of items in order")
		return FailOrder
	}
	return PlaceOrder
}

type OrderFailedAction struct{}

func (a *OrderFailedAction) Execute(eventCtx EventContext) EventType {
	order := eventCtx.(*OrderCreationContext)
	fmt.Println("Order failed, err:", order.err)
	return NoOp
}

type OrderPlacedAction struct{}

func (a *OrderPlacedAction) Execute(eventCtx EventContext) EventType {
	order := eventCtx.(*OrderCreationContext)
	fmt.Println("Order placed, items:", order.items)
	return NoOp
}

type ChargingCardAction struct{}

func (a *ChargingCardAction) Execute(eventCtx EventContext) EventType {
	shipment := eventCtx.(*OrderShipmentContext)
	fmt.Println("Validating card, shipment:", shipment)
	if shipment.cardNumber == "" {
		shipment.err = errors.New("Card number is invalid")
		return FailTransaction
	}
	return ShipOrder
}

type TransactionFailedAction struct{}

func (a *TransactionFailedAction) Execute(eventCtx EventContext) EventType {
	shipment := eventCtx.(*OrderShipmentContext)
	fmt.Println("Transaction failed, err:", shipment.err)
	return NoOp
}

type OrderShippedAction struct{}

func (a *OrderShippedAction) Execute(eventCtx EventContext) EventType {
	shipment := eventCtx.(*OrderShipmentContext)
	fmt.Println("Order shipped, address:", shipment.address)
	return NoOp
}

Let’s now define our state machine.

func newOrderFSM() *StateMachine {
	return &StateMachine{
		States: States{
			Default: State{
				Events: Events{
					CreateOrder: CreatingOrder,
				},
			},
			CreatingOrder: State{
				Action: &CreatingOrderAction{},
				Events: Events{
					FailOrder:  OrderFailed,
					PlaceOrder: OrderPlaced,
				},
			},
			OrderFailed: State{
				Action: &OrderFailedAction{},
				Events: Events{
					CreateOrder: CreatingOrder,
				},
			},
			OrderPlaced: State{
				Action: &OrderPlacedAction{},
				Events: Events{
					ChargeCard: ChargingCard,
				},
			},
			ChargingCard: State{
				Action: &ChargingCardAction{},
				Events: Events{
					FailTransaction: TransactionFailed,
					ShipOrder:       OrderShipped,
				},
			},
			TransactionFailed: State{
				Action: &TransactionFailedAction{},
				Events: Events{
					ChargeCard: ChargingCard,
				},
			},
			OrderShipped: State{
				Action: &OrderShippedAction{},
			},
		},
	}
}

Let’s put our order processing FSM to test.

func TestOrderFSM(t *testing.T) {
	orderFsm := newOrderFSM()

	// Define the context for order creation.
	creationCtx := &OrderCreationContext{
		items: []string{},
	}

	// Try to create an order with invalid set of items.
	err := orderFsm.SendEvent(CreateOrder, creationCtx)
	if err != nil {
		t.Errorf("Failed to send create order event, err: %v", err)
	}

	// The state machine should enter the OrderFailed state.
	if orderFsm.Current != OrderFailed {
		t.Errorf("Expected the FSM to be in the OrderFailed state, actual: %s",
			orderFsm.Current)
	}

	// Let's fix the order creation context.
	creationCtx = &OrderCreationContext{
		items: []string{"foo", "bar"},
	}

	// Let's now retry the same order with a valid set of items.
	err = orderFsm.SendEvent(CreateOrder, creationCtx)
	if err != nil {
		t.Errorf("Failed to send create order event, err: %v", err)
	}

	// The state machine should enter the OrderPlaced state.
	if orderFsm.Current != OrderPlaced {
		t.Errorf("Expected the FSM to be in the OrderPlaced state, actual: %s",
			orderFsm.Current)
	}

	// Let's now define the context for shipping the order.
	shipmentCtx := &OrderShipmentContext{
		cardNumber: "",
		address:    "123 Foo Street, Bar Baz, QU 45678, USA",
	}

	// Try to charge the card using an invalid card number.
	err = orderFsm.SendEvent(ChargeCard, shipmentCtx)
	if err != nil {
		t.Errorf("Failed to send charge card event, err: %v", err)
	}

	// The state machine should enter the TransactionFailed state.
	if orderFsm.Current != TransactionFailed {
		t.Errorf("Expected the FSM to be in the TransactionFailed state, actual: %s",
			orderFsm.Current)
	}

	// Let's fix the shipment context.
	shipmentCtx = &OrderShipmentContext{
		cardNumber: "0000-0000-0000-0000",
		address:    "123 Foo Street, Bar Baz, QU 45678, USA",
	}

	// Let's now retry the transaction with a valid card number.
	err = orderFsm.SendEvent(ChargeCard, shipmentCtx)
	if err != nil {
		t.Errorf("Failed to send charge card event, err: %v", err)
	}

	// The state machine should enter the OrderShipped state.
	if orderFsm.Current != OrderShipped {
		t.Errorf("Expected the FSM to be in the OrderShipped state, actual: %s",
			orderFsm.Current)
	}

	// Let's try charging the card again for the same order. We should see the
	// event getting rejected as the card has been already charged once and the
	// state machine is in the OrderShipped state.
	err = orderFsm.SendEvent(ChargeCard, shipmentCtx)
	if err != ErrEventRejected {
		t.Errorf("Expected the FSM to return a rejected event error")
	}
}

Here’s the output from running the test.

=== RUN   TestOrderFSM
Validating, order: OrderCreationContext [ items: , err: <nil> ]
Order failed, err: Insufficient number of items in order
Validating, order: OrderCreationContext [ items: foo,bar, err: <nil> ]
Order placed, items: [foo bar]
Validating card, shipment: OrderShipmentContext [ cardNumber: , address: 123 Foo Street, Bar Baz, QU 45678, USA, err: <nil> ]
Transaction failed, err: Card number is invalid
Validating card, shipment: OrderShipmentContext [ cardNumber: 0000-0000-0000-0000, address: 123 Foo Street, Bar Baz, QU 45678, USA, err: <nil> ]
Order shipped, address: 123 Foo Street, Bar Baz, QU 45678, USA
--- PASS: TestOrderFSM (0.00s)

This implementation passes in appropriate contexts (i.e. OrderCreationContext and OrderShipmentContext) along with the event during the call to SendEvent. Note that the event actions called as a part of the loop for a single event all get the same context and can reuse fields (for example, see the usage of the items field in both CreatingOrderAction and the OrderPlacedAction). One key situation that was very easily handled by the framework is a duplicate call for charging the card (see the last part of the test above). Since the state machine had already transitioned over to the OrderShipped state, the ChargeCard event was rejected straight away and we prevented the system from getting into an inconsistent state just by the help of the declarative state machine configuration.

Conclusion

State machines are a powerful concept which can bring order into systems with asynchronous and multi-source inputs. As shown in the examples in this post, state machines are capable of producing deterministic behavior in complex situations, and the state machine framework described in this post can nicely handle simple use-cases that you may come across in everyday life.

I’d love to hear your feedback!

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

How to Write Envoy Filters Like a Ninja!

2019-05-28T10:40:00+00:00

Envoy is a programmable L3/L4 and L7 proxy that powers today’s service mesh solutions including Istio, AWS App Mesh, Consul Connect, etc. At Envoy’s core lie several filters that provide a rich set of features for observing, securing, and routing network traffic to microservices.

In these set of posts, we’ll have a look at the basics of Envoy filters and learn how to extend Envoy by implementing custom filters to create useful features!

Part 1 - Introduction

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

Hand-crafting a Sidecar Proxy and Demystifying Istio

2019-03-10T19:37:00+00:00

The sidecar proxy pattern is an important concept that lets Istio provide routing, metrics, security, and other features to services running in a service mesh. In this post I’ll explain key techniques that power Istio and I’ll also show you a way to build a simple HTTP traffic-sniffing sidecar proxy.

Introduction

Service mesh implementations generally rely on a sidecar proxy that lets them control, observe, and secure applications. The sidecar proxy is a reverse proxy and all traffic flows through it before reaching the destination service. The proxy analyzes this traffic and produces useful statistics, and provides flexible routing capabilities. The proxy can also secure application traffic using mTLS.

In this post, we’ll build a simple sidecar proxy that can sniff HTTP traffic and produce statistics such as request size, response status, etc. We’ll then deploy a HTTP service in a Kubernetes Pod, configure the sidecar proxy, and examine the generated statistics.

Building an HTTP Traffic-Sniffing Proxy

Istio relies on Envoy for proxying network traffic. The Envoy proxy is packaged as a container and deployed beside the service container in a single Pod. In this post, we’ll be building a miniscule proxy in Golang that can sniff HTTP traffic.

Our proxy needs to listen on a TCP port for incoming HTTP requests and then forward those to a destination address. Since, in our case, both the proxy and the service reside in the same container, the destination host can be addressed via the loopback IP address (i.e. 127.0.0.1); however, we still need a port number to identify the destination service.

const (
        proxyPort   = 8000
        servicePort = 80
)

We can now define the base for the proxy. The proxy will listen to requests on proxyPort and forward requests to the servicePort. The proxy will finally print statistics after servicing each request.

// Create a structure to define the proxy functionality.
type Proxy struct{}

func (p *Proxy) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	// Forward the HTTP request to the destination service.
        res, duration, err := p.forwardRequest(req)

	// Notify the client if there was an error while forwarding the request.
        if err != nil {
                http.Error(w, err.Error(), http.StatusBadGateway)
                return
        }

	// If the request was forwarded successfully, write the response back to
	// the client.
        p.writeResponse(w, res)

	// Print request and response statistics.
        p.printStats(req, res, duration)
}

func main() {
	// Listen on the predefined proxy port.
        http.ListenAndServe(fmt.Sprintf(":%d", proxyPort), &Proxy{})
}

The most important part for a proxy is its capability to forward requests. Let’s first define this functionality in our proxy implementation.

func (p *Proxy) forwardRequest(req *http.Request) (*http.Response, time.Duration, error) {
	// Prepare the destination endpoint to forward the request to.
        proxyUrl := fmt.Sprintf("http://127.0.0.1:%d%s", servicePort, req.RequestURI)

	// Print the original URL and the proxied request URL.
        fmt.Printf("Original URL: http://%s:%d%s\n", req.Host, servicePort, req.RequestURI)
        fmt.Printf("Proxy URL: %s\n", proxyUrl)

	// Create an HTTP client and a proxy request based on the original request.
        httpClient := http.Client{}
        proxyReq, err := http.NewRequest(req.Method, proxyUrl, req.Body)

	// Capture the duration while making a request to the destination service.
        start := time.Now()
        res, err := httpClient.Do(proxyReq)
        duration := time.Since(start)

	// Return the response, the request duration, and the error.
        return res, duration, err
}

Now that we have the response from the proxied request, let’s define the logic to write it back to the client.

func (p *Proxy) writeResponse(w http.ResponseWriter, res *http.Response) {
	// Copy all the header values from the response.
        for name, values := range res.Header {
                w.Header()[name] = values
        }

	// Set a special header to notify that the proxy actually serviced the request.
        w.Header().Set("Server", "amazing-proxy")

	// Set the status code returned by the destination service.
        w.WriteHeader(res.StatusCode)

	// Copy the contents from the response body.
        io.Copy(w, res.Body)

	// Finish the request.
        res.Body.Close()
}

The final part for the proxy is to print statistics. Let’s go ahead and implement that.

func (p *Proxy) printStats(req *http.Request, res *http.Response, duration time.Duration) {
        fmt.Printf("Request Duration: %v\n", duration)
        fmt.Printf("Request Size: %d\n", req.ContentLength)
        fmt.Printf("Response Size: %d\n", res.ContentLength)
        fmt.Printf("Response Status: %d\n\n", res.StatusCode)
}

We have now built a fully functional HTTP traffic-sniffing proxy.

Build a Container Image for the Proxy

Istio packages Envoy and runs it as a container beside the service container. Let’s build a proxy container image that runs the above Go code to replicate Istio’s behavior.

# Use the Go v1.12 image for the base.
FROM golang:1.12

# Copy the proxy code to the container.
COPY main.go .

# Run the proxy on container startup.
ENTRYPOINT [ "go" ]
CMD [ "run", "main.go" ]

# Expose the proxy port.
EXPOSE 8000

To build the proxy container image, we can simply issue the following Docker command.

$ docker build -t venilnoronha/amazing-proxy:latest -f Dockerfile .

Setting up the Pod Network

We need to set up the Pod network to ensure that the sidecar proxy receives all application traffic so that it can analyze and forward it to the required destination. One way to achieve that is to ask the user to configure all the client services to point to the proxy port while the proxy points to the service port. This complicates the user experience. A better and a more transparent way to achieve this is by using the Netfilter/iptables component from the Linux kernel.

Kubernetes Networking 101

To understand it better, let’s list the network interfaces exposed by Kubernetes to a Pod.

$ kubectl run -i --rm --restart=Never busybox --image=busybox -- sh -c "ip addr"

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever

174: eth0@if175: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:11:00:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.5/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

As you observe, a Pod gets access to at least 2 network interfaces i.e. lo and eth0. Interface lo stands for loopback and eth0 for ethernet. A thing to note here is that these are virtual and not real interfaces.

Port Mapping With iptables

A simplest use for iptables is to map one port to another. We can leverage this to transparently route traffic to our proxy. Istio uses this exact concept to set up its Pod networking.

The idea here is map the service port (80) on the eth0 interface to the proxy port (8000). This will ensure that the traffic from outside the container is routed to the proxy whenever it tries to access the service via port 80. We let the lo interface to route Pod-internal traffic directly to the destination service i.e. without the hop to the proxy as shown in the above diagram.

The Init Container

Kubernetes let’s you run init containers prior to running regular containers in a Pod. Istio uses an init container to set up the Pod networking in order to set up the necessary iptables rules. Let’s do the same to route Pod-external traffic to the proxy.

#!/bin/bash

# Forward TCP traffic on port 80 to port 8000 on the eth0 interface.
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 8000

# List all iptables rules.
iptables -t nat --list

We can now create a Docker container with this initialization script.

# Use the latest Ubuntu image for the base.
FROM ubuntu:latest

# Install the iptables command.
RUN apt-get update && \
    apt-get install -y iptables

# Copy the initialization script into the container.
COPY init.sh /usr/local/bin/

# Mark the initialization script as executable.
RUN chmod +x /usr/local/bin/init.sh

# Start the initialization script on container startup.
ENTRYPOINT ["init.sh"]

To build the Docker image, we execute the following command.

$ docker build -t venilnoronha/init-networking:latest -f Dockerfile .

The Demo

We’ve built a proxy and an init container to set up Pod networking. It’s now time to put this to a test. For this, we’ll be using the httpbin container as the service.

The Deployment

Istio automatically injects the init container as well as the proxy. However, for our experiment, we can manually craft the Pod yaml.

apiVersion: v1
kind: Pod
metadata:
  name: httpbin-pod
  labels:
    app: httpbin
spec:
  initContainers:
  - name: init-networking
    image: venilnoronha/init-networking
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
  containers:
  - name: service
    image: kennethreitz/httpbin
    ports:
    - containerPort: 80
  - name: proxy
    image: venilnoronha/amazing-proxy
    ports:
    - containerPort: 8000

We’ve set up the init container with root privileges and have configured the proxy and the service as regular containers. To deploy this on a Kubernetes cluster, we can issue the following command.

$ kubectl apply -f httpbin.yaml

The Test

To test the deployment, let’s first identify the ClusterIP for the Pod. To do so, we can execute the following command.

$ kubectl get pods -o wide
NAME          READY     STATUS    RESTARTS   AGE       IP           NODE
httpbin-pod   2/2       Running   0          21h       172.17.0.4   minikube

We now need to generate traffic from outside the Pod. For this purpose, I’ll be using the busybox container to issue HTTP requests via curl.

Let’s first send a GET request to the httpbin service.

$ kubectl run -i --rm --restart=Never busybox --image=odise/busybox-curl \
          -- sh -c "curl -i 172.17.0.4:80/get?query=param"

HTTP/1.1 200 OK
Content-Length: 237
Content-Type: application/json
Server: amazing-proxy

Let’s now send a POST request.

$ kubectl run -i --rm --restart=Never busybox --image=odise/busybox-curl \
          -- sh -c "curl -i -X POST -d 'body=parameters' 172.17.0.4:80/post"

HTTP/1.1 200 OK
Content-Length: 317
Content-Type: application/json
Server: amazing-proxy

Let’s finally send a GET request to the /status endpoint.

$ kubectl run -i --rm --restart=Never busybox --image=odise/busybox-curl \
          -- sh -c "curl -i http://172.17.0.4:80/status/429"

HTTP/1.1 429 Too Many Requests
Content-Length: 0
Content-Type: text/html; charset=utf-8
Server: amazing-proxy

Note that we send the request to port 80 i.e. the service port and not the proxy port. The iptables rules ensure that this gets routed to the proxy first which then forwards the request to the service. Also, we see the extra header Server: amazing-proxy which we had added in our proxy implementation.

Proxy Statistics

Let’s now peek into the statistics produced by the proxy. To do so, we can run the following command.

$ kubectl logs httpbin-pod --container="proxy"

Original URL: http://172.17.0.4:80/get?query=param
Proxy URL: http://127.0.0.1:80/get?query=param
Request Duration: 1.979348ms
Request Size: 0
Response Size: 237
Response Status: 200

Original URL: http://172.17.0.4:80/post
Proxy URL: http://127.0.0.1:80/post
Request Duration: 2.026861ms
Request Size: 15
Response Size: 317
Response Status: 200

Original URL: http://172.17.0.4:80/status/429
Proxy URL: http://127.0.0.1:80/status/429
Request Duration: 1.191793ms
Request Size: 0
Response Size: 0
Response Status: 429

As you observe, we do see that the results from the proxy match to the requests we generated.

Conclusion

We implemented a simple HTTP traffic-sniffing proxy, which we wired in a Kubernetes Pod using the init container. We also looked at how iptables can provide flexible networking in order to provide a great user experience when dealing with proxies. Above all, we’ve learnt some key concepts that form Istio to what it is today.

If you’d like me to write about a particular topic in the service mesh domain, then feel free to reach out to me.

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

Seamless Cloud-Native Apps with gRPC-Web and Istio

2018-11-10T15:48:00+00:00

gRPC-Web enables web applications to access gRPC backends via a proxy like Envoy. Envoy serves as the default proxy for Istio, and, so, we can leverage Istio’s EnvoyFilter construct to create seamless, well connected, Cloud-Native web applications.

Introduction

In this post, I’ll walk you through the process of building a simple web application that replaces keywords in user-entered text with emojis by communicating with a gRPC backend via gRPC-Web and Istio.

Below is an outline of the steps we’ll follow to create the emoji application.

Define the protocol format using Protobuf
Compile the Protobuf definition to generate Go and JavaScript files
Build and test a Go based gRPC service that replaces keywords in input text with emojis
Create a web interface for the emoji service using gRPC-Web
Configure the EnvoyFilter and deploy the backend over Istio
Deploy the web application and test our emoji service

The Architecture

Let’s dive a little deeper to understand what the final architecture for the emoji service will look like.

Simply put, the web application will leverage the gRPC-Web library to send HTTP requests to the Istio Gatway whenever the user supplies some text. The Istio Gateway will then route the HTTP request to the Proxy sidecar running alongside the emoji service backend, which will then translate the HTTP call to a gRPC call using Envoy’s gRPC-Web filter.

Defining The Protocol Format

First, let’s define the protocol format using Protobuf.

syntax = "proto3";

package emoji;

service EmojiService {
  rpc Emojize (EmojizeRequest) returns (EmojizeReply);
}

message EmojizeRequest {
  string text = 1;
}

message EmojizeReply {
  string emojized_text = 1;
}

We define a simple service named EmojiService that understands a rpc call named Emojize which accepts an EmojizeRequest object and returns an instance of EmojizeReply.

The message named EmojizeRequest holds a string field named text to represent user-entered text. Also, EmojizeReply holds a string named emojized_text to represent the final output string wherein the server will have replaced known emoji keywords with an actual emoji character.

Compiling The Protobuf Definition

Let’s first create the project directory structure like grpc-web-emoji/emoji/ and put the above definition in a file named emoji.proto.

We can now compile the emoji.proto file and generate the necessary Go files.

$ protoc -I emoji/ emoji/emoji.proto --go_out=plugins=grpc:emoji

Similarly, we also generate the JavaScript files.

$ protoc -I emoji/ emoji/emoji.proto --js_out=import_style=commonjs:emoji \
         --grpc-web_out=import_style=commonjs,mode=grpcwebtext:emoji

At this point, you should land up with a directory structure like below.

── grpc-web-emoji
   └── emoji
       ├── emoji.pb.go
       ├── emoji.proto
       ├── emoji_grpc_web_pb.js
       └── emoji_pb.js

Building And Testing The Go Backend

Let’s now create a go program that implements the EmojiService API. To do so, we create a file named main.go with the following contents.

package main

import (
	"context"
	"log"
	"net"

	proto "github.com/venilnoronha/grpc-web-emoji/emoji"
	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
	emoji "gopkg.in/kyokomi/emoji.v1"
)

// server is used to implement the EmojiService interface
type server struct{}

// Emojize takes a input string via EmojizeRequest, replaces known keywords with
// actual emoji characters and returns it via a EmojizeReply instance.
func (s *server) Emojize(c context.Context, r *proto.EmojizeRequest)
			(*proto.EmojizeReply, error) {
	return &proto.EmojizeReply{EmojizedText: emoji.Sprint(r.Text)}, nil
}

func main() {
	// listen to TCP requests over port 9000
	lis, err := net.Listen("tcp", ":9000")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	log.Printf("listening on %s", lis.Addr())

	// register the EmojiService implementation with the gRPC server
	s := grpc.NewServer()
	proto.RegisterEmojiServiceServer(s, &server{})
	reflection.Register(s)
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

I’ve used the kyokomi/emoji library to do the heavy lifting i.e. converting keywords from input text into emojis.

Let’s start the server, like so:

$ go run -v main.go
2018/11/12 10:45:12 listening on [::]:9000

It’s now time to test the emoji service programmatically. We’ll do so by creating a new client implementation named emoji_client.go with the following contents.

package main

import (
	"log"
	"time"

	proto "github.com/venilnoronha/grpc-web-emoji/emoji"
	"golang.org/x/net/context"
	"google.golang.org/grpc"
)

func main() {
	// connect to the server
	conn, err := grpc.Dial("localhost:9000", grpc.WithInsecure())
	if err != nil {
		log.Fatalf("could not connect to the service: %v", err)
	}
	defer conn.Close()

	// send a request to the server
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()

	c := proto.NewEmojiServiceClient(conn)
	resp, err := c.Emojize(ctx, &proto.EmojizeRequest{
		Text: "I like :pizza: and :sushi:!",
	})
	if err != nil {
		log.Fatalf("could not call service: %v", err)
	}
	log.Printf("server says: %s", resp.GetEmojizedText())
}

We can now run the emoji service client as shown below.

$ go run emoji_client.go
2018/11/12 10:55:52 server says: I like 🍕  and 🍣 !

Voila! The gRPC emoji service works as expected, and it’s now time to get the web frontend up and running.

Creating The Web Interface Using gRPC-Web

First, let’s create a HTML page named index.html. The page displays a text editor to the user and calls a emojize function (which we’ll define later) to send the user-input to the backend emoji service. The emojize function will also consume the gRPC response from the backend service and update the user input field with the data given out by the server.

<!DOCTYPE html>
<html>
<body>
  <div id="editor" contentEditable="true" hidefocus="true" onkeyup="emojize()"></div>
  <script src="dist/main.js"></script>
</body>
</html>

Let’s put the JavaScript code for the web frontend in a file named client.js as shown below.

const {EmojizeRequest, EmojizeReply} = require('emoji/emoji_pb.js');
const {EmojiServiceClient} = require('emoji/emoji_grpc_web_pb.js');

var client = new EmojiServiceClient('http://192.168.99.100:31380');
var editor = document.getElementById('editor');

window.emojize = function() {
  var request = new EmojizeRequest();
  request.setText(editor.innerText);

  client.emojize(request, {}, (err, response) => {
    editor.innerText = response.getEmojizedText();
  });
}

Notice that the EmojiServiceClient connects with the backend emoji service at http://192.168.99.100:31380 and not http://localhost:9000. This is because the web application can’t directly speak with a gRPC backend, and, therefore, we’ll be deploying our backend emoji service over Istio. The Istio deployment will be running on Minikube with the IP address of 192.168.99.100 and the default Istio Ingress port exposed for HTTP is 31380.

We’ll now need a couple of libraries to generate the final dist/main.js which we reference in index.html. To do so, we use the following npm package.json configuration.

{
  "name": "grpc-web-emoji",
  "version": "0.1.0",
  "description": "gRPC-Web Emoji Sample",
  "devDependencies": {
    "@grpc/proto-loader": "^0.3.0",
    "google-protobuf": "^3.6.1",
    "grpc": "^1.15.0",
    "grpc-web": "^1.0.0",
    "webpack": "^4.16.5",
    "webpack-cli": "^3.1.0"
  }
}

At this point, we can install the libraries and generate dist/main.js using the following commands.

$ npm install
$ npx webpack client.js

Deploying The Backend Over Istio

We can now package the backend emoji service into a container and deploy it over Istio. We’d need to install the gRPC-Web EnvoyFilter here so that calls to the backend gRPC service can be translated to/from gRPC and HTTP.

Let’s start off by building the Docker image with the following Dockerfile.

FROM golang:1.11 as builder
WORKDIR /root/go/src/github.com/venilnoronha/grpc-web-emoji/
COPY ./ .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -v -o emoji-service main.go

FROM scratch
WORKDIR /bin/
COPY --from=builder /root/go/src/github.com/venilnoronha/grpc-web-emoji/emoji-service .
ENTRYPOINT [ "/bin/emoji-service" ]
CMD [ "9000" ]
EXPOSE 9000

We can build and push the image to Docker Hub like so:

$ docker build -t vnoronha/grpc-web-emoji .
$ docker push vnoronha/grpc-web-emoji

Next, we define the Kubernetes Service and Deployment configuration as shown below. Let’s call this backend.yaml.

apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    app: backend
spec:
  ports:
  - name: grpc-port
    port: 9000
  selector:
    app: backend
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      containers:
      - name: backend
        image: vnoronha/grpc-web-emoji
        imagePullPolicy: Always
        ports:
        - containerPort: 9000

Note that once we deploy this service over Istio, the grpc- prefix in the Service port name will allow Istio to recognize this as a gRPC service.

Since we want the gRPC-Web filter to be installed on the backend sidecar proxy, we need to install it prior to deploying the backend service. The EnvoyFilter configuration is shown below. We name this filter.yaml.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: grpc-web-filter
spec:
  workloadLabels:
    app: backend
  filters:
  - listenerMatch:
      listenerType: SIDECAR_INBOUND
      listenerProtocol: HTTP
    insertPosition:
      index: FIRST
    filterType: HTTP
    filterName: "envoy.grpc_web"
    filterConfig: {}

Next, we need to define an Istio Gateway to route HTTP traffic over to the backend service. To do so, we write the following configuration to a file named gateway.yaml.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend
spec:
  host: backend
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs
spec:
  hosts:
  - "*"
  gateways:
  - gateway
  http:
  - match:
    - port: 80
    route:
    - destination:
        host: backend
        port:
          number: 9000
        subset: v1
    corsPolicy:
      allowOrigin:
        - "*"
      allowMethods:
        - POST
        - GET
        - OPTIONS
        - PUT
        - DELETE
      allowHeaders:
        - grpc-timeout
        - content-type
        - keep-alive
        - user-agent
        - cache-control
        - content-type
        - content-transfer-encoding
        - custom-header-1
        - x-accept-content-transfer-encoding
        - x-accept-response-streaming
        - x-user-agent
        - x-grpc-web
      maxAge: 1728s
      exposeHeaders:
        - custom-header-1
        - grpc-status
        - grpc-message
      allowCredentials: true

Notice that we’ve defined a complex corsPolicy here, and it’s required for gRPC-Web to work correctly.

We can now simply deploy the above configurations in the following order.

$ kubectl apply -f filter.yaml
$ kubectl apply -f <(istioctl kube-inject -f backend.yaml)
$ kubectl apply -f gateway.yaml

Once the backend pod has started, we can verify that the gRPC-Web filter was correctly configured in its sidecar proxy like so:

$ istioctl proxy-config listeners backend-7bf6c8f67c-8lbm7 --port 9000 -o json
...
    "http_filters": [
        {
            "config": {},
            "name": "envoy.grpc_web"
        },
...

Deploying And Testing The Web Frontend

We’ve now reached the final stage of the experiment. We serve the web application using a HTTP server via Python.

$ python2 -m SimpleHTTPServer 8080
Serving HTTP on 0.0.0.0 port 8080 ...

Let’s head over to http://localhost:8080 to play with the emoji web interface.

If everything went well, you should have a fully functional gRPC-Web based web application, as shown below.

If you observe the Developer Tools on a browser like Chrome, you should see the gRPC-Web HTTP requests as shown below.

Conclusion

gRPC-Web provides a great way of bringing the benefits of gRPC services over to web applications. It currently needs a middle agent like the Istio Data Plane (i.e. Envoy Proxy) in order to translate data to/from HTTP and gRPC formats. However, once we have the infrastructure ready, building web applications can become a seamless experience for developers when working with gRPC.

References

The gRPC-Web Hello World Guide
The WebpageFx Emoji Cheat Sheet

Let me know what you think.

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

Envoy, gRPC, and Rate Limiting

2018-10-27T03:25:00+00:00

Envoy is a lightweight service proxy designed for Cloud Native applications. It’s also one of the few proxies that support gRPC, which is based on the H2 (HTTP/2) protocol. gRPC is a high performance RPC (Remote Procedure Call) framework and it supports a plethora of environments.

In this post, we’ll be building a Greeter application in C++ using gRPC and Protocol Buffers, building another gRPC application in Go that implements Envoy’s RateLimitService interface, and finally deploying Envoy as a proxy for the Greeter application while leveraging our Rate Limit service to create backpressure.

The gRPC Greeter Application

We start by installing gRPC and Protobuf, and build the Greeter application in C++. You can also build this application by picking one of the other languages listed in the docs; however, I’ll be using C++ for the purpose of this post.

Below is a schematic for the Greeter application.

Once we run the Greeter application, we should have the following output on the console.

$ ./greeter_server
Server listening on 0.0.0.0:50051

$ ./greeter_client
Greeter received: Hello world

Upgrading The gRPC Greeter Application

Let’s now enhance the Greeter application to respond with a request count prefix instead of the static “Hello” prefix. To do so, simply update the greeter_server.cc implementation as shown below.

 // Logic and data behind the server's behavior.
 class GreeterServiceImpl final : public Greeter::Service {
+  int counter = 0;
   Status SayHello(ServerContext* context, const HelloRequest* request,
                   HelloReply* reply) override {
-    std::string prefix("Hello ");
+    std::string prefix(std::to_string(++counter) + " ");
     reply->set_message(prefix + request->name());
     return Status::OK;
   }

Once you re-build and execute greeter_server, you should see the following output when you send requests via the greeter_client.

$ for i in {1..3}; do ./greeter_client; sleep 1; done
Greeter received: 1 world
Greeter received: 2 world
Greeter received: 3 world

A Simple Rate Limit Service

Next, we implement a simple Rate Limit service in Go by extending the Envoy’s RateLimitService proto interface. To do so, we create a Go project named rate-limit-service and vendor Envoy’s go-control-plane and its related dependencies. The go-control-plane project provides Go bindings for Envoy protos. We also create cmd/server/main.go and cmd/client/main.go for implementing the Rate Limit service later.

$ mkdir -p $GOPATH/src/github.com/venilnoronha/rate-limit-service/
$ cd $GOPATH/src/github.com/venilnoronha/rate-limit-service/
$ mkdir -p cmd/server/ && touch cmd/server/main.go
$ mkdir cmd/client/ && touch cmd/client/main.go

Once you’ve vendored all dependencies, you should land up with a project structure like below. Note that I’ve only highlighted the packages relevant for this experiment.

── rate-limit-service
   ├── cmd
   │   ├── client
   │   │   └── main.go
   │   └── server
   │       └── main.go
   └── vendor
       ├── github.com
       │   ├── envoyproxy
       │   │   ├── data-plane-api
       │   │   └── go-control-plane
       │   ├── gogo
       │   │   ├── googleapis
       │   │   └── protobuf
       │   └── lyft
       │       └── protoc-gen-validate
       └── google.golang.org
           ├── genproto
           └── grpc

The Rate Limit Service Server

We now create a simple gRPC Rate Limit server which limits every second request.

package main

import (
        "log"
        "net"
        "golang.org/x/net/context"
	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
	rls "github.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v2"
)

// server is used to implement rls.RateLimitService
type server struct{
	// limit specifies if the next request is to be rate limited
	limit bool
}

func (s *server) ShouldRateLimit(ctx context.Context,
		request *rls.RateLimitRequest) (*rls.RateLimitResponse, error) {
	log.Printf("request: %v\n", request)

	// logic to rate limit every second request
	var overallCode rls.RateLimitResponse_Code
	if s.limit {
		overallCode = rls.RateLimitResponse_OVER_LIMIT
		s.limit = false
	} else {
		overallCode = rls.RateLimitResponse_OK
		s.limit = true
	}

	response := &rls.RateLimitResponse{OverallCode: overallCode}
	log.Printf("response: %v\n", response)
        return response, nil
}

func main() {
	// create a TCP listener on port 50052
        lis, err := net.Listen("tcp", ":50052")
        if err != nil {
                log.Fatalf("failed to listen: %v", err)
        }
	log.Printf("listening on %s", lis.Addr())

	// create a gRPC server and register the RateLimitService server
        s := grpc.NewServer()
	rls.RegisterRateLimitServiceServer(s, &server{limit: false})
        reflection.Register(s)
        if err := s.Serve(lis); err != nil {
                log.Fatalf("failed to serve: %v", err)
        }
}

Let’s now start the RateLimitService server as shown below.

$ go run cmd/server/main.go
2018/10/27 00:35:28 listening on [::]:50052

The Rate Limit Service Client

We similarly create a RateLimitService client to validate the server’s behavior.

package main

import (
        "log"
	"time"
        "golang.org/x/net/context"
        "google.golang.org/grpc"
	rls "github.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v2"
)

func main() {
        // Set up a connection to the server
        conn, err := grpc.Dial("localhost:50052", grpc.WithInsecure())
        if err != nil {
                log.Fatalf("could not connect: %v", err)
        }
        defer conn.Close()
        c := rls.NewRateLimitServiceClient(conn)

        // Send a request to the server
        ctx, cancel := context.WithTimeout(context.Background(), time.Second)
        defer cancel()
	r, err := c.ShouldRateLimit(ctx, &rls.RateLimitRequest{Domain: "envoy"})
        if err != nil {
                log.Fatalf("could not call service: %v", err)
        }
        log.Printf("response: %v", r)
}

Let’s now test out the server/client interaction by firing up the client.

$ for i in {1..4}; do go run cmd/client/main.go; sleep 1; done
2018/10/27 17:32:23 response: overall_code:OK
2018/10/27 17:32:25 response: overall_code:OVER_LIMIT
2018/10/27 17:32:26 response: overall_code:OK
2018/10/27 17:32:28 response: overall_code:OVER_LIMIT

You should see a corresponding log on the server.

2018/10/27 17:32:23 request: domain:"envoy"
2018/10/27 17:32:23 response: overall_code:OK
2018/10/27 17:32:25 request: domain:"envoy"
2018/10/27 17:32:25 response: overall_code:OVER_LIMIT
2018/10/27 17:32:26 request: domain:"envoy"
2018/10/27 17:32:26 response: overall_code:OK
2018/10/27 17:32:28 request: domain:"envoy"
2018/10/27 17:32:28 response: overall_code:OVER_LIMIT

The Envoy Proxy

It’s now time to bring in the Envoy proxy which will route calls from the Greeter client to the Greeter server while testing the rate with our Rate Limit service implementation. Below is a diagram depicting our final deployment.

Proxy Configuration

We use the following Envoy configuration for registering both the Greeter and the RateLimitService servers and enabling Rate Limit checks. Note that the locally deployed services are referred via the docker.for.mac.localhost address as we’ll later deploy Envoy on Docker for Mac.

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 9211 # expose proxy on port 9211
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          access_log: # configure logging
            name: envoy.file_access_log
            config:
              path: /dev/stdout
          route_config:
            name: greeter_route # configure the greeter service routes
            virtual_hosts:
            - name: service
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                  grpc: {}
                route:
                  cluster: greeter_service
              rate_limits: # enable rate limit checks for the greeter service
                actions:
                - destination_cluster: {}
          http_filters:
          - name: envoy.rate_limit # enable the Rate Limit filter
            config:
              domain: envoy
          - name: envoy.router # enable the Router filter
            config: {}
  clusters:
  - name: greeter_service # register the Greeter server
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {} # enable H2 protocol
    hosts:
    - socket_address:
        address: docker.for.mac.localhost
        port_value: 50051
  - name: rate_limit_service # register the RateLimitService server
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {} # enable H2 protocol
    hosts:
    - socket_address:
        address: docker.for.mac.localhost
        port_value: 50052
rate_limit_service: # define the global rate limit service
  use_data_plane_proto: true
  grpc_service:
    envoy_grpc:
      cluster_name: rate_limit_service

Deploying The Envoy Proxy

To deploy the Envoy proxy, we copy the above configuration to a file named envoy.yaml. Then, we can build a Docker image with the following Dockerfile.

FROM envoyproxy/envoy:latest
COPY envoy.yaml /etc/envoy/envoy.yaml

To build the image, run the following command.

$ docker build -t envoy:grpc .
Sending build context to Docker daemon  74.75kB
Step 1/2 : FROM envoyproxy/envoy:latest
 ---> 51fc619e4dc5
Step 2/2 : COPY envoy.yaml /etc/envoy/envoy.yaml
 ---> c766ba3d7d09
Successfully built c766ba3d7d09
Successfully tagged envoy:grpc

We can then run the proxy, like so:

$ docker run -p 9211:9211 envoy:grpc
...
[2018-10-28 02:59:20.469][000008][info][main] [source/server/server.cc:456] starting main dispatch loop
[2018-10-28 02:59:20.553][000008][info][upstream] [source/common/upstream/cluster_manager_impl.cc:135] cm init: all clusters initialized
[2018-10-28 02:59:20.554][000008][info][main] [source/server/server.cc:425] all clusters initialized. initializing init manager
[2018-10-28 02:59:20.554][000008][info][config] [source/server/listener_manager_impl.cc:908] all dependencies initialized. starting workers

Updating The Greeter Client

Since we now want to route our Greeter client calls via Envoy, we update the server port from 50051 to 9211 in its implementation and rebuild it.

   GreeterClient greeter(grpc::CreateChannel(
-      "localhost:50051", grpc::InsecureChannelCredentials()));
+      "localhost:9211", grpc::InsecureChannelCredentials()));
   std::string user("world");
   std::string reply = greeter.SayHello(user);

The Final Test

At this point we have the Greeter server, the RateLimitService server and an instance of Envoy proxy running. It’s now time to validate the complete deployment. To do so, we can simply send a few Greeter requests via the updated Greeter client as shown below.

$ for i in {1..10}; do ./greeter_client; sleep 1; done
Greeter received: 4 world
14:
Greeter received: RPC failed
Greeter received: 5 world
14:
Greeter received: RPC failed
Greeter received: 6 world
14:
Greeter received: RPC failed
Greeter received: 7 world
14:
Greeter received: RPC failed
Greeter received: 8 world
14:
Greeter received: RPC failed

As you observe, alternate requests fail with a gRPC status code of 14 i.e. RPC failed, while 5 out of the 10 requests are successful. This shows that the Rate Limit service is limiting every second request as designed, and Envoy rightly terminates such requests on its end.

Conclusion

This post should give you a high level idea of how you can use Envoy as a proxy for your own applications, and should also help you understand how Envoy’s Rate Limit filter works with the gRPC protocol.

Thoughts are much appreciated. :)

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

Raw TCP Traffic Shaping with Istio 1.1.0

2018-10-19T13:00:00+00:00

Istio provides sophisticated routing mechanics via concepts like VirtualService, DestinationRule, Gateway, etc. Istio 1.0 enabled HTTP traffic shifting via weighted route definitions. I was able to contribute a similar feature for TCP/TLS services via my PRs on Envoy and on Istio. The feature in Envoy was released in 1.8.0 and the one in Istio will be available in the upcoming 1.1.0 release.

In this article, we’ll be looking at a simple TCP Echo Server written in Go, containerize it with Docker, deploy it over Kubernetes, and exercise Istio’s weighted TCP routing feature over it to understand its behavior in production services.

The TCP Echo Server

For the purpose of this post, we’ll be creating a simple TCP Server which listens to connections and prepends a simple prefix to the Client request data and returns it as a response. Below is a pictorial representation of the same.

Let’s now look at the Go code for the TCP Echo Server.

package main

import (
	"bufio"
	"fmt"
	"io"
	"net"
	"os"
)

// main serves as the program entry point
func main() {
	// obtain the port and prefix via program arguments
	port := fmt.Sprintf(":%s", os.Args[1])
	prefix := os.Args[2]

	// create a tcp listener on the given port
	listener, err := net.Listen("tcp", port)
	if err != nil {
		fmt.Println("failed to create listener, err:", err)
		os.Exit(1)
	}
	fmt.Printf("listening on %s, prefix: %s\n", listener.Addr(), prefix)

	// listen for new connections
	for {
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println("failed to accept connection, err:", err)
			continue
		}

		// pass an accepted connection to a handler goroutine
		go handleConnection(conn, prefix)
	}
}

// handleConnection handles the lifetime of a connection
func handleConnection(conn net.Conn, prefix string) {
	defer conn.Close()
	reader := bufio.NewReader(conn)
	for {
		// read client request data
		bytes, err := reader.ReadBytes(byte('\n'))
		if err != nil {
			if err != io.EOF {
				fmt.Println("failed to read data, err:", err)
			}
			return
		}
		fmt.Printf("request: %s", bytes)

		// prepend prefix and send as response
		line := fmt.Sprintf("%s %s", prefix, bytes)
		fmt.Printf("response: %s", line)
		conn.Write([]byte(line))
	}
}

To test this program, simply copy the code into a file named main.go and run it as shown below.

$ go run -v main.go 9000 hello
listening on [::]:9000, prefix: hello

We can now interact with this program over TCP via nc (Netcat). To send a request, we can use a BusyBox container as shown below.

$ docker run -it --rm busybox sh -c 'echo world | nc docker.for.mac.localhost 9000'
hello world

As you see, the request “world” was prepended with “hello” which resulted in the “hello world” response. Note that I’m executing the BusyBox container over Docker for Mac which is why I access the TCP Echo Server via docker.for.mac.localhost instead of localhost.

Containerizing The TCP Echo Server

Since we eventually want to run the TCP Echo Server over a Kubernetes cluster, let’s now put it in a container and publish the image to Docker Hub.

Firstly, we create a Dockerfile with the following contents.

# build a main binary using the golang container
FROM golang:1.11 as builder
WORKDIR /go/src/github.com/venilnoronha/tcp-echo-server/
COPY main.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main main.go

# copy the main binary to a separate container based on alpine
FROM alpine:3.8
RUN apk --no-cache add ca-certificates
WORKDIR /bin/
COPY --from=builder /go/src/github.com/venilnoronha/tcp-echo-server/main .
ENTRYPOINT [ "/bin/main" ]
CMD [ "9000", "hello" ]
EXPOSE 9000

We now build the container and publish the image to Docker Hub as shown below.

$ docker build -t vnoronha/tcp-echo-server:latest .
Sending build context to Docker daemon  60.93kB
...
Successfully built d172af115e18
Successfully tagged vnoronha/tcp-echo-server:latest

$ docker push vnoronha/tcp-echo-server:latest
The push refers to repository [docker.io/vnoronha/tcp-echo-server]
b4cc76510de6: Pushed
...
latest: digest: sha256:0a45b5a0d362db6aa9154717ee3f2b... size: 949

Deploying The TCP Echo Server On Kubernetes

Service Configuration

We want to deploy 2 versions of the TCP Echo Server, both with different prefixes to show the routing behavior. To do so, we create service.yaml with a Kubernetes Service and 2 Deployments for the 2 different versions of TCP Echo Server we want to run.

apiVersion: v1
kind: Service
metadata:
  name: tcp-echo-server
  labels:
    app: tcp-echo-server
spec:
  selector:
    app: tcp-echo-server
  ports:
  - port: 9000
    name: tcp
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tcp-echo-server-v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: tcp-echo-server
        version: v1
    spec:
      containers:
      - name: tcp-echo-server
        image: vnoronha/tcp-echo-server:latest
        args: [ "9000", "one" ] # prefix: one
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tcp-echo-server-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: tcp-echo-server
        version: v2
    spec:
      containers:
      - name: tcp-echo-server
        image: vnoronha/tcp-echo-server:latest
        args: [ "9000", "two" ] # prefix: two
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000

Deploying Minikube

Minikube serves as a great tool for locally developing on Kubernetes. I’ve started my Minikube instance with the following command.

$ minikube start --bootstrapper kubeadm       \
                 --memory=8192                \
                 --cpus=4                     \
                 --kubernetes-version=v1.10.0 \
                 --vm-driver=virtualbox
Starting local Kubernetes v1.10.0 cluster...
...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

Installing Istio

At the time of writing this article, Istio 1.1.0 wasn’t released. Therefore, I resorted to an Istio Daily Pre-Release to demonstrate the new feature. Please refer to the Istio Docs to learn to download and configure Istio.

Once configured, here’s an easy way to fully deploy Istio components.

$ kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
...
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created

$ kubectl apply -f install/kubernetes/istio-demo.yaml
namespace/istio-system created
...
destinationrule.networking.istio.io/istio-policy created
destinationrule.networking.istio.io/istio-telemetry created 

Deploying TCP Echo Servers With Istio Proxies

Since we want to demonstrate Istio’s routing mechanics, let’s deploy the tcp-echo-server with the Istio Proxy side-car as shown below.

$ kubectl apply -f <(istioctl kube-inject -f service.yaml)
service/tcp-echo-server created
deployment.extensions/tcp-echo-server-v1 created
deployment.extensions/tcp-echo-server-v2 created

We can verify that the services are running via the following commands.

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
tcp-echo-server-v1-78684f5697-sv5r5   2/2       Running   0          56s
tcp-echo-server-v2-74bf9999c8-hhhf9   2/2       Running   0          56s

$ kubectl logs tcp-echo-server-v1-78684f5697-sv5r5 tcp-echo-server
listening on [::]:9000, prefix: one

$ kubectl logs tcp-echo-server-v2-74bf9999c8-hhhf9 tcp-echo-server
listening on [::]:9000, prefix: two

Weighted TCP Routing With Istio

This is the final part of this exercise where we define a VirtualService, DestinationRule and a Gateway with weighted routes and verify the system behavior.

Routing Configuration

We create a DestinationRule with 2 subsets to represent the 2 versions of the TCP Echo Server. The Gateway allows traffic to access the services on TCP port 31400. Finally, the VirtualService specifies that 80% of the traffic must be routed to TCP Echo Server v1 and 20% to v2.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: destination
spec:
  host: tcp-echo-server
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 31400
      name: tcp
      protocol: TCP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: route
spec:
  hosts:
  - "*"
  gateways:
  - gateway
  tcp:
  - match:
    - port: 31400
    route:
    - destination:
        host: tcp-echo-server
        port:
          number: 9000
        subset: v1
      weight: 80
    - destination:
        host: tcp-echo-server
        port:
          number: 9000
        subset: v2
      weight: 20

Deploying The Route Configuration

To apply the configuration, copy it to a file named route-config.yaml and install it via the following command.

kubectl apply -f route-config.yaml
destinationrule.networking.istio.io/destination created
gateway.networking.istio.io/gateway created
virtualservice.networking.istio.io/route created

Verifying Istio’s TCP Routing Behavior

Let’s first determine the Ingress IP.

$ minikube ip
192.168.99.100

We can now send a few requests to the weight balanced TCP Echo Server via the Ingress.

$ for i in {1..10}; do
for> docker run -it --rm busybox sh -c '(date; sleep 1) | nc 192.168.99.100 31400'
for> done
one Sat Oct 20 04:38:05 UTC 2018
two Sat Oct 20 04:38:07 UTC 2018
two Sat Oct 20 04:38:09 UTC 2018
one Sat Oct 20 04:38:12 UTC 2018
one Sat Oct 20 04:38:14 UTC 2018
one Sat Oct 20 04:38:17 UTC 2018
one Sat Oct 20 04:38:19 UTC 2018
one Sat Oct 20 04:38:22 UTC 2018
one Sat Oct 20 04:38:24 UTC 2018
two Sat Oct 20 04:38:27 UTC 2018

As you see, about 80% of the requests have a prefix of “one” and the rest 20% have a prefix of “two”. This proves that the weighted TCP routes are indeed in effect.

The diagram below should give you a good idea as to how the Istio landscape for this demonstration looks like.

Cleanup

Just delete the Minikube deployment, like so:

$ minikube stop && minikube delete
Stopping local Kubernetes cluster...
Machine stopped.
Deleting local Kubernetes cluster...
Machine deleted.

Conclusion

As shown in this article, it’s quite easy to configure weighted TCP routes via the upcoming Istio 1.1.0 release. Also, this article should give you a good idea on constructing your own weighted TCP routes, and shaping your TCP traffic from the ground up!

As always, please feel free to reach out with questions or comments.

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.

Set sail a production-ready Istio Adapter

2018-10-07T11:49:00+00:00

So, you’ve walked through the Istio Mixer Adapter guide and want to now publish your own amazing adapter? This post will run you through the process of setting sail your own adapter on the seas of production.

Introduction

Depending on your knowledge of Go, Protobufs, gRPC, Istio, Docker and Kubernetes, you may find the process of publishing an Istio Mixer Adapter from being easy to tasking. This post assumes that you have some experience with these technologies, and that you’ve been able to complete at least one of the walk-throughs from the Istio Wiki.

For the purpose of this post, I’d be talking about building an Istio Mixer Adapter that consumes Metrics. Here’s a run-down of the steps we’ll be looking at:

The Istio Mixer - Adapter Interface Architecture
Creating An Out-Of-Tree Mixer Adapter
Publishing The Adapter To Docker Hub
Writing Kubernetes Config For The Adapter
Deploying And Testing The Adapter With Istio

Again, I will try my best to render all the important details in this post to bring your new adapter to life.

The Istio Mixer - Adapter Interface Architecture

Let’s first have a look at how the adapter is to be interfaced with the Istio Mixer. Kubernetes abstracts the interfacing to some extent; however, it’s important for us to understand this, at least, in a little detail.

Here’s a brief description of each of the elements in the architecture above.

Microservice is a user application that is deployed over Istio
Proxy is the Istio component, i.e. Envoy Proxy, that controls the network communication in the Service Mesh
Mixer refers to the Istio component that receives metric (and other) data from the Proxy and forwards it to other components, in this case, the Adapter
Adapter is the application we’re building which consumes metric data from the Mixer over a gRPC channel
Operator is an actor who is responsible for configuring the deployment, in this case, Istio and the Adapter

The important thing to note here is that each of these components run as seperate processes, and that they may be distributed over a network. Also, Mixer establishes a gRPC channel with the Adapter for the purpose of providing it with user configuration and metric data.

Creating An Out-Of-Tree Mixer Adapter

For brevity, I’m relying on the Istio Mixer Out Of Tree Adapter Walkthrough for you to follow and build a working out-of-tree mixer adapter. Below is an outline of the steps you’re required to follow to create a working out-of-tree Istio Mixer Adapter that consumes metric data.

Create config.proto to represent the Adapter configuration
Create the mygrpcadapter.go implementation which handles the HandleMetric(context.Context, *metric.HandleMetricRequest) (*v1beta11.ReportResult, error) gRPC API call
Generate the configuration files via go generate ./...
Create main.go which creates the gRPC server and listens to API calls
Write the sample_operator_config.yaml for the adapter
Test and validate your adapter by starting a local Mixer process
Configure the out-of-tree project
Vendor the necessary dependencies (using Go Modules, Glide, Dep, etc. this time)
Build and test the out-of-tree adapter by starting a local Mixer process

Publishing The Adapter To Docker Hub

Once you have the myootadapter project set up and tested locally, it’s time to build and publish the Adapter to a repository like Docker Hub. Please perform the following steps before proceeding.

Move the contents of the mygrpcadapter/testdata/ directory to under operatorconfig/
Create a file named Dockerfile to hold the steps to create the Docker image
Finally, create a file named mygrpcadapter-k8s.yaml under operatorconfig/ which we’ll use later to deploy the Adapter via Kubernetes

Once you’ve followed these steps, you should have a project structure like below.

── myootadapter
   ├── Dockerfile
   ├── glide.lock # Depending on your vendoring tool,
   ├── glide.yaml # you'd have some files like these.
   ├── mygrpcadapter
   │   ├── cmd
   │   │   └── main.go
   │   ├── config
   │   │   ├── config.pb.go
   │   │   ├── config.proto
   │   │   ├── config.proto_descriptor
   │   │   ├── mygrpcadapter.config.pb.html
   │   │   └── mygrpcadapter.yaml
   │   └── mygrpcadapter.go
   └── operatorconfig
       ├── attributes.yaml
       ├── metrictemplate.yaml
       ├── sample_operator_config.yaml
       ├── mygrpcadapter-k8s.yaml
       └── mygrpcadapter.yaml

Let’s now build the Docker image and publish it to Docker Hub.

Building The Docker Image

The multi-stage builds pattern can be used to build the Docker image. Copy the following contents to your Dockerfile.

FROM golang:1.11 as builder
WORKDIR /go/src/github.com/username/myootadapter/
COPY ./ .
RUN CGO_ENABLED=0 GOOS=linux \
    go build -a -installsuffix cgo -v -o bin/mygrpcadapter ./mygrpcadapter/cmd/

FROM alpine:3.8
RUN apk --no-cache add ca-certificates
WORKDIR /bin/
COPY --from=builder /go/src/github.com/username/myootadapter/bin/mygrpcadapter .
ENTRYPOINT [ "/bin/mygrpcadapter" ]
CMD [ "8000" ]
EXPOSE 8000

The CMD [ "8000" ] line tells Docker to pass 8000 as an argument to /bin/mygrpcadapter which is defined by the ENTRYPOINT [ "/bin/mygrpcadapter" ] line. Since we fix the gRPC listener port to 8000 here, we must also update the sample_operator_config.yaml to reflect the same. We do that by replacing address: "{ADDRESS}" with address: mygrpcadapter:8000.

Let’s also update the file_path to store the output on a volume which we’ll create later. Update file_path: "out.txt" to file_path: "/volume/out.txt". You’d then land up with a Handler configuration like below.

apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
 name: h1
 namespace: istio-system
spec:
 adapter: mygrpcadapter
 connection:
   address: "mygrpcadapter:8000"
 params:
   file_path: "/volume/out.txt"

Now, we run the following command from the myootadapter directory to build and tag the Docker image.

docker build -t dockerhub-username/mygrpcadapter:latest .

Publishing The Image To Docker Hub

First, login to Docker Hub via your terminal.

docker login

Next, push the image using the following command.

docker push dockerhub-username/mygrpcadapter:latest

Writing Kubernetes Config For The Adapter

Let’s now fill out the configuration for deploying the adapter via Kubernetes. Copy the following configuration to the mygrpcadapter-k8s.yaml file which we created earlier.

apiVersion: v1
kind: Service
metadata:
  name: mygrpcadapter
  namespace: istio-system
  labels:
    app: mygrpcadapter
spec:
  type: ClusterIP
  ports:
  - name: grpc
    protocol: TCP
    port: 8000
    targetPort: 8000
  selector:
    app: mygrpcadapter
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mygrpcadapter
  namespace: istio-system
  labels:
    app: mygrpcadapter
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mygrpcadapter
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      containers:
      - name: mygrpcadapter
        image: dockerhub-username/mygrpcadapter:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
        volumeMounts:
        - name: transient-storage
          mountPath: /volume
      volumes:
      - name: transient-storage
        emptyDir: {}

The above configuration defines a simple service with just a single replica which is created out of the image at dockerhub-username/mygrpcadapter:latest. The service can be referred to by name as mygrpcadapter and can be addressed to via port 8000. That’s how the address: "mygrpcadapter:8000" configuration in sample_operator_config.yaml refers to this particular deployment.

Also, notice these special annotations:

annotations:
  sidecar.istio.io/inject: "false"
  scheduler.alpha.kubernetes.io/critical-pod: ""

This tells the Kubernetes scheduler to not inject the Istio Proxy sidecar if automatic injection is in place. We do that because we don’t really need a Proxy in front of our Adapter. Also, the second annotation marks this pod as critical for the system.

We also create a transient volume named transient-storage which is used for storing the Adapter output i.e. the out.txt file. The following snippet from the above configuration enables us to do that.

    volumeMounts:
    - name: transient-storage
      mountPath: /volume
  volumes:
  - name: transient-storage
    emptyDir: {}

Deploying And Testing The Adapter With Istio

Again, for brevity, I’m relying on the project documentation for you to deploy Istio, run the Bookinfo sample application and to determine the ingress IP and port.

Deploying The Adapter

We can now deploy the Adapter via Kubernetes like so:

kubectl apply -f operatorconfig/

This should deploy the mygrpcadapter service under the istio-system namespace. You can verify this by executing the following command.

kubectl get pods -n istio-system

This would print a log like below.

NAME                                       READY     STATUS        RESTARTS   AGE
istio-citadel-75c88f897f-zfw8b             1/1       Running       0          1m
istio-egressgateway-7d8479c7-khjvk         1/1       Running       0          1m
.
.
mygrpcadapter-86cb6dd77c-hwvqd             1/1       Running       0          1m

You could also check the Adapter logs by executing the following command.

kubectl logs mygrpcadapter-86cb6dd77c-hwvqd -n istio-system

It should then print the following log.

listening on "[::]:8000"

Testing The Adapter

Execute the following command in your terminal, or hit the http://${GATEWAY_URL}/productpage URL in your browser to send a request to the Bookinfo deployment.

curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage

Verify the output in the /volume/out.txt file by accessing the Adapter container.

kubectl exec mygrpcadapter-86cb6dd77c-hwvqd cat /volume/out.txt

You should see an output like below.

HandleMetric invoked with:
  Adapter config: &Params{FilePath:/volume/out.txt,}
  Instances: 'i1metric.instance.istio-system':
  {
		Value = 1235
		Dimensions = map[response_code:200]
  }

Conclusion

Istio provides a standard mechanism to manage and observe microservices in the cloud. Mixer enables developers to easily extend Istio to custom platforms. And, I hope that this guide has given you a glimpse of the Istio Mixer - Adapter interfacing, and how to build a production-ready Adapter yourself!

Go, publish your own Istio Mixer Adapter! Feel free to use the Wavefront by VMware Adapter for Istio for reference.

Also, refer to this Wiki if you wish to publish your adapter on the Istio Adapters page.

Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions.