Frugal

A very fast dynamic Thrift serializer & deserializer without generating code.

Features

Code Generation Free

Traditional Thrift serializer and deserializer are based on generated code which is no longer needed since we can make use of struct field tags.

High Performance

Based on the test cases in frugal/tests, Frugal is about 2.5x to 3.7x faster than Apache Thrift (TBinaryProtocol) in the benchmark below.

There may be variations between different test cases. Feel free to share your test cases with us.

go version go1.23.6 linux/amd64

goos: linux
goarch: amd64
pkg: github.com/cloudwego/frugal/tests
cpu: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz

Marshal_ApacheThrift/small-4              3468714         346.1 ns/op    1684.32 MB/s           0 B/op           0 allocs/op
Marshal_ApacheThrift/medium-4              128386          9343 ns/op    1875.07 MB/s           0 B/op           0 allocs/op
Marshal_ApacheThrift/large-4                 7208        164521 ns/op    1845.68 MB/s         109 B/op           0 allocs/op
Marshal_Frugal/small-4                   13032746         92.45 ns/op    6306.09 MB/s           0 B/op           0 allocs/op
Marshal_Frugal/medium-4                    327564          3669 ns/op    4774.38 MB/s           0 B/op           0 allocs/op
Marshal_Frugal/large-4                      18751         64212 ns/op    4728.90 MB/s           0 B/op           0 allocs/op

Unmarshal_ApacheThrift/small-4            1548732         774.1 ns/op     753.15 MB/s        1120 B/op           4 allocs/op
Unmarshal_ApacheThrift/medium-4             42676         30665 ns/op     571.27 MB/s       44704 B/op         175 allocs/op
Unmarshal_ApacheThrift/large-4               2106        515642 ns/op     588.88 MB/s      775936 B/op        3030 allocs/op
Unmarshal_Frugal/small-4                  4963635         266.2 ns/op    2189.92 MB/s         544 B/op           1 allocs/op
Unmarshal_Frugal/medium-4                   99786         11321 ns/op    1547.45 MB/s       19908 B/op          57 allocs/op
Unmarshal_Frugal/large-4                     5838        197987 ns/op    1533.69 MB/s      349252 B/op         997 allocs/op

What can you do with Frugal ?

Use Frugal as Kitex serializer and deserializer

No more massive serialization and deserialization code, leads to a more tidy project. No more meaningless diff of generated code in code review.

Serialized and Deserialize struct generated by Thriftgo

If you have a Thrift file, and all you need is using Frugal to do serialization and deserialization. You can use thriftgo to generate Go struct, then you can use Frugal.

Serialization and deserialization on a customized Go struct

If you don’t want any Thrift files, and you want serialize or deserialize a customized Go struct. You can add some struct field tag to the Go struct, then you can use Frugal.

Usage

Using with Kitex

1. Update Kitex and Frugal

go get github.com/cloudwego/kitex@latest
go get github.com/cloudwego/frugal@latest

2. Generate code with -thrift frugal_tag option

Example:

kitex -thrift frugal_tag -service a.b.c my.thrift

If you don’t need codec code, you can use -thrift template=slim option to reduce generated code significantly.

kitex -thrift frugal_tag,template=slim -service a.b.c my.thrift

Note: Kitex detects Frugal support through frugal struct tags, so keep frugal_tag when enabling the Frugal codec.

3. Init clients and servers with Frugal codec

Use thrift.NewThriftCodecWithConfig to enable Frugal. Codec type priority: Frugal > FastCodec > Apache Thrift.

Client example:

package main

import (
    "context"

    "example.com/kitex_test/client/kitex_gen/a/b/c/echo"
    "github.com/cloudwego/kitex/client"
    "github.com/cloudwego/kitex/pkg/remote/codec/thrift"
)

func main() {
    codec := thrift.NewThriftCodecWithConfig(thrift.FrugalRead | thrift.FrugalWrite)
    cli := echo.MustNewClient("a.b.c", client.WithPayloadCodec(codec))
    ...
}

Server example:

package main

import (
    "log"

    "github.com/cloudwego/kitex/server"
    c "example.com/kitex_test/kitex_gen/a/b/c/echo"
    "github.com/cloudwego/kitex/pkg/remote/codec/thrift"
)

func main() {
    codec := thrift.NewThriftCodecWithConfig(thrift.FrugalRead | thrift.FrugalWrite)
    svr := c.NewServer(new(EchoImpl), server.WithPayloadCodec(codec))

    err := svr.Run()
    if err != nil {
        log.Println(err.Error())
    }
}

Note: Kitex automatically falls back to FastCodec or Apache Thrift for types that don’t have frugal struct tags, so it’s safe to enable Frugal globally.

Codec type options

Option	Description
thrift.FrugalRead	Use Frugal for deserialization
thrift.FrugalWrite	Use Frugal for serialization
thrift.FrugalReadWrite	Shorthand for FrugalRead | FrugalWrite
thrift.FastRead	Use FastCodec for deserialization
thrift.FastWrite	Use FastCodec for serialization
thrift.EnableSkipDecoder	Required when using Buffered transport (no Framed/TTHeader)

For Framed or TTHeader transport, FrugalRead | FrugalWrite is sufficient. For Buffered (PurePayload) transport, add EnableSkipDecoder:

codec := thrift.NewThriftCodecWithConfig(thrift.FrugalRead | thrift.FrugalWrite | thrift.EnableSkipDecoder)

Using with Thrift IDL

Prepare Thrift file

We can define a struct in Thrift file like below:

my.thrift:

struct MyStruct {
    1: string msg
    2: i64 code
}

Use Thriftgo to generate code

Now we have thrift file, we can use Thriftgo to generate Go code.

Recent thriftgo versions generate thrift struct tags that Frugal can read directly, including extra type information for list, set, enum, and maps containing them when needed. Add frugal_tag if you want explicit Frugal tags, or when the generated code will be used with Kitex’s Frugal codec:

thriftgo -r -o thrift -g go:frugal_tag,package_prefix=example.com/kitex_test/thrift my.thrift

If you don’t need codec code, you can use template=slim option to reduce generated code:

thriftgo -r -o thrift -g go:frugal_tag,template=slim,package_prefix=example.com/kitex_test/thrift my.thrift

Use Frugal to serialize or deserialize

Now we can use Frugal to serialize or deserialize the struct defined in thrift file.

Example:

package main

import (
    "github.com/cloudwego/frugal"

    "example.com/kitex_test/thrift"
)

func main() {
    ms := &thrift.MyStruct{
        Msg: "my message",
        Code: 1024,
    }
    ...
    buf := make([]byte, frugal.EncodedSize(ms))
    frugal.EncodeObject(buf, nil, ms)
    ...
    got := &thrift.MyStruct{}
    frugal.DecodeObject(buf, got)
    ...
}

Serialization and deserialization on a customized Go struct

Define a Go struct

We can define a struct like this:

type MyStruct struct {
    Msg     string
    Code    int64
    Numbers []int64 
}

Add Frugal tag to struct fields

Frugal tag is like frugal:"1,default,string", 1 is field ID, default is field requiredness, string is field type. Field ID is required. Requiredness is optional and defaults to default. Field type is usually optional, but required for list, set, enum, and maps containing them.

You can add Frugal tag to MyStruct like below:

type MyStruct struct {
    Msg     string  `frugal:"1,default"`
    Code    int64   `frugal:"2,default"`
    Numbers []int64 `frugal:"3,default,list<i64>"`
}

All types example:

type MyEnum int64

type Example struct {
 MyOptBool         *bool            `frugal:"1,optional"`
 MyReqBool         bool             `frugal:"2,required"`
 MyOptByte         *int8            `frugal:"3,optional"`
 MyReqByte         int8             `frugal:"4,required"`
 MyOptI16          *int16           `frugal:"5,optional"`
 MyReqI16          int16            `frugal:"6,required"`
 MyOptI32          *int32           `frugal:"7,optional"`
 MyReqI32          int32            `frugal:"8,required"`
 MyOptI64          *int64           `frugal:"9,optional"`
 MyReqI64          int64            `frugal:"10,required"`
 MyOptString       *string          `frugal:"11,optional"`
 MyReqString       string           `frugal:"12,required"`
 MyOptBinary       []byte           `frugal:"13,optional"`
 MyReqBinary       []byte           `frugal:"14,required"`
 MyOptI64Set       []int64          `frugal:"15,optional,set<i64>"`
 MyReqI64Set       []int64          `frugal:"16,required,set<i64>"`
 MyOptI64List      []int64          `frugal:"17,optional,list<i64>"`
 MyReqI64List      []int64          `frugal:"18,required,list<i64>"`
 MyOptI64StringMap map[int64]string `frugal:"19,optional"`
 MyReqI64StringMap map[int64]string `frugal:"20,required"`
 MyOptEnum         *MyEnum          `frugal:"21,optional,MyEnum"`
 MyReqEnum         *MyEnum          `frugal:"22,optional,MyEnum"`
}

Use Frugal to serialize or deserialize

Example:

package main

import (
    "github.com/cloudwego/frugal"
)

func main() {
    ms := &MyStruct{
        Msg: "my message",
        Code: 1024,
        Numbers: []int64{0, 1, 2, 3, 4},
    }
    ...
    buf := make([]byte, frugal.EncodedSize(ms))
    frugal.EncodeObject(buf, nil, ms)
    ...
    got := &MyStruct{}
    frugal.DecodeObject(buf, got)
    ...
}