Optional types in Go in done right

Suppose you have a problem: you’ve got relatively small go codebase (10 kloc without tests) to support & develop at your job. You started to add new features, writing tests, deploying it to production on daily basis, and… It keeps breaking after each deployment.

I’m simply dont know every possible problem this codebase has, hence going to talk about only one of them: optional fields for client/server implementations done right and done wrong, with examples.

Even with comprehensive amount of golang’s tools available, it’s hard to enforce good coding style with best practises available for every line of code ever written.

Approaches

So every program has required input and optional input. It’s definetly possible to build entire codebase using default zero-values for each type of input. Golang enforces you to write code in such way, and since (arguably) “good go code” means “code, what Google thinks scales well”, I suppose Google probably build their codebase using zero values. Their other projects, such as gRPC does exacly the same (github issue with discussion, explanation) which is also could count as evidence for my assumption. But different companies has different approaches and not everyone has 20 years of their expirience. Zero values could break compatability with old code. There’re two major ways to deal with what:

Using pointers:

type Data struct {
    id int `json:"id"`
    name *string `json:"name,omitempty"`
    description *string `josn:"description,omitempty"`
}

Using optional fields:

type Data struct {
    id int `json:"id"`
    name String `json:"name"`
    description String `json:"description"`
}

type String struct {
    V string
    Set bool
}

Even quick search gives a lot of results with pointers approach: one, two, three.

I would argue what first approach:

Does not scale: one forgotten nil check and everything crashes with “Invalid memory address or nil pointer dereference”.
Standart library does not support it. “encoding/gob” for example, does not trasmit pointers and flatten values they point to..
Pointers is not an abstraction for representing optional values. Pointers represent memory address and nothing more.

Someone might say: “Wait, aren’t you suppose to test what?”. Well, it depends. Sometimes it’s easy to write test for all combinations of input data, sometimes you’re inventing fuzzer, which may not be best approach. It’s not going to work in my scenario.

I’m not 100% sure about points below, and they’re themselves bad arguments, because trying to outsmart the compiler is terrible idea unless you profile and find bottlenecks, but I included them anyway for you to explain me if I’m wrong. As I understood topic after researching it:

Every pointer increase pressure on GC. Which is itself a bad argument, trying to outsmart compiler is terrible idea, but I included it anyway for you, dear reader, to correct me if I’m wrong.

Possible memory fragmentation? Does golang make more memory allocations for this code:

name := "Service"
desc := "Shrek's search"
data := Data{
  id: 123,
  name: &name,
  description: &description,
}

then for this one:

data := Data{
  id: 123,
  name: "Service",
  description: "Shrek's search",
}

Second approach has it’s own downsides too, you must know them if you’re going to use it:

Code may become verbose.
Requires you to write lot of boilerplate.
Memory overhead. It’s probably depend on specific language implementation, but on OS X x64 arch:

    type Int struct {
        I   int
        Set bool
    }
    ...
	b := true
	p("bool: ", unsafe.Sizeof(b)) // prints: "bool: 1"
	in := 4
	p("int: ", unsafe.Sizeof(in)) // prints: "int: 8"
	i := Int{}
	p("opt int:", unsafe.Sizeof(i)) // prints: "opt int: 16"

Why it works in such way (on x64 arch) is explained here. TL;DR: Memory allocated in consecutive packet of 8 bytes. Bool takes 1 byte, but located in struct allocation would take minimum 8 bytes. Counting bytes – probably bad idea on most of the cases according to Rob Pike.

I made simple solution for last point. Let me tell about it little bit.

Show me the code!

So, how to do better? My contibution: types template. Installation & usage & extension described in README.md. To give you foretaste, here’s an example of what’s possible with simple code generation (more examples):

package main

import (
	"encoding/json"
	"errors"
	"fmt"

	"github.com/hqhs/types"
)

var MissingRequiredFieldError = errors.New("Some of required fields are missing from provided data")

type Presenter interface {
	IsPresent() bool
}

type ExampleT struct {
	F types.Int64  `json:"f"`
	I types.Int64  `json:"i"`
	E types.String `json:"e"`
	L types.String `json:"l"`
	D types.Int    `json:"d"`
}

func (e *ExampleT) Validate() error {
	requiredF := []Presenter{e.F, e.I, e.E}
	for _, c := range requiredF {
		if !c.IsPresent() {
			return MissingRequiredFieldError
		}
	}
	return nil
}

func main() {
	payloadOk := `
{
  "f": 123,
  "i": 321,
  "e": "hello where!"
}
`
	t1 := &ExampleT{}
	handleErr(json.Unmarshal([]byte(payloadOk), t1))
	handleErr(t1.Validate())
	fmt.Printf("Unmarshaled struct: %+v\n", t1)
	payloadErr := `
{
  "f": 123,
  "i": 321
}
`
	t2 := &ExampleT{}
	handleErr(json.Unmarshal([]byte(payloadErr), t2))
	handleErr(t2.Validate())
	fmt.Printf("Unmarshaled struct: %+v\n", t2)
	/* output:
		Unmarshaled struct: &{F:123 I:321 E:hello where! L: D:0}
		panic: Some of required fields are missing from provided data
	    ...
		*stacktrace*
	*/
}

func handleErr(err error) {
	if err != nil {
		panic(err)
	}
}

Custom types generation? how does it work?

Code generation is done using gotemplate. Basically gotemplate parses your template, replace every “type T string” with “tipe ${YourType} optional${YourType}” and saves them to different files. Simple, yet powerful solution.

Links

// Memory allocator.
//
// This was originally based on tcmalloc, but has diverged quite a bit.

tmalloc man page.

Byte order fallacy