Looking in Go’s Mirror: How and When to use reflect
This article was originally published on gopheradvent.com
Go’s static typing is a headline feature of the language. It prevents whole classes of bugs, makes code easier to navigate and refactor, and makes it easier for linters to analyze. But there are times when it’s very constricting. What if we’re reading JSON files from disk with unknown structure? We can’t define a type ahead of time that will cover all cases. The reflect package gives us the power to handle this situation, and to do much more. We can write functions that handle arbitrary types. reflect is also the technology behind Go’s “magical” struct tags, which are often used in serialization. reflect can be intimidating to new Go programmers because it’s very generic and you lose access to many niceties in the language. But it doesn’t have to be. Let’s build some programs that use reflect as a way to demystify the package and illustrate the power and pitfalls that come with using it.
reflect package basics
Value, Type, and Kind
The fundamental concepts of reflection are the Value and Type structs, and the key starting points are the ValueOf and TypeOf functions. You can use the TypeOf function to get the Type of any variable in Go; ValueOf provides access to the underlying value, along with the type information.
var x float64 = 8.4
var y struct{ Field float64}{ Field: 1.2 }
var z any = x
fmt.Println(reflect.TypeOf(z))
z = y
fmt.Println(reflect.TypeOf(z))
val := reflect.ValueOf(z)
fmt.Println(val.Type())
fmt.Println(val)
// Prints:
// float64
// struct { Field float64 }
// {1.2}
// struct { Field float64 }
// {1.2}
Every Type and Value also has a Kind that provides the building block for handling arbitrary data structures with reflect. The Kind() function returns one of Go’s primitive types like Float64, Map, Array, and Struct.
Settability
reflect also lets us modify the Value that we have as long as the value is “settable”. It’s convenient to think of “settability” as similar to “is a pointer”. reflect needs a pointer at the value it’s referring to if it wants to modify it. Here’s an example:
var x float64 = 8.4
fmt.Println("x:", x)
v := reflect.ValueOf(x)
fmt.Println("v is settable:", v.CanSet())
v = reflect.ValueOf(&x)
fmt.Println("pointer is settable:", v.CanSet())
v = v.Elem()
fmt.Println("pointer's value is settable:", v.CanSet())
v.SetFloat(6.1)
fmt.Println("x:", x)
OK, let’s look at some programs using reflect! These programs will be missing some checks and cases in the interest of brevity. Don’t base your production code on this!
Deserializing structured data with reflect
Suppose you have a file with JSON in it, but you don’t know its structure:
{
"name": "Joe Blubaugh",
"location": "Singapore",
"job": {
"company": "Grafana",
"title": "Software Engineer"
}
}
The json.Unmarshal function uses reflection to deserialize data when it’s passed an any value as the destination. We can also inspect the result with reflect. The full example is available on goplay.tools, and the key element for our analysis is a function that recursively processes a reflect.Value according to the Kind:
func visit(val reflect.Value, indent int) string {
var s string
switch val.Kind() {
case reflect.Interface:
s += visit(val.Elem(), indent)
case reflect.Map:
s += indentIt("Map\n", indent)
iter := val.MapRange()
for iter.Next() {
s += indentIt("Key: "+iter.Key().String(), indent+1)
s += "\n"
s += indentIt("Value: "+visit(iter.Value(), indent+1), indent+1)
s += "\n\n"
}
case reflect.String:
s += val.String()
}
return s
}
The switch on the kind of that Value tells us how to handle it. If it’s a map, we iterate over the keys and visit each value, and if it’s an interface we dereference it to get the underlying value before handling it. Most kinds are left out here for brevity, since the data supplied only has JSON objects and string.
How does Unmarshal create the right types when building up the structure here? As it scans the JSON, it chooses appropriate types based on the structure of the data, as documented here. It uses the MakeMap, MakeSlice, and New functions to create each Value and assign the parsed data to them. Using those functions, we can build any data structure we want at runtime without a prior definition. We can even build new structs without a predeclared type definition by using the StructOf function.
If you’re clever with reflection, that means that you can parse a schema at runtime and then validate whether any data matches that schema. The JSON below is an example of our schema. It has a value for every required field, with the correct type set.
{
"name": "foo",
"address": [123, "Street Name", "City", 11111],
"age_days": 1,
"height_cm": 1.0
}
and here is a structure that doesn’t match our schema because of a missing field.
{
"name": "Joe Blubaugh",
"address": [101, "Foo Street", "Bazville", 10101],
"height_cm": 180.1
}
This program can parse the schema and validate other input data by checking that the Type for each field matches and that there are no missing fields in the input data. Notice how similar the validate function is to the visit function that we used to analyze the structure of arbitrary JSON.
// This program loads a schema defined in JSON and validates other JSON
// against the schema. Every field in the schema is required in the data,
// and the order and type of values in any JSON arrays must also match.
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
var schema []byte = []byte(`
{
"name": "foo",
"address": [123, "Street Name", "City", 11111],
"age_days": 1,
"height_cm": 1.0
}`)
// This data does not match the schema because of a missing "age_days" field.
var invalidData []byte = []byte(`
{
"name": "Joe Blubaugh",
"address": [101, "Foo Street", "Bazville", 10101],
"height_cm": 180.1
}`)
// This data is valid
var validData []byte = []byte(`
{
"name": "Baby Blubaugh",
"address": [101, "Foo Street", "Bazville", 10101],
"height_cm": 39,
"age_days": 91
}`)
var s, d any
err := json.Unmarshal(schema, &s)
if err != nil {
panic(err)
}
err = json.Unmarshal(invalidData, &d)
if err != nil {
panic(err)
}
fmt.Println("Invalid data passed validation?", validate(s, d))
err = json.Unmarshal(validData, &d)
if err != nil {
panic(err)
}
fmt.Println("Valid data passed validation?", validate(s, d))
}
func validate(schema, data any) bool {
s := reflect.ValueOf(schema)
d := reflect.ValueOf(data)
sKind := s.Kind()
dKind := d.Kind()
if sKind != dKind {
return false
}
switch s.Kind() {
case reflect.Pointer:
return validate(s.Elem(), d.Elem())
case reflect.Map:
// Do both maps have the same length? If not, the data can't be valid.
if len(s.MapKeys()) != len(d.MapKeys()) {
return false
}
iter := s.MapRange()
for iter.Next() {
key := iter.Key()
sValue := iter.Value()
dValue := d.MapIndex(key)
if dValue.IsZero() {
// The key isn't present in the data:
return false
}
if !validate(sValue, dValue) {
return false
}
}
case reflect.Slice:
// Our schema is restrictive: the slice *must* be present in both structures,
// they must have the same length, and the same order of data types.
if d.Len() != s.Len() {
return false
}
for i := 0; i < s.Len(); i++ {
sVal := s.Index(i)
dVal := d.Index(i)
if !validate(sVal, dVal) {
return false
}
}
// Skip reflect.Struct because the code to iterate over fields is complex,
// and json Unmarshalling into `any` never produces a struct.
// If types match for primitive values like ints, then this is valid.
}
return true
}
Play around yourself, modifying the schema and the data to understand better how that works.
Using struct tags to access metadata
In Go we often use struct tags on data transfer objects: HTTP body data, structs that are mapped to database tables, etc. reflect gives you access to those tags. Suppose we want to redact some fields (like a credit card number) from a struct before sending it to other functions. We could write this program using tags:
package main
import (
"errors"
"fmt"
"reflect"
)
func main() {
type PaymentForm struct {
ValueCents int
SKU string
CreditCardNumber string `redact:"true"`
}
f := PaymentForm{
ValueCents: 10_00,
SKU: "abe15f59",
CreditCardNumber: "5555555555555555",
}
fmt.Println("Unredacted:", f)
err := redact(&f)
if err != nil {
panic(err)
}
fmt.Println("Redacted:", f)
}
func redact(val any) error {
v := reflect.ValueOf(val)
if v.Kind() != reflect.Pointer {
return errors.New("Must pass a struct pointer to be set.")
}
v = v.Elem()
if v.Kind() != reflect.Struct {
return errors.New("Must pass a struct pointer to be set.")
}
t := v.Type()
for i := 0; i < v.NumField(); i++ {
value := v.Field(i)
field := t.Field(i)
shouldRedact := field.Tag.Get("redact") == "true"
if shouldRedact {
// This is simplified to assume that all redacted fields are strings.
// That's just for brevity in this example.
value.SetString("XXXXX")
}
}
return nil
}
The full program is also here on the playground
Let’s not overdo it
If you get really creative with reflect you can do all sorts of things. It becomes possible to write partially applied functions and implement other functional programming primitives. “Clever” programming is an awkward fit for Go, though. When you use reflect, your code can quickly become complex and recursive, which can make it harder to maintain and harder to explain to your teammates.
Reflection also impacts your program’s performance. Accessing type information for a value is slow compared to code generated by the compiler. Even type assertions at compile time are significantly faster than similar code using reflect, so it’s best to use the package only when you can’t use the language’s ordinary type handling features like interfaces and type assertions.
All that being said, reflection provides an essential tool that we should all be familiar with: handling data when we can’t predict its structure at build time. That makes it really essential when handling byte streams from reading data off of disk or as the result of network calls.
Additional reading
Rob Pike has an excellent blog post called The Laws of Reflection that expands on the basics above, providing more detail about the underlying implementation of interface types the reflect package uses to do its thing.