A New Go API for JSON

Introduction

JavaScript Object Notation (JSON) is a uncomplicated data interchange format, the most famous data format used on the internet, and extensively peruse and written by Go programs. Go has included assistance for JSON in the norm archive since fine before Go 1.0. We archetypal we wrote concerning support for JSON in Go additional than 15 years ago!

Over time, packages develop alongside the needs of their users, and encoding/json is no exception. Last year, we wrote concerning a new experimental API for JSON designed to bring long-awaited improvements and fixes.

After additional feedback and refinement, the new encoding/json/v2 and encoding/json/jsontext packages are stable and mostly accessible in Go 1.27! This article is an updated type of the article introducing the experiment, describing the problems alongside encoding/json, and an introduction to the new packages.

Problems alongside encoding/json

Overall, encoding/json has held up well. The idea of marshaling and unmarshaling arbitrary Go types with several default depiction in JSON, blended alongside the capability to customize the representation, has proven to be extremely flexible. However, in the years since its introduction, various users have identified many shortcomings.

Behavior flaws

There are assorted behavioral flaws in encoding/json:

• Imprecise handling of JSON syntax: Over the years, JSON has seen increased standardization in command for programs to correctly communicate. Generally, decoders have rotate into stricter at rejecting ambiguous inputs, to decrease the chance that two implementations volition have different (successful) interpretations of a particular JSON value.

  • encoding/json currently accepts invalid UTF-8, whereas the latest Internet Standard (RFC 8259) requires valid UTF-8. The default behavior should study an error in the existence of invalid UTF-8, instead of introducing silent data corruption, which may logic problems downstream.

  • encoding/json currently accepts objects alongside copy associate names. RFC 8259 does not define how to grip copy names, so an implementation is liberated to choose an arbitrary value, merge the values, discard the values, or study an error. The existence of a copy name results in a JSON value without a universally accepted upon meaning. This could be exploited by attackers in safety applications and has been exploited before (as in CVE-2017-12635). The default behavior should err on the flank of safety and refuse copy names.

• Leaking nilness of slices and maps: JSON is frequently used to communicate with programs using JSON implementations that do not authorize null to be unmarshaled into a data category expected to be a JSON gathering or object. Since encoding/json marshals a nil piece or map as a JSON null, this may guide to errors whenever unmarshaling by another implementations. A survey indicated that most Go users favor that nil slices and maps are marshaled as an bare JSON gathering or entity by default.

• Case-insensitive unmarshaling: When unmarshaling, a JSON entity associate name is resolved to a Go struct site name using a case-insensitive match. This is a amazing default, a possible safety vulnerability, and a achievement limitation.

• Inconsistent calling of methods: Due to an implementation detail, MarshalJSON methods declared on a pointer receiver are inconsistently called by encoding/json. While regarded as a bug, this cannot be fixed as too many applications depend on the current behavior.

API deficiencies

The API of encoding/json can be tricky or restrictive:

• It is difficult to correctly unmarshal from an io.Reader. Users frequently compose json.NewDecoder(r).Decode(v), which fails to refuse trailing junk at the end of the input.

• Options can be set on the Encoder and Decoder types, but cannot be used alongside the Marshal and Unmarshal functions. Similarly, types implementing the Marshaler and Unmarshaler interfaces cannot create use of the options and there is no way to plumb options downward the call stack. For example, the Decoder.DisallowUnknownFields choice loses its effect when calling a tradition UnmarshalJSON method.

• The Compact, Indent, and HTMLEscape functions compose to a bytes.Buffer instead of item additional elastic akin a []byte or io.Writer. This limits the usability of those functions.

Performance limitations

Setting apart inner implementation details, the community API commits it to certain achievement limitations:

• MarshalJSON: The MarshalJSON interface method forces the implementation to allocate the returned []byte. Also, the semantics necessitate that encoding/json verify that the outcome is valid JSON and additionally to reformat it to equivalent the stated indentation.

• UnmarshalJSON: The UnmarshalJSON interface method requires that a complete JSON value be provided (without any trailing data). This forces encoding/json to parse the JSON value to be unmarshaled in its entirety to decide anywhere it ends before it can call UnmarshalJSON. Afterwards, the UnmarshalJSON method itself must parse the provided JSON value again.

• Lack of streaming: Even although the Encoder and Decoder types operate on an io.Writer or io.Reader, they buffer the complete JSON value in memory. The Decoder.Token method for study idiosyncratic tokens is allocation-heavy and there is no corresponding API for penning tokens.

Furthermore, if the implementation of a MarshalJSON or UnmarshalJSON method recursively calls the Marshal or Unmarshal function, then the achievement becomes quadratic.

Trying to fix encoding/json directly

Introducing a new, incompatible important type of a bundle is a dense consideration. If possible, we should try to fix the existing package.

While it is comparatively uncomplicated to add new features, it is difficult to alter existing features. Unfortunately, these problems are inherent consequences of the existing API, making them practically unattainable to fix inside the Go 1 compatibility promise.

We could in regulation province distinct names, specified as MarshalV2 or UnmarshalV2, but that is tantamount to creating a parallel namespace inside the identical package. This leads us to encoding/json/v2 (henceforth called v2), where we can create these changes inside a distinct v2 namespace in difference to encoding/json (henceforth called v1).

Building on encoding/json/jsontext

Before discussing the v2 API, we archetypal current the encoding/json/jsontext package that lays the basis for improvements to JSON in Go.

JSON serialization in Go can be damaged downward into two chief components:

• syntactic functionality that is concerned alongside handling JSON according to its grammar, and
• semantic functionality that defines the association between JSON values and Go values.

We use the conditions “encode” and “decode” to depict syntactic functionality and the conditions “marshal” and “unmarshal” to depict semantic functionality. We aim to provision a apparent difference between functionality that is purely concerned alongside encoding versus that of marshaling.

This diagram provides an overview of this separation. Purple blocks portray types, during blue blocks portray functions or methods. The direction of the arrows about represents the stream of data. The base fractional of the diagram, implemented by the jsontext package, contains functionality that is lone concerned alongside syntax, while the high half, implemented by the json/v2 package, contains functionality that assigns semantic definition to syntactic data handled by the base half.

The essential API of jsontext is the following:

package jsontext type Encoder struct { ... } func NewEncoder(io.Writer, ...Options) *Encoder func (*Encoder) WriteValue(Value) error func (*Encoder) WriteToken(Token) error type Decoder struct { ... } func NewDecoder(io.Reader, ...Options) *Decoder func (*Decoder) ReadValue() (Value, error) func (*Decoder) ReadToken() (Token, error) type Kind byte type Value []byte func (Value) Kind() Kind type Token struct { ... } func (Token) Kind() Kind 

The jsontext bundle provides functionality for interacting alongside JSON at a syntactic flat and derives its name from RFC 8259, division 2 where the grammar for JSON data is exactly called JSON-text. Since it lone interacts alongside JSON at a syntactic level, it does not depend on Go reflection.

The Encoder and Decoder provide assistance for encoding and decoding JSON values and tokens. The constructors accept variadic options that power the particular behavior of encoding and decoding. Unlike the Encoder and Decoder types declared in v1, the new types in jsontext evade muddling the difference between syntax and semantics and run in a really streaming manner.

A JSON value is a complete component of data and is represented in Go as a named []byte. It is identical to RawMessage in v1, which is now a category pseudonym to jsontext.Value. A JSON value is syntactically composed of one or additional JSON tokens. A JSON token is represented in Go as the opaque Token type with constructors and accessor methods. It is akin to Token in v1 but is designed portray arbitrary JSON tokens without allocation.

To determine the essential achievement problems with the MarshalJSON and UnmarshalJSON interface methods, we need an productive way of encoding and decoding JSON as a streaming sequence of tokens and values. In v2, we current the MarshalJSONTo and UnmarshalJSONFrom interface methods that run on an Encoder or Decoder, allowing the methods’ implementations to procedure JSON in a purely streaming manner. Thus, the json bundle need not be liable for validating or formatting a JSON value returned by MarshalJSON, nor would it need to be liable for determining the boundaries of a JSON value provided to UnmarshalJSON. These responsibilities pertain to the Encoder and Decoder.

Introducing encoding/json/v2

Building on the jsontext package, we now current the encoding/json/v2 package. It is designed to fix the problems in v1, while remaining acquainted to users of the v1 package. Our goal is that usages of v1 volition run mostly the identical if immediately migrated to v2.

Here, we chiefly shield the high-level API of v2. For examples on how to use it, we advance readers to study the examples in the v2 package, the “Working alongside JSON” tutorial, or read Anton Zhiyanov’s blog covering the topic.

The essential API of v2 is the following:

package json func Marshal(in any, opts ...Options) (out []byte, err error) func MarshalWrite(out io.Writer, in any, opts ...Options) error func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error func Unmarshal(in []byte, out any, opts ...Options) error func UnmarshalRead(in io.Reader, out any, opts ...Options) error func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error 

The Marshal and Unmarshal functions have a signature akin to v1, but obtain options to configure their behavior. The MarshalWrite and UnmarshalRead functions directly run on an io.Writer or io.Reader, avoiding the need to temporarily build an Encoder or Decoder just to compose or peruse from specified types. The MarshalEncode and UnmarshalDecode functions operate on a jsontext.Encoder and jsontext.Decoder and is really the underlying implementation of the earlier mentioned functions. Unlike v1, options are a first-class disagreement to all of the marshal and unmarshal functions, greatly extending the elasticity and configurability of v2. There are several options available in v2 which are not covered by this article.

Type-specified customization

Similar to v1, v2 allows types to define their own JSON representation by satisfying particular interfaces.

type Marshaler interface { MarshalJSON() ([]byte, error) } type MarshalerTo interface { MarshalJSONTo(*jsontext.Encoder) error } type Unmarshaler interface { UnmarshalJSON([]byte) error } type UnmarshalerFrom interface { UnmarshalJSONFrom(*jsontext.Decoder) error } 

The Marshaler and Unmarshaler interfaces are identical to those in v1. The new MarshalerTo and UnmarshalerFrom interfaces allow a category to portray itself as JSON using a jsontext.Encoder or jsontext.Decoder. This allows options to be forwarded downward the call stack, since options can be retrieved via the Options accessor method on the Encoder or Decoder.

See the OrderedObject example for how to execute a tradition category that maintains the ordering of JSON entity members.

Caller-specified customization

In v2, the visitant of Marshal and Unmarshal can additionally specify a tradition JSON depiction for any arbitrary type, where caller-specified functions obtain precedence complete type-defined methods or the default depiction for a particular type.

func WithMarshalers(*Marshalers) Options type Marshalers struct { ... } func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers func WithUnmarshalers(*Unmarshalers) Options type Unmarshalers struct { ... } func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers 

MarshalFunc and MarshalToFunc construct a tradition marshaler that can be passed to a Marshal call using WithMarshalers to override the marshaling of particular types. Similarly, UnmarshalFunc and UnmarshalFromFunc support akin customization for Unmarshal.

The ProtoJSON example demonstrates how this characteristic allows serialization of all proto.Message types to be handled by the protojson package.

Behavior differences

While v2 aims to behave mostly the identical as v1, its behavior has changed in several ways to location problems in v1, most notably:

• v2 reports an error in the existence of invalid UTF-8.

• v2 reports an error if a JSON entity contains a copy name.

• v2 marshals a nil Go piece or Go map as an bare JSON gathering or JSON object, respectively.

• v2 unmarshals a JSON entity into a Go struct using a case-sensitive equivalent from the JSON associate name to the Go site name.

• v2 redefines the omitempty tag choice to omit a site if it would have encoded as an “empty” JSON value (which are null, "", [], and {}).

• v2 reports an error whenever trying to serialize a time.Duration, which currently has no default representation, but provides options to let the visitant decide.

For most behavior changes, there is a struct tag choice or caller-specified option that can configure the behavior to run under v1 or v2 semantics, or equal another caller-determined behavior. See “Migrating to v2” for additional information.

Performance optimizations

The Marshal achievement of v2 is approximately at parity alongside v1. Sometimes it is slightly faster, but another times it is slightly slower. The Unmarshal achievement of v2 is considerably faster than v1, with benchmarks demonstrating improvements of up to 10x.

In command to get greater achievement gains, existing implementations of Marshaler and Unmarshaler should migrate to additionally implement MarshalerTo and UnmarshalerFrom, so that they can advantage from handling JSON in a purely streaming manner. For example, recursive parsing of OpenAPI details in UnmarshalJSON methods significantly hurt achievement in a particular assistance of Kubernetes (see kubernetes/kube-openapi#315), while switching to UnmarshalJSONFrom improved achievement by orders of magnitude.

For additional benchmark details, see the go-json-experiment/jsonbench repository.

On top of structural improvements, we’re continuing to allocate in achievement optimizations in the implementation for cases anywhere the v2 implementation is slower than v1. You can anticipate additional improvements in Go 1.28.

Compatibility

The Go 1 compatibility promise is a center justify from the Go project that allows developers to build applications that volition remain the test of period and continue operating on forthcoming versions of Go.

The first encoding/json and new encoding/json/v2 packages are of way the two topic to the Go 1 compatibility promise. Applications using v1 volition continue to be supported indefinitely and need not always migrate to v2.

That said, whenever designing the v2 packages we wanted to do improved than keeping v1 supported. We wanted to justify that v1 and v2 are compatible alongside one another so that packages using distinct versions can activity together. Doing so allows a gentle ecosystem immigration fairly than forcing users to use one type or another because of the choices of their dependencies.

For instance, assume your bundle uses v2 to marshal a struct containing a site alongside a category defined in another package. That bundle may use the first v1 MarshalJSON method to define its marshaling behavior. No problem, v2 supports MarshalJSON methods, so marshaling volition continue operating as expected.

Vice versa, assume your bundle uses v1 to marshal the identical struct. The another bundle may use the v2 MarshalJSONTo method for improved performance. No issue current either, as the v1 implementation supports v2’s MarshalJSONTo method under the hood.

We’ve achieved this by designing the new packages specified that, under the hood, v1 is implemented in conditions of v2. As mentioned above, v2 supports a assortment of Options. Each behavior difference is controlled by among the these options. Thus, v1 can be implemented in conditions of v2 by passing the complete set of v1 compatibility options.

With this approach, backwards-compatible additions specified as new methods (e.g., MarshalJSONTo) or new struct tags (e.g., embed) are automatically supported to the advantage of existing v1 users.

Migration

Implementing v1 in conditions of v2 options has the additional advantage of allowing gradual immigration of callsites.

encoding/json documents the complete set of behavior differences between v1 and v2, and which choice controls that behaviors.

Many applications can migrate immediately to v2 alongside no apparent behavior changes, but applications that depend on v1 behavior can use the options to alter lone the behaviors they depend on.

For example, we can choose all v2 behavior apart from for allowing copy names:

json.Marshal(..., json.DefaultOptionsV2(), jsontext.AllowDuplicateName(true)) 

Alternatively, we can choose all v1 behavior apart from for disallowing copy names:

json.Marshal(..., jsonv1.DefaultOptionsV1(), jsontext.AllowDuplicateName(false)) 

The encoding/json/v2 Migration Guide goes into additional detail, covering a assortment of scenarios for approaching immigration of an use from v1 to v2.

More information

For additional references on operating alongside the new JSON packages, see:

• The “Working alongside JSON” tutorial,
• The encoding/json/v2 and encoding/json/jsontext bundle documentation,
• The encoding/json package’s v2 difference documentation,
• and the encoding/json/v2 Migration Guide.

A lengthy period coming

The preparedness for a new important type of encoding/json spanned years. In delayed 2020, spurred on by the inability to fix issues in the current package, Daniel Martí (one of the maintainers of encoding/json) archetypal drafted his thoughts on what a hypothetical v2 bundle should appearance like. Separately, following former activity on the Go API for Protocol Buffers, Joe Tsai was disappointed that the protojson package needed to use a tradition JSON implementation since encoding/json was neither capable of adhering to the stricter JSON norm that the Protocol Buffer specification required, nor of efficiently serializing JSON in a streaming manner.

Believing a brighter forthcoming for JSON was the two beneficial and achievable, Daniel and Joe joined forces to brainstorm on v2 and started to build a prototype (with the first code being a polished type of the JSON serialization logic from the Go protobuf module). Over time, a few others (Roger Peppe, Chris Hines, Johan Brandhorst-Satzkorn, Damien Neil, Austin Clements, and Michael Pratt) joined the attempt by providing scheme review, code review, and regression testing. Many of the first discussions are publically accessible in our recorded meetings and meeting notes. Later discussions are accessible in our encoding/json/v2 operating collection gathering notes.

This activity has been community since the beginning, and we increasingly engaged the wider Go community, first alongside a GopherCon talk and discussion posted in delayed 2023, formal recommendation posted in first 2025, and eventually adopting encoding/json/v2 as a Go test in Go 1.25 for wider-scale evaluation by all Go users.

The v2 attempt took concerning 6 years to attain this point, incorporating feedback from many contributors and additionally gaining valuable empirical cognition from use in manufacturing settings.

It’s value noting that it’s mostly been developed and promoted by people not employed by Google, demonstrating that the Go project is a collaborative endeavor with a thriving earth community dedicated to improving the Go ecosystem.

Thank you to everyone that contributed!