1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
|
# ffjson: faster JSON for Go
[](https://travis-ci.org/pquerna/ffjson)
`ffjson` generates static `MarshalJSON` and `UnmarshalJSON` functions for structures in Go. The generated functions reduce the reliance upon runtime reflection to do serialization and are generally 2 to 3 times faster. In cases where `ffjson` doesn't understand a Type involved, it falls back to `encoding/json`, meaning it is a safe drop in replacement. By using `ffjson` your JSON serialization just gets faster with no additional code changes.
When you change your `struct`, you will need to run `ffjson` again (or make it part of your build tools).
## Blog Posts
* 2014-03-31: [First Release and Background](https://journal.paul.querna.org/articles/2014/03/31/ffjson-faster-json-in-go/)
## Getting Started
If `myfile.go` contains the `struct` types you would like to be faster, and assuming `GOPATH` is set to a reasonable value for an existing project (meaning that in this particular example if `myfile.go` is in the `myproject` directory, the project should be under `$GOPATH/src/myproject`), you can just run:
go get -u github.com/pquerna/ffjson
ffjson myfile.go
git add myfile_ffjson.go
## Performance Status:
* `MarshalJSON` is **2x to 3x** faster than `encoding/json`.
* `UnmarshalJSON` is **2x to 3x** faster than `encoding/json`.
## Features
* **Unmarshal Support:** Since v0.9, `ffjson` supports Unmarshaling of structures.
* **Drop in Replacement:** Because `ffjson` implements the interfaces already defined by `encoding/json` the performance enhancements are transparent to users of your structures.
* **Supports all types:** `ffjson` has native support for most of Go's types -- for any type it doesn't support with fast paths, it falls back to using `encoding/json`. This means all structures should work out of the box. If they don't, [open a issue!](https://github.com/pquerna/ffjson/issues)
* **ffjson: skip**: If you have a structure you want `ffjson` to ignore, add `ffjson: skip` to the doc string for this structure.
* **Extensive Tests:** `ffjson` contains an extensive test suite including fuzz'ing against the JSON parser.
# Using ffjson
`ffjson` generates code based upon existing `struct` types. For example, `ffjson foo.go` will by default create a new file `foo_ffjson.go` that contains serialization functions for all structs found in `foo.go`.
```
Usage of ffjson:
ffjson [options] [input_file]
ffjson generates Go code for optimized JSON serialization.
-go-cmd="": Path to go command; Useful for `goapp` support.
-import-name="": Override import name in case it cannot be detected.
-nodecoder: Do not generate decoder functions
-noencoder: Do not generate encoder functions
-w="": Write generate code to this path instead of ${input}_ffjson.go.
```
Your code must be in a compilable state for `ffjson` to work. If you code doesn't compile ffjson will most likely exit with an error.
## Disabling code generation for structs
You might not want all your structs to have JSON code generated. To completely disable generation for a struct, add `ffjson: skip` to the struct comment. For example:
```Go
// ffjson: skip
type Foo struct {
Bar string
}
```
You can also choose not to have either the decoder or encoder generated by including `ffjson: nodecoder` or `ffjson: noencoder` in your comment. For instance, this will only generate the encoder (marshal) part for this struct:
```Go
// ffjson: nodecoder
type Foo struct {
Bar string
}
```
You can also disable encoders/decoders entirely for a file by using the `-noencoder`/`-nodecoder` commandline flags.
## Using ffjson with `go generate`
`ffjson` is a great fit with `go generate`. It allows you to specify the ffjson command inside your individual go files and run them all at once. This way you don't have to maintain a separate build file with the files you need to generate.
Add this comment anywhere inside your go files:
```Go
//go:generate ffjson $GOFILE
```
To re-generate ffjson for all files with the tag in a folder, simply execute:
```sh
go generate
```
To generate for the current package and all sub-packages, use:
```sh
go generate ./...
```
This is most of what you need to know about go generate, but you can sese more about [go generate on the golang blog](http://blog.golang.org/generate).
## Should I include ffjson files in VCS?
That question is really up to you. If you don't, you will have a more complex build process. If you do, you have to keep the generated files updated if you change the content of your structs.
That said, ffjson operates deterministically, so it will generate the same code every time it run, so unless your code changes, the generated content should not change. Note however that this is only true if you are using the same ffjson version, so if you have several people working on a project, you might need to synchronize your ffjson version.
## Performance pitfalls
`ffjson` has a few cases where it will fall back to using the runtime encoder/decoder. Notable cases are:
* Interface struct members. Since it isn't possible to know the type of these types before runtime, ffjson has to use the reflect based coder.
* Structs with custom marshal/unmarshal.
* Map with a complex value. Simple types like `map[string]int` is fine though.
* Inline struct definitions `type A struct{B struct{ X int} }` are handled by the encoder, but currently has fallback in the decoder.
* Slices of slices / slices of maps are currently falling back when generating the decoder.
## Reducing Garbage Collection
`ffjson` already does a lot to help garbage generation. However whenever you go through the json.Marshal you get a new byte slice back. On very high throughput servers this can lead to increased GC pressure.
### Tip 1: Use ffjson.Marshal() / ffjson.Unmarshal()
This is probably the easiest optimization for you. Instead of going through encoding/json, you can call ffjson. This will disable the checks that encoding/json does to the json when it receives it from struct functions.
```Go
import "github.com/pquerna/ffjson/ffjson"
// BEFORE:
buf, err := json.Marshal(&item)
// AFTER:
buf, err := ffjson.Marshal(&item)
```
This simple change is likely to double the speed of your encoding/decoding.
[![GoDoc][1]][2]
[1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg
[2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Marshal
### Tip 2: Pooling the buffer
On servers where you have a lot of concurrent encoding going on, you can hand back the byte buffer you get from json.Marshal once you are done using it. An example could look like this:
```Go
import "github.com/pquerna/ffjson/ffjson"
func Encode(item interface{}, out io.Writer) {
// Encode
buf, err := ffjson.Marshal(&item)
// Write the buffer
_,_ = out.Write(buf)
// We are now no longer need the buffer so we pool it.
ffjson.Pool(buf)
}
```
Note that the buffers you put back in the pool can still be reclaimed by the garbage collector, so you wont risk your program building up a big memory use by pooling the buffers.
[![GoDoc][1]][2]
[1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg
[2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Pool
### Tip 3: Creating an Encoder
There might be cases where you need to encode many objects at once. This could be a server backing up, writing a lot of entries to files, etc.
To do this, there is an interface similar to `encoding/json`, that allow you to create a re-usable encoder. Here is an example where we want to encode an array of the `Item` type, with a comma between entries:
```Go
import "github.com/pquerna/ffjson/ffjson"
func EncodeItems(items []Item, out io.Writer) {
// We create an encoder.
enc := ffjson.NewEncoder(out)
for i, item := range items {
// Encode into the buffer
err := enc.Encode(&item)
// If err is nil, the content is written to out, so we can write to it as well.
if i != len(items) -1 {
_,_ = out.Write([]byte{','})
}
}
}
```
Documentation: [![GoDoc][1]][2]
[1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg
[2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Encoder
## Tip 4: Avoid interfaces
We don't want to dictate how you structure your data, but having interfaces in your code will make ffjson use the golang encoder for these. When ffjson has to do this, it may even become slower than using `json.Marshal` directly.
To see where that happens, search the generated `_ffjson.go` file for the text `Falling back`, which will indicate where ffjson is unable to generate code for your data structure.
## Tip 5: `ffjson` all the things!
You should not only create ffjson code for your main struct, but also any structs that is included/used in your json code.
So if your struct looks like this:
```Go
type Foo struct {
V Bar
}
```
You should also make sure that code is generated for `Bar` if it is placed in another file. Also note that currently it requires you to do this in order, since generating code for `Foo` will check if code for `Bar` exists. This is only an issue if `Foo` and `Bar` are placed in different files. We are currently working on allowing simultaneous generation of an entire package.
## Improvements, bugs, adding features, and taking ffjson new directions!
Please [open issues in Github](https://github.com/pquerna/ffjson/issues) for ideas, bugs, and general thoughts. Pull requests are of course preferred :)
## Similar projects
* [go-codec](https://github.com/ugorji/go/tree/master/codec#readme). Very good project, that also allows streaming en/decoding, but requires you to call the library to use.
* [megajson](https://github.com/benbjohnson/megajson). This has limited support, and development seems to have almost stopped at the time of writing.
# Credits
`ffjson` has recieved significant contributions from:
* [Klaus Post](https://github.com/klauspost)
* [Paul Querna](https://github.com/pquerna)
* [Erik Dubbelboer](https://github.com/erikdubbelboer)
## License
`ffjson` is licensed under the [Apache License, Version 2.0](./LICENSE)
|
