Tracing Application Interactions with CockroachDB
Part 2 — Node CPU Profiles
Continuing from my blog “Tracing application Interactions with CockroachDB”, Next, I would like to dive into the cpu profiles and give some examples of how to collect them and to inspect the profiles offline and online. CockroachDB is written in go, which has a built-in profiling tool which can be used to examine and understand hot spots in the Cockroach runtime. You can examine profiles for CPU, Heap, Mutex, Block, Goroutines and Jemalloc.
For this blog, I will focus on CPU profiles which are accessible from the CockroachDB Console ../#/debug endpoint at debug/pprof/ui/cpu/[*anynode*]/flamegraph?node=1&seconds=5&labels=true. CPU profiles display the functions which consume the most CPU time. CPU samples over a time interval, the callstack of the cockroach running program.
By clicking on the link above, a 5 second CPU profile will download to your laptop or open for interactive use. You can increase the duration and access a flamegraph from the ui directly.
There are various drop downs which can be used to view the following modes
Alternatively, you can save these profiles for future investigation offline. The following commands can be used for saving and viewing the profiles using go tool pprof.
Save the profile
curl -kL “http://127.0.0.1:26258/debug/pprof/profile?seconds=5" — output pprof.node1
go tool pprof -http=”:8081" pprof.node1
Will launch a browser to the profile http://localhost:8081 for the same interactive experience as in the DBConsole mode. You can also launch interactive command line
go tool pprof pprof.node1
And use any subcommands such as top and peak to view the top cpu consumers for the pprof sample. For example,
Shows the breakdown of calls sorted by flat column indicating time spent and then we can inspect a specific call with peek on encodeBlock which is the highest time consumer
And I can see what the system is doing for a period of time for this sample which is compression, data encoding and writing to SST tables. And further to addPoint
If i wanted to I can go back to source code to view what the call is doing or have the source installed locally to inspect it. We can also look back at the local web display
So that wraps up this short tour on CPU profiles which can be helpful when trying to understand how cockroachDB runtime consumes CPU for a node.
