Skip to content

proposal: new Recorder types in runtime/pprof #74545

Open
@mknyszek

Description

@mknyszek

proposal: new Recorder types in runtime/pprof

Written primarily by me and @prattmic, but with input from the Go team. Consider this as a first take. We're hoping to shake out the details with the community.

Background

The runtime/pprof package provides foundational Go APIs to collect runtime diagnostic data from Go programs. It is critical to Go's production stack, with 22,000+ public package imports.

For the most part this package serves us well, and most profiles are neatly captured by a global Profile value. However, configuring each profile is messy. The sampling and collection configuration of each profile generally consists of some bespoke functions and/or global variables in the runtime and runtime/pprof packages. Meanwhile the profile format is captured by an opaque and slightly mysterious debug integer. Things get worse as that there's other data that overlaps with the profile data. For the worst case of this complexity, see, for example, Felix Geisendörfer's goroutine profile feature matrix).

On top of all this, CPU profiles have a completely different API, and yet another new API that has an accepted proposal, but has stagnated before being implemented.

Goal

The existing API surface has three main problems:

The goal of this proposal is then to create a template for runtime diagnostics APIs that is clear, composable, and extendible.

Proposal

There are 3 kinds of profiles we expose today:

  1. Snapshot profiles (goroutine profile, heap profile (inuse))
  2. Profiles containing data since program start (heap profile (allocs) and most other profiles)
  3. Profiles containing data for a specific window of time (CPU profiles)

The core idea behind this proposal is to create a suite of recorder types that exclusively expose profiles as either snapshot profiles or time-windowed profiles. Profiles containing data since program start are non-intuitive and consistently confusing, especially when coupled with the fact that profile configuration is in a different package. (For example, mutex and block profiles are completely empty by default, and the API exposed takes a snapshot of all data since program start. So, by default, the API confusingly produces exactly nothing.) Recorder types for snapshot profiles expose a Snapshot method which writes the profile to an io.Writer, while recorder types for time-windowed profiles expose Start and Stop methods to record over a specific time window. We propose having one recorder type for each kind of profile and a separate recorder type for custom profiles. These recorder types will accept a bespoke set of configuration options in their constructors, centralizing the configuration to a single package, and localizing configuration options to a specific recorder instance.

This proposal is intended to supersede https://github.com/golang/go/issues/42502.

API

Given the core idea, the actual proposed API changes are relatively simple and straightforward, although the API surface is somewhat large. Below is a full listing of all the different recorders we propose. We omit the thread creation profile because it's not useful. (See #6104.)

package runtime/pprof

type CPURecorder struct { ... }

func CPURecorderConfig struct {
    // Period sets the duration between profile samples.
    //
    // If no value is set, the sample period for the duration is implementation-defined.
    // This implementation-defined value is independent of [runtime.SetCPUProfileRate],
    // but the rate set here may be visible to consumers of [StartCPUProfile], so callers
    // are discouraged from using both CPURecorder and StartCPUProfile in the same
    // program.
    Period time.Duration
}

func NewCPURecorder(CPURecorderConfig) (*CPURecorder, error)

// Start applies the recorder's configuration and begins global collection of
// CPU samples.
//
// Returns an error if this recorder had already been started.
func (*CPURecorder) Start(io.Writer) error

// Stop completes collection of the CPU profile.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*CPURecorder) Stop() error

type AllocRecorder struct { ... }

func AllocRecorderConfig struct {
    // BytesPerSample sets the maximum number of bytes allocated between samples.
    //
    // If no value is set, the sample period for the duration is implementation-defined.
    // This implementation-defined value is independent of [runtime.MemProfileRate],
    // but the rate set here may be visible to consumers of [Profile.WriteTo].
    BytesPerSample int64
}

func NewAllocRecorder(AllocRecorderConfig) *AllocRecorder

// Start applies the recorder's configuration and takes a snapshot of the profile.
//
// Returns an error if this recorder had already been started.
func (*AllocRecorder) Start(io.Writer) error

// Stop takes a second snapshot and computes the difference with the profile
// taken at start. The resulting profile is written to the io.Writer provided
// to Start. It contains all sampled allocations made in the window of time between
// Start and Stop, which is useful for identifying sources of high allocation volume.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*AllocRecorder) Stop() error

type HeapRecorder struct { ... }

func HeapRecorderConfig struct {
    // None for now.
}

func NewHeapRecorder(HeapRecorderConfig) *HeapRecorder

// Snapshot writes a sampled profile of the live heap to the provided io.Writer.
//
// The sampling rate is controlled by [runtime.MemProfileRate].
func (*HeapRecorder) Snapshot(io.Writer) (int, error)

// Start applies the recorder's configuration and takes a snapshot of the sampled live heap.
//
// Returns an error if this recorder had already been started.
func (*HeapRecorder) Start(io.Writer) error

// Stop takes a second snapshot and computes the difference with the profile
// taken at start. The resulting profile is written to the io.Writer provided
// to Start. This delta is useful for identifying memory leaks, since memory leaks
// will quickly rise out of the noise with a large positive delta over time.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*HeapRecorder) Stop() error

type BlockRecorder struct { ... }

func BlockRecorderConfig struct {
    // EventsPerSample sets the number of goroutine block events between samples.
    //
    // If no value is set, the sample period for the duration is implementation-defined.
    // This implementation-defined value is independent of [runtime.SetBlockProfileRate],
    // but the rate set here may be visible to consumers of [Profile.WriteTo].
    EventsPerSample int
}

func NewBlockRecorder(BlockRecorderConfig) (*BlockRecorder, error)

// Start applies the recorder's configuration and takes a snapshot of the profile.
//
// Returns an error if this recorder had already been started.
func (*BlockRecorder) Start(io.Writer) error

// Stop takes a second snapshot and computes the difference with the profile
// taken at start. The resulting profile is written to the io.Writer provided
// to Start.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*BlockRecorder) Stop() error

type MutexRecorder struct { ... }

func MutexRecorderConfig struct {
    // EventsPerSample sets the maximum number of unlock events between samples.
    //
    // If no value is set, the sample period for the duration is implementation-defined.
    // This implementation-defined value is independent of [runtime.SetMutexProfileRate],
    // but the rate set here may be visible to consumers of [Profile.WriteTo].
    EventsPerSample int
}

func NewMutexRecorder(MutexRecorderConfig) (*MutexRecorder, error)

// Start applies the recorder's configuration and takes a snapshot of the profile.
//
// Returns an error if this recorder had already been started.
func (*MutexRecorder) Start(io.Writer) error

// Stop takes a second snapshot and computes the difference with the profile
// taken at start. The resulting profile is written to the io.Writer provided
// to Start.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*MutexRecorder) Stop() error

type GoroutineRecorder struct { ... }

func GoroutineRecorderConfig struct {
    Format GoroutineProfileFormat
}

// GoroutineProfileFormat is an enumeration of available formats for writing
// out the goroutine profile.
type GoroutineProfileFormat int

const (
    PprofGoroutineProfile GoroutineProfileFormat = iota // Default gzipped protobuf.
    TextGoroutineProfile                                // Legacy text profile.
    TracebackGoroutineProfile                           // Matches default traceback format.
)

func NewGoroutineRecorder(MutexRecorderConfig) (*GoroutineRecorder, error)

// Snapshot snapshots the state of all goroutines, assembles it into a profile,
// and writes the result to the provided io.Writer.
func (*GoroutineRecorder) Snapshot(io.Writer) (int, error)

// ProfileRecorder is a generic profile recorder that works for any profile.
//
// It does not provide as much customizability as the more specific types,
// but works with any Profile, include custom Profiles.
type ProfileRecorder struct { ... }
func ProfileRecorderConfig struct {
    // None for now.
}

func NewProfileRecorder(*Profile, ProfileRecorderConfig) (*ProfileRecorder, error)

// Start applies the recorder's configuration and takes a snapshot of the profile.
//
// Returns an error if this recorder had already been started.
func (*ProfileRecorder) Start(io.Writer) error

// Stop takes a second snapshot and computes the difference with the profile
// taken at start. The resulting profile is written to the io.Writer provided
// to Start.
//
// Returns an error on any failure to write to the io.Writer provided to Start,
// or if the recorder had not been started.
func (*ProfileRecorder) Stop() error

// Snapshot emits all profile data collected since program start until this point.
func (*ProfileRecorder) Snapshot(io.Writer) (int, error)

Rationale

For the general structure of the API, with bespoke types representing some configuration, the rationale can be found in the lengthy discussion around the FlightRecorder proposal. In short, we want to support multiple consumers with different configurations (hence configurations are values) and we want to give room for configuration options to grow (hence the many recorder types we propose, instead of just one to rule them all). In some ways this proposal is just applying the insights and lessons from the FlightRecorder proposal to the runtime/pprof package.

What is new in this proposal is our focus on delta profiles. We have two reasons for this.

First, delta profiles compose much more cleanly. Long-term, we want to move toward being able to compose multiple profile consumers, but specifically by having configuration options specify a minimum requirement on how much detail is present in the profile. This is much harder to do for profile data that has been collected since program start. Although such profile data is useful, the API that already exists is about as good as we can do anyway.

Second, delta profiles match the models of CPU profiling and runtime execution traces far more closely. This means a more uniform API surface across all our diagnostics, leading to better discoverability of diagnostics, diagnostic configuration options, and a space to grow additional configuration options for each profile type.

As mentioned earlier, we still have to make some exceptions to delta profiles. Certain profiles represent a single instant in time, like goroutine profiles. The heap profile also represents an instant, specifically with the live heap part of the profile. (The heap profile and alloc profile are really the same thing, so this part can get a bit tricky. We choose to continue to have separate types for them, and when taking one of these profiles, the other one simply comes along for the ride.)

Composing consumers

Note that with some of the API above, each consumer may set its own desired sampling rate. For this to compose, the runtime must adjust its internal sampling to the maximum of all requested sampling rates.

This seems simple at first glance, but comes with a significant complication. Namely, the pprof format only supports a single global sampling rate so there's no way to indicate that some samples have different weights. The tooling also all makes the same assumption as a result.

Long-term, we believe the fix to this will be random downsampling to the desired rate. For this to work correctly we will need to change the implementation to track the rate that each sample was collected under, as currently all this data is aggregated away. With this information, randomly decimating samples in each sample rate group to the requested sampling rate should be sufficient. Implementing this will likely require some significant restructuring to the sampling bucket infrastructure in the runtime.

However, we need not block the new APIs on this work. For one, we take note that, by and large, diagnostics consumers do not adjust the sampling rate, or only do so infrequently. And it's already the case today that the sampling rate parameters should not (or simply cannot) change while profiling is active.

Therefore, I propose the following near-term compromise: if a consumer requests a sampling rate that is identical to the current sampling rate, new consumers are allowed to subscribe. Otherwise, Start returns a descriptive error explaining the issue and the current sampling rate. This compromise is already a step forward because it would allow multiple concurrent profiling consumers at all, and supports the common case without much additional effort required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    LibraryProposalIssues describing a requested change to the Go standard library or x/ libraries, but not to a toolProposal

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions