Denial of service in go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
The grpc Unary Server Interceptor created by the otelgrpc package added the labels net.peer.sock.addr and net.peer.sock.port with unbounded cardinality. This can lead to the server's potential memory exhaustion when many malicious requests are sent. This leads to a denial-of-service.
02 / AFFECTED SOFTWARE
Affected packages
2333 explicit affected versions
03 / CONNECTIONS
Connected vulnerabilities
04 / EVIDENCE
Source records
### Summary The grpc Unary Server Interceptor [opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/9d4eb7e7706038b07d33f83f76afbe13f53d171d/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L327) ``` // UnaryServerInterceptor returns a grpc.UnaryServerInterceptor suitable // for use in a grpc.NewServer call. func UnaryServerInterceptor(opts ...Option) grpc.UnaryServerInterceptor { ``` out of the box adds labels - `net.peer.sock.addr` - `net.peer.sock.port` that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent. ### Details An attacker can easily flood the peer address and port for requests. ### PoC Apply the attached patch to the example and run the client multiple times. Observe how each request will create a unique histogram and how the memory consumption increases during it. ### Impact In order to be affected, the program has to configure a metrics pipeline, use [UnaryServerInterceptor](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/9d4eb7e7706038b07d33f83f76afbe13f53d171d/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L327), and does not filter any client IP address and ports via middleware or proxies, etc. ### Others It is similar to already reported vulnerabilities. * [GHSA-rcjv-mgp8-qvmr](https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-rcjv-mgp8-qvmr) ([open-telemetry/opentelemetry-go-contrib](https://github.com/open-telemetry/opentelemetry-go-contrib)) - [GHSA-5r5m-65gx-7vrh](https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh "GHSA-5r5m-65gx-7vrh") ([open-telemetry/opentelemetry-go-contrib](https://github.com/open-telemetry/opentelemetry-go-contrib)) - [GHSA-cg3q-j54f-5p7p](https://github.com/advisories/GHSA-cg3q-j54f-5p7p "GHSA-cg3q-j54f-5p7p") ([prometheus/client_golang](https://github.com/prometheus/client_golang)) ### Workaround for affected versions As a workaround to stop being affected, a view removing the attributes can be used. The other possibility is to disable grpc metrics instrumentation by passing [`otelgrpc.WithMeterProvider`](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/instrumentation/google.golang.org/grpc/otelgrpc/v0.45.0/instrumentation/google.golang.org/grpc/otelgrpc/config.go#L138) option with [`noop.NewMeterProvider`](https://pkg.go.dev/go.opentelemetry.io/otel/metric/noop#NewMeterProvider). ### Solution provided by upgrading In PR [#4322](https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4322), to be released with v0.46.0, the attributes were removed. ### References - [#4322](https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4322)
The grpc Unary Server Interceptor created by the otelgrpc package added the labels net.peer.sock.addr and net.peer.sock.port with unbounded cardinality. This can lead to the server's potential memory exhaustion when many malicious requests are sent. This leads to a denial-of-service.
OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. Starting in version 0.37.0 and prior to version 0.46.0, the grpc Unary Server Interceptor out of the box adds labels `net.peer.sock.addr` and `net.peer.sock.port` that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent. An attacker can easily flood the peer address and port for requests. Version 0.46.0 contains a fix for this issue. As a workaround to stop being affected, a view removing the attributes can be used. The other possibility is to disable grpc metrics instrumentation by passing `otelgrpc.WithMeterProvider` option with `noop.NewMeterProvider`.
05 / REFERENCES
Further evidence
- https://github.com/open-telemetry/opentelemetry-go-contrib
- https://github.com/open-telemetry/opentelemetry-go-contrib/blob/9d4eb7e7706038b07d33f83f76afbe13f53d171d/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L327
- https://github.com/open-telemetry/opentelemetry-go-contrib/blob/instrumentation/google.golang.org/grpc/otelgrpc/v0.45.0/instrumentation/google.golang.org/grpc/otelgrpc/config.go#L138
- https://github.com/open-telemetry/opentelemetry-go-contrib/commit/04c5dcbb5b35f14b4e6793b245919c72addbc7d0
- https://github.com/open-telemetry/opentelemetry-go-contrib/commit/b44dfc9092b157625a5815cb437583cee663333b
- https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4322
- https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-8pgv-569h-w5rw
- https://nvd.nist.gov/vuln/detail/CVE-2023-47108
- https://pkg.go.dev/go.opentelemetry.io/otel/metric/noop#NewMeterProvider
- https://github.com/CVEProject/cvelistV5/tree/main/cves/2023/47xxx/CVE-2023-47108.json