Crash when querying for new decisions #3704
Description
What happened?
Bouncer is unable to query new decisions.
I have captured the following logs from the LAPI pod (there are many of these dumps):
crowdsec-lapi-bddc95c64-sqsq4:/var/lib/crowdsec/data/trace# cat crowdsec-crash.196145974.txt
error: runtime error: invalid memory address or nil pointer dereference
version: v1.6.5-72b4354b
BuildDate: 2025-02-10_14:52:20
GoVersion: 1.23.6
Platform: docker
goroutine 814489169 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:26 +0x5e
github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).writeStackTrace(0x37c4000, {0x1f6b540, 0x37a13f0})
github.com/crowdsecurity/go-cs-lib@v0.0.16/trace/trace.go:152 +0x16e
github.com/crowdsecurity/go-cs-lib/trace.WriteStackTrace(...)
github.com/crowdsecurity/go-cs-lib@v0.0.16/trace/trace.go:42
github.com/crowdsecurity/crowdsec/pkg/apiserver.recoverFromPanic(0xc000baa100)
github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:100 +0x17c
panic({0x1f6b540?, 0x37a13f0?})
runtime/panic.go:785 +0x132
github.com/crowdsecurity/crowdsec/pkg/database.(*Client).QueryNewDecisionsSinceWithFilters(0xc0000418c0, {0x26ea008, 0xc0008fc500}, 0x0, 0xc00538a840)
github.com/crowdsecurity/crowdsec/pkg/database/decisions.go:314 +0x897
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1.(*Controller).StreamDecisionNonChunked(0xc000c9de00, 0xc000baa100, 0xc004c7efc0, {0x6?, 0x3?, 0x227749f?}, 0xc00538a840)
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1/decisions.go:364 +0x612
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1.(*Controller).StreamDecision(0xc000c9de00, 0xc000baa100)
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1/decisions.go:422 +0x23c
github.com/gin-gonic/gin.(*Context).Next(0xc000baa100)
github.com/gin-gonic/gin@v1.9.1/context.go:174 +0x2b
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers.(*Controller).NewV1.PrometheusBouncersMiddleware.func7(0xc000baa100)
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1/metrics.go:111 +0x191
github.com/gin-gonic/gin.(*Context).Next(0xc000baa100)
github.com/gin-gonic/gin@v1.9.1/context.go:174 +0x2b
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers.(*Controller).NewV1.PrometheusMiddleware.func4(0xc000baa100)
github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1/metrics.go:123 +0x12d
github.com/gin-gonic/gin.(*Context).Next(0xc000baa100)
github.com/gin-gonic/gin@v1.9.1/context.go:174 +0x2b
github.com/crowdsecurity/crowdsec/pkg/apiserver.NewServer.CustomRecoveryWithWriter.func4(0xc004c63f80?)
github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:114 +0x3f
github.com/gin-gonic/gin.(*Context).Next(...)
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000baa100)
github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xdd
github.com/gin-gonic/gin.(*Context).Next(...)
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0002fed00, 0xc000baa100)
github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x64e
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0002fed00, {0x26e4ab8, 0xc0001fca80}, 0xc0009f0a00)
github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1b2
net/http.serverHandler.ServeHTTP({0xc00538a420?}, {0x26e4ab8?, 0xc0001fca80?}, 0x6?)
net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc000bb65a0, {0x26e9fd0, 0xc000e50f90})
net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 218
net/http/server.go:3360 +0x485
From the logs LAPI pod logs:
time="2025-06-26T08:19:08Z" level=warning msg="stacktrace written to /var/lib/crowdsec/data/trace/crowdsec-crash.2363993725.txt, please join to your issue" │
│ time="2025-06-26T08:19:08Z" level=info msg="10.32.0.165 - [Thu, 26 Jun 2025 08:19:08 UTC] \"GET /v1/decisions/stream?startup=false HTTP/1.1 500 10.062071904s \"Crowdsec-Bouncer-Traefik-Plugin/1.X.X\" \"" │
│ time="2025-06-26T08:19:08Z" level=warning msg="CreateMetric: context canceled" │
│ time="2025-06-26T08:19:08Z" level=error msg="storing metrics snapshot for 'crodsecBouncer@10.32.0.141@10.32.0.134@10.32.0.139@10.32.0.135@10.32.0.161@10.32.0.154@10.32.0.160@10.32.0.162@10.32.0.150@10.32.0 │
│ time="2025-06-26T08:19:08Z" level=info msg="10.32.0.165 - [Thu, 26 Jun 2025 08:19:08 UTC] \"POST /v1/usage-metrics HTTP/1.1 500 9.993893801s \"Crowdsec-Bouncer-Traefik-Plugin/1.X.X\" \""
What did you expect to happen?
No crash.
How can we reproduce it (as minimally and precisely as possible)?
Needs further investigation.
Anything else we need to know?
No response
Crowdsec version
$ cscli version
version: v1.6.5-72b4354b
Codename: alphaga
BuildDate: 2025-02-10_14:57:34
GoVersion: 1.23.6
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.5-72b4354b-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlogOS version
# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.21.2
PRETTY_NAME="Alpine Linux v3.21"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
$ uname -a
Linux crowdsec-lapi-bddc95c64-sqsq4 5.15.0-1090-azure #99-Ubuntu SMP Thu May 22 21:15:50 UTC 2025 x86_64 LinuxEnabled collections and parsers
$ cscli hub list -o rawAcquisition config
```console
# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
# paste output here
On Windows:
C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
paste output here
Config show
$ cscli config show
# paste output herePrometheus metrics
$ cscli metrics
# paste output here