Redundant log reads with Loki source and structured metadata #3690
Description
What happened?
When using the Loki source with structured metadata in the log entries, the Loki log source reads log lines multiple times if the value of the structured metadata varies and multiple log entries are fetched with a single query. This results in a long time to (actually) process all log entries and consumes unnecessary resources.
This effect can be observed via the metrics command and the Lines read metric. In my case with differences in the structured metadata of every log entry, the number of new Lines read for n log entries is n*(n+1)/2, so e.g. for 100 lines I get 5.050 lines read, or for 1.000 lines I would get 500.500 lines read (haven't let it running that long yet though :D)
After adjust the log ingestion and removing the structured log data from the log entries, Crowdsec behaves as expected and processes every log entry only once.
What did you expect to happen?
No matter what structured metadata values are part of the Loki query response, the log lines should be read only once.
How can we reproduce it (as minimally and precisely as possible)?
I have prepared two shell scripts that ingest data into Loki that can be used to produce the behavior.
Scenario A (works as expected)
In this scenario everything works fine. It makes use of structured metadata (the third item in the log value), but the values do not vary (every trace_id has the same value).
Show script
#!/bin/bash
LOKI_URL="http://localhost:3100/loki/api/v1/push"
# Base timestamp, nano seconds will be added manually below
BASE_TS=$(date +%s)
# Scenario A
read -r -d '' PAYLOAD <<EOF
{
"streams": [
{
"stream": {
"some_label": "some_label_value"
},
"values": [
[ "${BASE_TS}000000001", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "1"} ],
[ "${BASE_TS}000000002", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "1"} ],
[ "${BASE_TS}000000003", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "1"} ]
]
}
]
}
EOF
curl -X POST -H "Content-Type: application/json" -d "${PAYLOAD}" "${LOKI_URL}"When ingesting these logs, you can observe this log entry from Crowdsec:
time="2025-06-19T16:07:46Z" level=debug msg="(timer:10s) 1 results / 3 entries result[0] (uri:http://loki:3100/loki/api/v1/query_range?direction=forward&end=1750349256436007389&limit=100&query=%some_label%3D%some_value%22%7D%0A&start=1750348773671562964)" component=lokiclient source="http://logging_loki:3100/" type=loki
The important bit here is the 1 results / 3 entries result[0], that will be different in Scenario B.
When downloading the Loki response via the provided URL, you get this payload:
Show response
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "1"
},
"values": [
[
"1750349252000000001",
"127.0.0.1 - - [Thu Jun 19 16:07:32 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
],
[
"1750349252000000002",
"127.0.0.1 - - [Thu Jun 19 16:07:32 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
],
[
"1750349252000000003",
"127.0.0.1 - - [Thu Jun 19 16:07:32 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 34294,
"linesProcessedPerSecond": 357,
"totalBytesProcessed": 288,
"totalLinesProcessed": 3,
"execTime": 0.008398,
"queueTime": 0.000129,
"subqueries": 0,
"totalEntriesReturned": 3,
"splits": 2,
"shards": 2,
"totalPostFilterLines": 3,
"totalStructuredMetadataBytesProcessed": 48
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 1,
"totalBatches": 3,
"totalLinesSent": 3,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 288,
"headChunkLines": 3,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 3,
"headChunkStructuredMetadataBytes": 48,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 978069,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}Scenario B (produces the described bug)
In this scenario, the trace_id has a different value for every log item and Crowdsec processes the 3 log items in this example as 6 lines (you can increase this quickly to large numbers by adding further log items).
Show script
#!/bin/bash
LOKI_URL="http://localhost:3100/loki/api/v1/push"
# Base timestamp, nano seconds will be added manually below
BASE_TS=$(date +%s)
# Base timestamp, nano seconds will be added manually below
BASE_TS=$(date +%s)
read -r -d '' PAYLOAD <<EOF
{
"streams": [
{
"stream": {
"some_label": "some_label_value"
},
"values": [
[ "${BASE_TS}000000001", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "1"} ],
[ "${BASE_TS}000000002", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "2"} ],
[ "${BASE_TS}000000003", "127.0.0.1 - - [$(date)] \"GET /index.html HTTP/1.1\" 200 1234", {"trace_id": "3"} ]
]
}
]
}
EOF
curl -X POST -H "Content-Type: application/json" -d "${PAYLOAD}" "${LOKI_URL}"In this case, you can observe the following log output:
time="2025-06-19T16:12:29Z" level=debug msg="(timer:10s) 3 results / 1 entries result[0] (uri:http://loki:3100/loki/api/v1/query_range?direction=forward&end=1750349539187198954&limit=100&query=%some_label%3D%some_value%22%7D%0A&start=1750349252000000004)" component=lokiclient source="http://loki:3100/" type=loki
time="2025-06-19T16:12:29Z" level=debug msg="(timer:100ms) 2 results / 1 entries result[0] (uri:http://loki:3100/loki/api/v1/query_range?direction=forward&end=1750349549185088928&limit=100&query=%some_label%3D%some_value%22%7D%0A&start=1750349537000000002)" component=lokiclient source="http://loki:3100/" type=loki
time="2025-06-19T16:12:29Z" level=debug msg="(timer:100ms) 1 results / 1 entries result[0] (uri:http://loki:3100/loki/api/v1/query_range?direction=forward&end=1750349549290321775&limit=100&query=%some_label%3D%some_value%22%7D%0A&start=1750349537000000003)" component=lokiclient source="http://loki:3100/" type=loki
As you can see here, it is sending 3 requests to Loki to fetch the log entries. The first request results in 3 results with 1 log entry, the second request in 2 results and the last request in 1 result.
Here you can also see the Loki query response of the three requests:
Show responses
[
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "1"
},
"values": [
[
"1750349537000000001",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
},
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "2"
},
"values": [
[
"1750349537000000002",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
},
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "3"
},
"values": [
[
"1750349537000000003",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 59997,
"linesProcessedPerSecond": 624,
"totalBytesProcessed": 288,
"totalLinesProcessed": 3,
"execTime": 0.0048,
"queueTime": 0.000073,
"subqueries": 0,
"totalEntriesReturned": 3,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 3,
"totalStructuredMetadataBytesProcessed": 48
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 3,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 288,
"headChunkLines": 3,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 3,
"headChunkStructuredMetadataBytes": 48,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 163359,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
},
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "2"
},
"values": [
[
"1750349537000000002",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
},
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "3"
},
"values": [
[
"1750349537000000003",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 58038,
"linesProcessedPerSecond": 604,
"totalBytesProcessed": 192,
"totalLinesProcessed": 2,
"execTime": 0.003308,
"queueTime": 0.000042,
"subqueries": 0,
"totalEntriesReturned": 2,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 2,
"totalStructuredMetadataBytesProcessed": 32
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 2,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 192,
"headChunkLines": 2,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 2,
"headChunkStructuredMetadataBytes": 32,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 124079,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
},
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"detected_level": "unknown",
"some_label": "some_value",
"trace_id": "3"
},
"values": [
[
"1750349537000000003",
"127.0.0.1 - - [Thu Jun 19 16:12:17 UTC 2025] \"GET /index.html HTTP/1.1\" 200 1234"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 16045,
"linesProcessedPerSecond": 167,
"totalBytesProcessed": 96,
"totalLinesProcessed": 1,
"execTime": 0.005983,
"queueTime": 0.000493,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 16
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 96,
"headChunkLines": 1,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 16,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 603541,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
]As you can see, the grouping of the log entries in the response is different and I think that's where the problem is coming from.
Anything else we need to know?
The exemplary log lines in my reproducer cannot be parsed by Crowdsec, because I simplified it for this ticket. But it does not matter if the log line is valid and can be parsed or not, the bug happens independent of the parsing.
Crowdsec version
$ cscli version
version: v1.6.8-f209766e
Codename: alphaga
BuildDate: 2025-03-25_15:56:53
GoVersion: 1.24.1
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.8-f209766e-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasou
rce_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlogOS version
I am running Crowdsec in Docker on Ubuntu.
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logoEnabled collections and parsers
$ cscli hub list -o raw
crowdsecurity/caddy-logs,enabled,1.1,Parse caddy logs,parsers
crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers
crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.3,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/sshd-logs,enabled,3.0,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios
crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.3,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios
crowdsecurity/CVE-2023-22518,enabled,0.3,Detect CVE-2023-22518 exploits,scenarios
crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios
crowdsecurity/CVE-2024-0012,enabled,0.1,Detect CVE-2024-0012 exploitation attempts,scenarios
crowdsecurity/CVE-2024-38475,enabled,0.1,Detect CVE-2024-38475 exploitation attempts,scenarios
crowdsecurity/CVE-2024-9474,enabled,0.1,Detect CVE-2024-9474 exploitation attempts,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.3,Apache - Path Traversal (CVE-2021-41773),scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.3,Apache - Path Traversal (CVE-2021-42013),scenarios
crowdsecurity/http-cve-probing,enabled,0.6,Detect generic HTTP cve probing,scenarios
crowdsecurity/http-generic-bf,enabled,0.9,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sap-interface-probing,enabled,0.1,Detect generic HTTP SAP interface probing,scenarios
crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios
crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/netgear_rce,enabled,0.4,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios
crowdsecurity/ssh-refused-conn,enabled,0.1,Detect sshd refused connections,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.3,detect w00tw00t,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/http_base,enabled,0.3,,contexts
crowdsecurity/base-http-scenarios,enabled,1.1,http common : scanners detection,collections
crowdsecurity/caddy,enabled,0.1,caddy support : parser and generic http scenarios,collections
crowdsecurity/http-cve,enabled,2.9,Detect CVE exploitation in http logs,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.6,sshd support : parser and brute-force detection,collectionsAcquisition config
```console
source: loki
log_level: debug
url: http://loki:3100/
query: |
{some_label="some_value"}
labels:
type: caddy
Config show
$ cscli config show
Global:
- Configuration Folder : /etc/crowdsec
- Data Folder : /var/lib/crowdsec/data
- Hub Folder : /etc/crowdsec/hub
- Simulation File : /etc/crowdsec/simulation.yaml
- Log Folder : /var/log
- Log level : info
- Log Media : stdout
Crowdsec:
- Acquisition File : /etc/crowdsec/acquis.yaml
- Parsers routines : 1
- Acquisition Folder : /etc/crowdsec/acquis.d
cscli:
- Output : human
- Hub Branch :
API Client:
- URL : http://0.0.0.0:8080/
- Login : localhost
- Credentials File : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
- Listen URL : 0.0.0.0:8080
- Listen Socket :
- Profile File : /etc/crowdsec/profiles.yaml
- Trusted IPs:
- 127.0.0.1
- ::1
- Database:
- Type : sqlite
- Path : /var/lib/crowdsec/data/crowdsec.db
- Flush age : 7d
- Flush size : 5000Prometheus metrics
$ cscli metrics
+--------------------------------------------------------------------------------------------------------------------------+
| Acquisition Metrics |
+--------------------------------+------------+--------------+----------------+------------------------+-------------------+
| Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | Lines whitelisted |
+--------------------------------+------------+--------------+----------------+------------------------+-------------------+
| loki:http://logging_loki:3100/ | 9 | - | 9 | - | - |
+--------------------------------+------------+--------------+----------------+------------------------+-------------------+
+----------------------------------------------+
| Local API Alerts |
+--------------------------------------+-------+
| Reason | Count |
+--------------------------------------+-------+
| crowdsecurity/http-crawl-non_statics | 40 |
+--------------------------------------+-------+
+------------------------------------------------------------------+
| Local API Decisions |
+--------------------------------------+----------+--------+-------+
| Reason | Origin | Action | Count |
+--------------------------------------+----------+--------+-------+
| http:bruteforce | CAPI | ban | 320 |
| http:crawl | CAPI | ban | 15000 |
| http:exploit | CAPI | ban | 13 |
| http:scan | CAPI | ban | 6359 |
| ssh:bruteforce | CAPI | ban | 2978 |
| crowdsecurity/http-crawl-non_statics | crowdsec | ban | 1 |
| generic:scan | CAPI | ban | 83 |
+--------------------------------------+----------+--------+-------+
+------------------------------------+
| Local API Metrics |
+--------------------+--------+------+
| Route | Method | Hits |
+--------------------+--------+------+
| /v1/heartbeat | GET | 23 |
| /v1/usage-metrics | POST | 1 |
| /v1/watchers/login | POST | 1 |
+--------------------+--------+------+
+-------------------------------------------+
| Local API Machines Metrics |
+-----------+---------------+--------+------+
| Machine | Route | Method | Hits |
+-----------+---------------+--------+------+
| localhost | /v1/heartbeat | GET | 23 |
+-----------+---------------+--------+------+
+-----------------------------------------------------+
| Parser Metrics |
+--------------------------+------+--------+----------+
| Parsers | Hits | Parsed | Unparsed |
+--------------------------+------+--------+----------+
| crowdsecurity/non-syslog | 9 | 9 | - |
+--------------------------+------+--------+----------+