How to Reduce Elasticsearch Log Storage Consumption With FluentBit

Is your Elasticsearch storage consumption driving you up the wall?

Is your ILM (Index Lifecycle Management) policy running perfectly, your indices are all nicely rolled over, and yet somehow you still had to extend server storage by several GB this week?

Are the developers casually enabling DEBUG logging “just for a few hours” while your disk usage graph is shooting through the roof?

Is your log pipeline powered by Fluent Bit?

Before marching up to the dev team asking them to disable that “temporary” debug log before the server SSD explodes, there's one more thing you can try.

Disable Kubernetes Annotations in Fluent Bit

If you are collecting Kubernetes logs using the Fluent Bit Kubernetes filter, you may be storing a lot of unnecessary metadata inside Elasticsearch.

Kubernetes annotations often contain information that nobody realistically searches for in logs, such as:

• CI/CD "managed by" metadata
• Deployment tooling annotations
• Helm release details
• Random automation tags
• Internal tracking metadata

In most environments, engineers mainly search using:

• Namespace names
• Pod names
• Container names
• Hostnames
• Log messages themselves

That extra annotation data adds up fast across thousands or millions of log entries and can quietly eat away at your Elasticsearch storage.

To disable them you can implement the following line in your Fluent Bit config filter:

Annotations Off

This disables Kubernetes pod annotations from being attached to each log record before being sent to Elasticsearch.

This is usually where a surprising amount of unnecessary storage usage comes from.

A full snippet of the Filter section is as follows:

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed
    Annotations         Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

Trimming unnecessary metadata can noticeably reduce index sizes and slow down storage growth over time. And sometimes, saving even a few GB per day is enough to stop you from asking uncomfortable questions during the next review meeting.

#logging #fluentbit #elasticsearch