Fluentd

Fluentd and Fluent-bit plays an integral role in the logging module, in collection and processing of data from the cluster and its transfer to the elasticsearch.

These tools can be described in short as follows:

• Fluentd is a data collector for unified logging that can store collected data in various destinations. In our module, we configure fluentd to write the data to elasticsearch
• Fluent Bit is a multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations

Requirements

• Kubernetes >= 1.20.0
• Kustomize >= v3.3.X
• prometheus-operator

Prometheus Operator is necessary since we configure a ServiceMonitor to make some metrics available from fluentd on prometheus

Image repository and tag

• Fluentd image: registry.sighup.io/sighup/fluentd-elasticsearch:v1.14.2-debian-1.0
• Fluentd repo: Fluentd on Giithub
• Fluentd documentation: Fluentd Homepage
• Fluent-bit image: fluent/fluent-bit:1.8.10
• Fluent-bit repo: Fluent-bit on Github
• Fluent-bit documentation: Fluent-bit Homepage

Configuration

• Listens on port 24231
• Resource limits are 1000m for CPU and 400Mi for memory
• Configured to send log data to Elasticsearch
• Metrics are scraped every 30s by Prometheus
• Integrates with elasticsearch-single and elasticsearch-triple

Deployment

You can deploy fluentd by running the following command in the root of the project:

kustomize build | kubectl apply -f -