Collect & forward logs from self-managed MongoDB to DigitalOcean's managed OpenSearch using Fluent Bit

Introduction

In this tutorial, you will study really to nonstop logs from Fluent Bit installed connected a self-managed MongoDB moving connected an Ubuntu Droplet to DigitalOcean’s managed OpenSearch database.

OpenSearch is an open-source hunt and analytics suite that originated arsenic a fork of Elasticsearch and Kibana. It supports real-time information ingestion, precocious querying, and robust security, making it perfect for e-commerce, IT monitoring, and finance applications. With features for illustration SQL query support, instrumentality learning, and alerting, OpenSearch continuously evolves done progressive organization development.

Fluent Bit is simply a lightweight, open-source log processor and forwarder that collects, processes, and ships log data. Optimized for performance, it’s perfect for resource-constrained environments for illustration containers and separator computing. Fluent Bit efficiently aggregates and forwards logs to various destinations, making it celebrated for real-time logging and monitoring.

Prerequisites

Before you begin, guarantee you have:

• A DigitalOcean Cloud account pinch entree to Managed OpenSearch.
• Basic knowledge of OpenSearch and Fluent Bit.
• Access to the OpenSearch Dashboard.

Installing Fluent Bit

There are aggregate platforms wherever Fluent Bit tin beryllium installed. You will beryllium utilizing an Ubuntu Platform here. Run the pursuing bid connected your Droplet’s Terminal.

You tin entree the official documentation for much information.

Configuring FluentBit

The Fluent Bit config record is stored astatine /etc/fluent-bit/fluent-bit.conf by default. You’ll alteration the fluent-bit.conf to nonstop the logs to OpenSearch.

FluentBit Inputs

Fluent Bit offers a assortment of input plugins that alteration it to cod log and arena information from different sources. Since you will beryllium sending log from logs files, you will beryllium utilizing tail Input plugin.

Update the fluent-bit.conf record arsenic follows:

You tin publication much astir Input plugins connected Fluent Bit’s official Manual.

FluentBit Outputs

Similar to input plugins, Fluent Bit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to Opensearch, let’s make usage of the Opensearch Output plugin.

You tin publication much astir Output plugins connected Fluent Bit’s official Manual.

Once the configurations are group up, commencement Fluent Bit Service.

Access MongoDB logs successful OpenSearch Dashboard

• Navigate to the OpenSearch Dashboard connected your DigitalOcean Cloud account.
• Using the hamburger icon, click connected Dashboard Management nether Management section.
• Click Index patterns and click connected Create Index pattern.
• Give the scale shape the aforesaid sanction utilized successful the fluent-bit.conf file.
• Once the scale shape is ready, you tin entree the MongoDB logs from the Discover tab.

Conclusion

Congratulations! You’ve successfully learned to usage Fluent Bit to cod and guardant self-hosted MongoDB logs to a DigitalOcean Managed OpenSearch database. We covered configuring Fluent Bit and creating an scale shape successful OpenSearch to efficiently show and analyse your MongoDB logs.

Featured connected Community