How FourKites runs a scalable and cost-effective log analytics solution to handle petabytes of logs

About this Video

• Video Title: How FourKites runs a scalable and cost-effective log analytics solution to handle petabytes of logs
• Channel: Trino
• Speakers: One unnamed speaker
• Duration: 00:28:19

Introduction

This video details FourKites' cost-effective and scalable log analytics solution. The speaker explains their approach to managing petabytes of logs generated by their 120+ microservices, highlighting the use of Trino as a core component alongside other technologies like Fluent Bit, AWS Lambda, S3, and Superset. The presentation covers data ingestion, transformation, storage, querying, security, and optimization strategies.

Key Takeaways

• Scalable Log Analytics Solution: FourKites built a log analytics solution to handle 6-8 terabytes of logs daily, using Trino as a distributed query engine.
• Cost-Effectiveness: The solution is significantly more cost-effective than third-party alternatives.
• Data Ingestion and Transformation: Fluent Bit ingests logs, which are then transformed by Lambda functions into a standardized format stored in S3.
• Data Storage and Querying: S3 serves as the storage layer, with Trino enabling efficient querying of log data. Superset provides a user interface for dashboards and alerts.
• Security: Row-level security is implemented using a custom column in the transformed data, allowing granular access control for partners.
• Optimizations: Compaction jobs improve query performance on historical data, while timeout settings and the creation of log subsets enhance overall efficiency. Lambda function optimization, data store optimization (reducing raw log retention), and UI optimization (using asynchronous mode in Superset) further enhance performance and reduce costs.