高吞吐、低成本日志系统方案ClickHouse + Filebeat/Fluentd

下面给你一套高吞吐、低成本日志系统方案ClickHouse Filebeat/Fluentd的最简可落地架构含组件分工、部署要点、建表示例和配置模板。一、整体架构极简版应用日志文件/容器日志 ↓ Filebeat / Fluent Bit采集、轻量过滤 ↓ Kafka可选削峰、解耦、重放 ↓ ClickHouse列式存储、高压缩、高吞吐写入/查询 ↓ Grafana / 自建后台可视化、检索、告警核心优势高吞吐ClickHouse 批量写入、向量化执行写入 TPS 远高于 ES低成本列式 高压缩通常 5–15 倍磁盘 / 内存成本远低于 ES适合日志时间序列、结构化 / 半结构化、聚合查询友好二、组件选型与分工1. 采集层Filebeat vs Fluent BitFilebeat优点Elastic 生态成熟、配置简单、对日志文件 / 多行支持好适合传统服务、文件日志、Java 应用Fluent Bit优点更轻量、性能更好、K8s / 容器原生、插件丰富适合K8s、云原生、高密部署推荐非 K8s 用FilebeatK8s 用Fluent Bit。2. 缓冲层Kafka强烈建议作用削峰、解耦、保证不丢、可重放、多下游复用不建议直接从采集器写 ClickHouse突发流量易写失败、无重试队列3. 存储层ClickHouse推荐引擎MergeTree家族MergeTree/ReplacingMergeTree/SummingMergeTree分区按天 / 小时分区日志典型索引时间戳 服务名 级别 traceId 等常用维度压缩默认LZ4即可高压缩用ZSTD三、ClickHouse 建表示例通用日志表CREATE TABLE IF NOT EXISTS logs ( timestamp DateTime64(3) CODEC(DoubleDelta, LZ4), service String CODEC(ZSTD), level String CODEC(ZSTD), trace_id String CODEC(ZSTD), span_id String CODEC(ZSTD), host String CODEC(ZSTD), path String CODEC(ZSTD), message String CODEC(ZSTD), json_fields Map(String, String) CODEC(ZSTD) ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (service, level, toStartOfHour(timestamp), timestamp) TTL timestamp INTERVAL 30 DAY DELETE; -- 自动删 30 天日志按需改要点DateTime64(3)毫秒级时间CODEC显式指定压缩提升压缩比PARTITION BY toDate(timestamp)按天分区删除 / 查询更快TTL自动过期降低存储成本四、Filebeat → Kafka 配置示例filebeat.ymlfilebeat.inputs: - type: log enabled: true paths: - /var/log/myapp/*.log multiline.type: pattern multiline.pattern: ^\[ multiline.negate: true multiline.match: after fields: service: my-java-app fields_under_root: true output.kafka: hosts: [kafka-1:9092, kafka-2:9092] topic: logs-myapp partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 1000000 processors: - add_host_metadata: ~ - add_cloud_metadata: ~Webfunny 全链路监控埋点平台 是一站式前端监控 用户行为埋点 大数据分析平台天然适配点位细查、用户行为回溯、批量导出等场景一体化架构监控 埋点同一套 SDK数据互通无壁垒私有化部署数据完全本地化满足企业合规要求高吞吐支撑基于 ClickHouse 构建亿级日志秒级查询全端覆盖H5 / 小程序 / APP / 鸿蒙全覆盖统一导出口径可定制强支持接口扩展、分布式锁、限流降级等企业级能力五、Fluent Bit → Kafka 配置示例fluent-bit.conf[SERVICE] Flush 1 Log_Level info Daemon off [INPUT] Name tail Path /var/log/myapp/*.log Parser json Tag logs.myapp Multiline On Multiline_Format regex Multiline_Pattern ^\[ DB /var/lib/fluent-bit/pos.db [OUTPUT] Name kafka Match logs.* Brokers kafka-1:9092,kafka-2:9092 Topics logs-myapp Compression gzip六、Kafka → ClickHouse 写入方案方案 AClickHouse Kafka Engine推荐最简单在 CH 建一张 Kafka 消费表 物化视图写入日志表。1建 Kafka 引擎表消费 topicCREATE TABLE logs_kafka ( raw String ) ENGINE Kafka SETTINGS kafka_broker_list kafka-1:9092,kafka-2:9092, kafka_topic_list logs-myapp, kafka_group_name clickhouse-log-group, kafka_format JSONAsString, kafka_skip_broken_messages 1;2建物化视图解析 JSON 写入logs表CREATE MATERIALIZED VIEW logs_mv TO logs AS SELECT toDateTime64(JSONExtractString(raw, timestamp), 3) AS timestamp, JSONExtractString(raw, service) AS service, JSONExtractString(raw, level) AS level, JSONExtractString(raw, trace_id) AS trace_id, JSONExtractString(raw, span_id) AS span_id, JSONExtractString(raw, host) AS host, JSONExtractString(raw, path) AS path, JSONExtractString(raw, message) AS message, JSONExtract(raw, json_fields, Map(String, String)) AS json_fields FROM logs_kafka;优点无额外组件CH 自己消费 Kafka运维简单。方案 B使用 Vector / Logstash更灵活适合需要复杂 ETL、多 sink、路由、 enrichment 场景流程Kafka → Vector/Logstash解析 / 清洗→ ClickHouse七、关键优化高吞吐 低成本1. ClickHouse 写入优化批量写入Kafka CH Kafka Engine 天然批量max_insert_block_size 1048576async_insert 1异步插入提升吞吐注意数据安全内存表 / Buffer 表做写入缓冲可选2. 存储与压缩优先ZSTD压缩比 LZ4 压缩比更高CPU 开销可接受按时间分区TTL 自动清理避免无限膨胀冷数据可使用 S3 分层存储CH 支持 S3 磁盘3. 采集端优化采集端开启gzip压缩多行日志正确配置避免乱序 / 截断采集器资源限制Filebeat/Fluent Bit 都很轻量八、典型查询示例-- 近 1 小时错误日志 SELECT * FROM logs WHERE level ERROR AND timestamp now() - INTERVAL 1 HOUR ORDER BY timestamp DESC LIMIT 100; -- 按服务统计 QPS/错误数 SELECT toStartOfMinute(timestamp) AS minute, service, count() AS total, countIf(level ERROR) AS errors FROM logs WHERE timestamp now() - INTERVAL 1 HOUR GROUP BY minute, service ORDER BY minute, service;九、与 ELK 对比简要表格维度ELK StackClickHouse Filebeat/Fluentd写入吞吐中高更高批量 向量化存储成本较高倒排索引低列式 高压缩全文检索强一般n-gram/Token 可做聚合 / 统计一般极强运维复杂度中ES 分片 / 集群敏感较低CH 简单无分片烦恼适合场景全文检索、日志 安全分析高吞吐、低成本、统计聚合为主十、落地建议如果你是Java 后端常见场景应用Spring Boot Logback/Log4j2 输出 JSON 日志采集Filebeat采集日志文件 → Kafka存储ClickHouse用 Kafka Engine 消费可视化Grafana ClickHouse 数据源做面板 / 检索