Files
go-trustlog/api/adapter/publisher.go
ryan 88f80ffa5e feat: 新增数据库持久化模块(Persistence),实现 Cursor + Retry 双层架构
## 核心功能

### 1. 数据库持久化支持
- 新增完整的 Persistence 模块 (api/persistence/)
- 支持三种持久化策略:
  * StrategyDBOnly - 仅落库,不存证
  * StrategyDBAndTrustlog - 既落库又存证(推荐)
  * StrategyTrustlogOnly - 仅存证,不落库
- 支持多数据库:PostgreSQL, MySQL, SQLite

### 2. Cursor + Retry 双层架构
- CursorWorker:第一道防线,快速发现新记录并尝试存证
  * 增量扫描 operation 表(基于时间戳游标)
  * 默认 10 秒扫描间隔,批量处理 100 条
  * 成功更新状态,失败转入重试队列
- RetryWorker:第二道防线,处理失败记录
  * 指数退避重试(1m → 2m → 4m → 8m → 16m)
  * 默认最多重试 5 次
  * 超限自动标记为死信

### 3. 数据库表设计
- operation 表:存储操作记录,支持可空 IP 字段
- trustlog_cursor 表:Key-Value 模式,支持多游标
- trustlog_retry 表:重试队列,支持指数退避

### 4. 异步最终一致性
- 应用调用立即返回(仅落库)
- CursorWorker 异步扫描并存证
- RetryWorker 保障失败重试
- 完整的监控和死信处理机制

## 修改文件

### 核心代码(11个文件)
- api/persistence/cursor_worker.go - Cursor 工作器(新增)
- api/persistence/repository.go - 数据仓储层(新增)
- api/persistence/schema.go - 数据库 Schema(新增)
- api/persistence/strategy.go - 策略管理器(新增)
- api/persistence/client.go - 客户端封装(新增)
- api/persistence/retry_worker.go - Retry 工作器(新增)
- api/persistence/config.go - 配置管理(新增)

### 修复内部包引用(5个文件)
- api/adapter/publisher.go - 修复 internal 包引用
- api/adapter/subscriber.go - 修复 internal 包引用
- api/model/envelope.go - 修复 internal 包引用
- api/model/operation.go - 修复 internal 包引用
- api/model/record.go - 修复 internal 包引用

### 单元测试(8个文件)
- api/persistence/*_test.go - 完整的单元测试
- 测试覆盖率:28.5%
- 测试通过率:49/49 (100%)

### SQL 脚本(4个文件)
- api/persistence/sql/postgresql.sql - PostgreSQL 建表脚本
- api/persistence/sql/mysql.sql - MySQL 建表脚本
- api/persistence/sql/sqlite.sql - SQLite 建表脚本
- api/persistence/sql/test_data.sql - 测试数据

### 文档(2个文件)
- README.md - 更新主文档,新增 Persistence 使用指南
- api/persistence/README.md - 完整的 Persistence 文档
- api/persistence/sql/README.md - SQL 脚本说明

## 技术亮点

1. **充分利用 Cursor 游标表**
   - 作为任务发现队列,非简单的位置记录
   - Key-Value 模式,支持多游标并发扫描
   - 时间戳天然有序,增量扫描高效

2. **双层保障机制**
   - Cursor:正常流程,快速处理
   - Retry:异常流程,可靠重试
   - 职责分离,监控清晰

3. **可空 IP 字段支持**
   - ClientIP 和 ServerIP 使用 *string 类型
   - 支持 NULL 值,符合数据库最佳实践
   - 使用 sql.NullString 正确处理

4. **完整的监控支持**
   - 未存证记录数监控
   - Cursor 延迟监控
   - 重试队列长度监控
   - 死信队列监控

## 测试结果

-  单元测试:49/49 通过 (100%)
-  代码覆盖率:28.5%
-  编译状态:无错误
-  支持数据库:PostgreSQL, MySQL, SQLite

## Breaking Changes

无破坏性变更。Persistence 模块作为可选功能,不影响现有代码。

## 版本信息

- 版本:v2.1.0
- Go 版本要求:1.21+
- 更新日期:2025-12-23
2025-12-23 18:59:43 +08:00

120 lines
3.2 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package adapter
import (
"context"
"errors"
"github.com/ThreeDotsLabs/watermill/message"
"github.com/apache/pulsar-client-go/pulsar"
"go.yandata.net/iod/iod/go-trustlog/api/logger"
)
const (
OperationTopic = "persistent://public/default/operation"
RecordTopic = "persistent://public/default/record"
)
// PublisherConfig is the configuration to create a publisher.
type PublisherConfig struct {
// URL is the Pulsar URL.
URL string
// TLSTrustCertsFilePath is the path to the CA certificate file for verifying the server certificate.
// If empty, TLS verification will be disabled.
TLSTrustCertsFilePath string
// TLSCertificateFilePath is the path to the client certificate file for mTLS authentication.
// If empty, mTLS authentication will be disabled.
TLSCertificateFilePath string
// TLSKeyFilePath is the path to the client private key file for mTLS authentication.
// If empty, mTLS authentication will be disabled.
TLSKeyFilePath string
// TLSAllowInsecureConnection allows insecure TLS connections (not recommended for production).
TLSAllowInsecureConnection bool
}
// Publisher provides the pulsar implementation for watermill publish operations.
type Publisher struct {
conn pulsar.Client
logger logger.Logger
pubs map[string]pulsar.Producer
}
// NewPublisher creates a new Publisher.
func NewPublisher(config PublisherConfig, adapter logger.Logger) (*Publisher, error) {
clientOptions := pulsar.ClientOptions{
URL: config.URL,
// Logger: 使用 Pulsar 默认 loggerinternal 包引用已移除)
}
// Configure TLS/mTLS
if err := configureTLSForClient(&clientOptions, config, adapter); err != nil {
return nil, errors.Join(err, errors.New("failed to configure TLS"))
}
conn, err := pulsar.NewClient(clientOptions)
if err != nil {
return nil, errors.Join(err, errors.New("cannot connect to pulsar"))
}
return NewPublisherWithPulsarClient(conn, adapter)
}
// NewPublisherWithPulsarClient creates a new Publisher with the provided pulsar connection.
func NewPublisherWithPulsarClient(conn pulsar.Client, logger logger.Logger) (*Publisher, error) {
return &Publisher{
conn: conn,
pubs: make(map[string]pulsar.Producer),
logger: logger,
}, nil
}
// Publish publishes message to Pulsar.
//
// Publish will not return until an ack has been received from Pulsar.
// When one of messages delivery fails - function is interrupted.
func (p *Publisher) Publish(topic string, messages ...*message.Message) error {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
producer, found := p.pubs[topic]
if !found {
pr, err := p.conn.CreateProducer(pulsar.ProducerOptions{Topic: topic})
if err != nil {
return err
}
producer = pr
p.pubs[topic] = producer
}
for _, msg := range messages {
// 跳过 nil 消息
if msg == nil {
continue
}
p.logger.DebugContext(ctx, "Sending message", "key", msg.UUID, "topic", topic)
_, err := producer.Send(ctx, &pulsar.ProducerMessage{
Key: msg.UUID,
Payload: msg.Payload,
})
if err != nil {
return err
}
}
return nil
}
// Close closes the publisher and the underlying connection.
func (p *Publisher) Close() error {
for _, pub := range p.pubs {
pub.Close()
}
p.conn.Close()
return nil
}