Files
go-trustlog/api/persistence/unit_test.go
ryan 88f80ffa5e feat: 新增数据库持久化模块(Persistence),实现 Cursor + Retry 双层架构
## 核心功能

### 1. 数据库持久化支持
- 新增完整的 Persistence 模块 (api/persistence/)
- 支持三种持久化策略:
  * StrategyDBOnly - 仅落库,不存证
  * StrategyDBAndTrustlog - 既落库又存证(推荐)
  * StrategyTrustlogOnly - 仅存证,不落库
- 支持多数据库:PostgreSQL, MySQL, SQLite

### 2. Cursor + Retry 双层架构
- CursorWorker:第一道防线,快速发现新记录并尝试存证
  * 增量扫描 operation 表(基于时间戳游标)
  * 默认 10 秒扫描间隔,批量处理 100 条
  * 成功更新状态,失败转入重试队列
- RetryWorker:第二道防线,处理失败记录
  * 指数退避重试(1m → 2m → 4m → 8m → 16m)
  * 默认最多重试 5 次
  * 超限自动标记为死信

### 3. 数据库表设计
- operation 表:存储操作记录,支持可空 IP 字段
- trustlog_cursor 表:Key-Value 模式,支持多游标
- trustlog_retry 表:重试队列,支持指数退避

### 4. 异步最终一致性
- 应用调用立即返回(仅落库)
- CursorWorker 异步扫描并存证
- RetryWorker 保障失败重试
- 完整的监控和死信处理机制

## 修改文件

### 核心代码(11个文件)
- api/persistence/cursor_worker.go - Cursor 工作器(新增)
- api/persistence/repository.go - 数据仓储层(新增)
- api/persistence/schema.go - 数据库 Schema(新增)
- api/persistence/strategy.go - 策略管理器(新增)
- api/persistence/client.go - 客户端封装(新增)
- api/persistence/retry_worker.go - Retry 工作器(新增)
- api/persistence/config.go - 配置管理(新增)

### 修复内部包引用(5个文件)
- api/adapter/publisher.go - 修复 internal 包引用
- api/adapter/subscriber.go - 修复 internal 包引用
- api/model/envelope.go - 修复 internal 包引用
- api/model/operation.go - 修复 internal 包引用
- api/model/record.go - 修复 internal 包引用

### 单元测试(8个文件)
- api/persistence/*_test.go - 完整的单元测试
- 测试覆盖率:28.5%
- 测试通过率:49/49 (100%)

### SQL 脚本(4个文件)
- api/persistence/sql/postgresql.sql - PostgreSQL 建表脚本
- api/persistence/sql/mysql.sql - MySQL 建表脚本
- api/persistence/sql/sqlite.sql - SQLite 建表脚本
- api/persistence/sql/test_data.sql - 测试数据

### 文档(2个文件)
- README.md - 更新主文档,新增 Persistence 使用指南
- api/persistence/README.md - 完整的 Persistence 文档
- api/persistence/sql/README.md - SQL 脚本说明

## 技术亮点

1. **充分利用 Cursor 游标表**
   - 作为任务发现队列,非简单的位置记录
   - Key-Value 模式,支持多游标并发扫描
   - 时间戳天然有序,增量扫描高效

2. **双层保障机制**
   - Cursor:正常流程,快速处理
   - Retry:异常流程,可靠重试
   - 职责分离,监控清晰

3. **可空 IP 字段支持**
   - ClientIP 和 ServerIP 使用 *string 类型
   - 支持 NULL 值,符合数据库最佳实践
   - 使用 sql.NullString 正确处理

4. **完整的监控支持**
   - 未存证记录数监控
   - Cursor 延迟监控
   - 重试队列长度监控
   - 死信队列监控

## 测试结果

-  单元测试:49/49 通过 (100%)
-  代码覆盖率:28.5%
-  编译状态:无错误
-  支持数据库:PostgreSQL, MySQL, SQLite

## Breaking Changes

无破坏性变更。Persistence 模块作为可选功能,不影响现有代码。

## 版本信息

- 版本:v2.1.0
- Go 版本要求:1.21+
- 更新日期:2025-12-23
2025-12-23 18:59:43 +08:00

363 lines
8.9 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// +build unit
package persistence
import (
"context"
"database/sql"
"testing"
"time"
_ "github.com/mattn/go-sqlite3"
)
// 这个文件包含不依赖外部复杂模块的纯单元测试
// TestDBConfigCreation 测试数据库配置创建
func TestDBConfigCreation(t *testing.T) {
t.Run("default config", func(t *testing.T) {
cfg := DefaultDBConfig("postgres", "test-dsn")
assertEqual(t, cfg.DriverName, "postgres")
assertEqual(t, cfg.DSN, "test-dsn")
assertEqual(t, cfg.MaxOpenConns, 25)
assertEqual(t, cfg.MaxIdleConns, 5)
assertEqual(t, cfg.ConnMaxLifetime, time.Hour)
assertEqual(t, cfg.ConnMaxIdleTime, 10*time.Minute)
})
t.Run("custom config", func(t *testing.T) {
cfg := DBConfig{
DriverName: "mysql",
DSN: "user:pass@/db",
MaxOpenConns: 50,
MaxIdleConns: 10,
ConnMaxLifetime: 2 * time.Hour,
ConnMaxIdleTime: 20 * time.Minute,
}
assertEqual(t, cfg.DriverName, "mysql")
assertEqual(t, cfg.MaxOpenConns, 50)
})
}
// TestTrustlogStatusEnum 测试存证状态枚举
func TestTrustlogStatusEnum(t *testing.T) {
tests := []struct {
status TrustlogStatus
expected string
}{
{StatusNotTrustlogged, "NOT_TRUSTLOGGED"},
{StatusTrustlogged, "TRUSTLOGGED"},
}
for _, tt := range tests {
assertEqual(t, string(tt.status), tt.expected)
}
}
// TestRetryStatusEnum 测试重试状态枚举
func TestRetryStatusEnum(t *testing.T) {
tests := []struct {
status RetryStatus
expected string
}{
{RetryStatusPending, "PENDING"},
{RetryStatusRetrying, "RETRYING"},
{RetryStatusDeadLetter, "DEAD_LETTER"},
}
for _, tt := range tests {
assertEqual(t, string(tt.status), tt.expected)
}
}
// TestPersistenceStrategyEnum 测试持久化策略枚举
func TestPersistenceStrategyEnum(t *testing.T) {
tests := []struct {
strategy PersistenceStrategy
expected string
}{
{StrategyDBOnly, "DB_ONLY"},
{StrategyDBAndTrustlog, "DB_AND_TRUSTLOG"},
{StrategyTrustlogOnly, "TRUSTLOG_ONLY"},
{PersistenceStrategy(999), "UNKNOWN"},
}
for _, tt := range tests {
assertEqual(t, tt.strategy.String(), tt.expected)
}
}
// TestDefaultPersistenceConfig 测试默认持久化配置
func TestDefaultPersistenceConfig(t *testing.T) {
cfg := DefaultPersistenceConfig(StrategyDBAndTrustlog)
assertEqual(t, cfg.Strategy, StrategyDBAndTrustlog)
assertEqual(t, cfg.EnableRetry, true)
assertEqual(t, cfg.MaxRetryCount, 5)
assertEqual(t, cfg.RetryBatchSize, 100)
}
// TestDefaultRetryWorkerConfig 测试默认重试工作器配置
func TestDefaultRetryWorkerConfig(t *testing.T) {
cfg := DefaultRetryWorkerConfig()
assertEqual(t, cfg.RetryInterval, 30*time.Second)
assertEqual(t, cfg.MaxRetryCount, 5)
assertEqual(t, cfg.BatchSize, 100)
assertEqual(t, cfg.BackoffMultiplier, 2.0)
assertEqual(t, cfg.InitialBackoff, 1*time.Minute)
}
// TestGetDialectDDL 测试不同数据库的 DDL 生成
func TestGetDialectDDL(t *testing.T) {
drivers := []string{"postgres", "mysql", "sqlite3", "sqlite", "unknown"}
for _, driver := range drivers {
t.Run(driver, func(t *testing.T) {
opDDL, cursorDDL, retryDDL, err := GetDialectDDL(driver)
if err != nil {
t.Fatalf("GetDialectDDL(%s) returned error: %v", driver, err)
}
if opDDL == "" {
t.Error("operation DDL should not be empty")
}
if cursorDDL == "" {
t.Error("cursor DDL should not be empty")
}
if retryDDL == "" {
t.Error("retry DDL should not be empty")
}
})
}
}
// TestDatabaseConnection 测试数据库连接(使用 SQLite 内存数据库)
func TestDatabaseConnection(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
err = db.Ping()
if err != nil {
t.Fatalf("failed to ping database: %v", err)
}
}
// TestDDLExecution 测试 DDL 执行
func TestDDLExecution(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
opDDL, cursorDDL, retryDDL, err := GetDialectDDL("sqlite3")
if err != nil {
t.Fatalf("failed to get DDL: %v", err)
}
// 执行 operation 表 DDL
_, err = db.Exec(opDDL)
if err != nil {
t.Fatalf("failed to execute operation DDL: %v", err)
}
// 执行 cursor 表 DDL
_, err = db.Exec(cursorDDL)
if err != nil {
t.Fatalf("failed to execute cursor DDL: %v", err)
}
// 执行 retry 表 DDL
_, err = db.Exec(retryDDL)
if err != nil {
t.Fatalf("failed to execute retry DDL: %v", err)
}
// 验证表是否创建成功
tables := []string{"operation", "trustlog_cursor", "trustlog_retry"}
for _, table := range tables {
var name string
err = db.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&name)
if err != nil {
t.Errorf("table %s was not created: %v", table, err)
}
}
}
// TestOperationTableStructure 测试 operation 表结构
func TestOperationTableStructure(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
opDDL, _, _, _ := GetDialectDDL("sqlite3")
_, err = db.Exec(opDDL)
if err != nil {
t.Fatalf("failed to create operation table: %v", err)
}
// 验证关键字段存在
requiredFields := []string{
"op_id", "op_actor", "doid", "producer_id",
"request_body_hash", "response_body_hash",
"client_ip", "server_ip", "trustlog_status",
}
rows, err := db.Query("PRAGMA table_info(operation)")
if err != nil {
t.Fatalf("failed to get table info: %v", err)
}
defer rows.Close()
foundFields := make(map[string]bool)
for rows.Next() {
var cid int
var name, typ string
var notnull, pk int
var dfltValue sql.NullString
err = rows.Scan(&cid, &name, &typ, &notnull, &dfltValue, &pk)
if err != nil {
t.Fatalf("failed to scan row: %v", err)
}
foundFields[name] = true
}
for _, field := range requiredFields {
if !foundFields[field] {
t.Errorf("required field %s not found in operation table", field)
}
}
}
// TestCursorTableInitialization 测试游标表初始化
func TestCursorTableInitialization(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
_, cursorDDL, _, _ := GetDialectDDL("sqlite3")
_, err = db.Exec(cursorDDL)
if err != nil {
t.Fatalf("failed to create cursor table: %v", err)
}
// 验证初始记录存在
var count int
err = db.QueryRow("SELECT COUNT(*) FROM trustlog_cursor WHERE id = 1").Scan(&count)
if err != nil {
t.Fatalf("failed to count cursor records: %v", err)
}
if count != 1 {
t.Errorf("expected 1 cursor record, got %d", count)
}
}
// TestRetryTableIndexes 测试重试表索引
func TestRetryTableIndexes(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
_, _, retryDDL, _ := GetDialectDDL("sqlite3")
_, err = db.Exec(retryDDL)
if err != nil {
t.Fatalf("failed to create retry table: %v", err)
}
// 验证索引存在
expectedIndexes := []string{
"idx_retry_status",
"idx_retry_next_retry_at",
}
rows, err := db.Query("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='trustlog_retry'")
if err != nil {
t.Fatalf("failed to get indexes: %v", err)
}
defer rows.Close()
foundIndexes := make(map[string]bool)
for rows.Next() {
var name string
err = rows.Scan(&name)
if err != nil {
t.Fatalf("failed to scan row: %v", err)
}
foundIndexes[name] = true
}
for _, idx := range expectedIndexes {
if !foundIndexes[idx] {
t.Errorf("expected index %s not found", idx)
}
}
}
// TestNullableFields 测试可空字段
func TestNullableFields(t *testing.T) {
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer db.Close()
opDDL, _, _, _ := GetDialectDDL("sqlite3")
_, err = db.Exec(opDDL)
if err != nil {
t.Fatalf("failed to create operation table: %v", err)
}
// 插入测试数据IP 字段为 NULL
ctx := context.Background()
_, err = db.ExecContext(ctx, `
INSERT INTO operation (
op_id, doid, producer_id, op_source, op_type,
do_prefix, do_repository, trustlog_status, timestamp
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
`, "test-001", "10.1000/repo/obj", "producer-001", "DOIP", "Create",
"10.1000", "repo", "NOT_TRUSTLOGGED", time.Now())
if err != nil {
t.Fatalf("failed to insert record with null IPs: %v", err)
}
// 查询验证
var clientIP, serverIP sql.NullString
err = db.QueryRowContext(ctx, "SELECT client_ip, server_ip FROM operation WHERE op_id = ?", "test-001").
Scan(&clientIP, &serverIP)
if err != nil {
t.Fatalf("failed to query record: %v", err)
}
if clientIP.Valid {
t.Error("clientIP should be NULL")
}
if serverIP.Valid {
t.Error("serverIP should be NULL")
}
}
// 辅助函数
func assertEqual(t *testing.T, got, want interface{}) {
t.Helper()
if got != want {
t.Errorf("got %v, want %v", got, want)
}
}