Distributed Database Sharding Strategies and Consistency Guarantees

1. Sharding Strategies

When a single database reaches the terabyte scale, sharding becomes inevitable. Common partitioning strategies include hash sharding, range sharding, and consistent hashing. Choosing the right shard key is the core of the design.

2. Go Sharding Implementation

The following example demonstrates a consistent hash sharding router implemented in Go, responsible for routing requests to the correct database shard.

Go Sharding Logic

go
package shard

import (
	"crypto/sha256"
	"fmt"
	"hash/crc32"
)

type Router struct {
	shards []string
}

func NewRouter(nodes []string) *Router {
	return &Router{shards: nodes}
}

// ShardKey calculates the target shard based on user ID
func (r *Router) ShardKey(userID string) string {
	h := crc32.ChecksumIEEE([]byte(userID))
	idx := int(h) % len(r.shards)
	return r.shards[idx]
}

// ShardKeyConsistent uses consistent hashing
func (r *Router) ShardKeyConsistent(userID string) string {
	hash := sha256.Sum256([]byte(userID))
	value := int(hash[0])<<24 | int(hash[1])<<16 |
		int(hash[2])<<8 | int(hash[3])
	idx := value % len(r.shards)
	// Virtual node support (simplified)
	return r.shards[idx]
}

func main() {
	nodes := []string{"shard-0", "shard-1", "shard-2", "shard-3"}
	router := NewRouter(nodes)

	userIDs := []string{"user_1001", "user_1002", "user_1003"}
	for _, uid := range userIDs {
		shard := router.ShardKeyConsistent(uid)
		fmt.Printf("User %s -> %s\n", uid, shard)
	}
}

SQL Query Example

sql
-- Create shard table
CREATE TABLE IF NOT EXISTS orders_0000 (
    id BIGINT PRIMARY KEY,
    user_id VARCHAR(64) NOT NULL,
    amount DECIMAL(12,2) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_user_id (user_id)
);

-- Cross-shard query (results merged at application layer)
-- The query runs independently on each shard
SELECT user_id, SUM(amount) as total
FROM orders_0000
WHERE created_at >= '2026-01-01'
GROUP BY user_id;

3. Consistency Guarantees

For distributed transactions, TCC or Saga patterns are recommended, combined with eventual consistency compensation mechanisms. For read-heavy scenarios, use read-write separation and cache warming strategies to improve performance.