# Admin Http Source: https://docs.springtail.io/admin-http # AdminServer Class ## Overview `AdminServer` provides an embedded HTTP administration interface for Springtail services. It runs as a singleton service, starts its own thread, and exposes a small HTTP API for health checks, configuration inspection, logging control, and dynamically registered administrative endpoints. The server automatically binds to an available port and publishes its address to Redis for service discovery. The server is built on top of `httplib::Server` and uses JSON for all request and response payloads. ## Features * **Embedded HTTP server** using the `cpp-httplib` library * **Dynamic route registration** for GET and POST handlers * **Built-in health and configuration endpoints** * **Centralized error handling and response formatting** with structured JSON responses * **Thread-safe dynamic route management** using shared mutexes * **Automatic service discovery** via Redis integration * **Logging control** through HTTP endpoints * **Single-threaded request processing** for simplicity * **Runs in a dedicated thread** * **Singleton-based lifecycle management** ## Lifecycle The AdminServer instance is created automatically when first accessed. Startup and shutdown are managed through the common Singleton infrastructure. On startup: * The HTTP server thread is started * Default routes are registered * The server binds to an IP and port * The bound address is published to Redis On shutdown: * The HTTP server is stopped * The server thread is joined * Resources are released cleanly ## Class Declaration ```cpp theme={null} class AdminServer : public Singleton ``` The `AdminServer` inherits from `Singleton` to ensure only one instance exists throughout the application lifetime. ## Public Interface ### Type Definitions #### `GetHandler` ```cpp theme={null} using GetHandler = std::function; ``` Handler function type for GET requests. **Parameters:** * `path`: The request path * `params`: URL query parameters * `json_response`: JSON object to populate with response data #### `PostHandler` ```cpp theme={null} using PostHandler = std::function; ``` Handler function type for POST requests. **Parameters:** * `path`: The request path * `params`: URL query parameters * `body`: Request body as string * `json_response`: JSON object to populate with response data ### Route Management Methods #### `register_get_route()` ```cpp theme={null} void register_get_route(const std::string& path, GetHandler &&handler) ``` Registers a handler for a specific GET request path. **Parameters:** * `path`: The URL path to handle (e.g., "/status") * `handler`: Handler function to execute for this path **Thread Safety:** Yes (uses unique lock) **Example:** ```cpp theme={null} AdminServer::get_instance()->register_get_route("/status", [](const std::string &path, const httplib::Params ¶ms, nlohmann::json &json_response) { json_response = {{"status", "running"}, {"uptime", get_uptime()}}; } ); ``` #### `deregister_get_route()` ```cpp theme={null} void deregister_get_route(const std::string& path) ``` Removes a previously registered GET handler. **Parameters:** * `path`: The URL path to deregister **Thread Safety:** Yes (uses unique lock) #### `register_post_route()` ```cpp theme={null} void register_post_route(const std::string& path, PostHandler &&handler) ``` Registers a handler for a specific POST request path. **Parameters:** * `path`: The URL path to handle * `handler`: Handler function to execute for this path **Thread Safety:** Yes (uses unique lock) **Example:** ```cpp theme={null} AdminServer::get_instance()->register_post_route("/restart", [](const std::string &path, const httplib::Params ¶ms, const std::string &body, nlohmann::json &json_response) { nlohmann::json request = nlohmann::json::parse(body); perform_restart(request["component"]); json_response = {{"status", "restarted"}}; } ); ``` #### `deregister_post_route()` ```cpp theme={null} void deregister_post_route(const std::string& path) ``` Removes a previously registered POST handler. **Parameters:** * `path`: The URL path to deregister **Thread Safety:** Yes (uses unique lock) #### `exists()` ```cpp theme={null} static bool exists() ``` Checks if the AdminServer singleton instance has been created. **Returns:** `true` if instance exists, `false` otherwise ## Built-in Endpoints The AdminServer provides several built-in endpoints for common administrative tasks: ### GET /health Returns server health status. **Response:** ```json theme={null} { "status": "up" } ``` ### GET /config Returns all application configuration settings. **Response:** ```json theme={null} { "status": "ok", "settings": { // All configuration properties } } ``` ### GET /logging Returns current logging configuration and statistics. **Response:** ```json theme={null} { "log_level": "info", "debug_level": 0, "module_masks": { /* ... */ } } ``` ### POST /logging Updates logging configuration dynamically. **Request Body:** ```json theme={null} { "log_level": "debug", // Optional: Set log level (trace, debug, info, warn, error, critical) "debug_level": 2, // Optional: Set debug level (0-n) "module_mask": { // Optional: Enable/disable specific module logging "module": "database", "value": true } } ``` **Response:** ```json theme={null} { "status": "ok", "result": { // Updated logging configuration } } ``` ## Custom Route Registration The users of this class may dynamically register and deregister administrative routes. ```cpp theme={null} GET Routes using GetHandler = std::function; ``` Routes are matched by exact path. ```cpp theme={null} POST Routes using PostHandler = std::function; ``` ## Error Handling All request handlers are executed inside a common error wrapper that: * Converts exceptions into JSON responses * Sets appropriate HTTP status codes * Logs errors consistently Supported Error Types * `HttpError` for application-level HTTP failures * JSON parsing errors * Standard C++ exceptions * Unknown exceptions ### `HttpError` Exception HttpError represents an HTTP-aware exception type. **Features**: * Custom error message * Explicit HTTP status code * Automatically translated into JSON error responses ```cpp theme={null} class HttpError : public Error ``` Custom exception class for HTTP-specific errors with status codes. #### Constructor ```cpp theme={null} explicit HttpError(const std::string &msg, uint32_t error_code = 400) ``` **Parameters:** * `msg`: Error message * `error_code`: HTTP status code (default: 400) #### Method ```cpp theme={null} uint32_t get_error_code() ``` Returns the HTTP error code associated with the exception. ## InternalHTTPServer Class `InternalHTTPServer` is a private nested class that extends `httplib::Server` to provide additional functionality. **Capabilities**: * Retrieve bound IP and port at runtime * Convert HTTP requests into a human-readable string format for debugging ### Methods #### `get_bind_ip_port()` ```cpp theme={null} std::string get_bind_ip_port() ``` Returns the IP address and port the server is bound to in the format "ip:port". Supports both IPv4 and IPv6. **Returns:** String in format "x.x.x.x:port" or empty string if socket is invalid #### `request_to_string()` (static) ```cpp theme={null} static std::string request_to_string(const httplib::Request& request) ``` Helper function for debugging that converts an HTTP request to a detailed string representation. **Parameters:** * `request`: The HTTP request to convert **Returns:** Multi-line string with all request details including method, path, headers, parameters, body, form data, and file uploads ## Private Implementation Details ### Constructor ```cpp theme={null} AdminServer() ``` The constructor performs the following initialization: 1. Configures the HTTP server with a single-threaded task queue 2. Registers all built-in endpoints (/health, /config, /logging) 3. Sets up wildcard dispatchers for GET and POST requests 4. Starts the server thread 5. Waits until the server is ready 6. Publishes the server address to Redis for service discovery ### Thread Management #### `_internal_run()` ```cpp theme={null} void _internal_run() override ``` Runs the HTTP server on the configured IP and port. The server binds to 0.0.0.0:0 by default, allowing the OS to select an available port. #### `_internal_thread_shutdown()` ```cpp theme={null} void _internal_thread_shutdown() override ``` Signals the server to stop accepting new requests and begin shutdown. ### Request Dispatching #### `_dispatch_get()` ```cpp theme={null} void _dispatch_get(const httplib::Request& req, httplib::Response& res) ``` Dispatches GET requests to registered handlers. Uses shared lock for thread-safe route lookup. **Behavior:** * Looks up handler in `_get_routes` map * Invokes handler if found * Throws `HttpError` with 404 status if no handler exists #### `_dispatch_post()` ```cpp theme={null} void _dispatch_post(const httplib::Request& req, httplib::Response& res) ``` Dispatches POST requests to registered handlers. Uses shared lock for thread-safe route lookup. **Behavior:** * Looks up handler in `_post_routes` map * Invokes handler if found * Throws `HttpError` with 404 status if no handler exists ### Error Handling #### `_wrap_error_handler()` ```cpp theme={null} template requires std::same_as, nlohmann::json> void _wrap_error_handler(httplib::Response& res, Func func, Args && ...args) ``` Generic wrapper that catches exceptions from handler functions and converts them to structured JSON error responses. **Caught Exceptions:** * `HttpError`: Returns custom error code with error message * `nlohmann::detail::exception`: Returns 500 with JSON parsing error details * `std::exception`: Returns 500 with standard exception message * `...` (catch-all): Returns 500 with unknown error status **Error Response Format:** ```json theme={null} { "status": "error_type", "error_message": "detailed error message" } ``` ### Member Variables ```cpp theme={null} InternalHTTPServer _svr; // HTTP server instance std::string _ip{"0.0.0.0"}; // Bind IP address uint16_t _port{0}; // Bind port (0 = auto-select) std::unordered_map _get_routes; // GET handler registry std::unordered_map _post_routes; // POST handler registry std::shared_mutex _mutex; // Protects route maps ``` ## Redis Integration On startup, the server publishes its network location to Redis using: * Instance ID * Instance key * Program name This allows external tools and services to discover the active admin endpoint dynamically. ## Usage Functionality: * The server listens on 0.0.0.0 by default * The port is assigned dynamically unless configured otherwise * All responses are JSON * Unregistered paths return HTTP 404 ### Basic Usage Example ```cpp theme={null} // Server automatically starts when first accessed AdminServer::get_instance(); // Register a custom GET endpoint AdminServer::get_instance()->register_get_route("/metrics", [](const std::string &path, const httplib::Params ¶ms, nlohmann::json &json_response) { json_response = { {"requests_processed", get_request_count()}, {"avg_response_time_ms", get_avg_response_time()} }; } ); // Register a custom POST endpoint AdminServer::get_instance()->register_post_route("/cache/clear", [](const std::string &path, const httplib::Params ¶ms, const std::string &body, nlohmann::json &json_response) { clear_cache(); json_response = {{"status", "cleared"}}; } ); ``` ### Handler with Error Handling ```cpp theme={null} AdminServer::get_instance()->register_post_route("/user/create", [](const std::string &path, const httplib::Params ¶ms, const std::string &body, nlohmann::json &json_response) { nlohmann::json request = nlohmann::json::parse(body); // Validate input if (!request.contains("username")) { throw HttpError("Missing username field", 400); } // Process request std::string username = request["username"]; if (!create_user(username)) { throw HttpError("User already exists", 409); } json_response = { {"status", "created"}, {"username", username} }; } ); ``` ### Using Query Parameters ```cpp theme={null} AdminServer::get_instance()->register_get_route("/search", [](const std::string &path, const httplib::Params ¶ms, nlohmann::json &json_response) { // Access query parameters from URL like /search?query=test&limit=10 std::string query = params.count("query") ? params.at("query") : ""; int limit = params.count("limit") ? std::stoi(params.at("limit")) : 10; auto results = perform_search(query, limit); json_response = { {"query", query}, {"results", results} }; } ); ``` ### Deregistering Routes ```cpp theme={null} // Remove a route when no longer needed AdminServer::get_instance()->deregister_get_route("/metrics"); AdminServer::get_instance()->deregister_post_route("/cache/clear"); ``` ## Service Discovery On startup, the AdminServer automatically publishes its listening address to Redis: **Redis Key Format:** ``` Hash: admin_console:{instance_id} Field: {instance_key}:{program_name} Value: {ip}:{port} ``` This allows other services to discover and connect to the admin interface dynamically. ## Thread Safety * Route registration/deregistration uses `std::unique_lock` for exclusive access * Route dispatching uses `std::shared_lock` for concurrent read access * Multiple GET/POST requests can be processed concurrently (read access) * Route modifications block all request processing (write access) * The server uses a single-threaded task queue, so handlers execute sequentially ## Design Considerations 1. **Single-threaded Processing**: The server uses a single-threaded task queue to simplify synchronization and avoid complex concurrency issues in handlers 2. **Dynamic Port Selection**: Binding to port 0 allows the OS to select an available port, avoiding conflicts 3. **Wildcard Matching**: The server registers wildcard patterns (`.*`) and uses custom dispatchers for flexible routing 4. **JSON-based Communication**: All responses are JSON for consistency and ease of parsing 5. **Centralized Error Handling**: The `_wrap_error_handler` template ensures consistent error responses across all endpoints ## Notes * The server automatically binds to `0.0.0.0` on an available port * All handlers must populate the `json_response` parameter * Throwing `HttpError` allows custom HTTP status codes * The server waits until ready before publishing to Redis * Built-in endpoints cannot be deregistered * Route paths are exact matches (no regex or wildcards in custom routes) # Architecture Source: https://docs.springtail.io/architecture ## Overview Springtail is a distributed, read-only database designed to scale out compute resources for servicing read-intensive workloads. Using a shared-storage layer, Springtail enables rapid scaling of stateless compute resources, allowing them to start up or shut down instantly. It ingests data from a Postgres primary database instance and stores that data in a proprietary table format. It uses a modified Postgres frontend to service database queries ensuring Postgres query compatibility. ## Components Springtail’s architecture includes several core components, each running independently, with the capability to recover independently. Each component stores data persistently in the shared filesystem and communicates with other components via Remote Procedure Calls (RPCs). ### Storage layer The storage layer is a distributed filesystem accessible by all components. This fault-tolerant filesystem stores data in multiple availability zones and can scale by adding additional storage nodes. Table data and metadata are stored as files within the filesystem. Data is written copy-on-write, enabling access to older versions of tables. ### Ingest pipeline Springtail ingests data from a primary Postgres database using the Postgres logical replication protocol. DDL changes such as table creation and table modification are not replicated by Postgres (using logical replication). Triggers are installed on the primary database to add DDL modifications to the replication stream (for `CREATE TABLE` | `ALTER TABLE` | etc). Once data is received by Springtail it is logged to the storage system making it durable, so that the primary database can release its resources (freeing data from its write-ahead log). Once the data is durable, transactions are extracted from the replication stream. Each transaction is isolated from the log and the operations that make up that transaction (e.g., `INSERT` | `UPDATE` | `DELETE` | etc.) are processed, updating the corresponding tables within the storage layer. Data within each table is stored in primary key order. When all operations for a transaction are processed, the system's latest transaction ID is advanced, resulting in a new version of the database. ```mermaid theme={null} flowchart LR subgraph Ingest-Pipeline direction TB top1[Ingest & Logging] --> m2[Transaction Processing] --> bottom1[Storage] end Primary-DB ---> top1 ``` ### Compute query node Data is queried via a Postgres frontend running on a compute node. Each compute node accesses the storage-layer in a read-only fashion to read table data and metadata. Multiple compute nodes can be run in parallel. Thanks to their stateless design, compute nodes can be rapidly scaled up or down. Springtail uses PostgreSQL’s **Foreign Data Wrapper (FDW)** interface to enable Postgres to access the table data stored within the shared filesystem, ensuring seamless compatibility with Postgres queries, as if querying a native Postgres instance. ```mermaid theme={null} flowchart LR subgraph Compute-Node direction TB top2[Postgres] --> mid[Foreign Data Wrapper] --> bottom2[Storage] end ``` ### Proxy One of Springtail’s core tenants is providing access to scale-out replication without requiring application changes or database migration. As such, Springtail introduces a proxy that acts as an intermediary between the customer application and the Springtail Foreign Data Wrapper frontend nodes. Applications connect to the Springtail proxy as they would to the primary Postgres database. The proxy then routes traffic appropriately—sending writes to the primary database and reads to Springtail’s compute nodes. The proxy parses the Postgres wire protocol and understands which queries the Springtail system can handle, and which need to be sent to the primary database. Additionally, the Springtail proxy provides load-balancing across the read-replicas and provides session-level connection pooling. ```mermaid theme={null} flowchart LR subgraph Compute-Node direction TB top2[Postgres] --> bottom2[Storage] end Application --> Proxy ---> top2 Proxy --> Primary-DB ``` # Building & Modules Source: https://docs.springtail.io/building ## Compiling the code Compiling the C++ code is fairly straightforward. All dependencies will be installed via vcpkg; note the first build may take a long time due to downloading and compiling all dependencies. Make sure all dependencies from the [Local Dev Environment](/local-dev-environment) setup have been installed; otherwise vcpkg deps will not compile. To do a debug build run: ```bash theme={null} springtail % ./debug.sh # first time build, setups up springtail/debug springtail % cd debug; make # compile source; or switch to appropriate module springtail % ./clean.sh. # does a make clean and removes the springtail/debug dir ``` ## Adding a C++ module * Create new subdirs under src/ and include/ * Add module to top level CMakeLists.txt: `add_subdirectory(src/module)` * Within module (`/src/module/`) create a test subdir and a CMakeLists.txt * Add source code and library dependencies to CMakeLists.txt * Add test source code (or helper utils) under `test/` subdir * Add to function `springtail_unit_test()` to `CMakeLists.txt` * E.g. ``` springtail_unit_test(test_replica_shutdown SOURCES test/test_replica_shutdown.cc INCLUDES ${CMAKE_SOURCE_DIR}/include LIBS springtail_proxy GTest::gtest_main ) ``` ## Accessing the postgres database on your machine Run the following command: `psql "dbname=springtail user=springtail host=/var/run/postgresql password=springtail”` # Client Session Source: https://docs.springtail.io/client-session # Client Session Architecture ## Overview The Client Session represents a connection from a PostgreSQL client (e.g., psql, application) to the Springtail proxy. It manages the client's state, handles incoming queries, coordinates with server sessions, and ensures proper query routing and response handling. ## Hierarchy The Client Session builds upon a base Session class which provides: * Basic connection management (socket, read/write buffers) * Message parsing and framing * Asynchronous I/O handling * Connection state tracking ## Key Responsibilities 1. **Query Reception**: Receive and parse PostgreSQL protocol messages from client 2. **Query Routing**: Determine which server session should execute the query 3. **State Management**: Track transaction state, prepared statements, portals, cursors 4. **Response Coordination**: Collect responses from server sessions and forward to client 5. **Session Replay**: Ensure new server sessions have correct session state 6. **Error Handling**: Manage error conditions and recovery ## State Machine ### Primary States ```mermaid theme={null} stateDiagram-v2 [*] --> STARTUP STARTUP --> AUTH_SERVER: client auth completes AUTH_SERVER --> READY: server auth completes READY --> QUERY: client sends query message QUERY --> READY: server sends ReadyForQuery STARTUP --> ERROR: fatal error / disconnect AUTH_SERVER --> ERROR: fatal error / disconnect READY --> ERROR: fatal error / disconnect QUERY --> ERROR: fatal error / disconnect ERROR --> [*] note right of STARTUP: Initial state, handling authentication note right of AUTH_SERVER: Waiting for server authentication note right of READY: Idle, waiting for next query note right of QUERY: Processing query (delegated to server session) note right of ERROR: Fatal error, connection closing ``` ### State Transitions #### STARTUP → AUTH\_SERVER * **Trigger**: Client authentication completes successfully * **Actions**: * Extract database and user credentials * Create primary server session * Initiate server authentication #### AUTH\_SERVER → READY * **Trigger**: Server authentication completes successfully * **Actions**: * Send authentication success to client * Set transaction status to idle * Prepare for query processing #### READY → QUERY (via Server Session) * **Trigger**: Client sends query message (Query, Parse, Bind, Execute, etc.) * **Actions**: * Validate query message * Determine target server session (primary vs replica) * Queue message for server session * Server session transitions to QUERY or DEPENDENCIES state #### Server Session QUERY → READY * **Trigger**: Server sends ReadyForQuery * **Actions**: * Update transaction status (idle, in-transaction, error) * Client session remains in READY state * Process next queued message if available #### ANY → ERROR * **Trigger**: Fatal error, client disconnect, or Terminate message * **Actions**: * Close connection * Clean up resources * Notify server sessions to shut down ## Query Processing Flow ### 1. Query Reception The client session receives PostgreSQL protocol messages from the client: **Message Types Handled:** * **Query (Simple Protocol)**: Multiple semicolon-separated SQL statements * **Parse**: Prepare a named or unnamed statement * **Bind**: Bind parameters to a prepared statement creating a portal * **Execute**: Execute a bound portal * **Describe**: Request description of a statement or portal * **Close**: Close a statement or portal * **Sync**: Synchronization point in extended protocol * **Flush**: Force pending responses to be sent ### 2. Query Analysis For each incoming query, the client session: 1. **Parse Query Text**: Extract SQL statements from message 2. **Classify Statements**: Determine query types (SELECT, INSERT, UPDATE, etc.) 3. **Analyze Dependencies**: Identify state-changing operations: * SET statements (session variables) * PREPARE statements (prepared statements) * DECLARE statements (cursors) * Transaction control (BEGIN, COMMIT, ROLLBACK, SAVEPOINT) 4. **Determine Read Safety**: Check if query modifies data or accesses replicated tables 5. **Determine Routing**: * Read-only queries → replica (if available and not in transaction) * Write queries → primary * Queries in transaction → same server as transaction ### 3. Server Session Selection The routing logic follows these rules: **In Transaction:** * All queries route to the server that started the transaction * Typically this is the primary server * Ensures transaction isolation and consistency **Outside Transaction:** * **Write Queries**: Always route to primary * **Read-Only Queries**: * Route to replica if available and database is ready * Fall back to primary if no replica available * Fall back to primary if in primary-only mode **Shadow Mode:** * Read-only queries sent to both primary and replica * Primary results returned to client * Replica results discarded (for testing/validation) ### 4. Session Replay (State Synchronization) When sending a query to a server session that hasn't processed queries from this client before, or when switching servers, the client session ensures the server has the correct session state through a process called "session replay." **Session State Includes:** * Session variables (SET statements like `work_mem`, `application_name`) * Prepared statements (PREPARE statements) * Cursors with hold (DECLARE WITH HOLD statements) **Replay Process:** ``` Client Session State: - work_mem = '64MB' - application_name = 'myapp' - Prepared statement 'stmt1' First Query to New Server: ↓ 1. Client session queries statement cache for replay history 2. Generates dependency messages containing: - SET work_mem = '64MB' - SET application_name = 'myapp' - PREPARE stmt1 AS ... 3. Server session receives dependencies first 4. Server session executes dependencies (server enters DEPENDENCIES state) 5. Server session transitions to QUERY state 6. Server session executes actual query ``` **Transaction Replay:** When switching from replica to primary mid-transaction (rare case): * Transaction-level statements are also replayed * Ensures transaction state is consistent on new server ### 5. Response Handling Server sessions send responses back to the client session via callbacks: **Response Types:** * **ParseComplete**: Parse succeeded * **BindComplete**: Bind succeeded * **CommandComplete**: Statement executed with result summary * **RowDescription**: Column metadata for query results * **DataRow**: Individual result row data * **ReadyForQuery**: Server ready for next command (includes transaction status) * **ErrorResponse**: Query failed with error details The client session forwards these responses directly to the client, maintaining protocol transparency. ## Server Session Callbacks Server sessions invoke callbacks on the client session to report completion and status: ### Message Response Callback Called when a message completes execution: **For Each Message:** * Server session executes all statements in the message * Sends completion notification to client session * Client session updates statement cache * Records success or failure status **Statement Cache Updates:** * Add prepared statements to cache * Add cursors to cache * Track transaction state changes * Remove closed statements/portals ### Ready For Query Callback Called when the server sends ReadyForQuery: **Transaction Status Update:** * Update transaction status (I/T/E) * Clear associated session if no longer in transaction * Perform transaction commit/rollback in cache **State Synchronization:** * Replay pending state to other server sessions * Ensure all server sessions have consistent session state * Only done when outside transactions ## Statement Cache Interaction The client session maintains a History Cache that tracks query statements and their properties: ### Session History Long-lived state that persists across transactions: * **SET statements**: Session variables (e.g., work\_mem, search\_path) * **PREPARE statements**: Named prepared statements * **DECLARE WITH HOLD**: Cursors that survive transaction end * **LISTEN statements**: Notification channels ### Transaction History Transaction-scoped state that exists only within a transaction: * All statements executed in the current transaction * Rolled back on ROLLBACK or error * Merged to session history on successful COMMIT * Organized by savepoint levels ### History Cache Operations **Adding Statements:** * Parse each query to determine type and properties * Track read-safety, dependencies, and side effects * Associate with current transaction or savepoint level * Store metadata for replay purposes **Replay for Server Sessions:** * Query cache for statements needed by a server session * Filter by session vs transaction scope * Filter by read-only vs read-write * Generate dependency messages in correct order **Transaction Operations:** * **Commit**: Merge transaction history to session history * **Rollback**: Discard entire transaction history * **Savepoint**: Create nested scope level * **Rollback to Savepoint**: Discard statements after savepoint ## Thread Safety The Client Session operates within a single-threaded asynchronous I/O model: * Each client session runs on a single I/O thread * No locking required within a client session * Server session callbacks execute on the same thread * Message queues use locks for cross-thread notifications * Failover notifications use mutex-protected queue ## Performance Considerations ### Message Batching Multiple messages are grouped together to reduce round trips between the proxy and PostgreSQL server. This is especially beneficial for extended protocol sequences (Parse/Bind/Execute/Sync). ### Read Replica Routing Read-only queries are offloaded to replica servers, reducing load on the primary and improving overall throughput. This works transparently as long as the client is not in a transaction. ### Connection Pooling Server sessions can be pooled and reused across different client sessions, avoiding the overhead of establishing new connections and re-authenticating. ### Statement Cache Optimization The statement cache minimizes the overhead of session replay by: * Only replaying statements not yet seen by a server * Tracking which servers have which state * Compacting history on transaction commit * Removing redundant statements ### Pipelined Execution Server sessions pipeline messages to PostgreSQL, allowing multiple messages to be sent before waiting for responses. This reduces latency and improves throughput. # Code Structure Source: https://docs.springtail.io/code-structure ## Root directories | Directory | Description | | -------------- | ----------------------------------------------------------------------------------------------- | | src/ | Source tree by module | | include/ | Header tree by module | | external/ | Where `vcpkg` installs its packages | | scripts/ | Helper scripts | | python/ | Python libraries for testing; coordinator for deployment; localcluster controller and utilities | | docker/ | Docker files, scripts | | cmake/ | CMake helper functions | | local-cluster/ | Local cluster setup scripts for running cluster using containers on local machine | | debug/ \* | Conditionally created by CMake for debug builds | | release/ \* | Conditionally created by CMake for production builds | ## Important files | Filename | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | CMakeLists.txt | Top level CMake file for building code. Add new modules/sub-dirs here. Other configuration should be local to each module | | vcpkg.json | List of C++ vcpkg dependencies; See [list of packages](https://vcpkg.io/en/packages) | | clean.sh / debug.sh / release.sh | Helper scripts, wrapper around CMake to clean, build debug or build release | | system.json | Properties JSON file for specifying hostnames, and other configuration info for modules | | python/testing/springtail.py | Initialize redis with properties specified in system.json.\* file; bring up system, bring down system, run checks on system | | python/testing/test\_runner.py | Run full system tests, uses [springtail.py](http://springtail.py) to bring up and down system, runs tests located in test\_cases | ## Modules | Name | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | common | Common utilities, springtail\_init() should be called in every main; logging, exceptions, concurrent queues, thread pool, etc. | | storage | Threaded storage layer; B-tree, extent and schema handling and definition | | pg\_repl | Low-level Postgres replication; connecting to primary and decoding replication messages | | pg\_log\_mgr | Consumes Postgres replication stream to log, process log, extracts transaction start/commit, inserts commit records into a queue to be consumed by the committer. The committer is responsible for applying log changes to the tables and committing them. | | write\_cache | Cache for holding dirty extents by XID; indexed by Table ID, Extent ID, XID and primary key value | | xid\_mgr | Persistently stores last committed XID | | benchmarks | Code for benchmarking file system performance (for distributed file systems) | | proto | gRPC protocol definitions | | sys\_tbl\_mgr | System table service & cache; accessed via gRPC; centralizes management of system table metadata | | proxy | Postgres Proxy that handles read/write splitting sending writes to primary, and reads to FDWs | | pg\_fdw | Postgres frontend query nodes implemented using Postgres Foreign Data Wrapper (FDW) extension; also includes the ddl\_daemon for managing DDL changes for the FDW (to create/alter foreign tables) | | redis | Redis helper classes | | admin\_http | Embedded internal http server | | pg\_ext | Postgres extension related | # Common Init Source: https://docs.springtail.io/common-init # Springtail Initialization and Service Lifecycle This module provides the bootstrapping, dependency ordering, lifecycle management, and shutdown coordination for Springtail services. It defines: * A generic service abstraction * Multiple initialization entry points (normal, daemon, test, custom) * Dependency-aware shutdown ordering * Centralized signal handling for daemon execution * Cross-service argument storage * Deterministic startup and shutdown sequencing This is infrastructure code. Touch it carefully. ## Core Concepts ### ServiceRunner ServiceRunner is the base class for all services participating in Springtail initialization. ``` class ServiceRunner { public: explicit ServiceRunner(const std::string &name); virtual ~ServiceRunner(); virtual bool start(); virtual void stop(); const std::string &get_name() const; }; ``` Responsibilities: * Encapsulates startup and shutdown logic for a service * Provides a human-readable service name * Enables ordered startup and reverse-order shutdown Lifecycle Rules: * `start()` is called in registration order * If any `start()` fails, already-started services are stopped in reverse order * `stop()` is always called during shutdown, even on failure paths ## Initialization Entry Points: ### Standard Standard initialization for non-daemon processes is done using `springtail_init()` function. ``` void springtail_init( bool load_redis = false, const std::optional &log_filename = std::nullopt, const std::optional &logging_mask = std::nullopt ); ``` What It Does: * Initializes logging * Sets up exception handling * Loads properties (environment + optional Redis) * Initializes telemetry * Starts Redis manager * Initializes property Redis cache Typical Usage: ``` springtail_init(false); ``` This initialization is used for short-lived utility tools that still need Springtail infrastructure. Here is an example usage: ``` springtail_init(false, std::nullopt, 0); // utility logic springtail_shutdown(); ``` Notes: * No Redis loading * No daemonization * Logging mask explicitly provided * Clean startup and teardown in a single process This is the good choice for admin tools, migrations, and diagnostics. ### Daemon process Initialization for long-running daemon processes is done using `springtail_init_daemon()` function. ``` void springtail_init_daemon( const std::string &program_name, bool daemonize, const std::optional &logging_mask = std::nullopt ); ``` What It Adds: * Optional daemonization via fork() * PID file creation * Signal-based shutdown coordination * Admin HTTP server startup * Coordinator thread startup Daemonization Behavior * If `daemonize == true`, `fork()` function is called and parent proce exits immediately, while child becomes session leader. * Standard file descriptors are STDIN/STDOUT/STDERR redirected to `/dev/null` * PID written to `/.pid` Here is an example usage by long-running services such as the Proxy server. ``` springtail_store_arguments( ServiceId::ProxyServerId, { {"force_shadow", std::any(force_shadow)}, {"force_primary", std::any(force_primary)} } ); springtail_init_daemon(argv[0], daemonize, LOG_PROXY); ProxyServer::start(); springtail_daemon_run(); springtail_shutdown(); ``` Flow Breakdown 1. Store cross-service arguments * Arguments are available to services during initialization * Stored before initialization so runners can retrieve them 2. Initialize daemon runtime * Optional daemonization * PID file creation * Logging and signal handling * Admin server and coordinator startup 3. Start the main service * `ProxyServer::start()` runs after infrastructure is ready 4. Block until shutdown * `springtail_daemon_run()` waits for signals 5. Graceful shutdown * Services stop in dependency order * Resources released deterministically ### Unit tests initialization Initialization of unit tests is done using `springtail_init_test()` function. ``` void springtail_init_test( const std::optional &logging_mask = std::nullopt ); ``` Characteristics: * Forces Redis loading * No daemon logic * Simplified logging setup Here is a usage example inside a unit test program: Minimal, predictable setup intended for tests. ``` springtail_init_test(); // test logic springtail_shutdown(); ``` Characteristics: * Redis is enabled * No daemon logic * No blocking signal loop * Can only be called once during the test program life time Use this for unit tests. ### Custom initialization Fully custom initialization is done using `springtail_init_custom()` function. ``` void springtail_init_custom( std::vector> &runners ); ``` When to Use: * Embedding Springtail inside another process * Only a subset of initialization services are needed * Non-standard startup ordering You are responsible for providing all required runners. Use this when you need full control over which services start and in what order. Here is an example: ``` std::vector> service_runners; service_runners.emplace_back(std::make_unique()); service_runners.emplace_back(std::make_unique()); service_runners.emplace_back(std::make_unique(true)); service_runners.emplace_back(std::make_unique( "test", std::nullopt, LOG_ALL )); service_runners.emplace_back(std::make_unique("test")); springtail_init_custom(service_runners); // custom logic springtail_shutdown(); ``` When to use this: * Embedding Springtail inside another application * Running a partial service stack * Testing startup order interactions * Avoiding Redis, admin servers, or coordinators With `springtail_init_custom()` responsibilities shift to you. You must provide all required runners and you control startup order. ## Argument Passing Between Services Arguments stored via `springtail_store_arguments()` are accessible during initialization and runtime. Example: ``` auto force_shadow = springtail_retreive_argument( ServiceId::ProxyServerId, "force_shadow", false ); ``` Guarantees: * Type-safe via std::any * Service-scoped * Optional or required retrieval * Fails on type mismatch ## Built-in Service Runners ### Logging `DefaultLoggingRunner`: Handles final logging and open telemetry shutdown. `LoggingRunner`: Initializes logging with optional overrides: * Log filename * Logging mask * Daemon mode awareness ### Properties `PropertiesRunner`: Loads configuration from: * Environment * Optional Redis * Optional override file `PropertiesCacheRunner`: Initializes the Redis-backed configuration cache. ### Daemon `DaemonRunner`: Handles process daemonization and PID management. ### Exception Handling `ExceptionRunner`: Initializes signal handlers and backtrace support. ### Redis `RedisMgrRunner`: Initializes and shuts down the Redis connection manager. ### Coordinator Interface `CoordinatorRunner`: Starts the coordinator background thread. ### Admin Interface `AdminServerRunner`: Starts and stops the admin HTTP server. ## Service Registration and Shutdown ### ServiceRegister ServiceRegister is a singleton responsible for: * Owning all active services * Enforcing startup order * Enforcing reverse shutdown order If startup fails midway: * Already-started services are stopped immediately * Initialization aborts ### Dependency Graph Springtail defines an explicit service dependency graph which enforces correct shutdown ordering of the internal singleton services. This prevents resource teardown races. It uses topological sort on the predefined dependency graph. The sort is executed during runtime and will cause a fatal error is any cycle is detected. When a singleton service is initialized with a specific service id, it will be marked is active. During shutdown, topologically sorted list of services will be shutdown in the order defined by their dependencies. ## Graceful Shutdown The shutdown is performed using `springtail_daemon_run()` function: ``` void springtail_daemon_run(); ``` It blocks until one of the shutdown signals is received: * SIGINT * SIGTERM * SIGQUIT * SIGUSR1 * SIGUSR2 Once triggered: * The signal handlers for these signals are restored * Shutdown proceeds in dependency order The shutdown process is done by `springtail_shutdown()` function: ``` void springtail_shutdown(); ``` It shuts down all registered services in topological order. Notes: * Uses precomputed dependency ordering * Calls each service’s registered shutdown function * Performs final Protobuf cleanup ## Cross-Service Argument Storage This functionality is needed when we have command line arguments that need to be passed to specific `Singleton` services. It allows arbitrary typed arguments to be stored for the service that needs them and then retrieved by that service during initialization. To store an argument, use the following function: ``` springtail_store_argument( ServiceId service_id, const std::string &arg_name, const T &value ); ``` To retrieve an argument, use the following function: ``` auto value = springtail_retreive_argument( service_id, "arg_name", true ); ``` Guarantees: * Type-safe retrieval via std::any * Optional enforcement of required arguments * Fails on type mismatch # Configuring PostgreSQL Source: https://docs.springtail.io/configuring-postgres ## Enabling logical replication on the primary Before Springtail can replicate from your primary database, logical replication must be enabled and a few WAL-related parameters configured on the primary. ### Set `wal_level` to `logical` For a self-hosted PostgreSQL instance, set `wal_level = logical` in `postgresql.conf` and restart the server. On Amazon RDS or Aurora PostgreSQL, set this via a parameter group instead. In the RDS console, click **Parameter groups**, then **Create parameter group** and create a new parameter group with the same family as the primary database (for Aurora PostgreSQL, select the parameter type **cluster parameter group**). This creates a clone of the default parameter group. Under **Edit** mode, search for the `rds.logical_replication` parameter and set it to `1`. After saving, associate the new parameter group with the primary database, applying the change immediately if possible (a reboot is required). ### Set `max_replication_slots` Set `max_replication_slots` to a value greater than or equal to the number of databases Springtail will replicate, since Springtail uses one replication slot per database. ### Set `max_slot_wal_keep_size` Set `max_slot_wal_keep_size` to bound the amount of WAL the primary retains for replication slots. This prevents the primary database from running out of disk space if a replica falls behind. Choose a value appropriate for the primary's available disk. ## Creating the Springtail user To read the logical replication stream, Springtail requires a user account in your existing primary PostgreSQL instance. We recommend creating a new user specifically for Springtail. #### Option 1: Superuser access Creating a user with `SUPERUSER` privileges allows Springtail to fully manage replication slots, publications, and triggers for the logical replication stream. In Amazon RDS, the user must be granted the `rds_superuser` role. To create this user, use the following command, replacing `` with your desired password: ```sql theme={null} CREATE USER springtail WITH PASSWORD '' LOGIN role rds_superuser ``` Make sure you use a strong password, and keep it handy when configuring your Springtail instance. #### Option 2: Read-only user with CREATE permission Alternatively, you can create a **read-only user** with restricted access. This user needs: * **CONNECT** access to the databases to be replicated. * **SELECT** access to tables within those databases for replication purposes. * **SELECT** access to the tables in the pg\_catalog schema. These are queried to determine schema of the replicated tables and by the proxy when a user authenticates. * (Optional) **CREATE** access to the databases to be replicated. Springtail will create a special schema on the databases to be replicated to host triggers and functions. If you do not grant CREATE access, you will need to run the setup scripts delivered by Springtail manually. > By default all users have access to the tables within the pg\_catalog schema that Springtail requires. However, if you are not using the default permissions, the Springtail user will require access to: > pg\_class, pg\_namespace, pg\_attribute, pg\_index, pg\_collation, pg\_type, pg\_constraint, pg\_database, pg\_roles * **REPLICATION** role to start replication and query the replication slot (see below for specific role based on your hosted environment). In this case, you will need to set up and manage replication slots, publications, and triggers manually. Springtail will provide scripts to perform these tasks, but you’ll be responsible for executing them correctly. To create this user, use the following command, replacing `` with your desired user name and `` with your desired password: ```sql theme={null} CREATE USER WITH PASSWORD '' ROLE rds_replication; ``` After creating the user, grant `CONNECT` access to each database: ```sql theme={null} GRANT CONNECT ON DATABASE TO ; ``` Finally, connect to each database and grant `SELECT` access to the desired resources: ```sql theme={null} # To grant access to a single table GRANT SELECT ON TO ; # To grant access to all of the existing and future tables in a schema GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; ``` Make sure you use a strong password, and keep it handy for [instance setup](/deployment). #### Option 1: Superuser access Creating a user with `SUPERUSER` privileges allows Springtail to fully manage replication slots, publications, and triggers for the logical replication stream. In Supabase, the user must be granted the `supabase_admin` role. To create this user, use the following command, replacing `` with your desired password: ```sql theme={null} CREATE USER springtail WITH PASSWORD '' LOGIN role supabase_admin ``` Make sure you use a strong password, and keep it handy when configuring your Springtail instance. #### Option 2: Read-only user Alternatively, you can create a **read-only user** with restricted access. This user needs: * **CONNECT** access to the databases to be replicated. * **SELECT** access to tables within those databases for replication purposes. * **SELECT** access to the tables in the pg\_catalog schema. These are queried to determine schema of the replicated tables and by the proxy when a user authenticates. > By default all users have access to the tables within the pg\_catalog schema that Springtail requires. However, if you are not using the default permissions, the Springtail user will require access to: > pg\_class, pg\_namespace, pg\_attribute, pg\_index, pg\_collation, pg\_type, pg\_constraint, pg\_database, pg\_roles * **REPLICATION** role to start replication and query the replication slot (see below for specific role based on your hosted environment). In this case, you will need to set up and manage replication slots, publications, and triggers manually. Springtail will provide scripts to perform these tasks, but you’ll be responsible for executing them correctly. To create this user, use the following command, replacing `` with your desired user name and `` with your desired password: ```sql theme={null} CREATE USER WITH PASSWORD '' ROLE supabase_replication_admin; ``` After creating the user, grant `CONNECT` access to each database: ```sql theme={null} GRANT CONNECT ON DATABASE TO ; ``` Finally, connect to each database and grant `SELECT` access to the desired resources: ```sql theme={null} # To grant access to a single table GRANT SELECT ON TO ; # To grant access to all of the existing and future tables in a schema GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO ; ``` Make sure you use a strong password, and keep it handy for [instance setup](/deployment). ## Replication slots, publications, triggers and functions To set up logical replication in each database, Springtail creates one publication, one replication slot, and two triggers that utilize three custom functions. * A publication in PostgreSQL captures a set of changes (inserts, updates, and deletes) made to a specific table or set of tables in a database that can be sent to subscribers for replication. This is what defines the set of tables being replicated and captures their changes. * A replication slot in PostgreSQL ensures that the changes made in the primary database are retained until they are consumed by a subscriber. It acts as a buffer that stores the changes for a subscriber until they are applied. This ensures that Springtail does not miss any changes to the data, even in the face of network disconnect, database restart, or other such issues. * The triggers are created to capture schema changes within the database by emitting custom messages into the replication stream that Springtail can use to keep it’s copy of the schemas up-to-date. It also ensures that tables created without a primary key are marked with `REPLICA IDENTITY FULL` which will ensure that the entire row is sent whenever updates or deletes are performed against the table. You will need `SUPERUSER` privileges or `CREATE` privileges on the databases to be replicated, in order to create these objects automatically. After a publication and replication slot are created, Springtail will install triggers and functions on each table that is part of the publication. This will be done inside a special schema named `__pg_springtail_triggers`. If you created a Springtail user with `SUPERUSER` privileges, Springtail will create these objects automatically when you set up your instance. Otherwise, you can either create the `__pg_springtail_triggers` schema manually, then grant full access to the Springtail user, or you can run the grant `CREATE` privilege to the Springtail user, and Springtail will create the schema in the first place automatically. ## Authenticating existing users in Springtail Springtail authenticates users when they log in to the Springtail Proxy. Springtail's Proxy performs user authentication based on the users that exist in the Primary database and only allows those users that have CONNECT access to query the replicated database. The Proxy connects to both the Primary database as well as the Springtail replica in order to perform read/write splitting (sending writes to the Primary database, while sending reads to the Springtail replica). The Proxy caches the set of users and the databases to which they have access, and refreshes this data every few seconds by querying the Primary database. Currently, no other access checks are performed on users accessing replicated data (i.e., row level permissions and table level access checks are not performed). The Proxy authentication is 2-phased. In the first phase, the client or application authenticates with the Proxy (the Proxy is acting as the server); in the second phase, the Proxy authenticates with the Primary / replica (the Proxy is acting as the client). The Proxy uses the same username/password for both phases of authentication and the password is verified in each phase; if either phase fails, the client is denied access. ```mermaid theme={null} sequenceDiagram participant Client participant Proxy participant Primary Client->>Proxy: Authenticate Request Proxy->>Primary: Authenticate Request Primary-->>Proxy: Authentication Response Proxy-->>Client: Authentication Response ``` When setting up a database instance, Springtail requires a list of users, including their username and password, that will be logging in via the Proxy. The username and password must match the Primary database's version (i.e., that user must be able to log into the primary using that username and password). The password can be supplied in one of three forms: plain text, MD5 hash, or SCRAM-SHA-256 hash. The MD5 and SCRAM hashes can be obtained by querying the `pg_shadow` database tables; unfortunately, this table is not accessible on AWS RDS or Aurora instances. If using a MD5 or SCRAM hash as the password, the Primary database must be setup to accept that form of authentication for that user. Using a SCRAM hash is the most secure, as the hash by itself is not sufficient to log into the Primary (it is combined with a client secret that is extracted by the Proxy when the client authenticates to the Proxy). # Database Instances and Pooling Source: https://docs.springtail.io/database-instances-and-pooling # Database Instance Management ## Overview This document describes how database instances are managed in the system, how replica sets and primary sets are organized, and how server-side session pools are configured and maintained. The descriptions are written at a high level and focus on runtime behavior. ## Instance Lifecycle * **Provisioning**: Instances are provisioned and registered with the system as active members of a pool. Each instance advertises its connection endpoint and any grouping or prefix metadata used by the system. * **Health and availability**: The system monitors instance availability. An instance may be marked inactive or scheduled for shutdown when it becomes unresponsive or is intentionally removed. * **Shutdown**: When an instance shuts down, no new sessions are allocated from it. Active sessions notified of instance shutdown and when terminated are explicitly released. * **Replacement**: New or replacement instances can be added to the pool; the system updates its routing and allocation state to include them. ## Replica Sets * **Purpose**: Replica sets group read-only instances that can serve replicated data. They provide horizontal capacity for read workloads and can be used for shadowing or replication-aware routing. * **Load distribution**: The system selects replica instances for new sessions based on load characteristics (e.g., number of active sessions per instance) to spread work across replicas. * **Membership changes**: Replica instances can be added or removed dynamically. When a replica is removed, pooled and active sessions associated with that replica are released or reallocated after the active session has been migrated to another replica. ## Primary Management * **Role of primary**: A primary instance serves write traffic and authoritative state. Primary management coordinates session allocation to the primary and may also manage a standby for failover or replication. * **Primary/standby relationship**: The primary set comprises one primary and zero or more standby instances. The standby may be held in reserve for failover or used for maintenance tasks depending on operational policy. * **Failover & replacement**: The system currently **DOES NOT** support primary failover. ## Session Pools * **Purpose**: Session pools hold reusable server-side sessions to reduce connection overhead and speed client allocations. * **Pool keys**: Sessions in the pool are tracked by a composite key (database identity and user identity) so that pooled sessions match incoming allocation requirements. * **Configurable parameters**: * Size limit: Maximum number of pooled sessions the pool will hold for resource control. * Timeout/expiration: How long idle pooled sessions remain eligible before being considered expired. * Soft limits: A threshold to decide when to evict older entries to enforce expiration policies. * **Eviction policy**: Pools use an age- or usage-based eviction strategy (LRU) to remove the least-recently-used sessions when limits are reached or when sessions expire. * **Session lifecycle within pool**: * Return: When a session finishes client work and is eligible for reuse, it is returned to the pool and tracked for future allocations. * Allocation: A future request that matches the pool key may reuse a pooled session if available; otherwise a new session is created. * Release: Sessions are removed from the pool and deallocated when they are closed, when the instance is no longer active, or as part of eviction. * **Safety and cleanup**: Pool operations are coordinated to be safe in concurrent environments. When instances are removed or shutdown, pooled sessions are released and cleaned up to avoid resource leaks. ## User and Database Invalidation * **Invalidation triggers**: When a user account or a database is removed, disabled, or otherwise marked invalid, the system treats this as an allocation constraint change and initiates cleanup for any affected pooled sessions. * **Pool eviction**: Pooled sessions associated with the invalid user or database are identified and removed from session pools so they cannot be reused for future allocations. Eviction is performed promptly to prevent mismatched credentials or routing. * **Active sessions**: Active sessions that reference the invalid user or database kept open and will be terminated when the associated client session terminates. ## Allocation Strategy * **Prefer reuse**: When a pooled session matching request criteria exists, it is preferred over creating a new session in order to reduce overhead. * **Least-loaded selection**: When selecting a replica for new allocations, the system favors the least-loaded instance to balance active session counts. * **Fallbacks**: If no pooled session is available or no replica is available, the client session will fallback to the primary. ## Summary Database instance management comprises provisioning, dynamic membership, and allocation policies that together enable scalable read and write routing. Session pools reduce connection overhead, while replica and primary sets allow controlled distribution of workloads and operational flexibility. # DDL Manager Source: https://docs.springtail.io/ddl-manager # DDL Manager (FDW) — Architecture and Runtime Behavior This document describes how the DDL Manager operates, how it initializes databases on the Foreign Data Wrapper (FDW) Postgres node, and how it continuously applies schema and security-related changes originating from the primary system. It focuses on runtime behavior and component interactions. ## 1. High-level responsibility The DDL Manager is the component responsible for keeping an FDW Postgres instance synchronized with: * **Logical schema structure** (schemas, tables, partition parents, foreign tables, user-defined enum types). * **Ongoing DDL mutations** (create/rename/drop objects and other supported schema changes). * **Security and ownership state** (roles, role memberships, row-level security policies, and table owners). It does this by: * Performing **initial bootstrap** of each replicated database on the FDW node. * Listening for and applying **incremental DDL change batches** emitted through Redis queues. * Running a periodic **synchronization loop** that replicates security/ownership metadata from the primary Postgres to the FDW Postgres. *** ## 2. Startup sequence and initialization ### 2.1 Startup entry and configuration On startup, the DDL Manager is initialized with: * A unique FDW instance identifier. * Credentials for a privileged “DDL manager user” that can create/alter/drop objects and manage roles/policies on the FDW node. * A proxy user password used when creating replicated login roles on the FDW node. * FDW connection settings (host/port and an optional database name prefix). The startup process also: * Initializes internal caches (notably a small connection cache for FDW DB connections). * Starts auxiliary services needed for correctness (such as collecting transaction/XID related information elsewhere in the system). * Registers Redis-based watchers for dynamic configuration/state changes (notably, which databases should be replicated and their lifecycle state). ### 2.2 System type discovery (FDW node bootstrap prerequisite) Before creating or importing anything, the DDL Manager populates an internal cache of Postgres system-defined types by querying the FDW Postgres instance. This lets it later generate correct table column types when producing CREATE statements for certain objects that must exist locally (e.g., partition parent tables). ### 2.3 Discover replicated databases The DDL Manager retrieves from central configuration/Redis the set of database IDs that should be replicated to this FDW instance. For each configured database ID, it creates an internal tracking entry and begins watching the database’s runtime state. ### 2.4 Database state-driven creation Databases are not always immediately created. Instead, the DDL Manager reacts to database state: * When a database transitions into the **running** state, the DDL Manager performs full initialization on the FDW node (drop/create, schema setup, imports, etc.). * When a database leaves the **running** state, the DDL Manager removes it from the FDW node and cleans up associated runtime metadata. This makes database creation/removal deterministic and tied to operational readiness. *** ## 3. Creating and initializing a replicated database on the FDW node When a database becomes eligible (running), the DDL Manager performs initialization in several broad phases. ### 3.1 Drop and recreate the FDW-side database The FDW Postgres node maintains a distinct database per replicated source database (optionally with a prefix applied). Initialization begins by: * Dropping the FDW-side database (if present). * Creating it again as a clean target for import and future updates. This ensures the FDW database is rebuilt consistently from a known-good baseline during initial sync. ### 3.2 Connect into the new FDW database and prepare prerequisites After creating the database, the DDL Manager connects into that database and performs prerequisite setup: * Ensures required extensions exist (environment-dependent extensions as configured). * Recreates the FDW extension in the database. * Recreates the foreign server object used as the import source, including FDW-specific options such as identifiers and the current schema “watermark” XID. ### 3.3 Determine which schemas to replicate The DDL Manager computes the set of schemas to replicate based on configuration. This can include: * Explicit schema allowlists. * Schema names derived from included tables. * An “all schemas” mode. If no schemas are configured for replication, the database initialization effectively becomes a no-op for schema import. ### 3.4 Create schemas locally on FDW node For each schema selected for replication, the DDL Manager ensures the schema exists on the FDW node. ### 3.5 Create local objects that must exist as regular tables (partition parents) Some partitioning constructs require local (non-foreign) tables to exist on the FDW node so that foreign leaf partitions can attach or behave correctly. During initialization, the DDL Manager generates and executes the necessary DDL so that: * Partition parent tables (and other required non-leaf partition objects, depending on the partition structure) exist as regular tables. * These objects carry identifying metadata so they can later be recognized and validated. ### 3.6 Create user-defined enum types (where applicable) If the replicated schema contains user-defined enum types that need to exist on the FDW node, they are created before importing foreign tables. This is necessary because imported foreign tables may reference those types, and Postgres requires the types to exist at parse/plan time. ### 3.7 Import foreign schema With schemas created (and prerequisite local objects in place), the DDL Manager imports the foreign schema into the FDW database. This creates foreign tables that map to primary-side tables (subject to partitioning rules and the object selection rules from configuration). ### 3.8 Initialize Row-Level Security (RLS) flags on FDW objects After import, the DDL Manager scans the system’s schema metadata to determine which tables require row-level security enabled and/or forced, then applies the corresponding RLS settings on the FDW node. ### 3.9 Record the baseline schema watermark (schema XID) Initialization is anchored to a committed transaction identifier (schema XID). After initialization succeeds: * The DDL Manager records this schema XID in Redis as the FDW’s applied watermark for that database. * It updates its internal per-database “latest applied” XID. This watermarking is critical for allowing incremental DDL application to resume correctly from the baseline. *** ## 4. Continuous DDL change replication (schema mutations) ### 4.1 DDL change production and queueing (upstream) DDL statements are produced upstream and coordinated through Redis. The overall coordination model is: * DDL changes are associated with a specific database and a specific transaction identifier (XID). * Changes are delivered to FDW instances through a queue keyed by FDW identity. * FDWs report progress by writing back the latest schema XID they have successfully applied. The DDL Manager acts as the FDW-side consumer that processes these queued change batches. ### 4.2 Main DDL processing loop The DDL Manager runs a dedicated long-lived thread that: * Blocks waiting for the next set of DDL batches from Redis. * Receives one or more entries, each belonging to a single database and schema XID, containing a list of DDL JSON objects. For each received batch: * The DDL Manager determines whether the XID has already been applied (based on its per-database latest applied watermark). * If already applied, it will acknowledge the entry without updating progress (to clear redundant queue items). * If not yet applied, it schedules the work for execution. ### 4.3 Per-database ordering and pending DDL behavior DDL batches are applied in schema XID order per database. If a database is not currently in the running state: * The DDL Manager stores incoming DDL batches in a per-database pending map keyed by XID. * When the database later transitions to running, pending entries greater than the current initialized watermark are queued for execution. This prevents applying schema mutations to databases that are not yet initialized or are being removed. ### 4.4 Thread pool execution model Actual DDL application is performed via a small thread pool that accepts per-database work items. Each work item corresponds to a specific database and a range of schema XIDs to apply. Key properties: * Work is grouped by database so that a single task can apply multiple XIDs in sequence. * Execution uses a database-scoped lock to ensure the database cannot be concurrently removed while DDL is being applied. * DDL statements are executed within a transaction on the FDW database so the batch is atomic from the FDW’s perspective. ### 4.5 Generating executable SQL and applying it Incoming DDL payloads are JSON objects describing the intended mutation. The DDL Manager: * Converts each supported DDL JSON object into one SQL statement (or a small set of statements where required). * Filters out unsupported or no-op operations (some objects may not require action on the FDW node depending on partitioning and object type). * Appends an update to the FDW server metadata so that the FDW database itself records the most recently applied schema XID. All statements for the scheduled batch are executed in-order within a single transaction. If the transaction succeeds: * The DDL Manager updates Redis with the new schema XID watermark for that database. * Its internal latest applied watermark advances accordingly. If the transaction fails: * The DDL Manager treats this as a fatal replication failure for that batch and triggers rollback/requeue behavior via Redis coordination so that the system can retry or recover. *** ## 5. Periodic security/ownership and policy synchronization Schema structure replication alone is insufficient for correct access control and query behavior. The DDL Manager therefore runs a separate periodic synchronization thread that reconciles FDW-side security state with primary-side security state. ### 5.1 Purpose of the sync thread The sync thread is responsible for ensuring the FDW node reflects the primary node’s: * Roles (create/alter/drop). * Role membership grants and revocations. * Row-level security policies (create/drop/update). * Table ownership changes. This is done independently of the incremental DDL queue and runs continuously at a configurable interval. ### 5.2 How the sync thread operates At each interval, for each active/running replicated database: * It connects to the primary Postgres database for that database name. * It connects to the corresponding FDW database. * It opens a transaction on both connections. * It queries the primary for detected diffs against a snapshot history (diff-oriented approach). * It applies the necessary changes to the FDW database. * It updates/cleans the primary-side diff history entries once changes are successfully applied. If any error occurs while applying changes: * Both transactions are rolled back. * The system relies on the next sync iteration to retry once underlying dependencies are satisfied (for example, if a table required for a policy is not yet present on the FDW node). ### 5.3 First-pass initialization behavior On the first successful sync pass after startup, the DDL Manager resets diff history tracking so that: * The FDW starts from a clean security/ownership baseline. * Subsequent sync iterations only apply incremental changes. After the initial sync completes successfully, the FDW instance transitions into the normal “running” state in system configuration. ### 5.4 Dependency awareness for policies and ownership changes Some security or ownership changes depend on objects existing in the FDW database (e.g., policies require the target table to exist). The sync process therefore: * Verifies necessary objects exist before applying certain changes. * If a required object does not exist yet, it removes or defers the diff entry so it does not repeatedly fail. * Uses transactional protection so partial changes do not leave FDW state inconsistent. *** ## 6. Lifecycle events and shutdown behavior ### 6.1 Database set changes The set of databases replicated by this FDW instance can change dynamically. When the configured database list changes: * Databases no longer present are removed (including dropping the FDW-side database and clearing local tracking). * Newly added databases are registered and initialized when they enter the running state. ### 6.2 Database state transitions When a database leaves running state: * It is removed from the FDW node. * DDL queues for that database are cleared. * Connection cache entries are evicted. * Local history metadata is removed on a best-effort basis. When a database enters running state: * It is fully re-initialized on the FDW node and then begins consuming queued DDL mutations. ### 6.3 Shutdown On shutdown, the DDL Manager: * Signals the sync thread to wake and exit. * Joins the main DDL thread and sync thread. * Stops accepting or processing new DDL work. * Removes Redis watchers and clears internal database tracking. * Transitions the FDW instance’s global state to stopped. *** ## 7. Summary of interactions with FDW Postgres The FDW Postgres node is treated as a managed replica target, and the DDL Manager uses it for: * Creating/dropping whole databases for replication targets. * Creating schemas, extensions, and foreign server definitions. * Importing foreign schema into local schemas. * Creating required local tables/types that must exist prior to foreign imports. * Applying transactional DDL batches generated from queued changes. * Periodically reconciling security/ownership configuration to mirror the primary. This design splits synchronization into two complementary paths: * **DDL queue path** for structural schema mutations in near-real-time. * **Periodic sync path** for security and ownership metadata that is handled as diff-based reconciliation. # Overview Source: https://docs.springtail.io/deployment This section covers deploying Springtail in a production environment. A production deployment groups the Springtail services into three node types: * **Ingestion Node** — ingest changes from the primary database. * **Proxy Node** — handle client connections and route queries. * **FDW Node(s)** — serve queries; there can be one or more depending on the scale of the deployment. All nodes share a Redis cache and a common storage layer underneath. ## High-level steps Deploying Springtail to production involves the following steps: 1. **[Configure PostgreSQL](/configuring-postgres)** — prepare the primary database for replication. 2. **[Configure Redis](/redis-configuration)** — set up the shared Redis cache used by all nodes. 3. **[Deploy with the Coordinator](/production-deployment)** — use the Springtail Coordinator to start Springtail, add and remove replica nodes, and stop it. For a local, single-machine setup intended for development and testing, see [Local Cluster Deployment](/local-cluster-deployment) instead. # Extension Support Source: https://docs.springtail.io/extension-support # PostgreSQL Extension Support System ## Overview Springtail provides a comprehensive PostgreSQL extension support system that enables loading and using PostgreSQL extensions (e.g., `pg_trgm`, PostGIS) within the Springtail replica. The system dynamically loads extension shared libraries, registers their types, operators, and operator classes, and provides a PostgreSQL-compatible runtime environment for executing extension functions. **Key Capabilities:** * Dynamic loading of PostgreSQL extension shared libraries (`.so` files) * Registration of extension types, operators, and operator classes * Invocation of extension functions through a compatibility layer * Support for GIN and GiST index opclasses (e.g., `gin_trgm_ops`, `gist_point_ops`) * PostgreSQL-compatible memory management and function call interface *** ## Architecture Components ### 1. Extension Registry (`PgExtnRegistry`) **Location:** `src/pg_ext/extn_registry.cc`, `include/pg_ext/extn_registry.hh` Central registry that manages all loaded extensions. Singleton pattern ensures global access. **Responsibilities:** * Load extension shared libraries using `dlopen()` * Query PostgreSQL system catalogs for extension metadata * Maintain mappings of OIDs to function pointers * Provide lookup APIs for types, operators, and opclass methods ### 2. PostgreSQL Compatibility Layer (`pg_ext/`) **Location:** `src/pg_ext/` and `include/pg_ext/` Minimal reimplementation of PostgreSQL internal APIs to provide a compatible runtime environment for extension functions. **Components:** * **fmgr**: Function manager - `DirectFunctionCall*()` wrappers * **memory**: Memory context management (`TopMemoryContext`, `palloc`, `pfree`) * **string**: String utilities (`text` type, `cstring_to_text()`) * **array**: PostgreSQL array handling * **numeric**: Numeric type support * **date/time**: Date and time types * **jsonb**: JSONB type support * **error**: Error reporting (`ereport`, `elog`) * **node**: PostgreSQL node types * **hash**: Hash functions * **list**: PostgreSQL list structures ### 3. Extension Initialization **Location:** `src/pg_repl/pg_copy_table.cc::init_pg_extn_registry()` Initialization flow during database setup that loads extensions configured in `system.json.settings`. **Process:** 1. Read extension configuration from `Properties::EXTENSION_CONFIG` 2. For each extension: * Load shared library (`.so` file) * Load types from `pg_type` system catalog * Load operators from `pg_operator` system catalog * Load opclasses from `pg_opclass` system catalog 3. Create extension type definitions in Springtail system tables *** ## Extension Loading Flow ```mermaid theme={null} flowchart TD Start["Application Startup"] Init["PgCopyTable::init_pg_extn_registry(db_id, xid)
Reads extension config from system.json
For each extension:"] Libs["PgExtnRegistry::init_libraries()
- dlopen() loads extension .so file
- Stores library handle in _library_map"] Types["_load_extn_types()
- Queries pg_type for extension types
- dlsym() loads type I/O functions (typinput, typoutput, typreceive, typsend)
- PgExtnRegistry::add_type()
- Creates extension types in Server::create_usertype()"] Ops["_load_extn_operators()
- Queries pg_operator for extension operators
- dlsym() loads operator implementation function
- PgExtnRegistry::add_operator()
- Stores operator_name → function_ptr mappings"] OpClasses["_load_extn_opclasses()
- Queries pg_opclass for GIN/GIST opclasses
- dlsym() loads support function (compress, penalty, union, etc.)
- PgExtnRegistry::add_opclass()
- Stores (opclass_name, support_number) → method mappings"] Start --> Init Init --> Libs Init --> Types Init --> Ops Init --> OpClasses ``` *** ## Configuration Extension configuration is stored in `system.json` under the `extension_config` key: ```json theme={null} { "extension_config": { "lib_path": "/usr/lib/postgresql/16/lib/", "": { "pg_trgm": {}, "postgis": {} } } } ``` **Fields:** * `lib_path`: Directory containing PostgreSQL extension `.so` files * ``: Database ID (as string) mapping to list of extension names * Extension names must match the `.so` filename (e.g., `pg_trgm` → `pg_trgm.so`) *** ## Extension Registry API ### Type Management ```cpp theme={null} // Add extension type and its I/O functions void add_type(const std::string& extension, uint32_t oid, const std::string& typinput, const std::string& typoutput, const std::string& typreceive, const std::string& typsend); // Get type metadata by OID PgType get_type_by_oid(uint32_t oid) const; // Get type I/O function by name void* get_type_func_by_type_name(const std::string& type_name) const; // Convert binary → Datum using typreceive Datum binary_to_datum(const std::span& value, Oid pg_oid, int32_t atttypmod) const; // Convert Datum → string using typoutput std::string datum_to_string(Datum value, Oid pg_oid) const; ``` ### Operator Management ```cpp theme={null} // Add extension operator void add_operator(const std::string& extension, uint32_t oid, const std::string& oper_name, const std::string& proc_name); // Get operator function by OID void* get_operator_func_by_oid(uint32_t oid) const; // Get operator function by operator name (e.g., "=", "<@") void* get_operator_func_by_oper_name(const char* oper_name) const; // Compare two values using extension operator static bool comparator_func(const ExtensionContext* context, const std::span& lhval, const std::span& rhval); ``` ### Operator Class Management ```cpp theme={null} // Add opclass method (e.g., GIN extractValue, GIST compress) void add_opclass(const std::string& extension, PgOpsClass opclass, PgOpsClassMethod method); // Get opclass method by name and support number PgOpsClassMethod get_opclass_method_by_method_name(const std::string& opclass_name, int support_number); // Invoke opclass method static Datum invoke_opclass_method(const std::string& opclass_name, int support_number, Datum value); ``` *** ## Usage Examples ### Example 1: Trigram Extraction (GIN) ```cpp theme={null} // Extract trigrams from text using pg_trgm extension #include std::vector extract_trigrams(const std::string& text) { auto registry = PgExtnRegistry::get_instance(); // Convert string to PostgreSQL text datum Datum text_datum = PointerGetDatum(cstring_to_text_auto(text.c_str())); // Get GIN extractValue method for gin_trgm_ops auto method = registry->get_opclass_method_by_method_name("gin_trgm_ops", GIN_EXTRACTVALUE); // Invoke extractValue function PGFunction func = (PGFunction)method.function_ptr; int32_t nentries = 0; Datum result = DirectFunctionCall3(func, text_datum, PointerGetDatum(&nentries), PointerGetDatum(nullptr)); // Process result... Datum* entries = (Datum*) DatumGetPointer(result); std::vector trigrams; for (int i = 0; i < nentries; i++) { // Unpack trigram integer to 3-byte string uint32_t trgm_int = DatumGetInt32(entries[i]); trigrams.push_back(unpack_trigram_int_to_string(trgm_int)); } return trigrams; } ``` ### Example 2: GIST Compress (Geometric) ```cpp theme={null} // Compress point for GIST index using gist_point_ops GistEntry compress_point(const Point& point) { auto registry = PgExtnRegistry::get_instance(); // Convert point to datum Datum point_datum = PointPGetDatum(&point); // Get GIST compress method auto method = registry->get_opclass_method_by_method_name("gist_point_ops", GIST_COMPRESS); // Setup GISTENTRY GISTENTRY entry; entry.key = point_datum; entry.leafkey = true; // Invoke compress function PGFunction func = (PGFunction)method.function_ptr; Datum compressed = DirectFunctionCall1(func, PointerGetDatum(&entry)); // Extract compressed result GISTENTRY* result = (GISTENTRY*) DatumGetPointer(compressed); return GistEntry{result->key, true}; } ``` ### Example 3: Extension Type Comparison ```cpp theme={null} // Compare two extension type values (e.g., PostGIS geometries) bool compare_extension_values(const std::span& lhs, const std::span& rhs, uint32_t type_oid, const char* operator_name) { ExtensionContext context; context.type_oid = type_oid; context.op_str = operator_name; return PgExtnRegistry::comparator_func(&context, lhs, rhs); } ``` *** ## System Catalog Queries The extension registry queries PostgreSQL system catalogs to discover extension metadata: ### Type Query ```sql theme={null} SELECT t.oid AS type_oid, split_part(t.typinput::regproc::text, '.', 2) AS type_input, split_part(t.typoutput::regproc::text, '.', 2) AS type_output, split_part(t.typreceive::regproc::text, '.', 2) AS type_receive, split_part(t.typsend::regproc::text, '.', 2) AS type_send FROM pg_type t WHERE t.oid IN ( SELECT objid FROM pg_depend d JOIN pg_extension e ON e.oid = d.refobjid WHERE e.extname = '' AND d.deptype = 'e' AND d.classid = 'pg_type'::regclass ); ``` ### Operator Query ```sql theme={null} SELECT opr.oid AS oper_oid, opr.oprname AS oper_name, proc.proname AS proc_name FROM pg_operator opr JOIN pg_proc proc ON proc.oid = opr.oprcode WHERE proc.oid IN ( SELECT objid FROM pg_depend d JOIN pg_extension e ON e.oid = d.refobjid WHERE e.extname = '' AND d.deptype = 'e' AND d.classid = 'pg_proc'::regclass ); ``` ### Opclass Query ```sql theme={null} SELECT am.amname AS access_method, opc.opcname AS opclass_name, ap.amprocnum AS support_number, p.proname AS support_function_name FROM pg_opclass opc JOIN pg_am am ON am.oid = opc.opcmethod JOIN pg_opfamily opf ON opf.oid = opc.opcfamily LEFT JOIN pg_amproc ap ON ap.amprocfamily = opf.oid LEFT JOIN pg_proc p ON p.oid = ap.amproc LEFT JOIN pg_extension ext ON ext.oid = ( SELECT refobjid FROM pg_depend WHERE objid = opc.oid ) WHERE am.amname IN ('gin', 'gist') AND ext.extname = ''; ``` *** ## Integration Points ### 1. Index Building **File:** `src/pg_log_mgr/indexer.cc` Uses extension registry to invoke opclass methods during index construction: * GIN: `extractValue` to tokenize values * GIST: `compress` to encode leaf entries ### 2. Index Scanning **File:** `src/pg_fdw/pg_fdw_mgr.cc` Uses extension registry for query execution: * GIN: `extractQuery` to extract search keys from query patterns * GIST: `consistent` to filter index entries (future) ### 3. Index Maintenance **File:** `src/sys_tbl_mgr/mutable_table.cc` Uses extension registry for incremental updates: * Applies opclass methods to new/modified rows * Maintains index consistency ### 4. Storage Layer **Files:** `src/storage/gist_helpers.cc`, `src/storage/mutable_btree.cc` Direct integration with extension registry for GIST operations: * `extract_gist_entry_from_tuple()`: Compress leaf values * `compute_gist_penalty()`: Calculate insertion cost * `compute_union()`: Merge predicates for internal nodes *** ## Implementation Notes ### Dynamic Linking * Uses `dlopen(RTLD_NOW | RTLD_GLOBAL)` for eager symbol resolution * RTLD\_GLOBAL required for cross-extension symbol visibility * Extension `.so` files must be compiled with PostgreSQL headers ### Memory Management * Extension functions expect PostgreSQL memory contexts * `pg_ext/memory.cc` provides `TopMemoryContext` and `palloc/pfree` * Memory contexts are simplified compared to full PostgreSQL ### Function Call Interface * `DirectFunctionCall*()` macros wrap extension function calls * FunctionCallInfo structure provides arguments and context * Collation and null handling supported ### Type System Integration * Extension types stored in Springtail system tables * Binary format compatibility with PostgreSQL wire protocol * Datum abstraction layer for type-agnostic operations *** ## Limitations and Future Work ### Current Limitations 1. **Limited pg\_ext Coverage**: Not all PostgreSQL internal APIs are implemented 2. **No Extension Updates**: Extension changes require restart 3. **Single-threaded dlopen**: Library loading not thread-safe 4. **No Unloading**: Extensions cannot be dynamically unloaded *** ### Common Issues **Issue:** `Failed to load library: undefined symbol` * **Cause:** Extension .so missing dependencies * **Fix:** Ensure all PostgreSQL libraries are in `LD_LIBRARY_PATH` **Issue:** `Failed to find function PGFunction ` * **Cause:** Function name mismatch or not exported * **Fix:** Verify function exists in .so using `nm -D .so` **Issue:** `No extension configuration found` * **Cause:** Missing or malformed `extension_config` in system.json * **Fix:** Add proper configuration with db\_id and extension names *** ## Related Documentation * [GIN Index Implementation](/gin-index-support) - GIN index with extension support * [GIST Index Implementation](/gist-index-support) - GIST index with extension support * pg\_ext Library Reference - PostgreSQL compatibility layer * Extension Registry API - Detailed API documentation # FAQ Source: https://docs.springtail.io/faq ### What problem does Springtail solve? Springtail addresses the challenges of scaling database performance without the need for complex and risky migrations. Traditional migrations often involve compatibility issues, extensive testing, and a significant time investment. Springtail eliminates these hurdles by providing scalable, distributed read replicas that dynamically adjust to demand, delivering high throughput when needed and reducing costs during periods of low usage. ### What databases does Springtail support? Springtail is known to work with vanilla PostgreSQL, Amazon RDS for PostgreSQL, Aurora PostgreSQL, and Supabase. ### Which versions of Postgres does Springtail support? Springtail is compatible with PostgreSQL versions 14.0 and higher. ### How will Springtail import my existing data? To import your data, simply grant Springtail replication access to your PostgreSQL database. Springtail will take a snapshot of your existing tables and enable logical replication to capture ongoing changes, ensuring a consistent and up-to-date view of your data. ### Do I need to install a Postgres extension for Springtail? No, Springtail does not require a PostgreSQL extension. ### What kind of performance can I expect? Springtail improves concurrent query performance. It offloads and scales read execution, which also frees up your existing primary database to process more write transactions in less time. Individual query latencies may be slightly slower or faster depending on the query workload. # Foreign Data Wrapper Source: https://docs.springtail.io/foreign-data-wrapper ## Overview Springtail's PostgreSQL Foreign Data Wrapper (FDW) lets a PostgreSQL backend query Springtail tables as if they were native relations. The implementation splits responsibility between the PostgreSQL-facing callbacks (planner/executor integration) and the Springtail manager layer that understands Springtail metadata, schemas, and storage semantics. This document intentionally excludes the DDL manager; it focuses on the runtime FDW path. ## Architecture Layers ### FDW-Facing Layer (`src/pg_fdw/pg_fdw.c`, `src/pg_fdw/multicorn_util.c`) * Implements PostgreSQL FDW planning and execution callbacks and translates planner/executor data structures into Springtail-friendly descriptors. * Parses foreign table/server options (`db_id`, `tid`, `schema_xid`, etc.) and builds the FDW-private state carried across callbacks. * Classifies predicates, determines projection lists, constructs `ForeignPath`/`ForeignScan` nodes, and provides EXPLAIN output that reflects pushdown decisions and visibility parameters. ### Springtail Manager Layer (`src/pg_fdw/pg_fdw_mgr.cc`) * Resolves table metadata, schemas, indexes, and user-defined types via `TableMgrClient` and shared-memory caches. * Maintains the mapping from PostgreSQL transaction IDs to Springtail XIDs, enforces snapshot visibility, and invalidates caches when `schema_xid` advances. * Chooses covering indexes, computes iterator bounds, and drives scan execution over Springtail `Table` iterators (including index-only scans when possible). * Supplies statistics and path-key metadata back to the FDW layer so PostgreSQL's optimizer can cost the foreign relation accurately. ## Springtail Manager Layer Internals ### Metadata and Index Discovery `PgFdwMgr::_create_scan_state` materializes a `PgFdwState` that caches column metadata, available indexes (primary plus ready secondaries), and attribute maps. `_compute_planning_metadata` discovers which indexes match equality predicates ("qual indexes") or join quals, so later callbacks can re-use that work without rebuilding state. This keeps planning deterministic even when multiple callbacks access the same relation. ### Transaction/XID Workflow A background thread (`PgFdwMgr::_internal_run`) polls Redis for the most recent `schema_xid`, asks `XidMgrClient` for committed XIDs, and forwards progress to `PgXidCollectorClient`. When PostgreSQL enters the FDW via `fdw_create_state`, the manager: 1. Ensures `_schema_xid` is monotonic and invalidates cached metadata if a higher schema version was requested. 2. Maps the calling PostgreSQL transaction (`pg_xid`) to the latest committed Springtail XID (`_trans_xid`), re-using the mapping if the same backend invokes multiple scans. 3. Blocks until the Springtail XID is at least as recent as `_last_xid` so snapshot reads never regress. 4. Stores the mapping until `fdw_commit_rollback` clears it at transaction end. This handshake guarantees that every scan inside a PostgreSQL transaction uses the same Springtail snapshot, which is critical for multi-table consistency. ### Schema and User-Type Cache Maintenance `_try_create_cache` connects the FDW process to shared-memory caches for table roots, schemas, and user types. Whenever `schema_xid` jumps forward inside `fdw_create_state`, the manager invalidates both the schema cache and the table cache to avoid serving stale column or index definitions. User-defined type lookups use an LRU cache; cache misses flow through `sys_tbl_mgr::Client::get_usertype` pinned to the caller's XID. ## PostgreSQL FDW Callback Responsibilities ### Planning Phase * **GetForeignRelSize**: builds/updates the plan state, retrieves row-count statistics from the manager, and caches qualifying/join indexes for later callbacks. Width estimation inspects the column metadata directly, so subsequent callbacks do not repeat catalog lookups. * **GetForeignPaths**: classifies predicates (pushable vs. local), evaluates selectivity, and constructs one or more `ForeignPath` entries. Costs incorporate Springtail-specific multipliers (primary vs. secondary index lookups, full scans, etc.) returned by the manager. * **GetForeignPlan**: converts the chosen path into a `ForeignScan`, capturing the target columns, pushed quals, remnant local quals, and the cached routing/snapshot info. The resulting plan node contains all data needed for execution without further catalog I/O. ### Execution Phase * **BeginForeignScan**: calls `PgFdwMgr::fdw_begin_scan`, which allocates a `PgFdwState`, records PostgreSQL attribute metadata, decides on index-only vs. table scans, and initializes iterator bounds plus filter structures derived from quals. * **IterateForeignScan**: repeatedly asks `fdw_iterate_scan` for the next tuple. The manager walks Springtail iterators (ascending or descending), applies residual filters inline, converts fields into PostgreSQL datums (including enum/extension mapping), and reports EOS when iterators meet. * **ReScanForeignScan**: invokes `fdw_reset_scan`, re-derives iterator bounds (qualifications may have changed in nested-loop rechecks), reapplies filters, and rewinds the iterators. * **EndForeignScan**: delegates to `fdw_end_scan`, which logs row statistics, clears tracing state, and deletes the `PgFdwState`. * **Utility callbacks** (`ExplainForeignScan`, `AnalyzeForeignTable`, `ImportForeignSchema`): each relies on manager helpers to enumerate indexes, produce human-readable filter descriptions, gather statistics, or synthesize DDL. ## Query Planning and Optimization ### Predicate Pushdown `multicorn_util.c` walks `baserestrictinfo` clauses, checking operator support, function volatility, and type compatibility through the manager's `_is_type_sortable` and `check_type_compatibility`. Pushable quals become part of the remote qualifier list passed to `PgFdwMgr`, which then converts them into constant fields for iterator bounds. Non-pushable quals remain as local filters inside `fdw_iterate_scan`. ### Column Projection The FDW builds an `attrs_used` bitmap from the target list, remote/local quals, join keys, and required visibility metadata. The manager translates that bitmap into a concrete field list, optionally sourced from the covering index schema. Minimizing projected columns reduces deserialization work even though Springtail scans operate inside the same process boundary (fewer field extractions and datum conversions). ### Index-Aware Planning and Execution * **Planning**: `_compute_planning_metadata` and `_get_index_quals` mark indexes that satisfy WHERE or JOIN equality clauses. `fdw_get_path_keys` then surfaces those matches (and their cardinality adjustments, such as `rows = 1` for unique hits) so PostgreSQL can cost access paths correctly. * **Execution**: `_init_quals` chooses the best available index (primary first, then secondaries or a planner-provided sort index), while `_set_scan_iterators` converts qual prefixes into `lower_bound`/`upper_bound` ranges—including special handling for `NOT_EQUALS`. If the projection list is fully covered by the chosen index, `fdw_begin_scan` enables index-only mode by pulling fields directly from the index schema, reducing I/O. `fdw_iterate_scan` applies any remaining filters inline so PostgreSQL only sees qualifying tuples. * **Sort Pushdown**: `fdw_can_sort` verifies whether a requested sort order aligns with the chosen index (respecting direction and NULLS FIRST rules). When it does, the manager records the sort index so execution can scan in that order, eliminating extra Sort nodes in PostgreSQL. ## Consistency and Snapshot Isolation * **schema\_xid** freezes the schema version used for metadata resolution. If a higher value is requested, caches are invalidated before scan state is built, ensuring column/index definitions match the caller's expectation. * **pg\_xid** (from PostgreSQL) is mapped to a Springtail XID via `_update_last_xid`, guaranteeing that every scan within a backend observes the same snapshot. The background thread keeps `_last_xid` monotonic and notifies the XID collector whenever a newer committed XID is observed. * **Replica safety** is achieved indirectly: scans block until the committed Springtail XID is at least as recent as `_last_xid`, preventing reads from replicas that are behind the requested snapshot. ## Execution Flow Summary 1. `fdw_init` wires up logging, properties, and clients; `init()` connects to caches and starts the background XID thread. 2. Planning callbacks gather statistics, candidate indexes, and width estimates without materializing full scan state. 3. `fdw_create_state` (triggered from `GetForeignPlan`) maps the PostgreSQL transaction to a Springtail XID and returns FDW-private data stored in the plan node. 4. `fdw_begin_scan` builds a `PgFdwState`, selects/initializes iterators, and determines index-only vs. table scans. 5. `fdw_iterate_scan` streams tuples by advancing iterators, applying residual quals, converting datums, and updating per-scan stats. 6. `fdw_reset_scan` and `fdw_end_scan` rewind or tear down the state; `fdw_commit_rollback` clears transaction mappings when PostgreSQL ends the transaction. ## Operational Considerations * **Qual selectivity**: Because iterator bounds depend on deterministic quals, ensuring predicates use supported operators (simple comparisons, immutable expressions) maximizes index usage. * **Projection discipline**: Wide projections prevent index-only scans; trimming SELECT lists (and avoiding unused columns in quals) keeps scans in index-only mode more often. * **Monitoring**: EXPLAIN output from `fdw_explain_scan` lists chosen indexes, filters, and scan types, making it easier to diagnose why a query fell back to full scans or local filtering. # GIN Index support Source: https://docs.springtail.io/gin-index-support ## Overview > **Note:** This feature is currently a work in progress. Index building and maintenance are complete; scan implementation is partially implemented. Currently in the branches: SPR-1035-GIN-support-2 (build), SPR-1035-GIN-support-2-scan (scan) GIN (Generalized Inverted Index) support enables efficient text similarity searches using trigram-based indexing. The implementation tokenizes text column values into 3-character trigrams and stores them in an inverted index structure, mapping each trigram to the rows containing it. The index schema stores entries as `(column_position, token, internal_row_id)` tuples, allowing multi-column GIN indexes where each column's trigrams are distinguished by position. Currently, only `gin_trgm_ops` opclass is supported, enabling `LIKE` and `ILIKE` query operators. *** ## Key Components ### Index Building | Component | Location | Purpose | | ------------------ | ----------------------------------- | --------------------------------------------------------------- | | Trigram Helpers | `src/pg_ext/trgm_helpers.cc` | Extracts trigrams from text values using opclass `extractValue` | | GIN Schema Builder | `src/sys_tbl_mgr/schema_helpers.cc` | Creates the 3-column GIN index schema | | GIN Index Root | `src/sys_tbl_mgr/mutable_table.cc` | Initializes BTree configured for GIN storage | | Index Builder | `src/pg_log_mgr/indexer.cc` | Full build with reconciliation | ### Index Scanning (in-progress) | Component | Location | Purpose | | --------------------- | ---------------------------------- | --------------------------------------------------------- | | Query Helpers | `src/pg_fdw/trgm_query_helpers.cc` | Extracts trigrams from query using opclass `extractQuery` | | GINSecondary Iterator | `src/sys_tbl_mgr/table.cc` | Iterates GIN index with row deduplication | | FDW Integration | `src/pg_fdw/pg_fdw_mgr.cc` | Routes LIKE/ILIKE operators to GIN index scan | ### GIN Index Schema | Field | Type | Description | | ------------------------------ | ------ | ----------------------------- | | `__springtail_idx_position` | UINT32 | Column position in the table | | `__springtail_gin_idx_token` | TEXT | Trigram token (3-byte string) | | `__springtail_internal_row_id` | UINT64 | Reference to the source row | ### Supported Operators | Operator | Strategy Number | Description | | --------------- | --------------- | --------------------------------- | | `LIKE` (`~~`) | 3 | Case-sensitive pattern matching | | `ILIKE` (`~~*`) | 4 | Case-insensitive pattern matching | *** ## Data Flow ### Index Building ```mermaid theme={null} flowchart TD subgraph Creation [INDEX CREATION] direction TB Create["Server::_create_index()
→ _check_gin_index_columns() validates opclass
→ _upsert_index_name() persists metadata"] BuildIdx["Indexer::_build_index()
→ Detects INDEX_TYPE_GIN
→ Calls _build_gin_index()"] BuildGin["Indexer::_build_gin_index()
(builds index at XID x1 in a separate Indexer thread)
→ create_gin_index_root() initializes BTree
→ For each row, for each indexed column:
→ extract_trgm_from_value() extracts trigrams
→ Insert (position, token, row_id) into BTree"] Create --> BuildIdx --> BuildGin end subgraph Recon [RECONCILIATION] direction TB ReconStep["committer triggers process_index_reconciliation()
→ Calls _reconcile_index()
→ Reconciles index using new mutations after XID x1
→ Insert (position, token, row_id) for inserts/updates
→ Remove (position, token, row_id) for deletes"] end subgraph Maint [INCREMENTAL MAINTENANCE] direction TB Apply["MutableTable::apply_mutation<INSERT/DELETE>()
→ index_mutation_handler() checks index type via _index_lookup
→ For GIN: extracts trigrams, inserts/removes tuples
→ For BTree: standard key-value operation"] end Creation --> Recon --> Maint ``` ### Index Scanning (in-progress) ```mermaid theme={null} flowchart TD Query["QUERY: SELECT * FROM t WHERE col LIKE '%pattern%'"] InitQuals["FDW::_init_quals()
→ Selects GIN index with matching column"] SetIter["FDW::_set_scan_iterators()
→ extract_gin_keys_from_string() extracts query trigrams
→ Table::begin(index_id, tokens)"] Iter["Table::Iterator (GINSecondary)
→ Iterates BTree entries matching tokens
→ Deduplicates rows via _visited_internal_row_ids
→ Resolves row location via look_aside_index
→ Returns matching table rows"] Query --> InitQuals --> SetIter --> Iter ``` *** ## Implementation ### Index Creation GIN index creation validates opclass before persisting metadata: ```cpp theme={null} // server.cc - Server::_check_gin_index_columns() for (auto& idx_column : index_info.columns()) { if (std::ranges::find(ALLOWED_GIN_OPS, idx_column.opclass()) == std::end(ALLOWED_GIN_OPS)) { LOG_ERROR("Unsupported opclass '{}' for GIN index column", idx_column.opclass()); return false; } } ``` ### Trigram Extraction (Build) Unpacks PostgreSQL packed trigram integers to 3-byte strings: ```cpp theme={null} // trgm_helpers.cc - unpack_trigram_int_to_string() std::string unpack_trigram_int_to_string(uint32_t v) { unsigned char b1 = (v >> 16) & 0xFF; unsigned char b2 = (v >> 8) & 0xFF; unsigned char b3 = v & 0xFF; return std::string({b1, b2, b3}); } // trgm_helpers.cc - extract_trgm_from_value() auto&& extract_func = PgExtnRegistry::get_instance() ->get_opclass_method_func_ptr_by_method_name(opclass, method_strategy_number); Datum result = extract_func(fcinfo); Datum *entries = reinterpret_cast(DatumGetPointer(result)); ``` ### Full Index Build and reconciliation Iterates table rows and inserts trigram tuples: ```cpp theme={null} // indexer.cc - Indexer::_build_gin_index() for (auto row_i = table->begin(); row_i != table->end(); ++row_i) { auto&& row = *row_i; for (int i = 0; i < idx_cols.size(); i++) { auto&& tokens = extract_trgm_from_value(col_field->get_text(&row), column.opclass(), GIN_EXTRACTVALUE); for (auto& token : tokens) { // Key: (column_position, trigram_token, internal_row_id) key_fields->at(0) = std::make_shared>(pos); key_fields->at(1) = std::make_shared>(token); key_fields->at(2) = std::make_shared>(internal_row_id); root->insert(std::make_shared(key_fields, nullptr)); } } } Reconciliation // indexer.cc - Indexer::_reconcile_index() auto&& tokens = extract_trgm_from_value(std::string(col_field->get_text(&row)), column.opclass(), GIN_EXTRACTVALUE); for (auto& token: tokens) { // Set idx_position, token, internal_row_id key_fields->at(0) = std::make_shared>(pos); key_fields->at(1) = std::make_shared>(token); key_fields->at(2) = std::make_shared>(internal_row_id_f->get_uint64(&row)); auto tuple = std::make_shared(key_fields, nullptr); --- // for inserts/updates idx_state._root->insert(tuple); and // For deletes idx_state._root->remove(tuple); } ``` ### Incremental Maintenance Mutation handler distinguishes GIN from BTree indexes: ```cpp theme={null} // mutable_table.cc - index_mutation_handler() if (index_lookup.at(index_id).index_type == constant::INDEX_TYPE_GIN) { auto&& tokens = extract_trgm_from_value(col_field->get_text(&row), column.opclass, GIN_EXTRACTVALUE); for (auto& token : tokens) { key_fields->at(0) = std::make_shared>(pos); key_fields->at(1) = std::make_shared>(token); key_fields->at(2) = std::make_shared>(internal_row_id); if constexpr (op == IndexOperation::Insert) idx.first->insert(tuple); else idx.first->remove(tuple); } } ``` ### Schema Creation GIN index schema defines three key columns: ```cpp theme={null} // schema_helpers.cc - create_gin_index_schema() SchemaColumn idx_position_c(constant::INDEX_POSITION_FIELD, 0, SchemaType::UINT32, 0, false); SchemaColumn idx_gin_token_c(constant::INDEX_GIN_TOKEN_FIELD, 0, SchemaType::TEXT, 0, false); SchemaColumn internal_row_id(constant::INTERNAL_ROW_ID, 0, SchemaType::UINT64, 0, false); return base_schema->create_index_schema({}, { idx_position_c, idx_gin_token_c, internal_row_id }, gin_index_keys, extension_callback); ``` ### Trigram Extraction (Query) Invokes opclass `extractQuery` for search pattern: ```cpp theme={null} // trgm_query_helpers.cc - extract_gin_keys_from_string() Oid procOid = get_opfamily_proc(opfamily, inputType, inputType, GIN_EXTRACTQUERY_PROC); Datum *keys = (Datum *) DatumGetPointer( FunctionCall7Coll(&flinfo, collation, queryDatum, PointerGetDatum(&nkeys), UInt16GetDatum(op_strategy_number), PointerGetDatum(&partial_matches), PointerGetDatum(&extra_data), PointerGetDatum(&nullFlags), PointerGetDatum(&searchMode))); ``` ### GIN Iterator (in-progress) Deduplicates rows and resolves physical location: ```cpp theme={null} // table.cc - GINSecondary::update_page() while (true) { auto&& index_row = *_btree_i; internal_row_id = _internal_row_id_f->get_uint64(&index_row); // Skip already visited rows (same row may match multiple trigrams) if (!_visited_internal_row_ids.contains(internal_row_id)) { _visited_internal_row_ids.emplace(internal_row_id); break; } ++_btree_i; } // Resolve row location via look-aside index _look_aside_key_fields->at(0) = std::make_shared>(internal_row_id); auto lookup_tuple = std::make_shared(_look_aside_key_fields, nullptr); auto&& lookup_i = look_aside_index->lower_bound(lookup_tuple); ``` ### FDW Query Routing (in-progress) Routes LIKE/ILIKE operators to GIN index: ```cpp theme={null} // pg_fdw_mgr.cc - _iter_start() case LIKE: case ILIKE: auto search_key = tuple->to_string(); auto tokens = extract_gin_keys_from_string(search_key, "gin_trgm_ops", 100, TRGM_LIKE_STRATEGY_NUMBER); state->iter_start.emplace(state->table->begin(state->index->id, state->index_only_scan, tokens)); state->iter_end.emplace(state->table->end(state->index->id, state->index_only_scan)); break; // pg_fdw_mgr.cc - _is_valid_qual() case TEXTOID: return (op == EQUALS || op == NOT_EQUALS || op == LIKE || op == ILIKE); ``` *** ## Path to Completion The following work remains to complete GIN index support: 1. **Scan Implementation** - The `GINSecondary` iterator currently iterates all index entries. It needs to: * Filter entries to only those matching the extracted query tokens * Implement proper token intersection logic (all query trigrams must match) 2. **Query Optimization** - The FDW currently uses hardcoded `gin_trgm_ops` and collation. This should be derived from the index metadata. 3. **Testing** - End-to-end testing of LIKE/ILIKE queries using GIN indexes. # GIST Index support Source: https://docs.springtail.io/gist-index-support ## Overview > **Note:** This feature is currently a work in progress. Index building and maintenance are partially implemented. Currently in branch: SPR-1036-GiST GiST (Generalized Search Tree) support enables efficient indexing for geometric data types, range types, and other complex data structures. GiST is a balanced tree structure that allows custom operator classes to define how data is organized and searched within the index. The implementation uses PostgreSQL's GiST opclass methods (compress, penalty, union, picksplit, consistent) to build and maintain a tree structure where: * **Leaf nodes** store the actual indexed values and references to table rows * **Internal (branch) nodes** store predicates (bounding boxes, ranges, etc.) that cover all entries in their subtrees Currently supports various GiST opclasses including `gist_point_ops` for geometric point data. *** ## Key Components ### Index Building | Component | Location | Purpose | | ------------------- | ----------------------------------- | --------------------------------------------------------------- | | GiST Helpers | `src/storage/gist_helpers.cc` | Opclass method invocation (compress, penalty, union, picksplit) | | GiST Schema Builder | `src/sys_tbl_mgr/schema_helpers.cc` | Creates GiST index schema | | GiST Index Root | `src/sys_tbl_mgr/mutable_table.cc` | Initializes MutableBTree configured for GiST storage | | Insertion Logic | `src/storage/mutable_btree.cc` | Penalty-based subtree selection and insertion | | Cache Layer | `src/storage/cache.cc` | GiST-specific page insertion with subtree selection | ### GiST Index Schema GiST indexes use the same schema structure as regular B-tree indexes: | Field | Type | Description | | ------------------------------ | ------- | ---------------------------------------------- | | Indexed columns | Various | One or more columns with GiST-compatible types | | `__springtail_internal_row_id` | UINT64 | Reference to the source row | **Internal nodes** additionally store: * **Predicate/Union keys**: Compressed representations (e.g., bounding boxes) covering all child entries * **Child page ID**: Reference to the child page ### GiST Data Structure ```cpp theme={null} struct GistEntry { std::vector keys; // Datum per indexed column (compressed) bool leafkey; // true for leaf entries, false for internal uint64_t internal_row_id; // For leaf: row pointer; for internal: child page ID }; ``` ### Supported Opclass Methods | Method | Support Number | Purpose | | ----------------- | -------------- | ----------------------------------------------------- | | `GIST_CONSISTENT` | 1 | Determine if entry satisfies query predicate | | `GIST_UNION` | 2 | Compute union/bounding predicate of entries | | `GIST_COMPRESS` | 3 | Convert leaf value to compressed index representation | | `GIST_DECOMPRESS` | 4 | Convert index representation back to original form | | `GIST_PENALTY` | 5 | Compute penalty for inserting entry into subtree | | `GIST_PICKSPLIT` | 6 | Split overflowing page into two | | `GIST_EQUAL` | 7 | Check equality of keys | | `GIST_DISTANCE` | 8 | Compute distance for KNN searches | *** ## Data Flow ### Index Building ```mermaid theme={null} flowchart TD Create["INDEX CREATION"] CreateIdx["Server::_create_index()
→ Validates opclass compatibility
→ _upsert_index_name() persists metadata"] Root["MutableTable::create_gist_index_root()
→ Creates schema via create_gist_index_schema()
→ Initializes MutableBTree with INDEX_TYPE_GIST
→ Stores opclass names for each indexed column"] BuildIdx["Indexer::_build_index()
→ Detects INDEX_TYPE_GIST
→ Calls build logic (to be implemented)"] Create --> CreateIdx --> Root --> BuildIdx ``` ### Index Insertion ```mermaid theme={null} flowchart TD Op["INSERT/UPDATE Operation"] Insert["MutableBTree::insert()
→ Detects INDEX_TYPE_GIST
→ extract_gist_entry_from_tuple()
— For each indexed column:
— make_datum_from_field() converts Springtail field to Datum
— Invokes GIST_COMPRESS via opclass method
→ Returns GistEntry with compressed keys"] PageInsert["Page::insert_gist()
→ Marks page as dirty
→ Delegates to StorageCache::Page::insert_gist()"] Choose["StorageCache::Page::gist_choose_subtree()
→ For each child page (internal nodes):
— read_branch_entry_from_row() extracts child predicate
— compute_gist_penalty() calculates insertion cost
— Selects child with minimum penalty
→ Returns iterator to chosen subtree"] CacheInsert["StorageCache::Page::insert_gist()
→ Inserts tuple into chosen extent/subtree
→ (Split logic and tree rebalancing: TBD)"] Op --> Insert --> PageInsert --> Choose --> CacheInsert ``` ### Incremental Maintenance ``` MutableTable::apply_mutation() → index_mutation_handler() checks index type via _index_lookup → For GIST: follows standard insertion path → For DELETE: removal logic (to be implemented) ``` *** ## Implementation ### GistEntry Extraction Converts table tuple to compressed GiST index entry: ```cpp theme={null} // gist_helpers.cc - extract_gist_entry_from_tuple() GistEntry extract_gist_entry_from_tuple(TuplePtr tuple, ExtentSchemaPtr schema, const std::vector& opclass_names) { GistEntry out; out.leafkey = true; for (std::size_t idx = 0; idx < opclass_names.size(); ++idx) { if (opclass_names[idx] == "EMPTY") continue; // Get raw datum from field FieldPtr field = tuple->field(idx); Datum raw = make_datum_from_field(field, tuple->row()); // Apply GIST_COMPRESS auto opclass_method = PgExtnRegistry::get_instance() ->get_opclass_method_by_method_name(opclass_names[idx], GIST_COMPRESS); if (opclass_method.function_ptr) { GISTENTRY entry; entry.key = raw; entry.leafkey = true; Datum compressed = DirectFunctionCall1(func, PointerGetDatum(&entry)); GISTENTRY *retval = (GISTENTRY *) DatumGetPointer(compressed); out.keys.push_back(retval ? retval->key : raw); } else { out.keys.push_back(raw); } } return out; } ``` ### Penalty Computation Determines best subtree for insertion using opclass penalty method: ```cpp theme={null} // gist_helpers.cc - compute_gist_penalty() double compute_gist_penalty(const GistEntry& existing_entry, const GistEntry& new_entry, const std::vector& opclass_names) { double total_penalty = 0.0; for (std::size_t idx = 0; idx < opclass_names.size(); ++idx) { auto opclass_method = PgExtnRegistry::get_instance() ->get_opclass_method_by_method_name(opclass_names[idx], GIST_PENALTY); if (!opclass_method.function_ptr) continue; GISTENTRY orig; orig.key = existing_entry.keys[idx]; orig.leafkey = existing_entry.leafkey; GISTENTRY newe; newe.key = new_entry.keys[idx]; newe.leafkey = new_entry.leafkey; float penalty = 0.0; DirectFunctionCall3(func, PointerGetDatum(&orig), PointerGetDatum(&newe), PointerGetDatum(&penalty)); total_penalty += penalty; } return total_penalty; } ``` ### Subtree Selection Chooses optimal child page for insertion based on penalty: ```cpp theme={null} // cache.cc - StorageCache::Page::gist_choose_subtree() Iterator gist_choose_subtree(const GistEntry& entry, ExtentSchemaPtr schema, const std::vector& opclass_names) { double best_penalty = std::numeric_limits::max(); auto best_it = _extents.end(); for (auto it = _extents.begin(); it != _extents.end(); ++it) { auto extent = it->make_safe_extent(_file, _database_id); auto &&row = (*extent)->back(); // Branch tuple at end GistEntry child = gist_helpers::read_branch_entry_from_row(row, schema, opclass_names); double p = gist_helpers::compute_gist_penalty(child, entry, opclass_names); if (p < best_penalty) { best_penalty = p; best_it = it; } } // Return iterator to chosen extent auto chosen_extent = best_it->make_dirty_safe_extent(_file, _database_id); return Iterator(this, best_it, std::move(chosen_extent), ...); } ``` ### Union Computation (for Internal Nodes) Computes bounding predicate covering all child entries: ```cpp theme={null} // gist_helpers.cc - compute_union() void compute_union(const std::vector& entries, GistEntry& union_entry, const std::vector& opclass_names) { union_entry.leafkey = false; // Union creates internal node entry for (std::size_t idx = 0; idx < opclass_names.size(); ++idx) { auto opclass_method = PgExtnRegistry::get_instance() ->get_opclass_method_by_method_name(opclass_names[idx], GIST_UNION); if (!opclass_method.function_ptr) continue; // Prepare GistEntryVector with all entries for this column GistEntryVector* vec = ...; // Allocate vec->n = entries.size(); for (size_t i = 0; i < entries.size(); ++i) { vec->vector[i].key = entries[i].keys[idx]; vec->vector[i].leafkey = entries[i].leafkey; } int out_size = 0; Datum result = DirectFunctionCall2(func, PointerGetDatum(vec), PointerGetDatum(&out_size)); union_entry.keys.push_back(result); } } ``` ### Schema Creation GiST index schema currently delegates to standard index schema: ```cpp theme={null} // schema_helpers.cc - create_gist_index_schema() ExtentSchemaPtr create_gist_index_schema(ExtentSchemaPtr base_schema, const std::vector& index_columns, uint64_t index_id, const ExtensionCallback& extension_callback, const Index& index) { // Currently uses standard index schema return create_index_schema(base_schema, index_columns, index_id, extension_callback); // Future: May need custom schema for internal node predicates } ``` ### MutableBTree Insertion ```cpp theme={null} // mutable_btree.cc - MutableBTree::insert() void MutableBTree::insert(TuplePtr value) { if (_index_type == constant::INDEX_TYPE_GIST) { LOG_INFO("Inserting value into GIST index"); // Extract and compress entry GistEntry entry = gist_helpers::extract_gist_entry_from_tuple( value, _leaf_schema, _opclass_names); // Insert via page NodePtr node = std::make_shared(nullptr, _root); node->page->insert_gist(entry, value); return; } // Standard B-tree insertion... } ``` *** ## Path to Completion The following work remains to complete GiST index support: ### 1. **Index Building** - Full initial index build * Implement `Indexer::_build_gist_index()` to iterate table and build initial tree * Handle tree construction with proper internal node creation using UNION * Implement reconciliation logic for mutations after initial build XID ### 2. **Page Splitting** - Handle overflow during insertion * Implement `compute_picksplit()` to split overflowing pages * Currently commented out in `gist_helpers.cc` * Use `GIST_PICKSPLIT` opclass method to partition entries * Update parent nodes with new predicates after split * Maintain tree balance ### 3. **Tree Navigation** - Complete subtree selection * Current implementation only handles single-level selection * Need recursive descent for multi-level trees * Proper leaf node detection and insertion ### 4. **Index Scanning** - Query execution using GiST index * Implement `GIST_CONSISTENT` method invocation for query predicates * Create GiST iterator for index scans * Support various search operators (`<@`, `&&`, `~`, etc.) * Implement KNN search using `GIST_DISTANCE` method ### 5. **Deletion and Maintenance** * Implement entry removal from GiST tree * Handle internal node updates when children change * Tree rebalancing and page merging * Vacuum support for GiST indexes ### 6. **Internal Node Management** * Proper storage and retrieval of internal node predicates * Union computation during page splits and merges * Predicate updates when child pages change ### 7. **Opclass Validation** * Implement validation in `Server::_check_gist_index_columns()` * Similar to GIN's `ALLOWED_GIN_OPS` list * Verify required methods are available for each opclass ### 8. **Testing** * End-to-end testing with `gist_point_ops` and geometric queries * Test files exist in `python/testing/proxy/tests/sql/gist.sql` * Validate tree structure and correctness * Performance testing with large datasets *** ## Current Status **Implemented:** * ✅ GistEntry structure and helper functions * ✅ Opclass method invocation framework (compress, penalty, union) * ✅ Basic insertion path with penalty-based subtree selection * ✅ Schema creation infrastructure * ✅ Field ↔ Datum conversion utilities **Partially Implemented:** * ⚠️ Page insertion (single-level only, no splits) * ⚠️ Union computation (implemented but not used in tree building) **Not Implemented:** * ❌ Full index building (Indexer integration) * ❌ Page splitting (picksplit) * ❌ Multi-level tree navigation * ❌ Index scanning and query execution * ❌ Deletion and maintenance * ❌ Internal node management *** ## Design Notes ### Differences from B-tree Indexes Unlike B-tree indexes which store (key, value) pairs: * **Leaf nodes** store compressed representations of indexed values * **Internal nodes** store predicates (unions/bounds) rather than actual values * **Insertion** uses penalty-based selection rather than key comparison * **Splitting** uses custom picksplit logic rather than median split ### Opclass Integration GiST heavily relies on PostgreSQL opclass methods: * `PgExtnRegistry` provides access to extension-defined opclass methods * Each indexed column can have a different opclass * Methods operate on `Datum` values (PostgreSQL's internal representation) * Conversion between Springtail's `Field` types and `Datum` is crucial ### Storage Considerations * **Leaf entries**: Store compressed keys + internal\_row\_id * **Branch entries**: Store union predicates + child page ID * Current schema treats both uniformly; may need distinction for internal nodes * Extent-based storage may need adaptation for variable-sized predicates # Git Workflow Source: https://docs.springtail.io/git-workflow This page describes the development process for working on Springtail, from issue creation through merge. ## Dev Process * **Issue Creation:** * Create a new issue in GitHub. * Clearly describe the feature's functionality and purpose. * Include relevant details, user stories, or acceptance criteria. * **Assign the issue to yourself if you will be working on it.** * **Branching:** * Once the issue is created, create a new development branch based on the `main` branch (`git checkout -b branch_name`). * Use a descriptive branch name that references the issue number (e.g., `123-feature` or `123-issue`). * **Push the newly created branch to your remote repository on GitHub (`git push -u`).** * If the above push command does not work, do `git push --set-upstream origin ` * If something is wrong and you need to delete remote branch, do `git push -d origin ` * To delete local branch, do `git branch -d ` * To see the list of remote branches and to verify that you have created a remote branch, do `git branch -r` * **Development:** * Implement the feature code within your development branch. * Make frequent commits with clear and concise commit messages that describe the specific changes made. * The commit message should reference the GitHub issue, e.g., * `git commit -m "Fix bug reported ref #123"` * Use unit tests to ensure the functionality of your code. * Run all tests locally prior to submitting a Pull Request. * From root of the build tree (e.g., in `springtail/debug/`) run `make test` * **Pull Request Creation:** * Once development is complete, create a Pull Request (PR) from your development branch to the `main` branch. * Reference the issue number in the PR title or description to link the feature to the initial request. * Provide a clear summary of the changes made and the purpose of the feature. * Request code reviews from other developers. * **Review and Merge:** * Address any feedback or suggestions received during the code review process. * Make necessary code changes and push them to your development branch. * Rerun all unit and integration tests. * Once the code is approved for merging, contact @Garth or @Craig to merge into main. * A **squash merge** option should be used to merge your changes into the `main` branch. This creates a single commit representing the entire feature development. * **Bring your branch up to date with main**: * Perform the following commands (best to have done a `git commit` locally before merging): ```bash theme={null} git pull git merge origin/main git push ``` * Pull in updates from a branch other than main * Perform the following commands: ```bash theme={null} git stash # optional - saves uncommited changes git pull git merge origin/ # will apply the changes from the branch git push # optional - only if you want to apply the same change to the remote branch git stash pop # optional - get the uncommited changes back and merge if needed ``` * If after merge command you want to revert to the pre-merge state, run `git reset --hard ORIG_HEAD` * To see the diff of the last commit, do `git diff HEAD^` * If you want a remote branch with the name different than your local branch: * Create branch: `git push --set-upstream origin :` * Push to remote branch: `git push origin HEAD:` * Cleanup remote branch view from git: `git remote prune origin` * Delete local branch: `git branch -D ` * Restore uncommitted, but not staged file: `git restore ` * Restore uncommitted, but staged file: `git restore --staged ` # Glossary Source: https://docs.springtail.io/glossary A single database that runs on a database instance. Database host machine, may contain multiple databases. Replication at the level of operations; describes the operation, the arguments and data (e.g., an `UPDATE` is applied to row X setting column Y = Z). Replication of the on-disk data structures after an operation has been applied. The customer's database within a database instance that is being replicated. The source of truth for data for all copies of that database. Log used by a database to log database operations. Used for recovery replay or replication. # Index Support Source: https://docs.springtail.io/index-support ## Overview The `Indexer` class is the core component responsible for building and managing secondary indexes on tables in the Springtail database system. It operates as a multi-threaded service within the `springtail::committer` namespace, handling index creation, deletion, and abort operations asynchronously via worker threads. Recovery and reconciliation operations are synchronous—the Committer waits for them to complete before proceeding. ### Responsibilities * **Building secondary indexes** by scanning table data * **Building look-aside indexes** to reduce write amplification (see [Look-Aside Index](#look-aside-index) for details) * **Handling index drops** - both immediate drops and drops while a build is in progress * **Recovering incomplete index operations** after system crashes or shutdowns * **Reconciling indexes** with data changes that occurred during the build phase * **Coordinating with committer** to signal when index operations are ready for commit ### Integration with Committer The Indexer is owned and managed by the `Committer` class: 1. **Initialization**: Committer creates the Indexer in `run()` with a configurable worker count 2. **Request routing**: Committer calls `process_requests()` after batching index requests across XIDs 3. **Recovery trigger**: Committer invokes `recover_indexes()` when it receives an `INDEX_RECOVERY_TRIGGER` message 4. **Reconciliation**: Committer calls `process_index_reconciliation()` when it receives a `RECONCILE_INDEX` message *** ## Key Components ### Index Status States The lifecycle of an index operation is tracked through three states: ```cpp theme={null} enum class IndexStatus { BUILDING, // Default state - index build is in progress DELETING, // Fresh drop request for an index not currently being built ABORTING // Drop requested while build is in progress }; ``` **State Transitions:** ```mermaid theme={null} flowchart TD BUILDING(["BUILDING"]) DELETING1(["DELETING"]) ABORTING(["ABORTING"]) Pending["pending reconciliation"] Recon["_reconcile_index()"] DELETING2(["DELETING
_drop()"]) ABORTING2(["ABORTING
_commit_build() (truncate)"]) BUILDING2(["BUILDING
_commit_build() (finalize)"]) DELETED1(["DELETED"]) DELETED2(["DELETED"]) READY(["READY"]) Start1["create_index"] --> BUILDING Start2["drop_index
(index NOT in _work_set)"] --> DELETING1 BUILDING -->|"drop_index (index IN _work_set)"| ABORTING BUILDING -->|build complete| Pending ABORTING --> Pending DELETING1 --> Pending Pending --> Recon Recon --> DELETING2 Recon --> ABORTING2 Recon --> BUILDING2 DELETING2 --> DELETED1 ABORTING2 --> DELETED2 BUILDING2 --> READY ``` **Notes:** * `create_index` → `BUILDING`: Default state when an index build is initiated * `drop_index` on index in `_work_set` → `ABORTING`: Build in progress, mark for abort * `drop_index` on index NOT in `_work_set` → `DELETING`: Fresh drop, no build in progress * `_reconcile_index()` checks the current status and decides: * `DELETING` → calls `_drop()` → `DELETED` * `ABORTING` → calls `_commit_build()` (truncate) → `DELETED` * `BUILDING` → calls `_commit_build()` (finalize) → `READY` ### Core Data Structures #### IndexParams Encapsulates all parameters needed for an index operation: ```cpp theme={null} struct IndexParams { uint64_t _db_id; // Database identifier uint64_t _xid; // Transaction ID when index was created proto::IndexProcessRequest _index_request; // Protobuf request containing index metadata + operation request IndexStatus _status = IndexStatus::BUILDING; // Current operation status }; ``` #### IndexState Captures the state of an index after initial build phase, used during reconciliation: ```cpp theme={null} struct IndexState { MutableBTreePtr _root; // B-tree root for the secondary index Key _key; // (db_id, index_id) identifier pair IndexParams _idx; // Original index parameters uint64_t _tid; // Table ID this index belongs to MutableBTreePtr _look_aside_root; // Look-aside index root (may be nullptr) }; ``` #### Key Type Unique identifier for work items: ```cpp theme={null} using Key = std::pair; // (db_id, index_id) ``` ### Internal Maps and Queues | Member | Type | Purpose | | --------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------- | | `_work_set` | `unordered_map` | Active index operations keyed by (db\_id, index\_id) | | `_queue` | `queue` | FIFO queue of keys for worker threads to process | | `_pending_idx_reconciliation_map` | `db_id → xid → list` | Indexes awaiting reconciliation, grouped by database and XID | | `_table_idx_map` | `db_id → table_id → list` | Maps tables to their in-progress index builds (for abort\_indexes) | | `_xid_ddl_counter_map` | `xid → atomic` | Tracks pending DDL operations per transaction | | `_look_aside_build_tracker` | `table_id → bool` | Prevents duplicate look-aside index builds when multiple indexes created concurrently | ### Synchronization Primitives | Mutex | Protects | | --------------------------------- | ------------------------------------------------------ | | `_m` | `_work_set`, `_queue`, condition variable coordination | | `_pending_reconciliation_map_mtx` | `_pending_idx_reconciliation_map` | | `_table_idx_map_mtx` | `_table_idx_map` | | `_xid_ddl_counter_map_mtx` | `_xid_ddl_counter_map` | | `_look_aside_mutex` | `_look_aside_build_tracker` | *** ## Data Flow ### Index Creation Flow ```mermaid theme={null} flowchart TD CB["Committer._commit_batch()
Collects index requests across batch XIDs, calls at final_xid"] PR["process_requests(db_id, final_xid, combined_index_requests)
Sets DDL counter = request count, routes each request by action type"] Build["build()
1. Skip if index already READY
2. Add key to _table_idx_map (abort tracking)
3. Add IndexParams to _work_set
4. Push key to _queue
5. Register with RedisDDL
6. Notify workers via _cv"] Worker["Worker: task()
Picks up key from queue, fetches IndexParams from _work_set,
calls _build() if status is BUILDING"] BuildFn["_build()
1. Invalidate table cache
2. Create B-tree root
3. Build look-aside (if first index)
4. Scan all rows
5. Insert into B-tree"] AddPending["_add_to_pending_reconciliation()
1. Add to pending map
2. Decrement DDL counter
3. If counter == 0: push IndexReconcileRequest to queue"] LogMgr["pg_log_mgr enqueues RECONCILE_INDEX message for pg_log_reader
(via IndexReconciliationQueueManager)"] CommitterRecv["Committer receives RECONCILE_INDEX message
Calls process_index_reconciliation()"] Recon["_reconcile_index()
Catches up with changes made during build:
- Invalidate old extents
- Populate new extents"] Commit["_commit_build()
1. Finalize B-tree
2. Update table roots
3. Set index state READY
4. Cleanup tracking maps"] CB --> PR --> Build --> Worker Worker --> BuildFn BuildFn -->|Returns IndexState with B-tree root| AddPending AddPending -->|Queue read by pg_log_mgr| LogMgr LogMgr -->|pg_log_reader pushes msg to committer queue| CommitterRecv CommitterRecv --> Recon --> Commit ``` ### Index Drop Flow ```mermaid theme={null} flowchart TD PR["process_requests()
action == 'drop_index'"] Drop["drop()"] NotIn["Index NOT in _work_set
Fresh drop: index exists and not being built"] IsIn["Index IS in _work_set (building)
Concurrent drop: build in progress"] CreateItem["Create work item with DELETING status
Push to queue, register RedisDDL"] MarkAbort["Mark existing item as ABORTING
Decrement DDL counter (no new work item)"] WorkerPicks["Worker picks up
Since !BUILDING: add to pending reconciliation directly"] BuildDetect["_build() detects ABORTING via _was_dropped() check every 1000 rows
Returns early with partial B-tree"] ReconDel["_reconcile_index()
Detects DELETING → calls _drop()"] ReconAbort["_reconcile_index()
Detects ABORTING → calls _commit_build()"] DropFn["_drop()
1. Truncate B-tree
2. Truncate look-aside if last secondary index
3. Set DELETED
4. Update roots"] CommitFn["_commit_build()
1. Truncate B-tree
2. Truncate look-aside if present
3. Set state DELETED"] PR --> Drop Drop --> NotIn Drop --> IsIn NotIn --> CreateItem --> WorkerPicks IsIn --> MarkAbort --> BuildDetect WorkerPicks -.->|Reconciliation pipe similar to create_index flow| ReconDel BuildDetect -.->|Reconciliation pipe similar to create_index flow| ReconAbort ReconDel --> DropFn ReconAbort --> CommitFn ``` ### Index Recovery Flow ```mermaid theme={null} flowchart TD Trigger["Committer receives INDEX_RECOVERY_TRIGGER
(pg_log_mgr pushes this message when the db gets added)"] Recover["recover_indexes(db_id)"] Cleanup["_cleanup_for_db()
Clears stale data:
- _work_set entries
- Pending recon map
- Table-index map"] GetUnfinished["sys_tbl_mgr::Server::get_unfinished_indexes_info()
Returns indexes in states:
- NOT_READY (incomplete builds)
- BEING_DELETED (incomplete drops)"] ForEach["For each XID with unfinished indexes:
Build IndexProcessRequest list:
- NOT_READY → action='create_index'
- BEING_DELETED → action='drop_index'"] PR["process_requests()
Schedules recovery work through normal build/drop paths"] Trigger --> Recover --> Cleanup --> GetUnfinished --> ForEach --> PR ``` ### Abort Indexes Flow (Table Resync) ```mermaid theme={null} flowchart TD Push["pg_log_reader pushes abort_index message into
committer queue for the table as part of table_resync processing"] PR["committer calls process_requests()
action == 'abort_index'"] Abort["abort_indexes()(db_id, table_id, xid)"] Steps["1. Decrement DDL counter
2. Find all index keys for (db_id, table_id) in _table_idx_map
3. For each key: set work_item._status = ABORTING"] Workers["Workers detect ABORTING status during build
via _was_dropped() checks
→ Clean up and mark DELETED"] Push --> PR --> Abort --> Steps --> Workers ``` *** ## Implementation Details ### Worker Thread Model The Indexer spawns a configurable number of worker threads at construction: ```cpp theme={null} Indexer::Indexer(uint32_t worker_count, ...) { CHECK_GT(worker_count, 0); for (auto i = 0; i != worker_count; ++i) { std::string thread_name = fmt::format("IndexWorker_{}", i); _workers.emplace_back([this](std::stop_token st) { task(st); }); pthread_setname_np(_workers.back().native_handle(), thread_name.c_str()); } } ``` **Worker Loop (`task()`):** 1. Wait on condition variable `_cv` for work in `_queue` 2. Pop key from queue 3. Fetch `IndexParams` from `_work_set` 4. If status is `BUILDING`: call `_build()` and add result to pending reconciliation 5. If status is `DELETING`/`ABORTING`: add directly to pending reconciliation (with null root) Workers use `std::jthread` with `std::stop_token` for graceful shutdown coordination. ### Index Build Process (`_build`) **Phase 1: Setup** 1. Invalidate table cache at the creation XID 2. Extract index column positions from `IndexInfo` (sorted by `idx_position`) 3. Get mutable table reference 4. Create empty B-tree root for the index **Phase 2: Look-Aside Index (if needed)** ```cpp theme={null} if (!look_aside_index) { std::unique_lock g(_look_aside_mutex); if (_look_aside_build_tracker.find(tid) == _look_aside_build_tracker.end()) { look_aside_index = mutable_table->create_look_aside_root(...); look_aside_index->init_empty(); _look_aside_build_tracker[tid] = true; build_look_aside = true; } } ``` * Look-aside maps `internal_row_id → (extent_id, row_id_within_extent)` * Only the first secondary index on a table builds it * `_look_aside_build_tracker` prevents race conditions with concurrent index creation **Phase 3: Table Scan** ```cpp theme={null} constexpr int DROP_CHECK_PERIOD = 1000; for (auto row_i = table->begin(); row_i != table->end(); ++row_i) { // Check for shutdown if (st.stop_requested()) { root->truncate(); return {nullptr, key, idx, tid, look_aside_index}; } // Check for concurrent drop every 1000 rows if (row_cnt % DROP_CHECK_PERIOD == 0 && _was_dropped(key)) { return {root, key, idx, tid, look_aside_index}; } // Insert (index_columns, internal_row_id) into B-tree root->insert(svalue); } ``` ### Index Reconciliation (`_reconcile_index`) After the initial build, changes made to the table during the build must be applied: **Extent Chain Processing:** ```cpp theme={null} auto next_eid = table->get_stats().end_offset; auto next_extent_result = table->read_extent_from_disk(next_eid); auto next_extent = next_extent_result.first; while (next_extent) { // Invalidate previous extent (remove old entries) if (prev_eid exists && not already invalidated) { for each row in prev_extent: idx_state._root->remove(svalue); invalidated_eids.insert(prev_eid); } // Populate with new extent entries for each row in next_extent: idx_state._root->insert(svalue); // Move to next extent in chain next_eid = next_extent_result.second; } ``` **Decision Logic:** | Work Item Status | Action | | ---------------- | ------------------------------------------------------------- | | `DELETING` | Call `_drop()` directly | | `ABORTING` | Call `_commit_build()` to truncate and mark deleted | | `BUILDING` | Process extent chain, then call `_commit_build()` to finalize | ### Index Commit (`_commit_build`) Finalizes an index build or abort: ```cpp theme={null} void _commit_build(MutableBTreePtr root, const Key& key, const IndexParams& idx, uint64_t end_xid, MutableBTreePtr look_aside_root) { // 1. Fetch latest work item state (may have changed to ABORTING) work_item = _work_set.at(key); _work_set.erase(key); // 2. Get current index state from metadata index_info = server->get_index_info(db_id, index_id, xid); // 3. Handle based on metadata state and work item status if (index_info.state == DELETED) { // Table resync occurred - just cleanup root->truncate(); root->finalize(); } else if (work_item.is_status(BUILDING)) { // Success path extent_id = root->finalize(); meta->roots.emplace_back(index_id, extent_id); server->update_roots(...); server->set_index_state(..., READY); } else if (work_item.is_status(ABORTING)) { // Abort path root->truncate(); root->finalize(); server->set_index_state(..., DELETED); } // 4. Cleanup tracking structures _remove_index_key(db_id, tid, key); } ``` ### DDL Counter Management The counter ensures the Committer only receives reconciliation notifications when ALL index operations for a transaction are complete: ```cpp theme={null} // Initialization in process_requests() _xid_ddl_counter_map[xid].store(index_requests.size()); // Decrement on each operation completion if (--_xid_ddl_counter_map[xid] == 0) { _xid_ddl_counter_map.erase(xid); // Notify Committer via queue _index_reconciliation_queue_mgr->push(db_id, std::make_shared(db_id, xid)); } ``` **Counter Decrement Points:** * `build()` - when index already READY (skip) * `drop()` - when marking existing work item as ABORTING * `abort_indexes()` - after marking all table indexes as ABORTING * `_add_to_pending_reconciliation()` - after build/worker processing completes ### Index Drop (`_drop`) Handles direct index deletion: ```cpp theme={null} void _drop(const Key& key, const IndexParams& idx, uint64_t end_xid) { // 1. Verify state and remove from work_set CHECK(idx._status == IndexStatus::DELETING); _work_set.erase(key); // 2. Validate index exists and is in BEING_DELETED state info = server->get_index_info(db_id, index_id, xid); if (info.id() == 0) return; // Invalid index // 3. Handle table-dropped case if (!table_exists) { server->set_index_state(..., DELETED); return; } // 4. Truncate the B-tree root->init(extent_id); root->truncate(); root->finalize(); // 5. Handle look-aside if this is the last secondary index if (is_last_index) { look_aside_root->truncate(); look_aside_root->finalize(); meta->roots.emplace_back(INDEX_LOOK_ASIDE, UNKNOWN_EXTENT); server->update_roots(...); } // 6. Mark as deleted and cleanup server->set_index_state(..., DELETED); _remove_index_key(...); } ``` ### Look-Aside Index **Structure:** ``` Key: internal_row_id (stable, never changes for a row) Value: (extent_id, row_id_within_extent) ``` **How it helps:** * Secondary indexes store `internal_row_id` instead of direct `(extent_id, row_offset)` pointers * When a data extent is rewritten, only the look-aside index needs to be updated with the new physical location * Secondary indexes remain unchanged, eliminating cascading updates **Lifecycle:** * **Created:** With the first secondary index on a table * **Updated:** During index reconciliation when extents change * **Dropped:** When the last secondary index on a table is dropped **Coordination for Concurrent Creation:** ```cpp theme={null} std::unique_lock g(_look_aside_mutex); if (_look_aside_build_tracker.find(tid) == _look_aside_build_tracker.end()) { // First thread to reach here builds the look-aside index _look_aside_build_tracker[tid] = true; build_look_aside = true; } // Other threads skip look-aside building ``` ### Thread Safety **Lock Ordering** (to prevent deadlocks): 1. `_m` (work\_set, queue) 2. `_table_idx_map_mtx` 3. `_pending_reconciliation_map_mtx` 4. `_xid_ddl_counter_map_mtx` 5. `_look_aside_mutex` **Patterns Used:** * `std::scoped_lock` for acquiring multiple locks atomically * `std::atomic` for DDL counters to minimize contention * Separate mutexes for independent data structures to allow parallelism ### Error Handling and Edge Cases | Scenario | Handling | | --------------------------------------- | ------------------------------------------------------ | | Index already READY at build time | Skip processing, decrement DDL counter | | Stop requested during build | Truncate partial B-tree, return null root for recovery | | Drop while building | Mark ABORTING, worker detects via periodic check | | Table dropped before index drop | Mark index DELETED without truncating | | Queue unavailable during reconciliation | Abort reconciliation (will retry on recovery) | | Table resync during build | Index marked DELETED via abort\_indexes | ### Public API Summary | Method | Description | | ------------------------------------------------------------- | -------------------------------------------------------- | | `process_requests(db_id, xid, requests)` | Entry point for index create/drop/abort operations | | `recover_indexes(db_id)` | Recovers incomplete operations after restart | | `process_index_reconciliation(db_id, reconcile_xid, end_xid)` | Finalizes pending index operations | | `abort_indexes(db_id, table_id, xid)` | Aborts all index builds for a table (used during resync) | # Ingest Nodes Source: https://docs.springtail.io/ingest-nodes The ingest nodes are responsible for synchronizing the replica data with the primary database data. There are two different key states of operation -- `INITIALIZE` and `RUNNING`. # Initialization When the system is started in the `INITIALIZE` state, the ingest node performs the following actions: 1. Connect to the logical replication slot and begin capturing the stream of mutations. 2. Capture the set of tables to synchronize and copy them to the replica storage. 3. Play forward any in-flight changes since the table snapshots were captured to ensure we have a consistent snapshot of the entire system. 4. Move into the RUNNING state. # Running When the system is started in the `RUNNING` state, the ingest node performs the following actions: 1. Connect to the logical replication slot and begin capturing the stream of mutations. 2. Perform any [recovery](/recovery) required to bring the system to a consistent snapshot. 3. Play forward any changes recorded into the local log since recovery began. 4. Move into a steady-state operation of [processing messages](/message-processing). # Integration Testing Source: https://docs.springtail.io/integration-testing The integration tests run a full Springtail instance and perform end-to-end testing of the ingest and query components by executing SQL and then verify correctness. The integration tests should be run from the `python/testing` directory via: ```bash theme={null} python3 test_runner.py ``` This will run all of the integration tests. # Integration test framework ## Overview Our SQL test runner operates using individual test files which each represent a single test case. These test cases can then be bundled together into test sets, which share a single global setup and cleanup phase. ## Running the tests Tests are run using `test_runner.py` which is located in `python/testing`. It requires a YAML configuration file with the following parameters: * `test_folder` * The path to the folder containing the test set directories. * `system_json_path` * The path to the Springtail system configuration JSON file. * `build_dir` * The path to the Springtail build directory. * `tmp_config_dir` * This a temporary directory for where to put an overlay generated system json file * `overlay_config` * A dictionary of overlays. The key represents the overlay name and a matching overlay file should be present in `overlays/.json` . For each overlay key we specify other parameters used by an overlay and interpreted by the testing framework * `configs` * In this section we specify what we want to run for each configuration. Each configuration contains a list of overlay and of the corresponding test sets that we want to run for this overlay. The `default` configuration is used if no command line parameters are passed in. The test runner configuration is stored in test\_runner\_config.yaml file and below is an example of what we expect to find there: ```yaml theme={null} # Folder holding the test cases test_folder: 'test_cases' # the location of the Springtail configuration file to use system_json_path: '../../system.json.test' # the location of the Springtail build build_dir: '../../debug' # the location for generated overlay Springtail configuration files tmp_config_dir: '/tmp/tmp_springtail_config' # overlay configuration, each overlay has a corresponding .json override file in # overlays directory # only overlays specified here can be used as paramerter with the script # -o option overlay_config: # uses a small repl log size to force more frequent log rotations small_log_rotate: # no parameters params: ~ # uses a small repl log size to force more frequent log rotations together with streaming postgres config small_log_rotate_with_streaming: params: postgres_config: logical_decoding_work_mem: 64 # uses a small data cache size to force more frequent eviction small_cache_size: # no parameters params: ~ # uses a small worker mem size to ensure streaming mode streaming_postgres_config: params: postgres_config: logical_decoding_work_mem: 64 # set the test config to run the test and verify steps against the proxy integration_test_config: params: use_proxy_for_test: true use_proxy_for_verify: true # special configuration for testing include schema include_schema_config: # no parameters params: ~ # config sections, contains full configuration of the run as we want to run it # each configuration specifies what we want to run without an overlay and # for each overlay mentioned, we specify which test sets we want to run it with configs: # configuration for nightly builds nightly: - overlay: ~ test_sets: 'all' - overlay: 'small_log_rotate' test_sets: ['recovery'] - overlay: 'small_log_rotate_with_streaming' test_sets: ['recovery'] - overlay: 'small_cache_size' test_sets: ['basic'] - overlay: 'streaming_postgres_config' test_sets: ['recovery', 'large_data'] - overlay: 'include_schema_config' test_sets: ['include_schema'] # configuration for github CI github_ci: - overlay: ~ test_sets: ['basic', 'framework', 'preload', 'enum_bits', 'complex', 'numeric', 'query_benchmark'] - overlay: 'small_log_rotate' test_sets: ['recovery'] - overlay: 'small_log_rotate_with_streaming' test_sets: ['recovery'] - overlay: 'small_cache_size' test_sets: ['basic'] - overlay: 'streaming_postgres_config' test_sets: ['recovery'] - overlay: 'include_schema_config' test_sets: ['include_schema'] # this configuration runs when there is nothing specified on the command line default: - overlay: ~ test_sets: ['basic', 'framework', 'preload', 'enum_bits', 'complex', 'numeric', 'query_benchmark'] - overlay: 'small_log_rotate' test_sets: ['recovery'] - overlay: 'small_log_rotate_with_streaming' test_sets: ['recovery'] - overlay: 'small_cache_size' test_sets: ['basic'] - overlay: 'streaming_postgres_config' test_sets: ['recovery', 'large_data'] - overlay: 'include_schema_config' test_sets: ['include_schema'] ``` There are several different ways to execute the test runner. ```bash theme={null} # run all tests with all overlays # -- this should be the default way to run the integration tests # it will executes the tests as specied in default config python3 test_runner.py # run all tests in a given test set without using any overlay python3 test_runner.py # run specific tests cases within a given test set without using any overlay python3 test_runner.py ... # run a specific overlay for given test set and test cases when specified python3 test_runner.py -o [ ...] # run a specific configuration, right now it can be one of 'nightly', # 'github_ci', or 'default' python3 test_runner.py -c ``` ## Test Case A test case is defined by a single file which has the following structure: ```bnf theme={null}
::= "##" ::= "metadata" | "test" | "verify" | "cleanup" ::= "###" ::= | | | # only valid in the "metadata" section ::= "autocommit" ("true" | "false") | "default_txn" | "sync_timeout" | "query_timeout" | "poll_interval" | "require_overlays" # only valid in the "test" section ::= "parallel" | "sequential" [ ] | "txn" | "load_csv" | "sleep" | "sync" | "recovery_point" | "force_recovery" | "restart" | "switch_db" | "streaming" # only valid in the "verify" sectiona ::= "schema_check"
| "table_exists"
("true" | "false") | "index_existt"
("true" | "false") | "switch_db" | "benchmark" # only valid in the "cleanup" section ::= "switch_db" # not fully specified here, but a SQL statement may span multiple lines ::= ';' ::= '--' ::= | |
| ``` ### Sections There are 2 required sections and 2 optional sections of every test case. #### Metadata (optional) The `metadata` section defines variables for the overall test case. * `autocommit` * Specifies if the sql commands are each individual transactions or not. If not, you can use `BEGIN` and `COMMIT` sql commands to perform a transaction within the test. Either way, a `COMMIT` is automatically issued at the end of the “test” section by the test runner. * The default value is `true` to maintain the behavior of the current tests. * `default_txn` * Specifies the name of the default transaction to use in sequential sections. This is only useful in very specific situations where you are using a mix of sequential and parallel sub-sections and want to ensure that the sequential sections use a specific transaction from the parallel sub-section without having to specify it with each sequential sub-section header. * The default value is `default` . * `sync_timeout` * Specifies the maximum time in seconds spent waiting for a sync to Springtail to complete. This impacts both the `sync` directive and the implicit sync that occurs after the `test` section is run. It may make sense to increase this if the test is performing actions that could take a long time to sync (e.g., loading a large amount of data or forcing a re-sync on a large table). * The default value is `3` seconds. * `query_timeout` * Specifies the maximum time to wait for a query to complete. If this timeout is exceeded by any query to the Springtail replica database then the test is considered failed. * The default value is `5` seconds. * `poll_interval` * Specifies the time between checks for the `sync` operation in seconds. Can be specified as a float for sub-second timing. * The default value is `0.001` seconds. * `require_overlays` * Specifies the list of overlays that are required to be able to run this test case #### Test (required) The `test` section defines the actual sql commands to run for this test case. There are also a number of directives available to better control the behaviors of the test. * `load_csv` * Specifies a CSV file and a table into which the CSV file will be loaded. Please ensure that the table exists before this directive is called since it will not create it. The CSV file is assumed to be in the same directory with the test case and it must contain headers which match the column names in the table it is being loaded into. * If your CSV file is large, please place a gzip compressed version of it in the public S3 bucket at `s3://public-share.springtail.io/test_files` — e.g., `s3://public-share.springtail.io/test_files/mushroom_overload.csv.gz` . The CSV files of that directory are synchronized locally before tests are run, allowing you to create a symlink to the appropriate CSV file from your test set’s directory. * `sleep` * Specifies that a given transaction should pause sending SQL statements for a given number of seconds. This is most useful when trying to enforce timings between different transactions in a parallel sub-section. * `sync` * Blocks execution until the previous SQL statements have been applied to the Springtail replica. This is almost never required since their is an implicit `sync` run at the end of every test section to ensure data is fully synced before running the verify section. * `txn` * Specifies that the following SQL statements should be executed within a single transaction with the provided logical name. These logical transactions are executed on separate database connections, meaning you can interleave SQL statements between transactions to ensure correctness in more complex situations. * By default SQL statements are part of the default transaction if none has been specified. * `parallel` * Specifies that the following statements are part of a parallel execution. In this sub-section, separate transactions (as defined by `txn` directives) are executed in parallel. If you do not specify a transaction, then the statements are executed in the default transaction. * `sequential` * Specifies that the following statements should be executed sequentially. This means that each statement will be run only when the previous has completed, even when switching between logical transactions. * `recovery_point` * Uses this instruction to get current XID and stores it as XID to recover to once `force_recovery` is triggered. * `force_recovery` * Brings down the system, reverts the committed XID to an earlier XID retrieved at `recovery_point`, and then brings the system back up to force it to recover data from the WAL. * `restart` * Forces the system to do full restart without resetting the database. * `swtich_db` * Changes the current transaction to use specified database. If there is no `txn` instruction preceding this statement, then database switch is applied to the default transaction, otherwise it is applied to the transaction specified by `txn` instruction that is preceding `switch_db` instruction. This forces all the further instructions or SQL statements to apply only to this database. * `streaming` * Enables streaming #### Verify (required) The `verify` section defines sql commands that are run against both the primary and the Springtail replica and then have their results compared to ensure that the test has run successfully. These can consist of `SELECT` statements or the following directives: * `schema_check` * Compares the table schemas between the primary and Springtail to ensure that they match. Also checks all active indexes (primary and secondary) * `table_exists` * Verifies that the specified table exists or does not exist on both primary and replica. * `index_exists` * Verifies that the specified index exists or does not exist on both primary and replica. * `swtich_db` * Changes the current transaction to use specified database.This forces all the further instructions or SQL statements to apply only to this database. #### Cleanup (optional) The `cleanup` section defines sql commands that are run against the primary after the verification step to revert any changes you don’t want visible to the next test in the test set. This section allows the following directives: * `swtich_db` * Changes the current transaction to use specified database.This forces all the further instructions or SQL statements to apply only to this database. ## Test Set A test set is composed of a set of test cases along with a global configuration. The test case is specified using a directory, the test cases are files within that directory that end with the `.sql` extension. The global configuration is a special file named `__config.sql` which uses a similar, but slightly different and simplified format from the test cases. ```bnf theme={null}
::= "##" ::= "metadata" | "setup" | "cleanup" ::= "###" ::= | | # only valid in the "metadata" section ::= "autocommit" ("true" | "false") | "live_startup" | "require_overlays" # only valid in the "setup" section ::= "load_csv"
| "add_db" | "switch_db" # only valid in the "cleanup" section ::= "switch_db" # not fully specified here, but a SQL statement may span multiple lines ::= ';' ::= '--' ::= | |
| ``` ### Global configuration The global configuration also has an optional `metadata` section which can contain the following instructions: * `autocommit` * Specifies if the sql commands are each individual transactions or not. If not, you can use `BEGIN` and `COMMIT` sql commands to perform a transaction within the test. Either way, a `COMMIT` is automatically issued at the end of the “test” section by the test runner. * The default value is `true` to maintain the behavior of the current tests. * `live_startup` * Specifies that the test set should run a background thread with the given sleep interval. This background thread will periodically issue an update to `background_control` table. * `require_overlays` * Specifies the list of overlays that are required to be able to run this test set The `setup` section of the global configuration specifies any SQL commands that should be run prior to Springtail starting. This allows us to test the initial sync of the primary database, as well as centralize the creation of any shared tables required by the individual test cases. It can have one initial instruction: * `add_db` * Add given database at the setup time. From that point on, this database will be available for `switch_db` instruction and data preload before the system starts. The `cleanup` section of the global configuration specifies any SQL commands that should be run after Springtail is shutdown. Ideally this would revert the database to it’s state prior to the test set running so that another test set can be run. ### Running a test set A test set execution is composed of the following steps: * Setup * Runs the `setup` from the global configuration to prepare the primary database for the test cases. * Start Springtail * Brings up the Springtail replica from scratch, also causing the initial sync of the primary database. * Execute test cases * Runs each test case in the test set. The test cases are run based on the lexicographical order of their file names. * Shutdown Springtail * Brings down the Springtail replica. * Cleanup * Runs the `cleanup` from the global configuration to prepare for the next test set. # Introduction Source: https://docs.springtail.io/introduction ## What is Springtail? Springtail scales read performance for existing PostgreSQL databases without requiring a migration or changes to your application. Elastic read scaling for your existing PostgreSQL database | Springtail It replicates data from your existing PostgreSQL instance and routes read queries to on-demand replica nodes. This allows you to add replicas for periods of high traffic and remove them when they are no longer needed, improving read performance while reducing infrastructure costs. ## Benefits Springtail provides significant advantages over traditional Postgres read replicas: * **Instant scale-up:** Springtail separates storage from compute, allowing additional replicas to spin up in seconds without requiring a new copy of the data. * **Cost-efficiency:** Only pay for what you use — reduce compute resources when they’re not needed, saving on peak provisioning costs. * **Built-in optimization:** Springtail's proxy handles load balancing and connection pooling, so you don’t need to manage these services separately. * **No downtime:** Bring replicas online quickly without the downtime of snapshot creation or synchronization delays. ## Key features Springtail enhances performance and efficiency through several key capabilities: * **Scalable compute:** Increase or decrease compute capacity as your workload changes, reducing unnecessary overhead. * **Scalable storage:** Independently scales storage from compute, allowing faster setup and efficient data access. * **Optimized ingest pipeline:** Reduces replication lag and ensures that reads receive the most up-to-date snapshot of the data. * **Built-in proxy:** Automatically manages load balancing and connection pooling, simplifying replica management. * **Custom storage format:** Optimizes storage for fast primary key access patterns, reducing latency on common database operations. ## Use cases Springtail replicas are well-suited for traditional read replica workloads: * **Nightly batch jobs:** Efficiently process recurring large-scale operations without impacting your primary database. * **ETL jobs:** Seamlessly handle extract, transform, and load processes with scalable compute. * **Read-heavy operations:** Handle large queries, such as SaaS dashboards or table scans, without overloading your primary instance. Any read-only transaction that might cause contention on your primary database is an ideal candidate for Springtail. ## Get started Ready to get started? Follow these steps: 1. Use the [Quickstart guide](/quickstart) to set up your first Springtail database instance. 2. Check out our How-to guides for tips on managing your Springtail instances. 3. Dive into the [Architecture](/architecture) to understand the underlying architecture of Springtail. # Local Cluster Deployment Source: https://docs.springtail.io/local-cluster-deployment # Local Cluster Deployment This section provides a step-by-step guide to deploying the system in a local cluster environment using Docker. This setup is ideal for development, testing, and small-scale production environments. ## Prerequisites * Docker * Docker Compose * Python3 * AWS CLI * Base container image for Springtail Service \* * More on this in the "Building a Base Image" section below. * Check out the `springtail` repository to your local machine. ## Cluster Building Flow ```mermaid theme={null} flowchart TD Z[Start Mock AWS Service] Y[Start Redis Service] X[Start Local Primary DB] Z --> A Y --> A X --> A A[Build Springtail Package] --> G[Upload Package to Local AWS S3] G --> F[Launch Bootstrap Container:
set secrets, set Redis config, amend shared Env files etc] F -- read shared Env files / Redis config --> M[Launch Ingestion Container] F -- read shared Env files / Redis config ---> N[Launch Proxy Container] F -- read shared Env files / Redis config --> Q[Launch FDW Container] M --> J[Join & Wait] N --> J Q --> J ``` ## Building a Base Image The Springtail service need a set of dependencies to run. These dependencies are packaged into a base container image, which is then used to build the service containers. The local cluster will assume the existence of such base image. All the steps below assume you are at the root of `springtail` repository. In the `local-cluster/env` directory, there is a `env.setup` file that defines the name of the base image to use. ``` IMAGE=local-cluster-img:latest ``` The base image (along with the controller and custom PostgreSQL images) is built automatically by `./cluster up` if it is not already present, so you normally do not need to build it by hand. To build or rebuild it manually, run the following from the repository root: ```shell theme={null} $ docker build -t local-cluster-img:latest -f docker/Dockerfile.local-cluster . ``` ## Working with Local Cluster ### Launching the Cluster 1. First build a springtail package. You can reuse an existing package if you have one though. ```shell theme={null} $ ./cluster build-package ``` 2. Start the local cluster. You can specify the full package path on your local machine (the tarball file generated from the previous step). This step also builds any missing images (base, controller, and custom PostgreSQL). ```shell theme={null} $ ./cluster up ``` To disable SSL for inter-service connections, append the `--disable-ssl` flag: ```shell theme={null} $ ./cluster up --disable-ssl ``` 3. It takes a few minutes (usually less than 2) to start the cluster. You can check the status by: ```shell theme={null} $ ./cluster status ``` ### Managing / Interacting with the Cluster Once all containers are up and running, you can `shell` into any container. For example: ```shell theme={null} $ ./cluster sh ``` The service names are: * `proxy` * `ingestion` * `fdw1` * `fdw2` * `controller` Other useful commands: ```shell theme={null} $ ./cluster logs # show logs for a service $ ./cluster restart # restart a service container $ ./cluster ls # list all services ``` By default, the `springtail-coordinator` service is not started. You can start it by (inside a container): ```shell theme={null} $ systemctl start springtail-coordinator ``` ### Host Ports The following services are exposed on the host: | Service | Host Port | | --------------- | --------- | | Primary DB | 15432 | | Redis | 16379 | | Proxy | 55432 | | FDW 1 | 45432 | | FDW 2 | 45433 | | AWS Mock (Moto) | 29999 | | Controller API | 19824 | Connect to the proxy from the host with any PostgreSQL client: ```shell theme={null} $ psql -h localhost -p 55432 -U postgres ``` ### Tearing Down the Cluster To tear down the Springtail containers, ```shell theme={null} $ ./cluster down ``` To stop a single service and its dependencies, pass its name: ```shell theme={null} $ ./cluster down proxy ``` To tear down everything including the supporting services (Redis, local AWS mock, local DB), ```shell theme={null} $ ./cluster down all ``` ## Internals This section explains what happens under the hood when you run the above commands. ### Building the Package The `./cluster build-package ` command does the following: Start a temporary build container from the image identified by `BASE_BUILDER_IMAGE_TAG` in the `cluster` script, which resides in the DevSupport ECR repo. If the BASE\_BUILDER\_IMAGE\_TAG is not available locally it will try to pull from the remote repo, thus requiring AWS SSO configured. It saves the package into the `` directory on the host machine. The package is a tarball file named `springtail--.tar.gz`. ### Starting the Cluster The `./cluster up ` command does the following: It sets the `` to the `PACKAGE_FILE_NAME` environment variable, making it available to the build process of all the containers. Then it starts the supporting local services, like the AWS mock, Redis, and a local PostgreSQL database serving as primary DB. Then it uploads the package to the local mock S3 service. Then it launches a bootstrap container that sets up shared environment files, secrets, and Redis configuration. Specifically, the bootstrap process will add new env vars into `./env/.env` file. Finally, it launches the main service containers: `proxy`, `ingestion`, and `fdw`, picking up all the env files in the `./env` directory. ### Service Initializations #### local-cluster-bootstrap This runs *before* any service container is started. It does the following: 1. Set up shared environment files in the `./env` directory, which will be picked up by all service containers; 2. Set up Redis configuration, creating a default user and password, and saving the config to `./env/redis.env`; 3. Set up secrets, creating a self-signed certificate for HTTPS, and saving the secrets to `./env/secrets.env`; In summary, this is setting up the supporting environment for the service containers to run. #### "CloudInit" Script For each service container, upon startup, it runs a `init-services` script, managed by supervisord, that does the following: 1. Download S3 package and extract it, moving the coordinator to the `/opt/springtail` directory as the bootstrap coordinator; 2. Make sure all necessary directories exist, creating them if needed; 3. For `fdw` specifically, it triggers an Ansible script to customize the PostgreSQL; This is akin to the EC2's `userdata` script (CloudInit), which is used to initialize a container. # Local Dev Environment Source: https://docs.springtail.io/local-dev-environment The local dev environment runs a PostgreSQL 16 instance and a development container with the full C++ toolchain (compiler, linker), plus PostgreSQL and Redis servers. Your local source tree is mounted into the container, so edits on your machine are built inside the container. There are two ways to bring it up: **Docker Compose** (recommended) or a **manual `docker run`**. ## Prerequisites * Docker and Docker Compose * Python 3 ### Installing a container runtime (macOS) Install Homebrew, then a container runtime: ```bash theme={null} /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # install if not already installed brew install colima docker docker-compose docker-Buildx ``` #### Colima (preferred) Install colima and docker (this is preferred); make sure you are using `macOS Virtualization.Framework` and `virtiofs` with colima. **NOTE**: Rosetta is a x86 translation layer, if not using Apple Silicon Mx then it may not be necessary. ```bash theme={null} # NOTE: 12 = GB memory, use less if needed # Also if Apple M4, pass --nested-virtualization to make gdb work colima start --memory 12 --cpu 5 --vm-type=vz --vz-rosetta # run colima status and check the output springtail % colima status INFO[0000] colima is running using macOS Virtualization.Framework INFO[0000] arch: aarch64 INFO[0000] runtime: docker INFO[0000] mountType: virtiofs ``` #### Docker Desktop If using Docker Desktop you may need to set some options to enable Rosetta and VirtioFS. See: [https://www.docker.com/blog/docker-desktop-4-25/](https://www.docker.com/blog/docker-desktop-4-25/) Go to Settings→General and select VirtioFS and Rosetta. ## Quick Start (Docker Compose) ### 1. Build the base Docker image From the repository root: ```shell theme={null} cd docker ./docker_base.sh ``` This builds the `springtail:base` image from `Dockerfile.base`, which includes a patched PostgreSQL 16 (built from source with RLS support for foreign tables), Redis, the C++ toolchain, and all Ansible-provisioned dependencies. ### 2. Start the dev environment ```shell theme={null} export SPRINGTAIL_SRC=/path/to/springtail # absolute path to your repo checkout docker compose up -d ``` This starts two services: | Service | Container | Description | Host Port | | ---------- | ---------------- | ---------------------------------------------- | ---------- | | `postgres` | `pg16` | PostgreSQL 16 with logical replication enabled | 5432 | | `dev` | `dev-springtail` | Development container with build tools | 2222 (SSH) | The dev container mounts your source tree at `/home/dev/springtail` and starts PostgreSQL, Redis, and SSH automatically via its entrypoint. ### 3. Build Springtail inside the container Shell into the dev container and run the debug build: ```shell theme={null} docker exec -it dev-springtail bash # Inside the container: cd ~/springtail ./vcpkg.sh # one-time: install C++ dependencies ./debug.sh # build debug binaries into ./debug/ ``` ### 4. Run the unit tests (C++ / CTest) ```shell theme={null} cd ~/springtail/debug make build_tests ctest ``` Or build and run in one step: ```shell theme={null} cmake --build debug --target check ``` The `check` target kills any running Springtail processes, installs SQL triggers, builds the tests, and runs them via CTest. ### 5. Run the integration tests The integration test runner is a Python script that exercises Springtail end-to-end against a real PostgreSQL instance. It must be run from its own directory: ```shell theme={null} cd ~/springtail/python/testing python3 test_runner.py ``` This runs the **default** test configuration, which includes the test sets `basic`, `framework`, `preload`, `enum_bits`, `complex`, `numeric`, `query_benchmark`, and `recovery` (with various overlay configurations). #### Common test\_runner.py options ```shell theme={null} # Run the default configuration (same as no arguments) python3 test_runner.py # Run a specific named configuration (e.g., nightly, github_ci_p1) python3 test_runner.py -c nightly # Run a single test set python3 test_runner.py basic # Run specific test cases within a test set python3 test_runner.py basic test_create.sql test_insert.sql # Run with a specific overlay python3 test_runner.py -o small_log_rotate recovery # Skip downloading test data from S3 (useful offline or in CI) python3 test_runner.py --skip-downloads # Output JUnit XML report python3 test_runner.py -j results.xml ``` Available test sets: `basic`, `complex`, `enum_bits`, `framework`, `include_schema`, `large_data`, `live_startup`, `numeric`, `policy_roles`, `preload`, `query_benchmark`, `recovery`, `text_tables`. Available overlays: `small_log_rotate`, `small_log_rotate_with_streaming`, `small_cache_size`, `streaming_postgres_config`, `integration_test_config`, `include_schema_config`. ### 6. Tear down ```shell theme={null} cd /path/to/springtail/docker docker compose down -v ``` ## Alternative: manual container (`docker run`) Instead of Docker Compose, you can build and run the dev container by hand. ### Build the image ```bash theme={null} git clone git@github.com:Springtail-inc/springtail.git cd springtail ``` From within the springtail dir, create the docker image and run it. * The `-p` options will map the ports from container to host. Use Postgres (5432), Redis (6379) and SSH (22) (only SSH is required). * The `-v` option will map your local springtail github source into the container (`source:target`). The source should be your local springtail dir, the target must be `/home/dev/springtail`. The source is the location at which you checked out and cloned the github source code; replace ``. * The `--privileged` and `--cap-add=SYS_PTRACE` options may be useful for debugging to attach to a running process ```bash theme={null} # from springtail root cd docker docker-buildx build --progress plain -t springtail:dev -f ./Dockerfile.base . # note: replace with the location of the cloned springtail dir # ~ is short for /Users/, so use one or the other docker run -p 2222:22 -p 6666:6379 -v ~/:/home/dev/springtail -d --name dev --shm-size=1g -it springtail:dev --privileged --cap-add=SYS_PTRACE ``` The Dockerfile.base edits the Postgres configuration and creates the script `entrypoint.sh` as the container entry point. This script ensures the Postgres and Redis services are running, and it creates the Postgres springtail user. The Dockerfile also edits the Postgres configuration as necessary. ### Logging in You should see 3 directories after logging in: ```bash theme={null} ssh -p 2222 dev@localhost # use password 'dev' ``` ```bash theme={null} dev@941221cd102e:~$ ls debug external springtail ``` The `springtail` directory should be the mounted directory from your local machine containing the code. The `debug` directory is where the code will be built within the container, and the `external` directory will host the `vcpkg` packages and source. ### Building From within the dev container; this will create symlinks to `../debug` and `../external` and will start the build. First it will download `vcpkg` and build it, then it will download the dependency packages and build them, lastly it will build the Springtail code. ```bash theme={null} # first time; go grab a coffee it may be a while... cd springtail ./debug.sh # subsequent times cd debug make ``` #### Troubleshooting the build * Check for the existence of the following two files, and make sure they are symlinks: * `debug -> /home/dev/debug/` * `external -> /home/dev/external/` ```bash theme={null} # make sure you are in the springtail dir cd /home/dev/springtail # look for debug and external ls -l ... lrwxr-xr-x 1 dev dev 16 Oct 4 23:29 debug -> /home/dev/debug/ ... lrwxr-xr-x 1 dev dev 19 Oct 4 23:29 external -> /home/dev/external/ # if they don't exist: ln -s /home/dev/debug ln -s /home/dev/external ``` * Sometimes running `./debug.sh` results in errors while building the external dependencies in vcpkg. If this happens, just try rerunning `./debug.sh` and see if you can make progress (i.e., check if it is failing on a different dependency, if so keep going). ## Operating the environment ### After Mac reboot After reboot if your container is no longer accessible, check if colima is running and if not start it: ```bash theme={null} > colima start --memory 16 --cpu 5 --vm-type=vz --vz-rosetta INFO[0000] starting colima INFO[0000] runtime: docker WARN[0000] Unable to enable Rosetta: Rosetta2 is not installed WARN[0000] Run 'softwareupdate --install-rosetta' to install Rosetta2 INFO[0000] starting ... context=vm INFO[0011] provisioning ... context=docker INFO[0012] starting ... context=docker INFO[0012] done ``` Now verify if the docker container is running: ```bash theme={null} docker container ls ``` If it did not find anything, the docker container needs to be restarted. Run the following command: ```bash theme={null} > docker container ls --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6bc0f4fe0fb3 springtail:dev "/usr/local/bin/entr…" 2 weeks ago Exited (255) 24 minutes ago 5432/tcp, 0.0.0.0:2222->22/tcp, [::]:2222->22/tcp, 0.0.0.0:6666->6379/tcp, [::]:6666->6379/tcp dev ``` The first number is the container id and this is what is needed to get container restarted. ```bash theme={null} docker container start ``` Now you can login into your container. If you try to do `docker run` command instead of `docker container start`, it will fail as follows: ```bash theme={null} > docker run -p 2222:22 -p 6666:6379 -v ~/springtail:/home/dev/springtail -d --name dev -it springtail:dev --priviliged --cap-add=SYS_PTRACE docker: Error response from daemon: Conflict. The container name "/dev" is already in use by container "6bc0f4fe0fb32c62945a411c67b4ddbaf522c6f4fcedbdc4a232d8ac0c9cee63". You have to remove (or rename) that container to be able to reuse that name. ``` ### Running out of space If you have trouble with disk space or RAM or CPU, you can restart `colima`: ```bash theme={null} # e.g. 300G disk, 20G RAM, and 6 CPUs colima start --disk 300 --memory 20 --vz-rosetta --cpu 6 --vm-type=vz ``` You would need to restart the docker container as specified above. ## VS Code Since you have mounted the `springtail` source dir into the container, the code it is building is the same code on your local machine. Any edits to the code on your local machine will be reflected in the container. So after you make a code change, just rebuild from the container (`cd debug; make`). ### Dev containers If you are using VSCode or Cursor (based on VSCode), and want more functionality to just work, it is best to have VSCode connect to the docker image and edit there. You can do this by installing the “Dev Containers” extension. And then access the Command Palette (cmd+shift+P) and run “Dev Containers: Attach to Running Container…”. You can then select your container. It will first connect as “root”, which is not what you want, so then run “Dev Containers: Open Container Configuration File” and set the contents to the following: ```json theme={null} { "remoteUser": "dev", "workspaceFolder": "/home/dev/springtail", } ``` You can then reconnect with “Dev Containers: Attach to Running Container…”. Note that in this environment, you can also install the “CMake” extension and run things like “CMake: Build” and get good language server integration. ### Clangd Additionally, in VSCode, I’d recommend installing the `clangd` extension, and also run `sudo apt install clangd` inside of your container. After installing the VSCode extension, you should get an option to disable “Microsoft Intellisense” (as both extensions provide the same functionality). The clangd extension should give much more accurate/faster support for “Find all references”. # Message Flow Overview Source: https://docs.springtail.io/message-flow-overview ## Overview The `ClientSession` and `ServerSession` classes in the Springtail project facilitate communication between the client and the database server. This communication is essential for managing queries, state transitions, and failover scenarios. ### ClientSession * **Manages Client Connections**: Handles incoming client requests and routes them to the appropriate `ServerSession`. * **State Management**: Tracks the state of the client connection (e.g., STARTUP, READY, QUERY). * **Failover Handling**: Manages failover notifications and transitions to replica servers when necessary. * **Query Parsing and Execution**: Processes client queries and forwards them to the `ServerSession`. ### ServerSession * **Manages Server Connections**: Handles communication with the database server. * **Batch Processing**: Queues and processes batches of messages from the `ClientSession`. * **Error Recovery**: Implements mechanisms to recover from errors and maintain session integrity. * **State Management**: Tracks the state of the server connection (e.g., STARTUP, DEPENDENCIES, QUERY). ## Communication Flow 1. **Client Request Handling**: * The `ClientSession` receives a request from the client. * It parses the request and determines the appropriate action (e.g., query execution, transaction management). 2. **Server Session Selection**: * The `ClientSession` selects or creates a `ServerSession` based on the request type and session state. * It may reuse an existing `ServerSession` or create a new one for failover scenarios. 3. **Message Forwarding**: * The `ClientSession` forwards the parsed query or command to the `ServerSession`. * The `ServerSession` processes the message and communicates with the database server. 4. **Response Handling**: * The `ServerSession` receives the response from the database server. * It forwards the response back to the `ClientSession`, which sends it to the client. 5. **Error and Failover Management**: * If an error occurs, the `ServerSession` attempts to recover and notifies the `ClientSession`. * In failover scenarios, the `ClientSession` transitions to a replica server and replays pending state. ## Message Path: Server -> Client Session This section describes how server-side connections are accepted, scheduled, processed, and how replies are returned to the client. 1. **Connection Acceptance**: * Server accepts new TCP connections and wraps them in connection objects * Creates an in-memory session to track the socket and connection state * Registers the session with the event infrastructure for monitoring 2. **Event Detection**: * Main event loop monitors registered sockets for readability * When a socket becomes readable, the client session associated with socket is obtained * All sockets associated with the client session (the client session and all server session sockets) are removed from the poll set * The client session is added to a runnable set; if the server session is in a RESET state, then there is no associated client session and it is added to the runnable set 3. **Worker Processing** * Runnable sessions are submitted to a worker pool * A worker thread picks up the session and begins processing * Worker reads incoming bytes, assembles protocol messages, and processes them 4. **Message Handling** * Client session first checks for data on the client session socket, then for data on the server session sockets * Client-facing sessions interpret the client protocol * Parse message payloads and forward commands to the server-side session * Manage transactional state and buffering as required 5. **Reply Delivery** * Server-side session receives responses from the database; the server session socket will have data available * Prepares outgoing buffers and writes to the client's socket * All traffic is logged by the request/response logging subsystem 6. **Session Re-queueing** * Worker clears temporary state after processing * Re-registers the session with the event loop if connection remains open * Performs cleanup if the session is closing or errors occur # Message Processing Source: https://docs.springtail.io/message-processing # Postgres Replication Message Processing (High-Level Flow) This document describes, at a high level, how a single Postgres replication message moves through the system: from a logical replication connection, into the log manager for durable staging, and then into message-stream parsing to produce structured events for downstream processing. ## 1. Major roles in the pipeline ### Replication connection (CDC ingress) A dedicated replication connection maintains a streaming session to Postgres and receives a continuous sequence of logical replication records. Each incoming record has: * An ordering position (log sequence position) * A message type (data change, transactional boundary, schema-related, etc.) * A raw payload, which may arrive fragmented across multiple network reads The replication connection is responsible for maintaining continuity, reconnect behavior, and producing a clean stream of replication payload bytes plus the metadata needed to order and acknowledge progress. ### Log manager (durable staging + coordination) The log manager sits immediately downstream of the replication connection. Its responsibilities are to: * Persist incoming replication messages to local durable storage in a sequential log format. * Ensure that acknowledgment back to Postgres happens only after durability guarantees are met. * Provide a stable “replayable” source of replication data for downstream consumers, including restart recovery after failures. This stage decouples ingestion reliability from downstream processing speed and isolates transient parsing or indexing failures from the replication session. ### Message stream processing (structured event extraction) Message-stream processing is the layer that takes the raw bytes from the staged log and interprets them into higher-level, typed events such as: * Transaction boundaries (begin/commit) * Row-level changes (insert/update/delete) * Metadata and schema-change related events * Other logical replication message categories relevant to the system It incrementally parses the byte stream, reconstructs message boundaries, and emits structured events to later stages. *** ## 2. End-to-end lifecycle of a single replication message ### Step 1: Receive message bytes over the replication stream The replication connection continuously reads from the logical replication stream. A “message” in this context may not be delivered as one contiguous read; payloads can be fragmented. The connection layer: * Collects incoming bytes * Preserves ordering * Associates each produced chunk with the message context necessary for later reassembly (including the total message length and current offset within the message) At this stage the data is still an uninterpreted replication payload (raw bytes plus positional metadata). ### Step 2: Forward to log manager for durable append Each message (or message fragment) is forwarded to the log manager, which appends it to a durable log file in a format that supports later sequential reading and recovery. Key properties of this append stage: * The log is written sequentially for throughput and simplicity. * Message framing information is preserved so downstream readers can locate message boundaries. * The log manager accounts for the fact that some messages may arrive in parts and ensures the log can still represent the original message correctly. ### Step 3: Acknowledge replication progress only after durability Acknowledging progress to Postgres is tied to durability, not just receipt. The log manager ensures that: * Replication progress is advanced only when the written bytes have been flushed to durable storage. * The “ack position” reflects committed progress in a way that is safe to resume from during restarts. This prevents situations where Postgres believes the consumer has safely processed data that was only buffered in memory and could be lost on crash. ### Step 4: Sequential log reading (decoupled consumer) Downstream processing reads replication data from the staged log rather than directly from the replication connection. This decoupling provides: * Replay capability after restart (the log is the source of truth) * Backpressure control (read rate can differ from ingest rate within bounded buffering) * A clean separation between “durable ingestion” and “semantic interpretation” # Operator class support Source: https://docs.springtail.io/operator-class-support ## Overview This document describes the implementation of operator class (opclass) support in Springtail, enabling the system to handle GIN and GiST secondary indexes in addition to the existing B-tree indexes. > **Status:** This feature is currently in development on branch `SPR-1090-gin-gist-base-3` and has not been merged to `main`. ### Background PostgreSQL uses operator classes to define the behavior of indexes for different data types. Each index type (B-tree, GIN, GiST) requires specific support functions identified by support numbers. For example: * **GIN indexes** use functions like `extractValue`, `extractQuery`, and `consistent` * **GiST indexes** use functions like `consistent`, `union`, `compress`, `decompress`, and `penalty` Previously, Springtail only supported B-tree secondary indexes. This implementation extends the system to: 1. Capture and store operator class metadata from PostgreSQL 2. Route index operations to the appropriate opclass-specific functions 3. Provide the foundation for building and maintaining GIN/GiST indexes ### Goals * Store `opclass` (operator class name) for each index column and `index_type` (btree, gin, gist) for each index * Enable dynamic invocation of opclass support functions via `OpClassHandler` * Prepare the indexer infrastructure to handle non-B-tree index types *** ## Implementation Details ### 1. New Data Structures #### OpClassHandler (`include/common/constants.hh`) ```cpp theme={null} struct OpClassHandler { using OpClassFunc = uintptr_t (*)(const std::string& opclass_name, int support_number, uintptr_t /*Datum*/ datum); OpClassFunc opclass_func = nullptr; ExtensionContext context = {}; }; ``` This handler encapsulates a callback for invoking opclass-specific functions. The `support_number` parameter identifies which support function to call (e.g., `GIST_CONSISTENT = 1`, `GIN_COMPARE = 1`). #### Index Type Constants ```cpp theme={null} static constexpr std::string_view INDEX_TYPE_GIN = "gin"; static constexpr std::string_view INDEX_TYPE_GIST = "gist"; static constexpr std::string_view INDEX_TYPE_BTREE = "btree"; ``` ### 2. Schema Extensions #### Replication Messages (`include/pg_repl/pg_repl_msg.hh`) Extended `PgMsgSchemaIndexColumn` with: ```cpp theme={null} std::string opclass; // operator class name (e.g., "tsvector_ops", "int4_ops") ``` Extended `PgMsgIndex` with: ```cpp theme={null} std::string index_type; // "gin", "gist", or "btree" ``` #### Internal Schema (`include/storage/schema.hh`) Extended `Index::Column` with: ```cpp theme={null} std::string opclass; ``` Extended `Index` with: ```cpp theme={null} std::string index_type; ``` #### System Tables (`include/sys_tbl_mgr/system_tables.hh`) **Indexes table** - Added column: | Column | Position | Type | | ------- | -------- | ---- | | OPCLASS | 6 | TEXT | **IndexNames table** - Added column: | Column | Position | Type | | ----------- | -------- | ---- | | INDEX\_TYPE | 8 | TEXT | ### 3. PostgreSQL Trigger Updates (`scripts/triggers.sql`) Modified the index creation trigger to extract opclass and index type from PostgreSQL system catalogs: ```sql theme={null} SELECT i.indexrelid AS index_oid, i.indclass AS indclass, am.amname AS index_type FROM pg_index i JOIN pg_class ic ON ic.oid = i.indexrelid JOIN pg_am am ON am.oid = ic.relam ... -- Extract opclass for each column SELECT opc.opcname AS opclass FROM unnest(ind_obj.indkey, ind_obj.indclass) WITH ORDINALITY AS u(attnum, opclass_oid, ord) JOIN pg_opclass opc ON opc.oid = u.opclass_oid ``` This captures: * `am.amname`: The access method name (btree, gin, gist, brin) * `opc.opcname`: The operator class name for each index column ### 4. MutableBTree Extensions (`include/storage/mutable_btree.hh`) Extended constructor to accept opclass handler and index type: ```cpp theme={null} MutableBTree(uint64_t database_id, const std::filesystem::path &file, const std::vector &keys, ExtentSchemaPtr schema, uint64_t xid, uint64_t max_extent_size, const ExtensionCallback &extension_callback = {}, const OpClassHandler &opclass_handler = {}, const std::string_view index_type = constant::INDEX_TYPE_BTREE); ``` New member variables: ```cpp theme={null} OpClassHandler _opclass_handler; std::string_view _index_type; ``` ### 5. Table Manager Updates #### MutableTable (`include/sys_tbl_mgr/mutable_table.hh`) Extended `create_index_root` signature: ```cpp theme={null} MutableBTreePtr create_index_root( uint64_t index_id, const std::vector& index_columns, const ExtensionCallback& extension_callback = {}, const OpClassHandler& opclass_handler = {}, const std::string_view index_type = constant::INDEX_TYPE_BTREE); ``` #### TableMgr (`include/sys_tbl_mgr/table_mgr.hh`) Extended `get_snapshot_table` to accept `OpClassHandler`: ```cpp theme={null} MutableTablePtr get_snapshot_table( uint64_t db_id, uint64_t table_id, uint64_t snapshot_xid, ExtentSchemaPtr schema, const std::vector& secondary_keys, const ExtensionCallback &extension_callback = {}, const OpClassHandler &opclass_handler = {}); ``` ### 6. Indexer Changes (`src/pg_log_mgr/indexer.cc`) The indexer now branches based on index type: ```cpp theme={null} if (idx._index_request.index().index_type() == constant::INDEX_TYPE_GIN) { //XXX: Build GIN INDEX } else { // Default - btree index builder root = mutable_table->create_index_root(index_id, idx_cols, {PgExtnRegistry::get_instance()->comparator_func}); // ... existing B-tree build logic } ``` Similar branching exists for: * Index invalidation during updates * Index population during reconciliation ### 7. Protobuf Schema Updates (`src/proto/sys_tbl_mgr.proto`) ```protobuf theme={null} message IndexColumn { string name = 1; int32 position = 2; int32 idx_position = 3; string opclass = 4; // NEW } message IndexInfo { ... string index_type = 10; // NEW } ``` *** ## Data Flow ### Index Creation ``` PostgreSQL CREATE INDEX ↓ DDL Trigger (triggers.sql) ↓ Extract: opclass, index_type from pg_opclass, pg_am ↓ Replication Message (PgMsgIndex) ↓ sys_tbl_mgr::Server::_create_index() ↓ Store in IndexNames (index_type) and Indexes (opclass) system tables ↓ Indexer loads Index metadata (includes index_type, opclass per column) ↓ Indexer builds index based on index_type ↓ Create MutableBTree with OpClassHandler ↓ For GIN/GiST: invoke opclass methods via OpClassHandler ``` *** ## Path to completion The core implementation is complete. The remaining blocker is a build failure in the unit test `src/pg_fdw/test/where_test.cc`. **Issue:** The test file imports both `table` and `mutable_table` headers simultaneously. These headers have conflicting dependencies—one pulls in custom Springtail extension-related imports while the other includes default PostgreSQL imports, causing symbol conflicts during compilation. **Resolution:** Avoid using `mutable_table` in the test. Instead, load the table data directly and use only the `Table` class for scan operations during testing. # Performance Testing Source: https://docs.springtail.io/performance-testing Performance testing covers two complementary approaches: the **load generator**, which measures ingest performance against the primary across runs to catch regressions, and **`perf`/`pgbench` profiling**, which captures CPU hotspots in performance-critical components as flame graphs. # Load generator The load generator script is available in `/python/performance/load_generator.py` It contains a config file `/python/performance/load_config.yaml` The config file looks like below ### Configuration sample ```yaml theme={null} system_json_path: '../../system.json.test' # Number of schemas to create num_schemas: 10 # Table configuration table_configuration: # Number of tables per schema num_tables: 10 # Number of columns per table (range) min_columns: 3 max_columns: 10 # Index configuration index_configuration: # Number of indexes per table (range) min_indexes: 0 max_indexes: 2 # Number of columns per index (range) min_columns_per_index: 1 max_columns_per_index: 2 # Load configuration load_configuration: # Number of inserts per table num_inserts: 500 # Number of updates per table num_updates: 250 # Number of deletes per table num_deletes: 125 # Whether to allow nulls in the data for inserts allow_nulls: rows: true cols: true # Whether to batch inserts batched_inserts: false # Operations to perform operations: ['create_table', 'create_index', 'insert_data', 'update_data', 'delete_data'] # Whether to use the previous configuration to load tables use_existing_config: true # Comparison threshold for the final aggregates, increase of the percentage will result in failure comparison_threshold: 15.0 file_configuration: base_dir: 'performance_test_output' # Directory for previous run files prev_run_dir: 'performance_test_output_prev_run' # Directory for output files output_files: dir: 'output_files' # Query info generated by the load_generator script query_info: 'query_info.csv' # Final report final_report: 'final_report.xlsx' # Final traces - Raw trace data final_traces: 'final_traces.csv' # Final aggregates final_aggregates: 'final_aggregates.csv' # Directory for meta files meta_files: dir: 'meta_files' # Run configuration run_config: 'run_config.csv' # SQL generated by the load_generator script load_sql: 'load.sql' # Table columns table_columns: 'table_columns.json' # Table columns table_columns_csv: 'table_columns.csv' # Directory for temporary files temporary_files: dir: 'temporary_files' # XID traces xid_traces: 'xid_traces.csv' # PG-XID traces pgxid_traces: 'pgxid_traces.csv' # XID to PG-XID mapping xid_pgxid_mapping: 'xid_pgxid_mapping.csv' # Merged traces merged_traces: 'merged_traces.csv' # PG-XID summary pg_xid_summary: 'pg_xid_summary.csv' use_s3: true metrics: ingest_total_time: label: 'Ingest total time (ms)' type: 'negative' primary_total_time: label: 'Primary total time (ms)' type: 'display' ingest_outperform_primary_percentage: label: 'Percentage where the ingest is faster than the primary' type: 'positive' ingest_outperform_primary_count: label: 'Number of times the ingest is faster than the primary' type: 'positive' primary_outperform_ingest_count: label: 'Number of times the primary is faster than the ingest' type: 'display' ``` ### Configuration details | Section | Config | Description | | ------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Config | system\_json\_path | Contains the path to the system JSON config.
This is used to read connection to the primary database | | DDL | num\_schema | Determines the number of schemas to be created | | | table\_configuration.num\_tables | Number of tables under each schema | | | table\_configuration.min\_columns / max\_columns | Min and max number of columns per table.
There are random column types that are created.
The tables always have id, created\_at and updated\_at columns.
The random types are: TEXT, INT, BIGINT, FLOAT, DOUBLE PRECISION, BOOLEAN, DATE, TIME, VARCHAR(255), CHAR(10), NUMERIC(10,2) | | | index\_configuration.min\_indexes / max\_indexes | Min and max indexes per table | | | index\_configuration.min\_columns\_per\_index / max\_columns\_per\_index | Number of columns per index | | DML | load\_configuration.num\_inserts | Number of inserts per table.
Follows batched\_inserts config to determine if inserts are done together or in batches | | | load\_configuration.num\_updates | Randomly selects 1-3 columns to update.
Uses ORDER BY with random column.
Uses generate\_values\_list for new values | | | load\_configuration.num\_deletes | Randomly deletes the number of rows specified.
Uses ORDER BY with random column | | | load\_configuration.allow\_nulls.rows | When inserts are done, this flag specifies if there can be NULL rows (except the id, created\_at, updated\_at columns).
Randomly selects 10-15% of the total number of INSERTS | | | load\_configuration.allow\_nulls.cols | When inserts are done, this flag specifies if there can be NULL values for the columns (except the id, created\_at, updated\_at columns).
Randomly select 10-15% of the columns | | Other | batched\_inserts | Sets if the number of inserts are done in a single transaction or if there are batches of inserts | | | operations | List of operations done.
Possible values: create\_table, create\_index, insert\_data, update\_data, delete\_data | | | use\_existing\_config | If set to true, it will look up the existing config file and recreate the same set of tables as before | | | use\_s3 | If set to true, the previous run configuration will be fetched from S3.
If not, any run that happened in local before will be copied over to a \_prev\_run folder and considered as the previous run.
If no previous run folder is present, it will be treated as a fresh run | | | comparison\_threshold | Threshold above which the performance is considered to have degraded.
See "Metrics" for more | | Files | base\_dir | Base output directory for the current run | | | prev\_run\_dir | Base output directory for the previous run | | | output\_files | Contains the following properties:
• dir (Main directory for the output files)
• query\_info (The CSV file containing the details about the queries)
• final\_report (The XLSX file containing the final aggregated report data)
• final\_traces (The CSV files containing the traces generated after running the load generator script)
• final\_aggregates (The CSV files containing the aggregate data like total time taken to be used in the final\_report) | | | meta\_files | Contains the following properties:
• dir (Main directory for the meta files)
• run\_config (The load generator configuration for the current run)
• load\_sql (A raw SQL that can be run to redo the steps in the current execution)
• table\_columns (A JSON file containing the information about the tables created as part of the current run, needed for existing\_config)
• table\_columns\_csv (CSV file with the information similar to table\_columns JSON) | | | temporary\_files | Contains the following properties:
• dir (Main directory for the temporary files)
• xid\_traces (Traces mapping XID to logs)
• pgxid\_traces (Traces mapping PGXID to logs)
• xid\_pgxid\_mapping (Mapping between XID to PGXID)
• merged\_traces (Trace file containing traces with both XID and PGXID mapped)
• pg\_xid\_summary (Summary file with time mapped from the other temporary files) | | Metrics | | Metrics section containing the different type of configured metrics.
There are 3 types as of now:
1. Negative (If the value goes down from previous run, it's considered good)
2. Positive (If the value goes up from previous run, it's considered good)
3. Display (Mostly used for Primary based metrics which are only used as a display) | ### Run command ```bash theme={null} cd /python/performance python3 load_generator.py -c load_config.yaml ``` # Profiling with `perf` and `pgbench` Performance testing of the system can be done by using tools like `perf` and `pgbench`. We use `perf` to attach to the process whose performance needs to be evaluated. Two performance critical components in the system are `proxy` (data access) and `pg_log_mgr_daemon` (for data replication). Here are the instructions for running performance test, collecting `perf` output, and post-processing it. 1. Start you test environment. Make sure the system is up, end to end. If you are only interested in looking at performance of the ingestion pipeline, running `proxy` is not required. It is better to run each part of the service in its own virtual machine so that the performance of different components does not interfere which the performance of the service whose you are trying to evaluate. 2. Log into the virtual machine of the service and run: ``` sudo perf record -g -p ``` 3. Generate load on the primary database directly or through the proxy: ``` # run pgbench init pgbench -i -s 10 -h -p -U # run pgbench load test pgbench -c 10 -j 10 -T 120 -h -p -U ``` * What we basically do here is running 10 database clients in 10 separate threads (one per thread) for 120 seconds. There are other options available for `pgbench`. All the information is available on this page: [pgbench](https://www.postgresql.org/docs/18/pgbench.html). When running directly against primary database, configure this database host name and port. When running against proxy, configure the host name and port that belong to proxy. 4. Once `pgbench` is done, stop the `perf record` command and post-process the data it collected using the following command: ``` sudo perf script -f > out.perf ``` 5. The output file we created in the previous step needs to be folded. There is a tool that does it: ``` # download the tool git clone https://github.com/brendangregg/FlameGraph.git # run the tool /FlameGraph/stackcollapse-perf.pl out.perf > out.folded ``` 6. The output file we produced in the previous step can now be used to view as a flame graph. Go to [https://www.speedscope.app/](https://www.speedscope.app/) and drag and drop the file. # PgExtnRegistry API Reference Source: https://docs.springtail.io/pgextnregistry-api-reference # PgExtnRegistry API Reference ## Overview `PgExtnRegistry` is a singleton class that manages PostgreSQL extension metadata and function pointers. It serves as the central registry for all loaded extensions, providing lookup and invocation APIs for extension types, operators, and operator classes. **Location:** `src/pg_ext/extn_registry.cc`, `include/pg_ext/extn_registry.hh` **Key Responsibilities:** * Load extension shared libraries via `dlopen()` * Maintain mappings from OIDs and names to function pointers * Provide type conversion utilities (binary ↔ Datum ↔ string) * Support opclass method invocation for GIN/GIST indexes * Enable extension operator comparisons *** ## Core Data Structures ### PgType Represents a PostgreSQL type with its I/O functions: ```cpp theme={null} struct PgType { uint32_t oid; // PostgreSQL type OID std::string typinput; // Text input function name (e.g., "int4in") std::string typoutput; // Text output function name (e.g., "int4out") std::string typreceive; // Binary receive function name (e.g., "int4recv") std::string typsend; // Binary send function name (e.g., "int4send") }; ``` ### PgOpsClass Represents an operator class (e.g., `gin_trgm_ops`, `gist_point_ops`): ```cpp theme={null} struct PgOpsClass { uint32_t oid; // OpClass OID from pg_opclass std::string name; // OpClass name (e.g., "gin_trgm_ops") std::string schema; // Schema name (e.g., "public") std::string access_method; // Access method: "gin" or "gist" std::string family; // OpFamily name }; ``` ### PgOpsClassMethod Represents a support function for an operator class: ```cpp theme={null} struct PgOpsClassMethod { uint32_t input_type_oid; // Input type OID std::string input_type; // Input type name uint32_t key_type_oid; // Key type OID (for index storage) std::string key_type; // Key type name int support_number; // Support proc number (1-11) std::string function_name; // Function name (e.g., "gin_extract_value_trgm") PGFunction function_ptr = nullptr; // dlsym() loaded function pointer }; ``` **Support Numbers (from `constants.hh`):** **GIN:** * `GIN_COMPARE = 1` - Compare function * `GIN_EXTRACTVALUE = 2` - Extract keys from indexed value * `GIN_EXTRACTQUERY = 3` - Extract keys from query * `GIN_CONSISTENT = 4` - Check if entry matches query * `GIN_COMPARE_PARTIAL = 5` - Partial match comparison * `GIN_TRICONSISTENT = 6` - Ternary consistency check * `GIN_OPTIONS = 7` - Index options **GIST:** * `GIST_CONSISTENT = 1` - Check consistency * `GIST_UNION = 2` - Union/merge predicates * `GIST_COMPRESS = 3` - Compress leaf value * `GIST_DECOMPRESS = 4` - Decompress value * `GIST_PENALTY = 5` - Calculate insertion penalty * `GIST_PICKSPLIT = 6` - Split page algorithm * `GIST_EQUAL = 7` - Equality check * `GIST_DISTANCE = 8` - Distance for KNN * `GIST_FETCH = 9` - Fetch tuple * `GIST_OPTIONS = 10` - Index options * `GIST_SORTSUPPORT = 11` - Sort support *** ## Library Management ### init\_libraries() Load an extension shared library and register it. ```cpp theme={null} void init_libraries(uint64_t db_id, const std::string& extension, const std::string& extension_lib_path); ``` **Parameters:** * `db_id`: Database ID * `extension`: Extension name (e.g., "pg\_trgm") * `extension_lib_path`: Full path to .so file (e.g., "/usr/lib/postgresql/16/lib/pg\_trgm.so") **Behavior:** * Calls `dlopen(lib_path, RTLD_NOW | RTLD_GLOBAL)` * Stores library handle in `_library_map[extension]` * Throws error if library cannot be loaded **Example:** ```cpp theme={null} auto registry = PgExtnRegistry::get_instance(); registry->init_libraries(16384, "pg_trgm", "/usr/lib/postgresql/16/lib/pg_trgm.so"); ``` *** ## Type Management APIs ### add\_type() Register an extension type and its I/O functions. ```cpp theme={null} void add_type(const std::string& extension, uint32_t oid, const std::string& typinput, const std::string& typoutput, const std::string& typreceive, const std::string& typsend); ``` **Parameters:** * `extension`: Extension name * `oid`: PostgreSQL type OID * `typinput`: Text input function name * `typoutput`: Text output function name * `typreceive`: Binary receive function name * `typsend`: Binary send function name **Behavior:** * Uses `dlsym()` to load all four I/O functions from extension library * Stores functions in `_type_func_name_to_func` map * Stores type metadata in `_type_oid_to_type` map **Example:** ```cpp theme={null} registry->add_type("pg_trgm", 16385, "gtrgmin", "gtrrgmout", "gtrgmrecv", "gtrgmsend"); ``` *** ### get\_type\_by\_oid() Retrieve type metadata by OID. ```cpp theme={null} PgType get_type_by_oid(uint32_t oid) const; ``` **Returns:** `PgType` structure or empty struct if not found **Example:** ```cpp theme={null} PgType type = registry->get_type_by_oid(16385); std::cout << "Type input function: " << type.typinput << std::endl; ``` *** ### get\_type\_func\_by\_type\_name() Get a type I/O function by name. ```cpp theme={null} PGFunction get_type_func_by_type_name(const std::string& type_name) const; ``` **Parameters:** * `type_name`: Function name (e.g., "int4in", "gtrgmrecv") **Returns:** Function pointer or `nullptr` if not found *** ### binary\_to\_datum() Convert binary wire format to PostgreSQL Datum using `typreceive` function. ```cpp theme={null} Datum binary_to_datum(const std::span& value, Oid pg_oid, int32_t atttypmod) const; ``` **Parameters:** * `value`: Binary data from PostgreSQL wire protocol * `pg_oid`: Type OID * `atttypmod`: Type modifier (e.g., varchar length) **Returns:** `Datum` representation of the value **Implementation:** ```cpp theme={null} Datum PgExtnRegistry::binary_to_datum(const std::span& value, Oid pg_oid, int32_t atttypmod) const { auto type = get_type_by_oid(pg_oid); auto typreceive = get_type_func_by_type_name(type.typreceive); // Create StringInfo for binary data StringInfoData string; initStringInfo(&string); appendBinaryStringInfoNT(&string, value.data(), value.size()); // Call typreceive function PGFunction typreceive_func = (PGFunction)typreceive; Datum result = DirectFunctionCall3(typreceive_func, PointerGetDatum(&string), ObjectIdGetDatum(0), Int32GetDatum(atttypmod)); return result; } ``` **Example:** ```cpp theme={null} // Convert binary point data to Datum std::span binary_point = get_binary_data(); Datum point_datum = registry->binary_to_datum(binary_point, POINTOID, -1); ``` *** ### datum\_to\_string() Convert Datum to string representation using `typoutput` function. ```cpp theme={null} std::string datum_to_string(Datum value, Oid pg_oid) const; ``` **Parameters:** * `value`: Datum to convert * `pg_oid`: Type OID **Returns:** String representation of the value **Implementation:** ```cpp theme={null} std::string PgExtnRegistry::datum_to_string(Datum value, Oid pg_oid) const { auto type = get_type_by_oid(pg_oid); auto typoutput = get_type_func_by_type_name(type.typoutput); // Call typoutput function PGFunction typoutput_func = (PGFunction)typoutput; Datum result = DirectFunctionCall1(typoutput_func, value); const char* str = DatumGetCString(result); return std::string(str); } ``` **Example:** ```cpp theme={null} // Convert point datum to string "(1.5,2.3)" std::string point_str = registry->datum_to_string(point_datum, POINTOID); ``` *** ## Operator Management APIs ### add\_operator() Register an extension operator. ```cpp theme={null} void add_operator(const std::string& extension, uint32_t oid, const std::string& oper_name, const std::string& oper_proc); ``` **Parameters:** * `extension`: Extension name * `oid`: Operator OID from `pg_operator` * `oper_name`: Operator symbol (e.g., `=`, `<@`, `&&`) * `oper_proc`: Implementation function name **Behavior:** * Uses `dlsym()` to load operator function from extension library * Stores in `_oper_name_to_func` and `_proc_name_to_func` maps * Stores OID mappings in `_oper_oid_to_name` and `_proc_oid_to_name` **Example:** ```cpp theme={null} registry->add_operator("pg_trgm", 3636, "%", "similarity"); ``` *** ### get\_operator\_func\_by\_oid() Get operator function by OID. ```cpp theme={null} PGFunction get_operator_func_by_oid(uint32_t oid) const; ``` **Returns:** Function pointer or `nullptr` if not found *** ### get\_operator\_func\_by\_oper\_name() Get operator function by operator symbol. ```cpp theme={null} PGFunction get_operator_func_by_oper_name(const char* oper_name) const; ``` **Parameters:** * `oper_name`: Operator symbol (e.g., `=`, `%`, `<@`) **Returns:** Function pointer or `nullptr` if not found **Example:** ```cpp theme={null} void* similarity_func = registry->get_operator_func_by_oper_name("%"); ``` *** ### get\_operator\_func\_by\_proc\_name() Get operator function by procedure name. ```cpp theme={null} PGFunction get_operator_func_by_proc_name(const std::string& proc_name) const; ``` **Example:** ```cpp theme={null} void* func = registry->get_operator_func_by_proc_name("similarity"); ``` *** ### comparator\_func() Compare two values using an extension operator. ```cpp theme={null} static bool comparator_func(const ExtensionContext* context, const std::span& lhval, const std::span& rhval); ``` **Parameters:** * `context`: Contains `type_oid` and `op_str` (operator name) * `lhval`: Left-hand value (binary format) * `rhval`: Right-hand value (binary format) **Returns:** Boolean result of comparison **Implementation:** ```cpp theme={null} bool PgExtnRegistry::comparator_func(const ExtensionContext* context, const std::span& lhval, const std::span& rhval) { auto extn_registry = PgExtnRegistry::get_instance(); // Convert binary to Datum Datum left_datum = extn_registry->binary_to_datum(lhval, context->type_oid, -1); Datum right_datum = extn_registry->binary_to_datum(rhval, context->type_oid, -1); // Get operator function auto operator_func = extn_registry->get_operator_func_by_oper_name(context->op_str); // Invoke operator PGFunction operator_func_ptr = (PGFunction)operator_func; Datum result = DirectFunctionCall3(operator_func_ptr, left_datum, right_datum, ObjectIdGetDatum(0)); return DatumGetBool(result); } ``` **Example:** ```cpp theme={null} // Compare two PostGIS geometries for equality ExtensionContext ctx; ctx.type_oid = GEOMETRYOID; ctx.op_str = "="; bool equal = PgExtnRegistry::comparator_func(&ctx, geometry1_binary, geometry2_binary); ``` *** ## Operator Class APIs ### add\_opclass() Register an operator class method. ```cpp theme={null} void add_opclass(const std::string& extension, PgOpsClass opclass, PgOpsClassMethod method); ``` **Parameters:** * `extension`: Extension name * `opclass`: OpClass metadata * `method`: Method metadata including support number and function name **Behavior:** * Uses `dlsym()` to load support function * Stores in nested map: `_opclass_function_map[opclass_name][support_number]` **Example:** ```cpp theme={null} PgOpsClass gin_trgm; gin_trgm.name = "gin_trgm_ops"; gin_trgm.access_method = "gin"; PgOpsClassMethod extractvalue; extractvalue.support_number = GIN_EXTRACTVALUE; extractvalue.function_name = "gin_extract_value_trgm"; registry->add_opclass("pg_trgm", gin_trgm, extractvalue); ``` *** ### get\_opclass\_method\_by\_method\_name() Retrieve an opclass method by name and support number. ```cpp theme={null} PgOpsClassMethod get_opclass_method_by_method_name(const std::string& opclass_name, int support_number); ``` **Parameters:** * `opclass_name`: OpClass name (e.g., "gin\_trgm\_ops") * `support_number`: Support procedure number (e.g., `GIN_EXTRACTVALUE = 2`) **Returns:** `PgOpsClassMethod` structure with function pointer, or empty struct if not found **Example:** ```cpp theme={null} auto method = registry->get_opclass_method_by_method_name("gin_trgm_ops", GIN_EXTRACTVALUE); if (method.function_ptr) { PGFunction func = (PGFunction)method.function_ptr; // Use function... } ``` *** ### get\_opclass\_method\_func\_ptr\_by\_method\_name() Get opclass method function pointer. ```cpp theme={null} PGFunction get_opclass_method_func_ptr_by_method_name(const std::string& opclass_name, int support_number) const; ``` **Returns:** Function pointer or `nullptr` if not found **Example:** ```cpp theme={null} void* func = PgExtnRegistry::get_opclass_method_func_ptr_by_method_name("gist_point_ops", GIST_COMPRESS); ``` *** ## Internal Data Structures ### Registry Maps ```cpp theme={null} private: // Library handles std::unordered_map _library_map; // extension_name → dlopen() handle // Type mappings std::unordered_map _type_oid_to_type; // type_oid → PgType std::unordered_map _type_func_name_to_func; // function_name → function_ptr // Operator mappings std::unordered_map _oper_oid_to_name; // operator_oid → operator_name std::unordered_map _proc_oid_to_name; // proc_oid → proc_name std::unordered_map _oper_name_to_func; // operator_name → function_ptr std::unordered_map _proc_name_to_func; // proc_name → function_ptr // Opclass mappings std::unordered_map> _opclass_function_map; // opclass_name → (support_number → PgOpsClassMethod) ``` *** ## Usage Patterns ### Pattern 1: Type I/O Conversion ```cpp theme={null} // Binary (wire protocol) → Datum → String std::span binary_data = ...; Datum datum = registry->binary_to_datum(binary_data, type_oid, -1); std::string text = registry->datum_to_string(datum, type_oid); ``` ### Pattern 2: Opclass Method Invocation ```cpp theme={null} // Get method, check validity, invoke auto method = registry->get_opclass_method_by_method_name(opclass, support_num); if (method.function_ptr) { PGFunction func = (PGFunction)method.function_ptr; Datum result = DirectFunctionCall1(func, input_datum); // Process result... } ``` ### Pattern 3: Extension Operator Comparison ```cpp theme={null} // Setup context ExtensionContext ctx; ctx.type_oid = custom_type_oid; ctx.op_str = "="; // Compare bool result = PgExtnRegistry::comparator_func(&ctx, lhs_binary, rhs_binary); ``` *** ## Thread Safety **Current Status:** Not thread-safe * `dlopen()` and `dlsym()` calls are not synchronized * Registry maps are accessed without locks * Singleton instance is not thread-safe **Recommendations:** * Initialize all extensions before starting worker threads * Treat registry as read-only after initialization * Add mutex protection if runtime extension loading is needed *** ## Error Handling All lookup methods log errors and return `nullptr` / empty struct on failure: ```cpp theme={null} void* func = registry->get_operator_func_by_oid(invalid_oid); // Logs: "Failed to find operator function by oid: " // Returns: nullptr ``` **Best Practice:** Always check for `nullptr` before using function pointers: ```cpp theme={null} void* func = registry->get_type_func_by_type_name("some_func"); if (!func) { // Handle error return; } PGFunction pg_func = (PGFunction)func; // Safe to use ``` *** ## Performance Considerations 1. **Function Pointer Caching:** Frequently-used functions should be cached at call site 2. **OID Lookups:** Type/operator OID lookups involve map lookups; cache results when possible 3. **Datum Conversions:** Binary ↔ Datum conversions invoke extension functions; minimize conversions 4. **dlsym() Cost:** Symbol lookups are relatively expensive; done once during registration *** # Postgres Compatibility Source: https://docs.springtail.io/postgres-compatibility Springtail achieves Postgres compatibility through three core mechanisms: 1. The use of the Postgres Foreign Data Wrapper as a frontend within the compute node. 2. The ability of the Springtail Proxy to route queries to the primary database that it does not support. These typically include queries that modify data. For example, DDL operations like `CREATE TABLE` | `ALTER TABLE` | `UPDATE` | `DELETE` | `DROP` | etc. 3. The use of database triggers on the primary database to replicate schema changes (DDL changes) that are not otherwise replicated through the Postgres Logical Replication Protocol. With these mechanisms, Springtail achieves full PostgreSQL compatibility. However, a few limitations exist for replicating specific data types and tables imposed by the constraints of the Postgres Logical Replication Protocol. ### Data limitations Certain types of tables cannot be replicated to Springtail due to limitations within the Postgres Logical Replication Protocol. These include: * Tables with generated columns; columns defined as functions over other columns * Tables or databases not using UTF-8 encodings * Tables or databases not using C or C.UTF-8 collations Certain operations require a full resync of table data, again, usually due to the lack of information present in the replication stream. While the table is being synced, queries for that table will be redirected to the primary. These operations include: * The modification of the primary key (this requires re-indexing the table) * A column changing from nullable to not-nullable * A change to the column’s type * Adding a new column with a default value # Postgres Replication Source: https://docs.springtail.io/postgres-replication ## What is Postgres database replication? PostgreSQL database replication is a process of creating and maintaining multiple copies of a database across different servers. This technique is crucial for ensuring data availability, improving performance, and enhancing disaster recovery capabilities. ## High-level use cases Database replication serves several important purposes: Ensures continuous access to data even if one server fails. Distributes read queries across multiple servers to improve performance. Provides a backup solution in case of data loss or system failure. Allows for running complex queries on a replica without affecting the primary database's performance. ## How databases work with write-ahead logging Write-Ahead Logging (WAL) is a crucial component that is used by many databases to ensure data is not lost during a crash. The log also makes it much simpler to perform replication between database instances. Here’s how it works: * The WAL records all changes made to the database before they are written to the actual data files. * Each WAL record contains the details of the changes made within each transaction and is assigned a new, unique, Log Sequence Number (LSN) to identify that change. * The WAL can be replayed upon restarting after a crash to ensure data integrity. It also enabled point-in-time recovery and replication. * For replication, WAL records are sent from the primary server to replicas, allowing them to replay the changes and stay in sync. For WAL configuration options, refer to the official PostgreSQL documentation: [Write Ahead Log](https://www.postgresql.org/docs/16/runtime-config-wal.html) ## Physical vs. logical replication PostgreSQL supports two main types of replication: **physical** and **logical**, each with distinct advantages and use cases. ### **Physical replication** Physical replication involves replicating physical data blocks after changes are applied. It is beneficial for large-scale replication because it copies data at the block level, making it faster. Additionally, physical replication supports streaming replication for near real-time updates. However, it replicates the entire database cluster, which includes all databases and system tables, making it less flexible for partial replication or replication across different PostgreSQL versions. Changes to the physical layout of data blocks across versions can break compatibility. ### Logical replication Logical replication exposes a logical representation of changes made within transactions, e.g., an update X was applied to row Y, rather than replicating the result stored within physical data blocks. This allows for the replication of specific tables or databases, and supports cross-version replication, which makes it more flexible for complex replication scenarios. However, logical replication can be slower for large-scale replication, requires more configuration and management, and does not support certain database modifications, such as schema changes involving `ALTER TABLE`. ## Logical replication in Postgres Today, logical replication is the preferred replication mechanism due to the cross-version compatibility and the ease of setup. However, it does have some limitations which are described below. Configuring logical replication in Postgres involves several steps and must be performed at a per-database level, not at the instance level. ### Configuration file (wal\_level for logical repl) To enable logical replication, you need to set the `wal_level` parameter to **logical** in the **postgresql.conf** file on the publisher server. This ensures that the WAL contains enough information for logical decoding. ### Publications and subscriptions **Publications** are created on the publisher (source) database and define the set of tables/schemas for which changes should be replicated. It is possible to create a publication that covers all new tables created within a database or schema. **Subscriptions** are set up on the subscriber (target) database and specify which publications to receive. ### Replication slot In logical replication, the WAL is used to capture logical changes to the database, and these changes are sent to the replication slot used by the logical replication process. Each change captured by logical decoding will receive a new LSN. These LSNs allow the logical replication client (like a subscriber) to track its position in the stream and ensure that no data is missed during replication. The replication slot is the mechanism that keeps track of changes that need to be sent to subscribers. It ensures that WAL records are not removed until all changes have been replicated. Specifically, the replication slot tracks the LSN of the WAL records that have been sent to the replica, as well as those that are acknowledged by the replica. ### Output plugin The output plugin in Postgres plays a vital role in logical replication by converting the changes captured from the WAL into a format that can be consumed by replication subscribers or external systems. It handles decoding the logical changes, formatting them, and streaming them in a structured format. It also manages replication flow by handling feedback from the subscriber, such as acknowledgment messages or LSNs. The output plugin is specified when creating the replication slot and is tied to that replication slot. While Postgres provides a default plugin (**pgoutput**), custom plugins can be created to stream changes in other formats (e.g., JSON — [wal2json](https://github.com/eulerto/wal2json)), making Postgres integration with other systems flexible and efficient. ### Important considerations Only once a specific LSN has been acknowledged can the record be freed from the WAL (thus freeing up disk space). If a replica takes too long to acknowledge a record, the WAL on the primary may continue growing in size. It is important for the primary to maintain the WAL records until acknowledged by the replica, otherwise the replica could lose data if it crashed before applying the changes. Similarly, it is important for the replica to quickly acknowledge records once they’ve been applied, minimizing the amount of data the primary must retain. The distance between the last LSN sent to the replica, and the last LSN acknowledged by replica contributes to replica lag. The time between these events is the time by which the replica is behind the primary. Minimizing this time is important for consistency. For example, if a write is issued to a primary and a read is issued to the replica, it is often desirable for that read to observe that write. If there is a large replica lag, there is a greater amount of time in which the reader may observe an older value. ## Limitations of logical replication While logical replication is powerful, it has some limitations in terms of what operations are replicated and which are not. ### Terminology To understand these limitations, it’s important to distinguish between Data Manipulation Language (DML) and Data Definition Language (DDL) operations: * **DML:** Operations that manipulate data (`INSERT` | `UPDATE` | `DELETE`) * **DDL:** Operations that define or modify the database structure (`CREATE TABLE` | `ALTER TABLE` | etc.) Logical replication primarily replicates DML operations: * `INSERT` | `UPDATE` | `DELETE` operations on tables * `TRUNCATE` operations (configurable) Most DDL operations are not automatically replicated, including: * Schema changes (`CREATE TABLE` | `ALTER TABLE`) * Index changes (both primary and secondary indexes) * Sequence operations All replicated tables must contain a primary index or must be designated as `REPLICA IDENTITY FULL` (within an `ALTER TABLE`) which will treat the entire row as the primary key. See the official PostgreSQL documentation on [Logical Replication Restrictions](https://www.postgresql.org/docs/16/logical-replication-restrictions.html) for more information about limitations. ### Handling non-replicated operations As noted above, operations that modify the schema (the structure of the tables) are typically not replicated using logical replication. There are a couple of ways to manage operations that aren’t automatically replicated: custom triggers or manual synchronization. Custom triggers can be set up to replicate certain operations not covered by logical replication. When an operation that modifies the schema (e.g., `ALTER TABLE`), a trigger (SQL function) can be called. Within this function one can extract the name of the table being modified. Unfortunately, the exact schema change is not available, but it is possible to query the current schema for that table. If setting up a custom replica, it may then be possible to compare the new schema to the old schema to determine the changes that occurred. While it is also possible to detect new index creation, or altered indexes via triggers, it becomes more complicated to automatically make the changes on the replica. Typically the index must be dropped and recreated. Recreating an index may require resorting the entire table (e.g., for a primary index). Manual synchronization is another method for synchronizing schema changes. It requires the changes to be manually applied to both the publisher and subscriber. When doing so, careful planning is required to coordinate schema changes to ensure consistency across all instances. E.g., when manually handling schema changes: * `ALTER TABLE`: Apply changes manually to both publisher and subscriber. * `CREATE TABLE`: Create the table on the subscriber before setting up replication for it. * Primary indexes: Changes must be manually applied to maintain consistency. * Secondary indexes: Can be created or dropped independently on the subscriber for query optimization. ## The replication protocol for consumers Understanding the replication protocol is crucial for consumers of PostgreSQL’s logical replication. The type and format of the messages that are generated by the primary and sent to the replica depend on the Output Plugin specified with the replication slot. The following sections assume the use of the default **pgoutput** plugin. ### Message types The replication protocol primarily uses two types of messages: * **XLogData**: Contains the actual data change messages from the WAL. The XLogData message contains the starting LSN, the ending LSN and the server’s timestamp. It also encodes the actual data change message. There is a message for each type of logical modification (e.g., an `UPDATE` | `INSERT` | `BEGIN` | `COMMIT` message). See the official PostgreSQL documentation on [Logical Replication Message Formats](https://www.postgresql.org/docs/16/protocol-logicalrep-message-formats.html) for more details on each message type. * **KeepAlive**: Sent periodically to maintain the connection and provide status updates. Typically the replica should respond immediately to keep the connection to the primary alive. See the official PostgreSQL documentation on [Streaming Replication Protocol](https://www.postgresql.org/docs/16/protocol-replication.html) for additional information. ### Streaming vs. full transaction messages PostgreSQL supports two modes of sending replication messages: 1. Full transaction mode 2. Streaming mode #### Full transaction mode In full transaction mode, messages are sent when a transaction is committed. No messages for aborted transactions are ever sent. This makes replication simpler, as all data can be replayed. Its biggest downside is that the primary DB must buffer all changes for a transaction until the commit occurs before it can send any data to the replica. If the transaction is large, this can result in buffering a large number of log records to disk, increasing resource utilization and latency. A full transaction starts with a `BEGIN` message and ends with a `COMMIT` message. The Postgres configuration parameter: `logical_decoding_work_mem`, controls how much data is buffered before switching to streaming mode (if streaming mode is enabled). #### Streaming mode In streaming mode, the primary sends messages to the replica as they occur, allowing for real-time replication. This means that changes within a transaction are not buffered until a commit occurs, so ABORTs must be sent in the replication stream. This makes it slightly more complicated for the replica, as it must now handle ABORTed transactions. Streaming messages were introduced in protocol version 2 (in Postgres version 14); as such, if replication is initiated with protocol version 1, stream messages are disabled. At the message level, streaming mode introduces `STREAM START`, `STREAM STOP`, `STREAM COMMIT`, and `STREAM ABORT` messages. Since operations are streamed prior to the commit or abort, streaming mode must also support sub-transactions (generated by `SAVEPOINTs`), as well as interleaved transactions. Messages between a `STREAM START` and `STREAM STOP` belong to the same parent transaction. The `STREAM START` will indicate the Postgres Transaction ID or XID of the parent transaction. After a `STREAM STOP` message, either a `STREAM START`, `STREAM COMMIT`, `STREAM ABORT` or full transaction will follow. To support sub-transactions in streaming mode, the normal replication messages are modified to contain an XID to indicate the (sub) transaction to which they belong. In streaming mode, a sub-transaction is indicated through a different XID than that of the parent. The parent (or top-level) transaction ID is indicated by the `STREAM START` operation. If a sub-transaction is aborted, then an explicit `STREAM ABORT` message will be issued containing both the parent and sub-transaction XID. It is possible to have nested sub-transactions by nesting `SAVEPOINT` commands. Replication messages will always contain the XID of the latest active sub-transaction. It is possible to abort multiple sub-transactions at once (e.g., rolling back to an early `SAVEPOINT`), in this case, multiple `STREAM ABORT` messages will be generated, one for each sub-transaction that is aborted. As well, when a `SAVEPOINT` is released, messages will contain the XID of the latest previous `SAVEPOINT` or the parent transaction (if nesting is fully unrolled). The `STREAM COMMIT`, unlike the abort, only specifies the parent XID; it implicitly commits all non-aborted sub-transactions. Both `STREAM COMMIT` and `STREAM ABORT` messages are sent after a `STREAM STOP`. ### Acknowledging last synced LSN Consumers must periodically acknowledge the latest LSN they’ve successfully processed. This acknowledgment is crucial for several reasons: * It allows the primary server to free up WAL space. Postgres can be configured to only hold onto so much WAL space before it starts reclaiming records. So it is important for the replica to make sure the log doesn’t grow to this point. See the official PostgreSQL documentation on the [max\_slot\_wal\_keep\_size](https://www.postgresql.org/docs/16/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) configuration option. * It helps with the tracking of replication lag. * It ensures that the consumer doesn't miss any data in case of disconnection. The latest LSN that the primary database is aware of can be queried by looking at the `pg_replication_slots` table in Postgres. The column `confirmed_flush_lsn` holds the last flushed LSN that Postgres has received from the replica. ### Fast-forwarding on idle databases For databases with infrequent changes, the replication protocol includes a mechanism to “fast-forward” the stream. Even if the database is idle, the current LSN on the primary may be advancing due to internal cleanup or other activities that occur within the database which don’t result in logical records being replicated. As such, if the LSN is not acknowledged then the primary database will not release the space held by WAL records. So it is important for the replica to periodically request the latest LSN, and acknowledge that LSN so that the primary can reclaim resources. # Postgres Stub layer Source: https://docs.springtail.io/postgres-stub-layer # pg\_ext: PostgreSQL Compatibility Library ## Overview The `pg_ext` library provides a minimal reimplementation of PostgreSQL internal APIs, enabling Springtail to execute functions from PostgreSQL extensions without requiring a full PostgreSQL backend. It creates a compatible runtime environment for extension code that expects PostgreSQL's data types, memory management, and function call interface. **Location:** Source files in `src/pg_ext/`, Headers in `include/pg_ext/` **Purpose:** * Provide PostgreSQL-compatible data structures (Datum, text, arrays, etc.) * Implement function call interface (`DirectFunctionCall*()`) * Emulate PostgreSQL memory context system * Support extension-defined types and operators *** ## Architecture ### Core Components ``` Source files (src/pg_ext/): Header files (include/pg_ext/): ├── fmgr.cc ├── fmgr.hh ├── memory.cc ├── memory.hh ├── string.cc ├── string.hh ├── array.cc ├── array.hh ├── numeric.cc ├── numeric.hh ├── date.cc ├── date.hh ├── jsonb.cc ├── jsonb.hh ├── error.cc ├── error.hh ├── node.cc ├── node.hh ├── hash.cc ├── hash.hh ├── list.cc ├── list.hh ├── parser.cc ├── parser.hh ├── heaptuple.cc ├── heaptuple.hh ├── pqformat.cc ├── pqformat.hh ├── bit.cc ├── bit.hh ├── float.cc ├── float.hh ├── guc.cc ├── guc.hh ├── extn_registry.cc ├── extn_registry.hh └── extn_parser.cc └── extn_parser.hh ``` **Components:** * **fmgr**: Function manager (DirectFunctionCall\* wrappers) * **memory**: Memory contexts (palloc, pfree, TopMemoryContext) * **string**: Text type and string utilities * **array**: PostgreSQL array handling * **numeric**: Numeric type support * **date**: Date/time types * **jsonb**: JSONB type support * **error**: Error reporting (ereport, elog) * **node**: PostgreSQL node types * **hash**: Hash functions * **list**: PostgreSQL list structures * **parser**: libpg\_query integration * **heaptuple**: Heap tuple representation * **pqformat**: Wire protocol formatting * **bit**: Bit string operations * **float**: Float type utilities * **guc**: GUC (configuration) stubs * **extn\_registry**: Extension registry (covered separately) * **extn\_parser**: Extension SQL parsing *** ## Function Manager (fmgr) ### Overview Provides macros and functions for calling PostgreSQL functions with a compatible calling convention. **Files:** `src/pg_ext/fmgr.cc`, `include/pg_ext/fmgr.hh` ### Key Macros #### LOCAL\_FCINFO() Allocates stack-local FunctionCallInfo structure: ```cpp theme={null} #define LOCAL_FCINFO(name, nargs) \ FunctionCallInfoBaseData name##data; \ FunctionCallInfo name = &name##data ``` **Usage:** ```cpp theme={null} LOCAL_FCINFO(fcinfo, 2); // Allocate fcinfo for 2 arguments ``` #### InitFunctionCallInfoData() Initialize FunctionCallInfo structure: ```cpp theme={null} void InitFunctionCallInfoData(FunctionCallInfo fcinfo, FmgrInfo* flinfo, int nargs, Oid collation, void* context, void* resultinfo); ``` ### DirectFunctionCall Family Convenience functions for calling PostgreSQL functions: #### DirectFunctionCall1() ```cpp theme={null} Datum DirectFunctionCall1(PGFunction func, Datum arg1); ``` **Implementation:** ```cpp theme={null} Datum DirectFunctionCall1(PGFunction func, Datum arg1) { LOCAL_FCINFO(fcinfo, 1); InitFunctionCallInfoData(*fcinfo, nullptr, 1, 0, nullptr, nullptr); fcinfo->args[0].value = arg1; fcinfo->args[0].isnull = false; return func(fcinfo); } ``` **Example:** ```cpp theme={null} // Call int4in("12345") PGFunction int4in_func = (PGFunction)get_type_func("int4in"); Datum result = DirectFunctionCall1(int4in_func, CStringGetDatum("12345")); int32_t value = DatumGetInt32(result); // value = 12345 ``` #### DirectFunctionCall2() ```cpp theme={null} Datum DirectFunctionCall2(PGFunction func, Datum arg1, Datum arg2); ``` **Example:** ```cpp theme={null} // Call int4add(42, 10) Datum result = DirectFunctionCall2(int4add_func, Int32GetDatum(42), Int32GetDatum(10)); // result = 52 ``` #### DirectFunctionCall3() ```cpp theme={null} Datum DirectFunctionCall3(PGFunction func, Datum arg1, Datum arg2, Datum arg3); ``` #### Collation Variants For functions that require collation (string operations): ```cpp theme={null} Datum DirectFunctionCall1Coll(PGFunction func, Oid collation, Datum arg1); Datum DirectFunctionCall2Coll(PGFunction func, Oid collation, Datum arg1, Datum arg2); Datum DirectFunctionCall3Coll(PGFunction func, Oid collation, Datum arg1, Datum arg2, Datum arg3); Datum DirectFunctionCall7Coll(PGFunction func, Oid collation, ...); // For GIN extractQuery ``` **Example:** ```cpp theme={null} // Case-insensitive text comparison Datum result = DirectFunctionCall2Coll(texteq_func, DEFAULT_COLLATION_OID, text1_datum, text2_datum); ``` ### Datum Type Conversions Macros for converting between C types and Datum: ```cpp theme={null} // Integer conversions #define Int32GetDatum(x) ((Datum)(x)) #define DatumGetInt32(x) ((int32_t)(x)) #define Int64GetDatum(x) ((Datum)(x)) #define DatumGetInt64(x) ((int64_t)(x)) // Pointer conversions #define PointerGetDatum(x) ((Datum)(x)) #define DatumGetPointer(x) ((void*)(x)) // Boolean conversions #define BoolGetDatum(x) ((Datum)((x) ? 1 : 0)) #define DatumGetBool(x) ((bool)(x)) // Float conversions #define Float4GetDatum(x) /* implementation */ #define DatumGetFloat4(x) /* implementation */ #define Float8GetDatum(x) /* implementation */ #define DatumGetFloat8(x) /* implementation */ // String conversions #define CStringGetDatum(x) PointerGetDatum(x) #define DatumGetCString(x) ((char*)DatumGetPointer(x)) // Object ID #define ObjectIdGetDatum(x) ((Datum)(x)) #define DatumGetObjectId(x) ((Oid)(x)) ``` *** ## Memory Management ### Overview PostgreSQL uses a memory context system for allocation tracking and bulk deallocation. `pg_ext` provides a simplified implementation. **Files:** `src/pg_ext/memory.cc`, `include/pg_ext/memory.hh` ### MemoryContext Class ```cpp theme={null} class MemoryContext { public: MemoryContext(MemoryContext* parent, std::string_view name, size_t init_size, size_t max_size); void* alloc(size_t size); void free(void* ptr); void reset(); // Free all allocations void delete_context(); private: std::string _name; size_t _init_size; size_t _max_size; MemoryContext* _parent; std::map> _blocks; std::unordered_map> _large_allocs; }; ``` ### Global Memory Context ```cpp theme={null} extern MemoryContext TopMemoryContext; ``` The global `TopMemoryContext` is the root of all memory contexts: ```cpp theme={null} // In memory.cc MemoryContext TopMemoryContext(nullptr, "TopMemoryContext", 8192, 1048576); ``` ### Allocation Functions #### palloc() Allocate memory from a memory context: ```cpp theme={null} void* palloc(size_t size); void* palloc0(size_t size); // Zero-initialized ``` **Implementation:** ```cpp theme={null} void* palloc(size_t size) { return TopMemoryContext.alloc(size); } void* palloc0(size_t size) { void* ptr = palloc(size); memset(ptr, 0, size); return ptr; } ``` **Example:** ```cpp theme={null} // Allocate 1024 bytes char* buffer = (char*)palloc(1024); // Use buffer... pfree(buffer); ``` #### pfree() Free memory allocated by palloc: ```cpp theme={null} void pfree(void* ptr); ``` #### repalloc() Reallocate memory: ```cpp theme={null} void* repalloc(void* ptr, size_t size); ``` ### Memory Blocks Internal structure for memory management: ```cpp theme={null} struct MemoryBlock { size_t size; // Total size of block size_t pos; // Current position char* memory; // Allocated memory MemoryBlock(size_t size); size_t remaining() const { return size - pos; } }; ``` ### Usage Notes * Extension functions expect `palloc()` for allocations * Memory contexts are simplified compared to PostgreSQL * No support for memory context switching or hierarchies * All allocations go to `TopMemoryContext` *** ## String and Text Types ### Overview PostgreSQL's `text` type is a variable-length binary string with a 4-byte header. **Files:** `src/pg_ext/string.cc`, `include/pg_ext/string.hh` ### Text Structure ```cpp theme={null} struct varlena { int32_t vl_len_; // Length including header (bit 0 = compressed flag) char vl_dat[1]; // Flexible array member }; typedef struct varlena text; ``` ### String Utilities #### cstring\_to\_text() Convert C string to PostgreSQL text: ```cpp theme={null} text* cstring_to_text(const char* str); text* cstring_to_text_with_len(const char* str, int len); ``` **Implementation:** ```cpp theme={null} text* cstring_to_text(const char* str) { int len = strlen(str); return cstring_to_text_with_len(str, len); } text* cstring_to_text_with_len(const char* str, int len) { text* result = (text*)palloc(len + VARHDRSZ); SET_VARSIZE(result, len + VARHDRSZ); memcpy(VARDATA(result), str, len); return result; } ``` **Example:** ```cpp theme={null} text* my_text = cstring_to_text("Hello, World!"); ``` #### cstring\_to\_text\_auto() Wrapper that handles const correctness: ```cpp theme={null} text* cstring_to_text_auto(const char* str); ``` #### text\_to\_cstring() Convert text to C string: ```cpp theme={null} char* text_to_cstring(const text* t); ``` **Implementation:** ```cpp theme={null} char* text_to_cstring(const text* t) { int len = VARSIZE_ANY_EXHDR(t); char* result = (char*)palloc(len + 1); memcpy(result, VARDATA_ANY(t), len); result[len] = '\0'; return result; } ``` **Example:** ```cpp theme={null} char* cstr = text_to_cstring(my_text); printf("%s\n", cstr); pfree(cstr); ``` ### Macros ```cpp theme={null} // Get size of varlena (excluding header) #define VARSIZE_ANY_EXHDR(ptr) (VARSIZE_ANY(ptr) - VARHDRSZ) // Get data pointer #define VARDATA_ANY(ptr) ((char*)(ptr) + VARHDRSZ) #define VARDATA(ptr) VARDATA_ANY(ptr) // Set size (including header) #define SET_VARSIZE(ptr, len) (((varlena*)(ptr))->vl_len_ = (len)) // Variable header size #define VARHDRSZ sizeof(int32_t) ``` *** ## Array Support ### Overview Support for PostgreSQL's array type. **Files:** `src/pg_ext/array.cc`, `include/pg_ext/array.hh` ### ArrayType Structure ```cpp theme={null} struct ArrayType { int32_t vl_len_; // varlena header int ndim; // Number of dimensions int32_t dataoffset; // Offset to data Oid elemtype; // Element type OID int* dim; // Dimension sizes int* lbound; // Lower bounds char* data; // Element data }; ``` ### Array Functions ```cpp theme={null} // Get array element Datum array_get_element(ArrayType* array, int n); // Get number of elements int array_length(ArrayType* array); // Iterate array void array_foreach(ArrayType* array, void (*callback)(Datum, void*), void* context); ``` *** ## Numeric Type ### Overview PostgreSQL's arbitrary-precision numeric type. **Files:** `src/pg_ext/numeric.cc`, `include/pg_ext/numeric.hh` ### Numeric Structure ```cpp theme={null} struct NumericData { int ndigits; // Number of digits int weight; // Weight of first digit int sign; // Sign (positive/negative/NaN) int dscale; // Display scale int16_t* digits; // Digit array }; typedef NumericData* Numeric; ``` ### Numeric Functions ```cpp theme={null} // Convert numeric to int64 int64_t numeric_to_int64(Numeric num); // Convert numeric to double double numeric_to_double(Numeric num); // Convert int64 to numeric Numeric int64_to_numeric(int64_t value); ``` *** ## Date and Time Types ### Overview PostgreSQL date/time types and operations. **Files:** `src/pg_ext/date.cc`, `include/pg_ext/date.hh` ### Types ```cpp theme={null} typedef int32_t DateADT; // Days since 2000-01-01 typedef int64_t Timestamp; // Microseconds since 2000-01-01 00:00:00 typedef int64_t TimestampTz; // Timestamp with timezone typedef int64_t TimeADT; // Microseconds since midnight typedef Interval IntervalADT; // Time interval struct Interval { int64_t time; // Microseconds int32_t day; // Days int32_t month; // Months }; ``` ### Date/Time Functions ```cpp theme={null} // Current timestamp Timestamp GetCurrentTimestamp(); // Date arithmetic Timestamp timestamp_add_interval(Timestamp ts, Interval* interval); // Conversions DateADT timestamp_to_date(Timestamp ts); Timestamp date_to_timestamp(DateADT date); ``` *** ## Error Handling ### Overview PostgreSQL error reporting system. **Files:** `src/pg_ext/error.cc`, `include/pg_ext/error.hh` ### Error Levels ```cpp theme={null} #define DEBUG5 10 #define DEBUG4 11 #define DEBUG3 12 #define DEBUG2 13 #define DEBUG1 14 #define LOG 15 #define NOTICE 18 #define WARNING 19 #define ERROR 20 #define FATAL 21 #define PANIC 22 ``` ### ereport Macro ```cpp theme={null} #define ereport(level, rest) \ do { \ if (level >= ERROR) { \ springtail_error_handler rest; \ } else { \ springtail_log_handler rest; \ } \ } while(0) ``` **Usage:** ```cpp theme={null} if (invalid_input) { ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("Invalid input value: %d", value))); } ``` ### elog Macro ```cpp theme={null} #define elog(level, ...) \ do { \ if (level >= ERROR) { \ throw std::runtime_error(fmt::format(__VA_ARGS__)); \ } else { \ LOG_INFO(__VA_ARGS__); \ } \ } while(0) ``` **Usage:** ```cpp theme={null} elog(ERROR, "Failed to process request: %s", error_msg); ``` *** ## StringInfo (pqformat) ### Overview Dynamic string buffer for building messages. **Files:** `src/pg_ext/pqformat.cc`, `include/pg_ext/pqformat.hh` ### StringInfoData Structure ```cpp theme={null} struct StringInfoData { char* data; // Buffer int len; // Current length int maxlen; // Allocated size int cursor; // Read cursor }; typedef StringInfoData* StringInfo; ``` ### StringInfo Functions ```cpp theme={null} // Initialize void initStringInfo(StringInfoData* str); void resetStringInfo(StringInfoData* str); // Append operations void appendStringInfo(StringInfoData* str, const char* fmt, ...); void appendStringInfoChar(StringInfoData* str, char ch); void appendBinaryStringInfo(StringInfoData* str, const char* data, int datalen); void appendBinaryStringInfoNT(StringInfoData* str, const char* data, int datalen); // Enlarge buffer void enlargeStringInfo(StringInfoData* str, int needed); ``` **Example:** ```cpp theme={null} StringInfoData buf; initStringInfo(&buf); appendStringInfo(&buf, "Value: %d, Name: %s", 42, "test"); appendStringInfoChar(&buf, '\n'); // Use buf.data pfree(buf.data); ``` *** ## Node Types ### Overview PostgreSQL's node type system for parse trees and plans. **Files:** `src/pg_ext/node.cc`, `include/pg_ext/node.hh` ### Node Structure ```cpp theme={null} struct Node { NodeTag type; // Node type identifier }; enum NodeTag { T_Invalid = 0, T_List, T_Integer, T_String, T_SelectStmt, // ... many more }; ``` ### Node Functions ```cpp theme={null} Node* copyObject(const Node* node); bool equal(const Node* a, const Node* b); char* nodeToString(const Node* node); ``` *** ## List Support ### Overview PostgreSQL's doubly-linked list implementation. **Files:** `src/pg_ext/list.cc`, `include/pg_ext/list.hh` ### List Structure ```cpp theme={null} struct ListCell { union { void* ptr_value; int int_value; Oid oid_value; } data; ListCell* next; }; struct List { NodeTag type; // T_List int length; ListCell* head; ListCell* tail; }; ``` ### List Functions ```cpp theme={null} // Create List* list_make1(void* datum1); List* list_make2(void* datum1, void* datum2); List* lappend(List* list, void* datum); List* lcons(void* datum, List* list); // Access void* linitial(const List* list); void* lsecond(const List* list); void* llast(const List* list); // Iteration #define foreach(cell, list) \ for ((cell) = list_head(list); (cell) != NULL; (cell) = lnext(list, cell)) ``` **Example:** ```cpp theme={null} List* mylist = NIL; mylist = lappend(mylist, some_pointer); mylist = lappend(mylist, another_pointer); ListCell* cell; foreach(cell, mylist) { void* item = lfirst(cell); // Process item... } ``` *** ## Integration with Springtail ### Type System Integration Extension types are stored in Springtail's type system through `create_usertype()`: ```cpp theme={null} // In pg_copy_table.cc PgMsgUserType msg; msg.oid = enum_type_oid; msg.name = extn_type_name; msg.type = constant::USER_TYPE_EXTENSION; server->create_usertype(db_id, {xid, 0}, msg); ``` ### Comparator Integration Extension operators are used in Springtail comparisons: ```cpp theme={null} // In field.cc ExtensionContext ctx; ctx.type_oid = field->type_oid; ctx.op_str = operator_symbol; return PgExtnRegistry::comparator_func(&ctx, lhs_binary, rhs_binary); ``` *** ## Limitations ### Not Implemented PostgreSQL features NOT implemented in pg\_ext: 1. **Transaction Management**: No MVCC, snapshot isolation 2. **Catalog Access**: No pg\_catalog queries 3. **SPI (Server Programming Interface)**: No SQL execution from C functions 4. **Triggers**: No trigger mechanism 5. **Full Memory Context Tree**: Simplified single-level context 6. **Signal Handling**: No PostgreSQL signal handlers 7. **Shared Memory**: No PostgreSQL shared memory segments 8. **TOAST**: No out-of-line storage for large values 9. **Vacuum**: No MVCC cleanup 10. **Write-Ahead Log**: No WAL integration ### Compatibility Notes * Extension functions must not rely on PostgreSQL backend state * SPI-using extensions will NOT work * Trigger-based extensions will NOT work * Extensions accessing pg\_catalog may fail * Extension functions assuming PostgreSQL memory semantics may leak memory *** ## Adding New pg\_ext Support To add support for a new PostgreSQL feature: 1. **Identify Required APIs**: Determine what PostgreSQL functions the extension uses 2. **Create Stub Implementation**: Add stubs in appropriate `src/pg_ext/*.cc` file 3. **Add Type Definitions**: Add necessary structs/types in `include/pg_ext/*.hh` 4. **Implement Core Logic**: Implement minimal functionality needed by extensions 5. **Test with Extension**: Verify extension functions work correctly **Example: Adding Support for Range Types** 1. Create header file `include/pg_ext/range.hh`: ```cpp theme={null} // include/pg_ext/range.hh struct RangeType { Oid rangetypid; char flags; Datum lower; Datum upper; }; Datum range_in(const char* str, Oid rangetypid); char* range_out(RangeType* range); bool range_contains(RangeType* range, Datum element); ``` 2. Create source file `src/pg_ext/range.cc`: ```cpp theme={null} // src/pg_ext/range.cc #include Datum range_in(const char* str, Oid rangetypid) { // Parse range string "[lower,upper)" // Create RangeType structure // Return as Datum } // ... implement other functions ``` *** # Production Deployment Source: https://docs.springtail.io/production-deployment Production deployment is managed by the Springtail Coordinator. Please see the [Coordinator documentation](https://github.com/Springtail-inc/springtail/tree/main/python/coordinator/README.md) for details on how to start Springtail, add and remove replica nodes, and stop Springtail. # Proxy Nodes Source: https://docs.springtail.io/proxy-nodes # Proxy Overview ## System Implementation Overview This system is designed to seamlessly integrate a **primary write database** with a read-replica system, **Springtail**, through intelligent routing and robust session management. The goal is to offload read traffic while maintaining data consistency and transactional integrity. At a high-level this involves splitting read and write accesses such that writes are performed at the Primary and reads are performed at the replicas. The system is composed of the following components: * **Client Session:** The connection from the client to the Proxy. * **Server Session:** The connection from the Proxy to a database instance. For every client session there exists a: * **Primary Session:** The connection from the Proxy to the primary database instance. * **Replica Session:** The connection from the Proxy to the replica (if a replica is available). The diagram below shows the message path from a client through the proxy into the client-facing session and branching to primary and replica server sessions. ```mermaid theme={null} flowchart LR Client[Client] Proxy[Proxy Server] CSession[Client Session] SPrimary["Server Session (Primary)"] SReplica["Server Session (Replica)"] Client -->|connect/request| Proxy Proxy -->|accept/dispatch| CSession CSession -->|request| SPrimary CSession -->|request| SReplica SPrimary -->|response| CSession SReplica -->|response| CSession CSession -->|response| Client ``` *** ### Data Routing Engine (Read/Write Splitting) The core function of the system is the read/write splitting, which determines the destination of each query: 1. **Query Parsing:** The engine performs uses a Postgres parser to parse every incoming SQL query to determine the operation type (read vs. write) and to assess if the operation is supported by the Springtail read replicas. 2. **Traffic Direction:** * **Writes (Primary Route):** All write operations (`INSERT`, `UPDATE`, `DELETE`) are directed exclusively to the **primary** database. * **Reads (Springtail Route):** All supported read operations (`SELECT`) are directed to the **Springtail** replica pool. * **Logging/Testing:** A mirrored path is implemented where, under specific testing conditions, all queries can be sent to both the primary and Springtail to **log and measure Springtail's performance and result consistency** against the primary. *** ### Session State and Load Management The system maintains session level consistency across Primary and Replica connections and ensures load is balanced across replicas. * **Session State:** To prevent errors, state-dependent operations like **prepared statements** and **SET search\_path** are **mirrored**—meaning they are executed on both the primary and replica even if the primary is the source of truth for the resulting data change. * **Replica Load Balancing:** The system incorporates a **round-robin/load balancing** component to distribute read requests evenly across the available **Springtail replicas**. This maintains high availability and efficiency. If the Springtail system scales up, or scales down, the Proxy is aware of the changes in replica nodes. * **Scale-up:** New sessions will be load-balanced across the new replica nodes. * **Scale-down:** Existing sessions on the node being removed will fail-over to another replica if one exists, or will fail-over to the Primary (if no other replica exists). *** ### Transactional Integrity and Session Switching The implementation must maintain the integrity of sessions and multi-step transactions, even when switching destinations. Read/write splitting happens on a transaction level. If a transaction starts on a replica and an UPDATE occurs, the Proxy will transition the transaction to the Primary. The remainder of operations will occur on the Primary until that transaction is completed (committed or rolled-back). The system has the following requirements: 1. **Transaction Tracking:** The system actively monitors the state of every **user session and ongoing transaction** started on Springtail. 2. **Dynamic Switch-Over:** If a transaction being executed on a Springtail replica suddenly calls for a **write operation** or any unsupported operation, the system initiates an immediate **switch-over** to the primary database. 3. **Session State Replay:** Crucially, before the primary accepts the switched transaction, the system must retrieve and **apply all current session-level settings** (e.g., timezone, transaction isolation level) that were active on the Springtail session to the new primary session. This ensures that the transaction completes with the exact same configuration it started with. *** ### Security and Access Layer The implementation establishes secure and reliable client connections using: * **Authentication:** The system enforces user identity verification using the standard PostgreSQL authentication mechanisms. It first validates the user against the Primary database before allowing access to the replica. * **Connection Termination:** Client traffic is received at a routing layer responsible for parsing and directing subsequent database operations. All connections are (optionally) encrypted with TLS. * **User Management:** All proxy users must be pre-registered with the Proxy (and must exist on the Primary). The Proxy will periodically validate that the user has connection access to each replicated database. # Query Node Lifecycle Source: https://docs.springtail.io/query-node-lifecycle # FDW Node Lifecycle (Coordinator, Postgres, XID Subscriber, DDL Manager) This document describes the runtime lifecycle of an FDW node and how its major components are started, monitored, drained, and stopped. It focuses on the operational behavior of the FDW service as a whole, including the local Postgres process, the XID Subscriber, and the DDL Manager. ## 1. Components on an FDW node An FDW node runs a small set of cooperating processes: * **Postgres (FDW database instance)**\ The local Postgres instance that hosts FDW-side databases, schemas, foreign tables, and any locally-materialized objects required for correct partitioning and metadata behavior.\ **NOTE:** the version of the local Postgres node is a custom-patched build. It is patched to allow row-level security policies on foreign-tables. * **XID Subscriber**\ A background service that participates in transaction/XID coordination for the system. On the FDW node it runs alongside the DDL Manager and helps the overall platform reason about replication progress and safe advancement. * **DDL Manager**\ The service responsible for: * Initializing and maintaining replicated databases on the FDW Postgres instance. * Importing schemas into FDW databases. * Applying ongoing DDL changes delivered asynchronously. * Running periodic synchronization for security/ownership-related replication. * **Coordinator (supervisor/launcher)**\ The service wrapper responsible for starting and supervising the above processes, restarting them on failure, and executing controlled shutdown workflows (including draining). *** ## 2. Startup lifecycle ### 2.1 Coordinator boot and environment readiness When the FDW node starts, the coordinator performs basic environment validation (paths, logging configuration, required configuration availability). In production environments it may also update installed binaries and ensure the FDW extension artifacts are present on the host. ### 2.2 Ensuring a runnable FDW-capable Postgres Before starting FDW-specific Springtail services, the node ensures the local Postgres instance is running. If it is not running, the coordinator starts it and waits until it is healthy. The FDW node’s Postgres must be healthy before any downstream components can function, because: * The DDL Manager must connect to Postgres to create FDW-side databases and apply DDL. * The system expects the FDW node to become query-ready only after Postgres is usable. ### 2.3 Dependency wait: ingestion readiness After Postgres is running, the coordinator waits for upstream ingestion-side dependencies to become reachable. This includes the services that provide the metadata and coordination information needed by the FDW node to initialize and track replication progress. This step prevents the FDW node from starting replication logic before the rest of the system can serve the required metadata and coordination APIs. ### 2.4 Starting FDW services in order Once dependencies are ready, the FDW node starts its Springtail services in a defined order: 1. **Postgres** (if not already running) 2. **XID Subscriber** 3. **DDL Manager** This order ensures: * The XID Subscriber and DDL Manager can immediately connect to Postgres. * Replication coordination can begin before DDL application tries to advance state. ### 2.5 Service enters steady-state running After successful startup: * The coordinator marks the service as running. * Ongoing supervision begins (see Monitoring and Recovery). *** ## 3. Steady-state behavior ### 3.1 Continuous supervision and health checks While running, the coordinator: * Monitors liveness heartbeats/timeout signals for the FDW-related services. * Periodically checks that each component is alive. * Tracks database-state changes for informational and operational visibility. * Triggers restarts when failures are detected, with protection against tight crash loops. ### 3.2 DDL Manager activity during steady state In steady state, the DDL Manager performs two concurrent responsibilities: * **Incremental schema change application**\ Consumes queued change batches and applies them transactionally to the FDW Postgres instance, then records progress. * **Periodic synchronization loop**\ Periodically reconciles role membership, policies, and ownership-related state from the primary into the FDW databases. ### 3.3 XID Subscriber activity during steady state The XID Subscriber runs continuously and participates in the broader coordination mechanisms for XID tracking and progress. Operationally, it is treated as a first-class FDW component that must be running and healthy for the FDW node to be considered healthy. *** ## 4. Failure handling and restart behavior ### 4.1 Failure detection sources The coordinator can detect component failures via: * Explicit liveness timeout signals (e.g., missed heartbeats). * Direct process health checks indicating a component is not alive. * Failure notifications delivered through a pub/sub mechanism. ### 4.2 Restart policy When the coordinator detects failures: * It attempts to restart the FDW service components. * It tracks repeated failures to detect instability. * If a component repeatedly fails too many times within a short window, the coordinator shifts from immediate restart to a backoff-and-retry model to avoid constant churn. This approach keeps the FDW node available when possible, but prevents uncontrolled restart loops when a persistent configuration or environmental issue exists. *** ## 5. Controlled shutdown via draining FDW nodes support a controlled shutdown sequence designed to avoid interrupting active client traffic. ### 5.1 Enter draining state A controlled shutdown begins by transitioning the FDW node’s service state to **draining**. This is an administrative signal that the FDW node should stop serving new work and prepare to stop safely. ### 5.2 Wait for client connections to drain After entering **draining**, the coordinator waits until the FDW Postgres instance has **no active client connections** remaining (i.e., all proxy/client sessions have disconnected). The goal is to avoid: * Terminating long-running queries mid-flight. * Cutting off active SQL sessions that depend on the FDW node. ### 5.3 Stop FDW services Once connections have drained to zero, the coordinator shuts down FDW services in a controlled manner. This brings down: * The DDL Manager * The XID Subscriber * The FDW Postgres instance if it is managed as part of the FDW service ### 5.4 Transition to stopped state After shutdown completes, the coordinator ensures the FDW node is in the **stopped** state. If the state transition does not occur promptly, it forces the state to stopped to guarantee that external controllers observe a consistent final state. ### 5.5 Cleanup of FDW replication coordination state As part of stopping, the FDW node cleans up replication coordination tracking associated with the FDW identity, such as: * Pending DDL queue entries for this FDW node. * Per-FDW progress tracking entries used for DDL/XID coordination. * Other FDW-specific bookkeeping data that should not persist across a node stop/replacement. This cleanup reduces the risk of: * Stale work being applied after restart in an unexpected context. * Inaccurate minimum-progress calculations caused by dead FDW participants. ### 5.6 Coordinator remains alive (optional idle mode) In some operational modes, after all FDW services are stopped the coordinator may remain running and enter an idle loop. This allows the node to remain manageable and responsive to external state changes (for example, a subsequent command to restart services) without requiring the coordinator itself to exit. *** ## 6. Summary state model An FDW node typically transitions through these phases: 1. **Boot / initialization** 2. **Postgres running and healthy** 3. **Dependencies reachable** 4. **XID Subscriber running** 5. **DDL Manager running** 6. **Steady-state supervision** 7. **Draining (on controlled shutdown)** 8. **Stopped (services down, state finalized, coordination cleaned up)** This lifecycle ensures the FDW node becomes operational only when dependencies are available, stays resilient under failures via restart logic, and can be taken offline safely through draining. # Query Nodes Source: https://docs.springtail.io/query-nodes # FDW Node Architecture Overview An FDW node is a managed replication-and-query endpoint that hosts a dedicated Postgres instance and a small set of Springtail services responsible for keeping that instance synchronized with the primary system. The node is designed to be self-starting, self-monitoring, and capable of controlled shutdown to avoid disrupting active client traffic. ## Core components ### Postgres (FDW database instance) The local Postgres service on the FDW node is the execution environment for: * FDW-side databases and schemas * Foreign tables that reference primary-side data * Select locally-created objects required to support schema shape, partitioning, and metadata consistency This Postgres instance is the target that all schema and security replication converges on. ### FDW extension (query interface inside Postgres) The FDW node’s Postgres instance is extended with a Springtail-provided Foreign Data Wrapper implemented as a **shared library** loaded into Postgres (installed as a Postgres extension). This extension provides the read path by: * Integrating with Postgres’ FDW APIs so queries can reference foreign tables normally. * Interpreting and planning **read-only** access through the foreign-table definitions created during schema import. * Acting as the boundary where SQL executed on the FDW node is translated into remote reads against the primary-side data source(s), while preserving Postgres semantics for query parsing and execution. In short: Postgres provides the SQL surface area, and the FDW shared library provides the mechanism that makes those SQL statements resolve to remote data. ### Coordinator (process supervisor) The coordinator is the control-plane process on the node. It is responsible for: * Starting the node’s components in the correct order * Waiting for upstream dependencies to be available before enabling replication services * Monitoring liveness/health and restarting components when failures occur * Orchestrating controlled state transitions (including draining before shutdown) ### XID Subscriber The XID Subscriber is a background service that participates in the system’s transaction/XID coordination. On an FDW node it runs alongside the DDL Manager and supports consistent progress tracking and coordination with the rest of the platform. ### DDL Manager The DDL Manager is the data-plane replication worker for schema and access-control state on the FDW node. It is responsible for: * Bootstrapping FDW-side databases (creating databases/schemas, installing prerequisites, and importing foreign schema) * Applying ongoing DDL change batches delivered asynchronously * Running periodic synchronization of roles, role memberships, policies, and ownership metadata from the primary into the FDW node ## How the pieces fit together At runtime, the coordinator ensures Postgres is healthy, then brings up the XID Subscriber and DDL Manager. The DDL Manager continuously updates the FDW Postgres instance so that the set of schemas and foreign tables available for query remains aligned with the primary. When client queries arrive, Postgres parses and plans them normally, and the FDW shared library implements the foreign-table access that performs remote reads. The coordinator monitors all components for liveness and takes corrective action (restart or controlled shutdown) when required, making the FDW node behave as a resilient, operationally-managed service. # Quickstart Source: https://docs.springtail.io/quickstart This quickstart is the fastest way to get a locally running version of Springtail up for testing. It uses the **local cluster**, a full multi-node Springtail deployment running under Docker Compose. For an explanation of what each step does, the cluster's internals, and the full set of management commands, see [Local Cluster Deployment](/local-cluster-deployment). ## Prerequisites * Docker and Docker Compose * Python 3 * AWS CLI ## Launch the cluster Run all commands from the `local-cluster/` directory: ```shell theme={null} cd local-cluster ``` Build a package and start the cluster. The first `./cluster up` builds any missing images, so it may take a few minutes: ```shell theme={null} ./cluster build-package /tmp/springtail-packages ./cluster up /tmp/springtail-packages/springtail--.tar.gz ``` Check that the cluster is up: ```shell theme={null} ./cluster status ``` Connect to the proxy with any PostgreSQL client: ```shell theme={null} psql -h localhost -p 55432 -U postgres ``` ## Tear down ```shell theme={null} # Stop the Springtail services, keeping the primary DB and dependencies ./cluster down # Stop everything and remove all data volumes ./cluster down all ``` # Recovery Source: https://docs.springtail.io/recovery # Replication Log Recovery (High-Level Process) This document describes the recovery process for the replication logging pipeline. It focuses on how replication messages are durably logged, how committed transactions are identified and tracked, and how the log manager starts in a recovery-oriented mode to find the most recent safe commit point before resuming normal ingestion. ## 1. What is persisted during normal operation ### 1.1 Logging replication messages (durable staging) As replication messages arrive from the upstream Postgres replication stream, they are appended to a local replication log on disk. This log is written sequentially and is designed to support: * Restart safety (ability to resume after crash) * Ordered replay (the log is read back in the same message order) * Message boundary reconstruction (messages may be fragmented during transport) The replication log acts as the durable “source of truth” for downstream processing, insulating the rest of the system from connection interruptions and process crashes. ### 1.2 Logging committed transactions (committed XIDs) Alongside (or derived from) replication message logging, the system identifies **commit boundaries** and records the fact that a transaction has committed. Conceptually, this produces a durable record of: * The transaction identifier (XID) that reached commit * The corresponding position in the replication stream/log that makes that commit “safe” * Any minimal metadata required to re-establish correct ordering and restart positions The key purpose of persisting commit information is to establish an unambiguous recovery point: after restart, the system can find the last fully committed transaction that is guaranteed present in the local log. *** ## 2. Why recovery must anchor on commits Replication streams deliver changes in transactional order, but correctness depends on respecting commit semantics: * Changes that occur before commit should not be considered durable/applicable as a completed unit until the commit is observed. * A crash may occur after some data has been written but before it is flushed, or after it is flushed but before higher-level state is updated. * A restart must safely choose a point that avoids “losing” committed work and avoids “inventing” commits that were never durably captured. Therefore, recovery is driven by locating the most recent **committed** transaction that is known to be safely represented in the local replication log. *** ## 3. Startup behavior in recovery mode When the log manager starts, it may enter a recovery-oriented startup path if it detects that: * A replication log already exists from a previous run, and/or * The previous run did not shut down cleanly, and/or * There is evidence that downstream consumers may not have fully processed all logged data In this recovery state, the immediate goal is not to start consuming new replication messages, but to **reconcile the log** and establish a correct resume point. *** ## 4. Recovery scanning: finding the latest committed entry ### 4.1 Sequential scan of the replication log Recovery proceeds by scanning the existing replication log from a known start point (typically the beginning of the active log segment or the last known safe offset). The scan treats the log as an ordered stream of framed replication messages. During the scan, the recovery logic: * Reconstructs message boundaries (including messages that were logged in parts) * Interprets message types sufficiently to detect transactional structure * Tracks transaction lifecycle markers (begin, changes, commit) The scan does not need to fully re-apply data changes. Its primary objective is to locate **the last commit record that is complete and consistent**. ### 4.2 Validating commit completeness Because a crash can happen mid-write, recovery must be conservative. It treats the “latest committed entry” as valid only if: * The commit marker is fully present in the log (not truncated) * The log framing around it is consistent * The commit can be understood as a complete boundary in the message stream If the scan encounters a partial/truncated message at the end of the file, the recovery process treats that tail as unsafe and does not advance the “latest committed” point beyond the last verified commit. ### 4.3 Establishing the safe resume point Once the scan completes, recovery produces a safe resume point consisting of: * The latest committed transaction identifier (XID) * The corresponding durable log position (or equivalent marker) associated with that commit This resume point is then used to: * Restart downstream processing at a consistent boundary * Determine what portion of the log is safe to keep and what tail may need to be truncated/ignored * Ensure that acknowledgments back to the upstream replication source align with what is durably captured *** ## 5. Transition from recovery to normal operation After determining the latest committed entry, the log manager transitions to normal operation: 1. **Finalize log state** * Any unsafe trailing log region after the last valid commit is treated as not authoritative. * The system ensures the active log is in a consistent state for appends and reads. 2. **Resume downstream replay** * Message processing can resume from the last committed boundary forward. * Any transactions after the last committed boundary are treated as incomplete and will be re-derived from the upstream stream as needed. 3. **Reconnect and continue ingestion** * The replication connection can be re-established to continue streaming from the correct upstream position. * New incoming messages are appended after the recovered safe boundary. *** ## 6. Outcome guarantees This recovery approach provides the following guarantees: * **No loss of committed work** that was durably logged: recovery anchors on the last verified commit present on disk. * **No reliance on in-memory state**: decisions are based on the persisted log and commit markers. * **Safe handling of truncated tails**: partial messages at the end of the log are not treated as committed progress. * **Consistent transactional boundaries**: resumption occurs at commit boundaries, preserving transaction semantics for downstream consumers. *** ## 7. Summary Recovery is driven by two persisted facts: 1. Replication messages are durably staged in a sequential replication log. 2. Committed transactions (XIDs) are detectable and tracked so the system can identify the last safe commit. On startup, the log manager enters a recovery state when needed, scans the replication log to find the most recent fully committed entry, and uses that commit boundary as the safe point from which to resume normal ingestion and message processing. # Redis Source: https://docs.springtail.io/redis # Redis Springtail has two main purposes for Redis. First is to store configuration in it and second, as an IPC mechanism for sharing the states between server processes and queuing DDL updates that need to be processed. All storage keys in Redis are prefixed with the database instance id that the data structure belongs to. ## Redis configuration database The configuration database has the following data structures: 1. **\:instance\_state** -- this is a hash table that has Postgres database id as a key and Springtail state of that database as a value. During normal operation the state of the database is set to `running`. When a new database is first introduced for replication, its state will be set to `initialize`. 2. **\:db\_config** -- this is a hash table that has Postgres database id a a key and Springtail configuration data for this database as a value. Here are the data stored per database: ``` { "name": "", "replication_slot": "", "publication_name": "", "include": { "schemas": [] } } ``` 3. **\:fdw** -- this is a hash table that has an FDW id as a key and an FDW configuration as a value. Here are the data stored in this hash table per FDW: ``` { "db_prefix": "", "fdw_user": "", "host": "", "password": "", "port": , "proxy_password": "", "state": "", "sync_seconds": } ``` * During normal operation the state of FDW will be set to `running`. 4. **\:instance\_config** -- this is a hash table that has several predefined keys that determine springtail instance configuration. Here is the list of all the keys together with the JSON structures associated with each key: ``` "primary_db": "{ "host": "", "port": }" "system_settings": "{ "logging": { "log_level": "debug", "log_level_debug": 3, "log_path": "/opt/springtail/logs/", "log_file_size": 1000000000, "log_file_count": 100, "log_pattern": "[%Y-%m-%d %T.%e %z] [%^%l%$] [%s:%#:%!] [thread %t] %v", "log_modules": ["all"], "log_rotation_enabled": true, "pid_path": "/opt/springtail/pids" }, "iopool": { "threads": 10, "filehandles": 30 }, "write_cache": { "rpc_config": { "server_port": 5051, "server_worker_threads": 8, "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem", "server_key": "/home/dev/springtail/ca_certs/server_key.pem", "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem", "ssl": false, "client_connections": 8, "client_cert": "/home/dev/springtail/ca_certs/client_cert.pem", "client_key": "/home/dev/springtail/ca_certs/client_key.pem", "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem" }, "disk_storage_dir": "/opt/springtail/data/write_cache", "NOTE": "memory for write cache 640MB and 512MB", "memory_high_watermark_bytes": 671088640, "memory_low_watermark_bytes": 536870912 }, "storage": { "table_dir": "table", "data_cache_size": 32768, "page_cache_size": 16384, "btree_cache_size": 512, "max_extent_per_page": 16, "vacuum_config": { "enabled": true, "vacuum_dir": "vacuum", "global_file_size_threshold": 102400, "hole_punch_block_size": 4096, "max_entries_in_memory": 100000 }, "metrics_update_freq_sec": 10, "io_request_queue_size": 16 }, "redis": { "host": "localhost", "port": 6379, "user": "default", "password": null, "ssl": false, "db": 1, "config_db": 0, "keep_alive_secs": 30, "pool": { "connections": 10, "max_idle_secs": 300, "max_connection_lifetime_secs": 0 } }, "log_mgr": { "replication_log_path": "repl_logs", "transaction_log_path": "xact_logs", "log_size_rollover_threshold": 134217728, "archive_logs": true, "indexer_worker_threads": 1, "reader_queue_mem_high_watermark": 134217728, "reader_queue_mem_low_watermark": 100663296, "committer_fsync_interval_ms": 500, "committer_fsync_worker_threads": 1, "rpc_config": { "server_port": 5052, "server_worker_threads": 32, "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem", "server_key": "/home/dev/springtail/ca_certs/server_key.pem", "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem", "ssl": false, "client_connections": 8, "client_cert": "/home/dev/springtail/ca_certs/client_cert.pem", "client_key": "/home/dev/springtail/ca_certs/client_key.pem", "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem" } }, "sys_tbl_mgr": { "rpc_config": { "server_port": 5053, "server_worker_threads": 16, "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem", "server_key": "/home/dev/springtail/ca_certs/server_key.pem", "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem", "ssl": false, "client_connections": 8, "client_cert": "/home/dev/springtail/ca_certs/client_cert.pem", "client_key": "/home/dev/springtail/ca_certs/client_key.pem", "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem" }, "cache_size": 4096, "roots_shm_cache_size": 10485760, "schema_shm_cache_size": 10485760, "usertype_shm_cache_size": 4194304 }, "proxy": { "NOTE": "cert and key required for ssl; modes = [primary|normal|shadow]; pg_shadow for testing", "enable_ssl": false, "port": 8888, "keep_alive_port": 0, "threads": 4, "cert_path": null, "key_path": null, "mode": "normal", "shadow_log_path": null, "pool_size_limit": 0, "pool_timeout_limit": 0, "pool_expiration_interval_secs": 0, "user_mgr_sleep_interval_secs": 15 }, "otel": { "enabled": false, "remote": false, "host": "127.0.0.1", "port": 4318, "metrics_export_interval_millis": 60000, "metrics_export_timeout_millis": 30000, "remote_log_level": "info" }, "extension_config": { "lib_path": "/usr/lib/postgresql/16/lib/", "1": { "cube": { "version": "1.5" }, "hstore": { "version": "1.8" }, "pg_trgm": { "version": "1.6" } } }, "aws_users_override": [ { "role": "replication", "username": "springtail", "password": "springtail", "type": "text" }, { "role": "fdw_superuser", "username": "springtail", "password": "springtail", "type": "text" }, { "role": "proxy_to_fdw", "username": "springtail", "password": "springtail", "type": "text" }, { "role": "database", "username": "springtail_proxy", "password": "springtail_proxy", "type": "text" } ] }" "id": "" "database_ids": "[]" "hostname:proxy": "" "hostname:ingestion": "" ``` * `system_settings` contains an example configuration. 5. **\:fdw\_ids** - this is a set that contains a list of ids for FDWs configured for this database instance. 6. **\:fdw\_dbs** - this is a hash table that maps FDW id to a JSON array of database ids supported by this FDW instance: ``` [, , ..., ] ``` 7. **\:admin\_console** - this is a hash table that daemon processes use to publish their admin IP address and port for HTTP access. The key of this table is in the format `:` and the value is an IP address and port number that admin process listens on. Here is an example: ``` "test:pg_log_mgr_daemon": "0.0.0.0:47057" "test:pg_xid_subscriber_daemon": "0.0.0.0:50361" "test:pg_ddl_daemon": "0.0.0.0:42641" "test:proxy": "0.0.0.0:58435" ``` * Here instance key is needed to differentiate between FDW instances because both `pg_xid_subscriber_daemon` and `pg_ddl_daemon` exist in each FDW. 8. **\:coordinator\_state** - this is a hash table that contains the state of all coordinators. Coordinator is a process that is used to monitor that Springtail processes are alive in each container. It is also responsible for loading up to date libraries and executables into the container. The key is a combination of instance type and instance key, `:`, and the value is the state of that specific instance. Here is an example of the data stored in this hash table: ``` "proxy:default": "running" "ingestion:default": "running" "fdw:fdw1": "running" "fdw:fdw2": "running" ``` ## Redis data database This database contains data structures used by Springtail system at runtime. It contains the following data structures: 1. **\:queue:ddl:xid:\:\** -- Queue of DDL operations for a given XID coming out of the LogParser. 2. **\:queue:index:ddl:xid:\:\** -- Queue of DDL index operations for a given XID coming out of the GC1 LogParser. 3. **\:hash:ddl:pc** -- HASH of pre-commit DDL operations. Stored with a key of `:`. 4. **\:queue:ddl:fdw:\** -- Queue of DDL changes for the FDW to process coming out of the Committer. 5. **\:hash:ddl:fdw** -- Hash set of schema XIDs per FDW. Hash key: `:`, value: ``. 6. **\:queue:sync\_tables:\** -- Queue for table sync requests; value is the table OID/TID. 7. **\:string:log\_resync:\** -- Key / value for log mgr resync point; value is `:`. 8. **\:hash:sync\_table\_ops:\** -- Hash table holding the table operations (drop, create and update\_roots) for each table sync. Each entry in the hash is stored as a JSON array of JSON objects. 9. **\:hash:liveness** -- Hash set for tracking liveness of a daemon. This hash is used by coordinator. 10. **\:set:db\_tables:\** -- Set holding schema.table names for each database. Value is a list of `.`. 11. **\:hash:idx:ddl:pc** -- Hash set of pre-commit DDL operations for index mutations. Stored with a key of `:`. 12. **\:hash:invalid\_tables** -- Hash set of excluded items for a given ``. Key: ``, value: ``. 13. **\:fdw\_min\_xids** -- Hash set of min XIDs per `:` for a given ``. Key: `:`, value: ``. 14. **\:fdw\_pids** -- Set of new FDW processes that have not started querying the database yet. Value: `::`. 15. **\:set:db\_index\_xids:\** -- Set holding index XIDs for each db id. Value: ``. 16. \*\*\:vacuum\_cutoff\_xids -- Hash of vacuum cutoff XIDs per db\_id for a given ``. Key: ``, Value: ``. # Redis Configuration Source: https://docs.springtail.io/redis-configuration # Configuration in Redis Two Redis databases are used by Springtail. * **Database 0:** This is the configuration database used to hold configuration data that is fairly static. * **Database 1:** This is the data database used to hold runtime data and queues. The configuration database must be initialized prior to starting up the system. The data database will be populated as the system runs. ## Database 0 (config DB) Snippet below shows the config distilled from the configuration tables and pushed to Redis, in most case we use both database instance ID and database config ID. * Database Instance Config (HSET) * key = `:instance_config` * value = Hash | field key | value | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | id | `` | | primary\_db | JSON: `{host: “”, port: }` | | database\_ids | JSON: `` “\[…]” (array of bigints) | | system\_settings | JSON: See example below, | | system\_settings\_gitsha | String: The 7-byte short GitSHA of the **latest** commit of the `prod.system.settings.json` , used for version checking | | hostname:proxy | (either missing or set to the complete value, when rebooting, need to delete first, so coordinator won’t grab the wrong values) | | hostname:ingestion | (either missing or set to the complete value, when rebooting, need to delete first, so coordinator won’t grab the wrong values) | **system\_settings** JSON example (**NOTE:** this is an example see `prod.system.settings.json` for all settings) ```json theme={null} { "logging": { "log_level": "debug", "log_path": "/opt/springtail/logs", "log_file_size": 104857600, "log_file_count": 10, "log_pattern": "[%Y-%m-%d %T.%e %z] [%^%l%$] [%s:%#:%!] [thread %t] %v", "log_modules": [ "all" ], "pid_path": "/opt/springtail/pids", "log_rotation_enabled": true }, "io_pool": { "threads": 10, "filehandles": 30 }, "redis": { "keep_alive_secs": 30, "pool": { "connections": 10, "max_idle_secs": 300, "max_connection_lifetime_secs": 0 } }, "write_cache": { "rpc_config": { "server_cert": "/opt/springtail/certs/server_cert.pem", "server_key": "/opt/springtail/certs/server_key.pem", "server_trusted": "/opt/springtail/certs/ca_cert.pem", "client_cert": "/opt/springtail/certs/client_cert.pem", "client_key": "/opt/springtail/certs/client_key.pem", "client_trusted": "/opt/springtail/certs/ca_cert.pem", "server_port": 55051, "server_worker_threads": 8, "ssl": true, "client_connections": 8 }, "io_threads": 8 }, "sys_tbl_mgr": { "rpc_config": { "server_cert": "/opt/springtail/certs/server_cert.pem", "server_key": "/opt/springtail/certs/server_key.pem", "server_trusted": "/opt/springtail/certs/ca_cert.pem", "client_cert": "/opt/springtail/certs/client_cert.pem", "client_key": "/opt/springtail/certs/client_key.pem", "client_trusted": "/opt/springtail/certs/ca_cert.pem", "server_port": 55053, "server_worker_threads": 16, "ssl": true, "client_connections": 8 }, "cache_size": 4096, "roots_shm_cache_size": 10485760 }, "storage": { "table_dir": "table", "data_cache_size": 16384, "page_cache_size": 16384, "btree_cache_size": 512, "max_extent_per_page": 16 }, "log_mgr": { "replication_log_path": "repl_logs", "transaction_log_path": "xact_logs", "log_size_rollover_threshold": 134217728, "archive_logs": true, "rpc_config": { "server_cert": "/opt/springtail/certs/server_cert.pem", "server_key": "/opt/springtail/certs/server_key.pem", "server_trusted": "/opt/springtail/certs/ca_cert.pem", "client_cert": "/opt/springtail/certs/client_cert.pem", "client_key": "/opt/springtail/certs/client_key.pem", "client_trusted": "/opt/springtail/certs/ca_cert.pem", "server_port": 55052, "server_worker_threads": 32, "ssl": true, "client_connections": 8 } }, "proxy": { "enable_ssl": true, "port": 5432, "threads": 4, "cert_path": "/opt/springtail/certs/server_cert.pem", "key_path": "/opt/springtail/certs/server_key.pem", "mode": "primary", "shadow_log_path": "/shadow_logs/proxy_shadow.log", "log_level": 2, "keep_alive_port": 10080, "use_pg_shadow": false, "pool_size_limit": 10, "pool_timeout_limit": 5, "pool_expiration_interval_secs": 300 }, "otel": { "enabled": true, "host": "localhost", "port": 4318, "metrics_host": "localhost", "metrics_port": 4318 } } ``` * Database Config (HSET) * key = `:db_config` * field\_key = `` * field\_value = JSON blob. * For the `“include”` section: * if it is empty or `"*"` in `"schemas"` section, we can assume the equivalence of Postgres `FOR ALL TABLES`; all tables will be replicated, including all future tables * for each entry in `“schemas”` we assume equivalence of Postgres `FOR TABLES IN SCHEMA`; all tables in that schema will be replicated, including all future tables * for each entry in `"tables"` we assume equivalence of Postgres `FOR TABLE`; only that table and its descendants will be replicated (no wildcards allowed). A `"tables"` section when `"schemas": ["*"]` is not valid. JSON: ```json theme={null} { "id": "", "name": "", "replication_slot": "", "publication_name": "", // example include sections // example 1: schemas only "include": { "schemas": ["*"], // * = all schemas (default); otherwise list specific schemas } // example 2: schemas with tables "include": { "schemas": ["schemaA", "schemaB", ...], // all current and future tables in schema 'schemaA' and 'schemaB' "tables": [{"schema": "schemaC", "table": "tableA"}, ...], // also include table 'tableA' from schema 'schemaC' } } ``` * FDW IDs (Set) * key = `:fdw_ids` * values = `` E.g., ```bash theme={null} SADD fdw-ids:1234 "fdw-01" "fdw-02" SMEMBERS fdw-ids:1234 SREM fdw-ids:1234 "fdw-01" ``` * FDW Config (HSET) * key = `:fdw` * field\_key = `` * field\_value = JSON: `{”host”: ”ip”, “port”: , “state”: “”, “sync_seconds” }`, state = `initialize`, `running`, `draining`, `stopped` * State transitions: * The www service will set the FDW state to `initialize` upon startup * The Springtail service (the ddlmgr) will set FDW state to `running` after it has completed initialization * The www service will set FDW state to `draining` upon scale-down * The Springtail service (the coordinator) will set FDW state to `stopped` once all connections completely drained from FDW * Note: * For each FDW instance, make sure the delete the field\_key then set to the right value, so the coordinator won’t grab the wrong values. * The Springtail service does not interpret `fdw_ip_as_opaque_id` . It should ignore the `initialize` state, which is essentially a state when the FDW scaling is requested but not yet available. * If state is `stopped` safe to remove from Redis * FDW user and password are set in AWS secrets mgr with role `fdw_superuser` * DB State (HSET) * Contains state for each DB within the instance * key = `:instance_state` * field\_key = `` * field\_value = one of: `initialize` `startup` `synchronizing` `running` `failed` `stopped` `removed` * Set to `initialize` for each db upon Launching, the very very first launching, after copy tables has been run the system will set the state to running. * If a table copy is required the state will change to `synchronizing` while the copy table is in progress and then will change back to running * If the db is restarted (or instance is restarted) then the state should be set to `startup` * Admin Console (HSET) * Contains ip and port pair for each daemon for admin console access * key = `:admin_console` * field\_key = `:` * field\_value = `:` * Coordinator State (HSET) * Contains state of coordinator on each instance * key = `:coordinator_state` * field\_key = `:` * field\_value = one of: `startup`, `running`, `reload`, `reloading`, `shutdown`, `dead` * `running` set after startup is complete or after reload is successful; set by coordinator * `dead` set after shutdown is complete; set by coordinator * `startup`, `reload`, `shutdown` set by external API to trigger coordinator action to move to running (`startup`) or dead (`shutdown`); reload causes a reload of system daemons * If the state is set to `shutdown` the coordinator will be shutdown immediately, so this `shutdown` flag is set in the FDW Clean up cronjob, before tearing down the FDWs. * `reload` should cause a reinstall of a running system.  I think the main difference is that `startup` requires the coordinator to be restarted manually, while `reload` is picked up while the coordinator is running and should do the same thing, but appears broken. * Include Schemas Change (HSET) * Include schemas changes *after* a DB Instance is live * key = `:include_changes` * field\_key = `` * field\_value = New list of schemas, can be either `["*"] or ["schema_1", "schema_2", ..., "schemaN"]` ## Database 1 (data) * FDW → DB list (HSET) * Mapping of FDW id → list of database ids the FDW supports. * key = `:fdw_dbs` * field\_key = `FDW id` * field\_value = comma/JSON list of database ids * Namespace include-changes (HSET) * Per-database namespace restriction lists used by change filters. * key = `:include_changes` * field\_key = `` * field\_value = list of namespaces (wildcard \["-"] or explicit list) * Pending include-changes (HSET) * Pending schema include/exclude changes staged for the log manager. * key = `:pending_include_changes` * field\_key = `` * field\_value = JSON object with add/remove lists * DDL queue by XID (QUEUE) * Queue of DDL operations for a specific XID emitted by the log parser. * key = `:queue:ddl:xid::` * queue entries = serialized DDL operations for the given ``, ``, `` * Index DDL queue by XID (QUEUE) * Queue of index-related DDL operations for a specific XID. * key = `:queue:index:ddl:xid::` * queue entries = serialized index DDL ops * Pre-commit DDL hash (HSET) * Hash storing pre-commit DDL operations (indexed by db/xid). * key = `:hash:ddl:pc` * field\_key = `db_id:xid` (or similar) * field\_value = serialized pre-commit DDL ops * DDL queue for FDW (QUEUE) * Queue of DDL changes for FDW processing. * key = `:queue:ddl:fdw:` * queue entries = tasks for FDW id * DDL FDW schema\_xids (HSET) * Hash of schema\_xids per FDW (tracking per fdw:db). * key = `:hash:ddl:fdw` * field\_key = `:` * field\_value = schema\_xid * Daemon liveness (HSET) * Liveness timestamps for daemons (used by coordinator). * key = `:hash:liveness` * field\_key = `:` * field\_value = timestamp * Coordinator liveness pub/sub (PUBSUB) * Pub/sub channel used to notify coordinator of dead daemons. * key = `:pubsub:liveness_notify` * message = `:` * DB table set (SET) * Set of schema.table names for a given database id. * key = `:set:db_tables:` * members = quoted(schema).quoted(table) * DB table changes pub/sub (PUBSUB) * Pub/sub channel notifying the proxy of table changes. * key = `:pubsub:db_table_changes` * message = `:::
` * Index precommit DDL hash (HSET) * Hash for pre-commit index DDL operations. * key = `:hash:idx:ddl:pc` * field\_key = `db_id:xid` (or similar) * field\_value = serialized index DDL ops * Invalid tables hash (HSET) * Hash of excluded/invalid tables for a db\_instance. * key = `:hash:invalid_tables` * field\_key = `` * field\_value = JSON describing exclusion * FDW min XID (HSET) * Minimum XIDs per fdw:db\_id for an instance (tracking progress). * key = `:fdw_min_xids` * field\_key = `:` * field\_value = min xid * FDW PIDs (SET) * Set of new FDW process ids that have not started querying yet. * key = `:fdw_pids` * members = `::` * DB index XIDs (SET) * Set holding index XIDs for each db id. * key = `:set:db_index_xids:` * members = xid values * Table sync state (HSET) * Set of Table IDs (TID/OID) and the corresponding xmin:xmax and in progress pg xids as determined by `pg_current_snapshot()` * key = `set:sync_table_state::` * field\_key = OID (TID) * field\_value = xmin:xmax:xid1,xid2,…. xids=list of in progress xids * Table sync queue (QUEUE); for GC only * Queue of table IDs that require resyncing * key = `queue:sync_tables:` * value = list of `` (OIDs) * Log Mgr resync point (STRING) * Resync point for the log mgr, for it to start reprocessing replication log messages after it was stalled * key = `string:log_resync:` * value = `filename:offset` * Vacuum cutoff XIDs (HSET) * Hash of vacuum cutoff XIDs per db\_id for an instance. Set of XIDs at which index is ddl is received and is being processed by the Indexer * key = `:vacuum_cutoff_xids` * field\_key = `` * field\_value = set of XIDs # Scaling Postgres Source: https://docs.springtail.io/scaling-postgres Modern databases provide multiple approaches to scaling and managing greater load and query throughput. Traditionally, scaling methods are categorized into two main types: 1. **Vertical scaling** - Scaling up by increasing resources on a single machine 2. **Horizontal scaling** - Scaling out by adding more machines (nodes) This guide will discuss the pros and cons of each approach. ## Vertical scaling (scaling up) Vertical scaling, or scaling up, involves increasing the resources of a single database instance. In cloud environments like Amazon RDS, this typically means: 1. Increasing CPU capacity 2. Adding more RAM 3. Upgrading to faster storage (e.g., from gp2 to io1 in AWS) 4. Expanding storage capacity In a cloud environment, CPU and RAM are commonly bundled as a machine instance type and are typically scaled by doubling the instance type (e.g., going from a Large to X-Large). ### Advantages of vertical scaling * Avoids modifications to application logic or database schema * Maintains data consistency and ACID properties * Suitable for workloads that require strong consistency and complex transactions ### Disadvantages of vertical scaling * Performance ceiling: There's a limit to how much a single machine can be scaled up * Cost inefficiency: Larger instances often come with exponential cost increases * Downtime during upgrades: Scaling up often requires instance restarts * Limited fault tolerance: A single instance remains a single point of failure ## Horizontal scaling (scaling out) Horizontal scaling involves distributing the database load across multiple servers. Few truly horizontally scalable databases exist, but these databases run on multiple nodes, typically distributing queries across multiple compute nodes and using a mix of shared storage layer along with locally cached data. Traditional databases like PostgreSQL are not designed for native horizontal scaling, but some strategies can be employed: 1. **Read replicas** - Offload read operations to multiple read-only database instances 2. **Sharding** - Splitting data across multiple database instances based on a partition key 3. **Connection pooling** - Efficiently manage and distribute database connections ### Advantages of horizontal scaling * Scale beyond the limits of a single machine * Improved fault tolerance and availability * More cost-effective at large deployments ### Disadvantages of horizontal scaling * Increased complexity in application logic and database management * Potential for data inconsistency across nodes * May require significant changes to existing applications * May require migration from existing database ## Migrating to horizontally scalable solutions When vertical scaling reaches its limits, organizations may need to migrate to a horizontally scalable solution. This process can be disruptive: Migration often requires significant downtime for data transfer and verification. Code may need to be rewritten to accommodate new data access patterns. Ensuring data integrity across a distributed system is complex. New bottlenecks may emerge in a distributed environment. Managing a distributed database requires new skills and tools. Both in terms of new infrastructure and engineering effort. ## Springtail's approach Springtail recognizes the challenges in migrating an application to a new database. There are often compatibility issues with the new database that require extensive testing, in addition to the time and complexity of performing the migration. Springtail solves this problem by leveraging a scalable set of distributed read-replicas. The set of Springtail replicas can dynamically grow or shrink as demand necessitates; providing throughput when needed, and reducing costs during periods of low usage. Learn more about Springtail's architecture [here](/architecture). # Server Session Source: https://docs.springtail.io/server-session # Server Session Documentation ## Overview The Server Session represents a connection between the Springtail proxy and a PostgreSQL backend server. It is responsible for managing the lifecycle of the server connection, processing client queries, and ensuring proper state synchronization between the client and the server. The Server Session acts as the intermediary between the Client Session and the PostgreSQL server, handling query execution, batching, dependency resolution, and error recovery. ## Responsibilities 1. **Connection Management**: Establish and maintain a connection to the PostgreSQL server, including authentication and session setup. 2. **Query Execution**: Process queries received from the Client Session, ensuring proper routing and execution. 3. **Batch Processing**: Handle batches of messages from the Client Session, including dependency resolution and pipelining. 4. **State Synchronization**: Replay session state (e.g., session variables, prepared statements) to ensure consistency when switching servers or recovering from failover. 5. **Error Handling**: Manage errors during query execution, including extended protocol error recovery and transaction rollback. 6. **Response Coordination**: Collect responses from the PostgreSQL server and forward them to the Client Session. *** ## State Transitions The Server Session operates as a state machine, transitioning between states based on the current operation and the responses from the PostgreSQL server. ### States 1. **STARTUP**: The initial state where the server session is being established and authenticated. 2. **READY**: The idle state where the server session is waiting for the next batch of messages from the Client Session. 3. **DEPENDENCIES**: The state where the server session is processing dependency messages (e.g., session replay) before executing the actual query batch. 4. **QUERY**: The state where the server session is processing the main query batch. 5. **EXTENDED\_ERROR**: The state entered when an error occurs during extended protocol processing. The server session discards all messages until a synchronization point (Sync) is reached. 6. **RESET\_SESSION**: The state where the server session is resetting its state to prepare for reuse. 7. **ERROR**: The state entered when a fatal error occurs, leading to session termination. ### State Transitions #### STARTUP → READY * **Trigger**: Server authentication completes successfully. * **Actions**: * Initialize session parameters. * Transition to READY state to wait for client messages. #### READY → DEPENDENCIES * **Trigger**: A batch with dependency messages is received. * **Actions**: * Process dependency messages (e.g., session replay). * Transition to QUERY state once dependencies are resolved. #### READY → QUERY * **Trigger**: A batch with no dependencies is received. * **Actions**: * Begin processing the query batch. #### DEPENDENCIES → QUERY * **Trigger**: All dependency messages in the batch are processed. * **Actions**: * Transition to QUERY state to process the main query batch. #### QUERY → READY * **Trigger**: The query batch is fully processed, and the server sends a ReadyForQuery message. * **Actions**: * Transition back to READY state to wait for the next batch. #### QUERY → EXTENDED\_ERROR * **Trigger**: An error occurs during extended protocol processing (e.g., Parse, Bind, Execute). * **Actions**: * Discard all messages in the batch until a Sync message is received. * Transition to EXTENDED\_ERROR state to wait for recovery. #### EXTENDED\_ERROR → QUERY * **Trigger**: A Sync message is processed, and the server sends a ReadyForQuery message. * **Actions**: * Transition back to QUERY state to resume normal processing. #### ANY → RESET\_SESSION * **Trigger**: The session is being reset for reuse. * **Actions**: * Clear session state and prepare for reuse. * Send message to Postgres server to reset session state (e.g., DISCARD ALL, etc) * Once response is received session can be added back to a session pool indexed by user/database. #### ANY → ERROR * **Trigger**: An error occurs, or the session is terminated. * **Actions**: * Close the server connection and clean up resources. * Client session is notified. * If the error is fatal, the client session is terminated and error sent to client, otherwise a failover to another session is attempted. *** ## Client Session API The Server Session interacts with the Client Session through a set of well-defined callbacks. These callbacks allow the Server Session to notify the Client Session about query results, errors, and state changes. ### Key Callbacks 1. **Message Response**: Notifies the Client Session when a query message has been processed. * Includes success or failure status. * Allows the Client Session to forward results or handle errors. 2. **Ready For Query**: Notifies the Client Session when the server is ready for the next query. * Includes the transaction status (idle, in-transaction, or error). * Allows the Client Session to update its state and send the next batch. 3. **Error Notification**: Notifies the Client Session about errors during query processing. * Includes error details and the affected query. 4. **Session Replay Completion**: Notifies the Client Session when dependency messages (e.g., session replay) have been processed. * Allows the Client Session to send the main query batch. *** ## Message Flow 1. Messages are received from the Client Session and added to the batch queue. 2. The Server Session processes messages in order, resolving dependencies first. 3. Responses are collected from the PostgreSQL server and forwarded to the Client Session. *** ## Batch Processing The Server Session processes messages from the Client Session in batches. A batch is a group of messages that can be sent to the PostgreSQL server in a single pipeline. ### Batch Structure A batch consists of: 1. **Dependency Messages**: Messages required to synchronize session state (e.g., SET statements, PREPARE statements). 2. **Query Messages**: The main query or queries to be executed. 3. **Control Messages**: Messages like Sync and Flush that control the flow of the batch. ### Batch Lifecycle 1. **Queueing**: The Client Session queues a batch of messages for the Server Session. 2. **Dependency Processing**: If the batch contains dependencies, they are processed first. 3. **Query Execution**: The main query messages are processed after dependencies are resolved. 4. **Completion**: The batch is marked complete when all messages are processed, and the server sends a ReadyForQuery message. *** ## Dependency Handling Dependencies are messages that must be processed before the main query batch. These include: 1. **Session Variables**: SET statements that configure session-level parameters. 2. **Prepared Statements**: PREPARE statements that define reusable queries. 3. **Cursors**: DECLARE statements for cursors that survive transaction boundaries. ### Dependency Resolution 1. Dependencies are sent to the server before the main query batch. 2. The Server Session transitions to DEPENDENCIES state while processing dependencies. 3. Once all dependencies are processed, the Server Session transitions to QUERY state. *** ## Error Handling The Server Session implements robust error handling to ensure proper recovery and client notification. ### Simple Protocol Errors 1. The server sends an ErrorResponse message. 2. The Server Session forwards the error to the Client Session. 3. The Server Session transitions back to READY state after processing the error. ### Extended Protocol Errors 1. An error occurs during Parse, Bind, or Execute. 2. The Server Session enters EXTENDED\_ERROR state. 3. All messages in the batch are discarded until a Sync message is received. 4. The Server Session processes the Sync message and transitions back to QUERY state. ### Fatal Errors 1. A fatal error occurs (e.g., connection failure, protocol violation). 2. The Server Session transitions to ERROR state. 3. The session is terminated, and resources are cleaned up. *** ## Summary The Server Session is a critical component of the Springtail proxy, responsible for managing the connection to the PostgreSQL server, processing client queries, and ensuring proper state synchronization. Its design ensures efficient query execution, robust error handling, and seamless integration with the Client Session. # Session Management Source: https://docs.springtail.io/session-management # Session Management ## Lifecycle The Proxy is session based. A session is a client connection. It exists from the time a client connects until the client disconnects (or is disconnected by a fatal error). A session is represented by a: * **Client Session:** that is connected from the client host to the Proxy * 2 **Server Sessions:** connected from the Proxy to a database instance. They are: * A **Primary Session** is connected to the primary database instance * A **Replica Session** is connected with a Springtail replica node; however if no replica nodes are available there may just be a single Primary Session present. When a session is closed by the client, the associated Server Sessions are returned to a session pool indexed by the user and the database. These are re-used when possible if the same user connects to the same database. A pool may expire a session after a period of time. In summary: 1. A client connects to the proxy, a new client session is created. 2. The client session authenticates the user with the Primary database. 3. A query is received from the client 4. The query is parsed, if the query is a **read-only** query, a replica server is selected as follows: * Either a pooled server session is selected or, * A server session is created, and the user is authenticated 5. This continues until the client disconnects or a fatal error occurs that disconnects the session A replica session may fail-over to another replica if the set of replica nodes are down-sized. If this happens, a new replica is chosen and the existing session state is replayed onto the new replica prior to it serving any queries. ## Session State Session state is preserved in a **History Cache**. The History Cache maintains the set of operations that transform client session state, e.g., Prepared Statements, SET operations, DISCARD or RESET operations. The cache is necessary to replay stateful operations when switching between the Primary and Replica Sessions. E.g., if a `SET search_path ...` operation comes in then it must be replayed on both Primary and Replica Sessions for future operations to be valid. Session state is maintained both at the Session level as well as at the Transaction level. Some state is transactional, meaning it only survives the current transaction and has no effect after the transaction has committed. This state is necessary when switching nodes within a transaction. Session state survives a transaction and must be replayed after a transaction has ended, or if a session is failed over to a new replica. ### Identifying Replay State Every operation is identified by an ID that is assigned once the operation has been successfully applied at a database instance (either primary or replica). Each server session maintains the ID of the last operation it has executed. Thus, the applicable session history can be fetched that has not yet been applied on that database instance. The session state is periodically compacted to remove duplicates or to remove operations that would have no effect due to a future operation (e.g., a RESET or a DISCARD or a CLOSE). ### Session History Long-lived state that persists across transactions: * **SET statements**: Session variables (e.g., work\_mem, search\_path) * **PREPARE statements**: Named prepared statements * **DECLARE WITH HOLD**: Cursors that survive transaction end * **LISTEN statements**: Notification channels ### Transaction History Transaction-scoped state that exists only within a transaction: * All statements executed in the current transaction * Rolled back on ROLLBACK or error * Merged to session history on successful COMMIT * Organized by savepoint levels ### Cache Operations **Adding Statements:** * Parse each query to determine type and properties * Track read-safety, dependencies, and side effects * Associate with current transaction or savepoint level * Store metadata for replay purposes **Replay for Server Sessions:** * Query cache for statements needed by a server session * Filter by session vs transaction scope * Filter by read-only vs read-write * Generate dependency messages in correct order **Transaction Operations:** * **Commit**: Merge transaction history to session history * **Rollback**: Discard entire transaction history * **Savepoint**: Create nested scope level * **Rollback to Savepoint**: Discard statements after savepoint # Shared Memory Caches Source: https://docs.springtail.io/shared-memory-caches # Shared Memory Cache Subsystem ## Overview The Shared Memory Cache Subsystem enables efficient inter-process communication (IPC) between the XID Manager daemon, the PgXidSubscriberMgr daemon, and PostgreSQL Foreign Data Wrapper (FDW) processes. This subsystem allows multiple FDW processes to share system metadata and transaction mutation data without repeated RPC calls to backend services. ### Key Benefits * **Zero-copy data sharing** between processes using Boost Interprocess shared memory * **Reduced RPC overhead** by caching frequently accessed metadata locally * **Transaction-aware caching** with XID (transaction ID) history tracking * **Automatic staleness detection** and cache refresh mechanisms * **LRU-based memory management** to prevent unbounded growth ## Architecture ### High-Level Data Flow ```mermaid theme={null} flowchart TD XID["XID Manager (xid_mgr_daemon)
Manages transaction commitments"] Sub["PgXidSubscriberMgr (daemon, separate thread)
- Receives XID notifications
- Creates 5 shared memory caches
- Populates caches on transaction commit
- Worker threads fetch metadata proactively"] subgraph SHM [Shared Memory IPC] direction TB Roots["SHM_CACHE_ROOTS (table metadata)"] Schemas["SHM_CACHE_SCHEMAS (table schemas)"] UserTypes["SHM_CACHE_USERTYPES (user-defined types)"] TableIds["SHM_CACHE_TABLE_IDS (pending mutations)"] Extents["SHM_CACHE_EXTENTS (mutation records)"] end FDW["PgFdwMgr (FDW process)
- Runs in PostgreSQL
- Opens existing caches
- Queries pending XIDs
- Applies mutations"] XID -->|"gRPC: XidPushResponse"| Sub Sub --> SHM SHM -->|opens and reads| FDW ``` ### Components | Component | Role | Process Type | | ---------------------- | -------------------------------------------------- | ---------------------- | | **PgXidSubscriberMgr** | Cache producer - creates and maintains caches | Daemon thread | | **PgFdwMgr** | Cache consumer - opens and uses caches | PostgreSQL FDW process | | **ShmCache** | Shared memory abstraction using Boost Interprocess | Library class | | **MsgCache** | Multi-index container with LRU eviction | Template class | | **XidMgrClient** | Streams XID commit notifications | gRPC client | ## Cache Types The subsystem manages **5 distinct shared memory caches**, each serving a specific purpose: ### 1. Roots Cache (SHM\_CACHE\_ROOTS) **Purpose:** Stores table root/metadata information **Key Properties:** * XID history: **Enabled** * Tracks schema changes across transactions * Contains table root objects with metadata pointers **Data Structure:** ```cpp theme={null} Key: (DbId, TableId) Value: vector sorted by XID ``` **Location:** * Created: `pg_xid_subscriber_mgr.cc:53` * Used by: `sys_tbl_mgr::Client::get_roots()` ### 2. Schema Cache (SHM\_CACHE\_SCHEMAS) **Purpose:** Stores table schema definitions **Key Properties:** * XID history: **Disabled** * Contains column definitions, types, constraints * Updated on DDL operations **Data Structure:** ```cpp theme={null} Key: (DbId, TableId) Value: vector ``` **Location:** * Created: `pg_xid_subscriber_mgr.cc:56` * Used by: `sys_tbl_mgr::Client::get_schema()` ### 3. UserType Cache (SHM\_CACHE\_USERTYPES) **Purpose:** Stores user-defined type information **Key Properties:** * XID history: **Disabled** * Contains custom type definitions * Required for schema deserialization **Data Structure:** ```cpp theme={null} Key: (DbId, TypeId) Value: vector ``` **Location:** * Created: `pg_xid_subscriber_mgr.cc:61` * Used by: `sys_tbl_mgr::Client::get_usertype()` ### 4. Table IDs Cache (SHM\_CACHE\_TABLE\_IDS) **Purpose:** Maps (DbId, XID) to list of TableIds modified in each transaction **Key Properties:** * XID history: **Enabled** * Tracks pending mutations per transaction * Critical for read-after-write consistency **Data Structure:** ```cpp theme={null} Key: (DbId, Xid) Value: serialized proto::XidPushResponse Fields: {db_id, xid, has_schema_changes, real_commit, table_ids[]} ``` **Location:** * Created: `pg_xid_subscriber_mgr.cc:64` * Used by: `PgFdwMgr::_get_table():645-654` **Usage Pattern:** ```cpp theme={null} // Query pending XIDs _table_ids_shm_cache->get_pending_xids(db_id) // Fetch mutation metadata _table_ids_shm_cache->find(db_id, xid) ``` ### 5. Extents Cache (SHM\_CACHE\_EXTENTS) **Purpose:** Stores mutation records (extents) from WriteCacheClient **Key Properties:** * XID history: **Disabled** * Contains actual data mutations * Populated before applying to base tables **Data Structure:** ```cpp theme={null} Key: (DbId, ExtentId) Value: vector ``` **Location:** * Created: `pg_xid_subscriber_mgr.cc:68` * Used by: `WriteCacheClient` for mutation staging ## Detailed Component Documentation ### PgXidSubscriberMgr **Header:** `include/pg_fdw/pg_xid_subscriber_mgr.hh:22-105` **Implementation:** `src/pg_fdw/pg_xid_subscriber_mgr.cc` #### Responsibilities 1. **Cache Creation** (`init()` method): * Creates all 5 shared memory caches with configured sizes * Removes stale caches from previous runs * Initializes XID history cleaners for enabled caches 2. **XID Push Notification Handling** (`task()` method): * Subscribes to XidMgrClient push stream * Receives `proto::XidPushResponse` messages * Dispatches based on `real_commit` flag 3. **Asynchronous Population** (worker threads): * Configurable thread pool size * Fetches metadata on transaction commit * Calls `Client::get_roots()`, `get_schema()`, `get_usertype()` * Automatically populates caches via registered ShmCache instances ### PgFdwMgr **Header:** `include/pg_fdw/pg_fdw_mgr.hh:133-468` **Implementation:** `src/pg_fdw/pg_fdw_mgr.cc` #### Responsibilities 1. **Cache Opening** (`_try_create_cache()` method): * Opens existing caches created by PgXidSubscriberMgr * Verifies cache liveness with `is_alive()` check * Registers caches with Client and WriteCacheClient singletons 2. **Mutation Application** (`_get_table()` method): * Queries pending XIDs since last committed * Fetches mutation metadata from table\_ids\_cache * Retrieves extent data from WriteCache * Applies mutations to base table as ChangeSet 3. **Lifecycle Management**: * Lazy initialization on first query * Automatic cache reopening if stale * Background thread for XID updates **Key Points:** * Opens in read mode (size=0 parameter) * Handles case where PgXidSubscriberMgr hasn't started yet * Checks `is_alive()` to detect stale caches * Registers with appropriate singleton clients **Performance Considerations:** * Only processes `XIDs <= snapshot_xid` (transaction isolation) * Limits to `MAX_WRITE_CACHE_EXTENTS` (10) per query * Resets pending XIDs if too many extents found * Avoids redundant extent fetches ### ShmCache Class **Header:** `include/sys_tbl_mgr/shm_cache.hh:73-360` **Implementation:** `src/sys_tbl_mgr/shm_cache.cc` #### Purpose Provides a shared memory abstraction using Boost Interprocess for inter-process data sharing. #### Key Features 1. **Dual-mode construction**: * **Create mode:** `ShmCache(type, size, xid_history)` with size > 0 * **Open mode:** `ShmCache(type, 0, xid_history)` opens existing cache 2. **XID tracking** (if enabled): * Tracks committed XID per database * Maintains list of pending XIDs * Records XID history for schema changes 3. **Keep-alive mechanism**: * Caches must be kept alive every 60ms * `is_alive()` checks if cache is stale * Automatic reconnection if stale detected **Usage:** * PgXidSubscriberMgr calls `keep_alive()` every 60ms * PgFdwMgr checks `is_alive()` before using cache * If stale, PgFdwMgr reopens cache ### MsgCache Template **Header:** `include/sys_tbl_mgr/msg_cache.hh:31-323` #### Purpose Multi-index container with LRU eviction, providing both fast lookup and insertion-order tracking. **Configuration:** * Free memory limit: **30%** (trigger eviction) * Free memory watermark: **50%** (stop eviction) * Retention: Top **10%** frequently accessed items protected (`msg_cache.hh:236`) ## Synchronization and Concurrency ### Thread Safety Strategy #### Mutex Hierarchy ```cpp theme={null} // PgFdwMgr (pg_fdw_mgr.hh:298) std::shared_mutex _shm_cache_mutex; // Protects cache pointers // ShmCache (shm_cache.hh:96) boost::interprocess::named_sharable_mutex* _mutex; // Per-cache lock // PgXidSubscriberMgr (pg_xid_subscriber_mgr.hh:93) std::mutex _mutex; // Protects populate job queue std::condition_variable _cv; // Worker coordination ``` ## Configuration ### Configuration Properties **File:** Properties singleton (commonly loaded from YAML/JSON config) **Schema:** ```json theme={null} { "sys_tbl_mgr": { "roots_shm_cache_size": 1073741824, "schema_shm_cache_size": 536870912, "usertype_shm_cache_size": 268435456, "table_ids_shm_cache_size": 536870912, "extents_shm_cache_size": 1073741824, "rpc_config": { "server_worker_threads": 4 } } } ``` **Parameters:** | Parameter | Type | Default | Description | | -------------------------- | ------- | ------- | --------------------------------------------- | | `roots_shm_cache_size` | size\_t | 1 GB | Size of roots cache in bytes | | `schema_shm_cache_size` | size\_t | 512 MB | Size of schema cache in bytes | | `usertype_shm_cache_size` | size\_t | 256 MB | Size of usertype cache in bytes | | `table_ids_shm_cache_size` | size\_t | 512 MB | Size of table IDs cache in bytes | | `extents_shm_cache_size` | size\_t | 1 GB | Size of extents cache in bytes | | `server_worker_threads` | size\_t | 4 | Number of worker threads for async population | ## Usage Examples ### Example 1: Querying Table with Pending Mutations **Scenario:** PostgreSQL FDW process executes `SELECT * FROM table WHERE id = 123` with snapshot XID 1000. **Steps:** ```cpp theme={null} // 1. PgFdwMgr::_get_table() is called auto table = _get_table(db_id, table_id, snapshot_xid=1000); // 2. Fetch base table from SysTableMgr auto base_table = SysTableMgr::get_instance()->get_table(db_id, table_id, 1000); // 3. Query pending XIDs from table_ids_cache auto pending_xids = _table_ids_shm_cache->get_pending_xids(db_id); // Result: [950, 975, 1050] // 4. Filter XIDs <= snapshot_xid // Process: 950, 975 (skip 1050) // 5. For XID 950: auto msg = _table_ids_shm_cache->find(db_id, 950); proto::XidPushResponse response; response.ParseFromString(msg); // response.table_ids() = [10, 25, 30] // 6. Check if table_id matches if (table_id == 25) { // 7. Fetch extents from WriteCache auto extents = WriteCacheClient::get_instance()->get_extents(db_id, table_id, 950); // 8. Apply extents as ChangeSet ChangeSet cs; for (auto& extent : extents) { cs.add_extent(extent); } base_table->apply_changeset(cs, 950); } // 9. Repeat for XID 975 // ... similar logic ... // 10. Return merged table return base_table; ``` **Result:** Query sees all committed data plus uncommitted mutations from XIDs 950 and 975, ensuring read-after-write consistency. ### Example 3: Cache Staleness Detection and Recovery **Scenario:** PgXidSubscriberMgr crashes and restarts while FDW processes are running. **Steps:** ```cpp theme={null} // 1. PgFdwMgr detects stale cache bool ShmCache::is_alive() const { auto now = std::chrono::steady_clock::now(); auto elapsed = now - _last_keep_alive; return elapsed < std::chrono::milliseconds(60); } // 2. PgFdwMgr::_try_create_cache() checks liveness if (!_roots_shm_cache || !_roots_shm_cache->is_alive()) { // 3. Reopen cache _roots_shm_cache = std::make_shared( sys_tbl_mgr::SHM_CACHE_ROOTS, 0, true); // Open mode // 4. Re-register with Client sys_tbl_mgr::Client::get_instance()->use_roots_cache(_roots_shm_cache); } // 5. PgXidSubscriberMgr restarts PgXidSubscriberMgr::start(); // 6. New caches created // Old shared memory segments removed // Fresh XIDs streamed from XID Manager // 7. FDW processes reopen caches on next query // Seamless recovery with minimal downtime ``` #### 4. LRU Eviction Strategy **Problem:** Unbounded cache growth causes OOM **Solution:** Multi-index container with sequenced index for LRU **Benefits:** * Automatic eviction at 30% free memory * Stops eviction at 50% free memory watermark ### Debugging Commands ```bash theme={null} # List shared memory segments ipcs -m # Remove stale shared memory ipcrm -m # Check cache sizes ls -lh /dev/shm/ # Monitor cache usage watch -n 1 'ipcs -m -u' # Verify PgXidSubscriberMgr is running ps aux | grep pg_xid_subscriber_mgr # Check XID Manager connection netstat -an | grep ``` \## Design Patterns ### 1. Producer-Consumer with Lazy Initialization **Pattern:** * **Producer:** PgXidSubscriberMgr creates and maintains caches * **Consumer:** PgFdwMgr opens and uses caches * **Trigger:** Query execution in PostgreSQL FDW process **Benefits:** * Loose coupling between producer and consumer * Graceful handling of startup ordering * Automatic recovery on producer restart ## References ### Source Files | File | Description | | ------------------------------------------- | ----------------------------------- | | `include/pg_fdw/pg_xid_subscriber_mgr.hh` | PgXidSubscriberMgr class definition | | `src/pg_fdw/pg_xid_subscriber_mgr.cc` | PgXidSubscriberMgr implementation | | `include/pg_fdw/pg_fdw_mgr.hh` | PgFdwMgr class definition | | `src/pg_fdw/pg_fdw_mgr.cc` | PgFdwMgr implementation | | `include/sys_tbl_mgr/shm_cache.hh` | ShmCache class definition | | `src/sys_tbl_mgr/shm_cache.cc` | ShmCache implementation | | `include/sys_tbl_mgr/msg_cache.hh` | MsgCache template definition | | `src/proto/xid_manager.proto` | XidPushResponse protobuf definition | | `include/write_cache/write_cache_client.hh` | WriteCacheClient class definition | ### External Dependencies | Library | Purpose | Documentation | | ------------------ | ---------------------- | -------------------------------------------------------------------------------------------------------- | | Boost Interprocess | Shared memory IPC | [boost.org/doc/libs/interprocess](https://www.boost.org/doc/libs/1_82_0/doc/html/interprocess.html) | | Boost Multi-Index | Multi-index containers | [boost.org/doc/libs/multi\_index](https://www.boost.org/doc/libs/1_82_0/libs/multi_index/doc/index.html) | | Protocol Buffers | Serialization | [developers.google.com/protocol-buffers](https://developers.google.com/protocol-buffers) | | gRPC | RPC framework | [grpc.io](https://grpc.io) | ### Related Documentation * PostgreSQL Foreign Data Wrapper API * Springtail Transaction Management * XID Manager Architecture * WriteCache Design # Singleton Services Source: https://docs.springtail.io/singleton-services # Singleton Template Class ## Overview The `Singleton` template class provides a thread-safe singleton pattern implementation with optional threading support and lifecycle management. This class is part of the `springtail` namespace and is designed to ensure that only one instance of a derived class exists throughout the application's lifetime. ## Features * **Thread-safe initialization** using `std::call_once` * **Optional threading support** with automatic thread naming * **Service registration** integration with centralized shutdown management through `ServiceRegistry` * **Protected lifecycle hooks** for derived class customization * **Deleted copy/move semantics** to prevent duplication * **Atomic shutdown flag** for safe thread termination ## Class Template ```cpp theme={null} template class Singleton ``` The template parameter `T` should be the derived class that inherits from `Singleton` (CRTP pattern). ## Public Interface ### Static Methods #### `get_instance()` ```cpp theme={null} static T* get_instance() ``` Returns the singleton instance of type `T`. Creates the instance on first call using thread-safe initialization. **Returns:** Pointer to the singleton instance **Thread Safety:** Yes **Example:** ```cpp theme={null} MyService* service = MyService::get_instance(); ``` #### `shutdown()` ```cpp theme={null} static void shutdown() ``` Performs cleanup and destruction of the singleton instance. Can be called multiple times safely (only executes once). This method: 1. Sets the shutdown flag 2. Signals thread termination if a thread exists 3. Joins the thread 4. Calls internal cleanup 5. Deletes the instance **Thread Safety:** Yes ### Instance Methods #### `start_thread()` ```cpp theme={null} void start_thread() ``` Starts a dedicated thread for the singleton instance. The thread executes the `_internal_run()` method. Thread name is automatically set based on the service ID or type name. **Example:** ```cpp theme={null} MyService::get_instance()->start_thread(); ``` ## Protected Interface Derived classes should override these methods to customize behavior: ### `_internal_run()` ```cpp theme={null} virtual void _internal_run() ``` Main execution function for the singleton's thread. Must be overridden if `start_thread()` is used. Should periodically check `_is_shutting_down()` to enable graceful termination. **Example:** ```cpp theme={null} void _internal_run() override { while (!_is_shutting_down()) { // Perform work std::this_thread::sleep_for(std::chrono::milliseconds(100)); } } ``` ### `_internal_thread_shutdown()` ```cpp theme={null} virtual void _internal_thread_shutdown() ``` Called during shutdown before the thread is joined. Use this to wake up blocking operations (e.g., notify condition variables). **Example:** ```cpp theme={null} void _internal_thread_shutdown() override { _condition.notify_all(); } ``` ### `_internal_shutdown()` ```cpp theme={null} virtual void _internal_shutdown() ``` Called during shutdown after the thread has been joined. Use this for cleanup of resources. **Example:** ```cpp theme={null} void _internal_shutdown() override { _connection.close(); } ``` ### `_is_shutting_down()` ```cpp theme={null} bool _is_shutting_down() const ``` Returns whether the singleton is in the shutdown process. **Returns:** `true` if shutting down, `false` otherwise ### Constructor ```cpp theme={null} explicit Singleton(ServiceId service_id = ServiceId::ServiceInvalidId) ``` Protected constructor that can only be called by derived classes. Optionally registers the service for centralized shutdown management. **Parameters:** * `service_id`: Service identifier for registration (default: `ServiceId::ServiceInvalidId`) ### Static Utility Methods #### `_assert_instance()` ```cpp theme={null} static void _assert_instance() ``` Asserts that the singleton instance has been created. Terminates if not. #### `_has_instance()` ```cpp theme={null} static bool _has_instance() ``` Checks if the singleton instance exists. **Returns:** `true` if instance exists, `false` otherwise ## ServiceId Enum `ServiceId` uniquely identifies a logical service within the Springtail system. These identifiers are used for: * Dependency ordering * Thread naming * Coordinated shutdown * Service registration The `ServiceId` enum defines service identifiers for various components in the system: ```cpp theme={null} enum class ServiceId : int32_t { ServiceInvalidId = -1, ServiceRegisterId = 0, DatabaseMgrId, UserMgrId, ProxyServerId, // ... additional service IDs ServiceCountId } ``` `ServiceInvalidId` is reserved for non-registered or internal services. ## Helper Functions ### `get_type_name()` ```cpp theme={null} template std::string get_type_name() ``` Extracts the type name from compiler-generated strings. **Returns:** String representation of type `T` ### Service Registration ```cpp theme={null} void springtail_register_service(ServiceId service_id, ShutdownFunc fn) const std::string& springtail_get_service_name(ServiceId id) ``` Functions for registering services and retrieving service names. ### `springtail_register_service()` Registers a service shutdown callback with the global service registry. * Services are shut down in dependency order. * Registration is idempotent and enforced per service ID. * Typically invoked automatically by `Singleton`. Registered shutdown functions are executed during `springtail_shutdown()`. ### `springtail_get_service_name()` Returns a human-readable name for a given `ServiceId`. Used for: * Logging * Thread naming * Debug output If a service ID is invalid, `"Invalid"` is returned. ## Singleton Framework The `Singleton` template provides a thread-safe, lifecycle-aware singleton implementation designed for long-lived system services. ### Key Properties * Lazy exactly-once initialization * Exactly-once shutdown * Optional background thread * Dependency-aware cleanup * Integrated with service registry ## Singleton Lifecycle ### Instance Creation * The instance is created on first call to `get_instance()`. * Creation is guarded by `std::call_once`. * The derived class must have a default constructor. ### Thread Management Singletons may optionally run a dedicated thread. To enable threading: * Call `start_thread()`. * Implement `_internal_run()`. Thread behavior: * Thread name is derived from `ServiceId`. * If no valid `ServiceId` exists, the type name is used. * Names are truncated to 15 characters to satisfy pthread limits. ### Shutdown Semantics Shutdown is global and idempotent. When `shutdown()` is called: 1. `_shutting_down` is set to true. 2. `_internal_thread_shutdown()` is invoked. 3. The thread is joined (if started). 4. `_internal_shutdown()` is invoked. 5. The instance is destroyed. Shutdown occurs exactly once, even if called multiple times. ## Overrides in Derived Classes Derived classes may need to implement overrides for the following hooks. Some of these functions are for running and shutting down the internal thread, while another is simply for internal shutdown. ### `_internal_run` Executed in the background thread. Must be overridden if `start_thread()` is used. *** ### `_internal_thread_shutdown` Used to wake or unblock the background thread prior to joining. Typical use cases: * Notifying condition variables * Closing file descriptors * Signaling event loops It does not have to be overriden, but if it is, it should signal termination for the thread run by this singleton service. ### `_internal_shutdown` Final cleanup hook. Used for: * Releasing resources * Stopping dependent components * Final state updates If there are other threads owned by this service, this is the place to signal their termination and join them. ## Shutdown State Checking ### `_is_shutting_down` Returns `true` if shutdown has been initiated. Intended for use inside `_internal_run()` loops to allow cooperative termination. ## Constructor Behavior The `Singleton` constructor accepts an optional `ServiceId`. If a valid `ServiceId` is provided: * The service is automatically registered for shutdown. * The shutdown order is dependency-aware and is a part of the initialization framework. Default value for service id is `ServiceInvalidId`. This value ensures that the service **won't be registered** with initialization framework. It will be up to the user to figure out when and how to invoke `shutdown()` function. ## Type Name Utility ### `get_type_name` Returns a demangled type name extracted from `__PRETTY_FUNCTION__`. Used internally for: * Thread naming fallback * Diagnostics Not intended as a stable ABI feature. ## Typical Usage Pattern 1. Define a service class inheriting from `Singleton`. 2. If you want this class to be a part of the global shutdown sequence, then you need to do the following: * create a new service id in `ServiceId` enum * in the constructor pass this new service id to `Singleton` constructor * in `init.cc` file, setup dependencies for this service on the other existing services, for the very least it should depend on `ServiceRegisterId` because this is the id of the service that provides initialization and shutdown for Springtail base functionality * in `init.cc` file, add the name of the new service to service names map `dependencies_names` 3. Optionally start a background thread. 4. Let the framework manage lifecycle and shutdown if the service id is assigned, otherwise manage it yourself. Shutdown is coordinated globally via `springtail_shutdown()`. Here is the usage example. ```cpp theme={null} class MyService : public springtail::Singleton { friend class springtail::Singleton; protected: MyService() : Singleton(springtail::ServiceId::MyServiceId) { // Initialize your service } void _internal_run() override { while (!_is_shutting_down()) { // Process work process_requests(); std::this_thread::sleep_for(std::chrono::milliseconds(100)); } } void _internal_thread_shutdown() override { // Wake up any blocking operations _request_queue.notify_all(); } void _internal_shutdown() override { // Cleanup resources _cleanup(); } public: void process_request(const Request& req) { // Public API } private: void process_requests() { // Internal processing } }; // Usage int main() { springtail_init(false); // Get instance and start thread MyService::get_instance()->start_thread(); // Use the service MyService::get_instance()->process_request(request); // Shutdown (typically called from a signal handler) springtail_shutdown(); } ``` ## Thread Safety * Instance creation is thread-safe via `std::call_once` * Shutdown is thread-safe via `std::call_once` * The `_shutting_down` flag is atomic for safe cross-thread access * Derived classes are responsible for their own thread-safety ## Design Considerations * No double initialization * No double shutdown * Deterministic shutdown order * Thread-safe lifecycle transitions * No dependency cycles allowed * Uses the Curiously Recurring Template Pattern (CRTP) for static polymorphism * Derived classes should friend `Singleton` to allow private constructor access * All copy and move operations are explicitly deleted * Instance is created on first `get_instance()` call * Services can self-register for centralized shutdown This framework is designed for: * Long-lived system services * Daemons and infrastructure components * Deterministic startup/shutdown # Storage Layout Source: https://docs.springtail.io/storage-layout # Springtail Storage Layer This document describes the storage layout and organization of data in Springtail's storage engine, from the low-level extent structure up through tables and indexes. ## Overview Springtail uses a multi-version concurrency control (MVCC) storage system with copy-on-write semantics. Data is organized hierarchically: * **Extents**: Fixed-size blocks containing rows of data with a fixed-width schema * **Fields**: Type-specific accessors for reading/writing column values within rows * **BTrees**: B+ tree indexes composed of extents, used for both primary and secondary indexes * **Tables**: Logical collections of data organized by a primary BTree and zero or more secondary indexes * **Cache**: Unified page cache managing access to extents across all tables ## Extent Structure An extent is the fundamental storage unit in Springtail. Each extent consists of three components stored sequentially on disk: ### 1. Extent Header The header (serialized via `ExtentHeader::pack()`) contains metadata about the extent: ``` Offset Size Field ------ ---- ----- 0 8 xid - Transaction ID when extent was written 8 8 prev_offset - Location of previous extent version (for MVCC) 16 4 row_size - Fixed size of each row in bytes 20 4 field_count - Number of fields in schema 24 N field_types - Array of field type bytes (1 byte per field) 24+N 1 type - Extent type (leaf/branch, root flag) ``` **Field Type Encoding**: Each field type byte encodes both the Springtail type (lower 7 bits) and nullable flag (top bit). **Extent Types**: * `LEAF` (0x00): Contains actual data rows * `BRANCH` (0x01): Contains index keys and child extent pointers * Root flag (0x02): Indicates this extent is a tree root ### 2. Fixed Data The fixed data section contains a contiguous array of fixed-width rows. Each row has exactly `row_size` bytes and stores: * **Fixed-width values**: Integers, floats stored inline at specific byte offsets (at the beginning of the row) * **Variable-data offsets**: 4-byte offsets into the variable data section for TEXT, BINARY, NUMERIC, and EXTENSION types * **Boolean fields**: Packed as bits after all fixed-width fields * **Null bitmaps**: Packed bits indicating which nullable fields are NULL (after boolean fields) * **Undefined bitmaps**: Packed bits for replication tracking (indicates unchanged fields). Note: undefined bitmaps are only present in write cache extents, not in on-disk table extents. Row layout is determined by the `ExtentSchema` (see `src/storage/schema.cc:79-162`), which places fixed-width fields first, then boolean bits, null bits, and finally undefined bits (if applicable) at the end of the row. **Example row structure for on-disk extents** (simplified): ``` [int64_field: 8 bytes] [text_offset: 4 bytes] [int32_field: 4 bytes] [bool_field: 1 bit] [null_bits: N bits] ``` **Example row structure for write cache extents** (simplified): ``` [int64_field: 8 bytes] [text_offset: 4 bytes] [int32_field: 4 bytes] [bool_field: 1 bit] [null_bits: N bits] [undefined_bits: M bits] ``` ### 3. Variable Data Variable-length values (TEXT, BINARY, NUMERIC, EXTENSION types) are stored in a separate variable data section managed by the `VariableData` class. This provides: * **Length-prefixed storage**: Each value stored as `[4-byte length][data]` * **Automatic deduplication**: Hash table detects duplicate values within an extent * **Chunked allocation**: Data organized in 4KB chunks to avoid vector reallocation * **Global offset addressing**: Fixed data stores 4-byte offsets into this global space The deduplication hash (in `Extent::_variable_hash`) maps `string_view` → offset, ensuring identical strings share storage. ## Field Abstraction The `Field` class hierarchy provides type-safe access to column values within logical rows (often extent rows, but also replication log messages, constant values, etc.): ### Field Types **ExtentField** (`include/storage/field.hh:508`): Accesses data in an `Extent::Row` by: * Storing byte offset within the fixed row data * For booleans: storing bit position and bitmask * For nullable fields: tracking null bitmap offset and bit * For variable data: reading offset from fixed data, then dereferencing into variable section **ConstField**: Represents constant values (for query predicates, default values) **PgLogField**: Reads data from PostgreSQL replication log messages **MutableField** (`include/storage/field.hh:325`): Extends `Field` with `set_*` methods for writing values ### Field Operations Fields support: * Type-specific `get_*` and `set_*` methods (e.g., `get_int64()`, `set_text()`) * Comparison operations (`less_than()`, `equal()`) used for index searches * Null and undefined checks via bitmask operations * Copying values between different row sources ## BTree Organization Tables and indexes are organized as B+ trees composed of extents: ### BTree Structure (`include/storage/btree.hh`) **Read-Only BTree**: * Immutable snapshot at a specific XID * Root extent identified by offset * Branch extents contain: `[sort_key_fields...][child_extent_offset: uint64]` * Leaf extents contain: actual table data or index entries * Binary search within extents for O(log n) lookups **Mutable BTree** (`include/storage/mutable_btree.hh`): * Copy-on-write modifications at target XID * In-memory page cache with configurable size * Dirty pages flushed hierarchically (children before parents) * Supports insert, remove, upsert operations * Auto-splits extents when they exceed size threshold ### BTree Extent Types **Branch Extents**: * Schema: sort key columns + `child_offset` field * Each row represents separator key + pointer to child extent * Used for navigation during tree traversal **Leaf Extents**: * For primary index: full table rows with all columns * For secondary index: index columns + extent\_id + row\_id for lookup ### BTree Iterator The `BTree::Iterator` performs in-order traversal: * Maintains path from root to current leaf (via `Node` objects) * `++` operator: advances within extent or navigates to next leaf via parent * `--` operator: similar backward traversal * Supports binary search via `lower_bound()`, `upper_bound()`, `find()` ## Table Organization ### Table Structure (`include/sys_tbl_mgr/table.hh`) A `Table` represents a logical collection of rows with: **Primary Index** (`_primary_index`): * BTree storing full row data, sorted by primary key * Schema includes all table columns * Branch extents use primary key fields for navigation * Leaf extents contain complete rows **Secondary Indexes** (`_secondary_indexes`): * Map from index\_id → (BTree, column\_positions) * Schema: index columns + internal\_row\_id * Used for queries with non-primary-key predicates * Maintained alongside primary index **Look-Aside Index** (`_look_aside_index`): * Maps internal\_row\_id → (extent\_id, row\_position) * Enables secondary index lookups to find actual rows * Schema: `[internal_row_id][extent_id][row_offset]` ### Table Iterator `Table::Iterator` supports multiple iteration modes: **Primary iteration**: Traverses BTree, loads data extents from extent\_id pointers **Secondary iteration**: Uses secondary index + look-aside for data access **Index-only iteration**: Returns index entries without fetching full rows ### Table Metadata Tables maintain metadata in a "roots" file: ``` Schema: [index_id: uint64][extent_id: uint64][last_internal_row_id: uint64] ``` One row per index tracks the root extent for each snapshot. ## Mutable Tables ### MutableTable (`include/sys_tbl_mgr/mutable_table.hh`) Provides write operations on tables: **Operations**: * `insert()`: Add new row (append or positioned by primary key) * `update()`: Modify existing row (copy-on-write) * `remove()`: Delete row from extent and indexes * `upsert()`: Insert or update based on primary key existence **Write Strategies**: * Empty table: Use `_empty_page` for first extent * Append: Add to last extent if sorted correctly * Lookup: Use primary index to find target extent * Direct: Write to specified extent\_id (from write cache) **Index Maintenance**: * Dirty pages trigger `_flush_handler()` callback * Invalidates old index entries via `_invalidate_indexes()` * Populates new entries via `_flush_and_populate_indexes()` * Maintains consistency between primary and secondary indexes ### Finalization `MutableTable::finalize()`: 1. Flushes all dirty pages (primary data + all index BTrees) 2. Returns `TableMetadata` with new root extent offsets 3. Optionally calls `sync_data_and_indexes()` for durability ## Storage Cache ### Cache Architecture (`include/storage/cache.hh`) The `StorageCache` provides unified page management: **Page States**: * `CLEAN`: Read from disk, unmodified, can be evicted * `DIRTY`: Modified by MutableTable, needs flush * `MUTABLE`: Actively being modified * `FLUSHING`: Asynchronous write in progress * `INVALID`: Evicted or replaced **Page Structure**: * Wraps one or more `Extent` objects * Tracks extent\_id, cache\_id, file path * Reference counting for multi-user access * LRU eviction for clean pages **Cache Operations**: * `get()`: Retrieve page by (database\_id, file, extent\_id, xid) * `get_or_create()`: Get existing or create new mutable page * Copy-on-write: CLEAN pages copied to MUTABLE on first write * Hierarchical flushing: Dirty pages written with updated child pointers ### SafePagePtr RAII wrapper managing page lifecycle: * Acquires page from cache on construction * Releases back to cache on destruction * Supports read (shared) and write (exclusive) locks * Transparent access to underlying `Page` object ## MVCC and Versioning ### Copy-on-Write Semantics All modifications create new extent versions: 1. Read extent at access\_xid 2. Copy to new MUTABLE extent 3. Apply modifications 4. Flush with target\_xid in header 5. Update parent extent with new child pointer ### Extent Chains Extents form version chains via `prev_offset`: ``` extent@xid=100 → extent@xid=50 → extent@xid=20 → ... ``` Old versions remain for queries at earlier XIDs (cleaned by Vacuumer). ### Snapshot Isolation * `Table` objects snapshot at specific XID * BTree navigation uses extent headers to find correct version * Time-travel queries access historical extent versions * Write transactions create new snapshot at target\_xid ## Schema Evolution Schemas are versioned alongside data: **ExtentSchema** (`include/storage/schema.hh`): * Defines column order, types, nullability * Calculates fixed row layout and field offsets * Generates Field accessors for reading/writing **Schema Changes**: * New columns: Added with default values or NULL * Dropped columns: Marked as non-existent in schema * Type changes: Handled via schema conversion during page reads * Schema metadata tracked per XID range ## File Organization Tables stored in directories: ``` {table_base}/{db_id}/{table_id}/{snapshot_xid}/ ├── raw - Primary table data extents ├── {index_id}.idx - Secondary index extents (one file per index) ├── look_aside - Look-aside index for row lookup └── roots - Index root metadata (system tables only) ``` Each file is an append-only sequence of extents written at increasing offsets. ## Summary Springtail's storage layer provides: * **Extent-level**: Compact fixed+variable row storage with deduplication * **Field-level**: Type-safe column access with null/undefined tracking * **Index-level**: B+ tree organization with copy-on-write modifications * **Table-level**: Primary + secondary indexes with look-aside support * **Cache-level**: Unified page management with MVCC snapshots This architecture enables efficient read-only OLTP-style queries while supporting MVCC for time-travel and concurrent access. # Syncing Tables Source: https://docs.springtail.io/syncing-tables # Table Synchronization Architecture ## Overview Springtail synchronizes tables from a primary PostgreSQL database using a sophisticated architecture that coordinates table copying with an ongoing three-stage replication pipeline (Log Writer → Log Reader → Committer). The PgCopyTable component operates independently to perform bulk table copies while the pipeline ensures consistency by tracking PostgreSQL transaction IDs (XIDs) and using snapshot-based visibility rules to determine which mutations should be applied during and after the copy. This document describes the architecture for synchronizing tables into Springtail, with particular emphasis on: * How the replication pipeline is stalled during table copies * How PostgreSQL XIDs (pgxids) are tracked and used * How snapshot-based visibility ensures correct synchronization points * How log replay interacts with table synchronization ## Key Components ### PgLogMgr **Location:** `src/pg_log_mgr/pg_log_mgr.{cc,hh}` The orchestration hub for Springtail's replication pipeline. Manages a three-stage processing pipeline: 1. **Log Writer Thread** - Connects to PostgreSQL replication stream and writes to log files 2. **Log Reader Thread** - Reads log files and parses transactions (via PgLogReader) 3. **Committer** - Processes committed transactions and coordinates garbage collection **Additional Coordination Thread:** * **Copy Thread** - Coordinates table synchronization requests from Redis queue, using PgCopyTable to perform the actual copying while ensuring consistent snapshots through pipeline coordination **State Machine:** ``` STATE_STARTUP → Initial state upon startup STATE_STARTUP_SYNC → Full sync required after startup STATE_RUNNING → Normal operational state STATE_SYNC_STALL → Stalls pipeline during sync STATE_SYNCING → Actively syncing tables STATE_REPLAYING → Replaying logs after sync STATE_STOPPED → Shutdown state ``` **Key Responsibilities:** * Manages state transitions during table synchronization * Coordinates pipeline stalling via STALL messages in the logger queue * Executes table copies via PgCopyTable * Assigns Springtail XIDs to transactions ### PgCopyTable **Location:** `src/pg_repl/pg_copy_table.{cc,hh}` A separate component (not part of the main replication pipeline) that handles the actual table copying using PostgreSQL's binary COPY protocol. Invoked by PgLogMgr's Copy Thread and interacts with the pipeline through SyncTracker to ensure consistent snapshots. **Key Features:** * Multi-threaded: Uses 4 worker threads to copy tables in parallel * Transaction isolation: Captures PostgreSQL snapshots (xmin/xmax/xips) at copy time * Schema preservation: Extracts full table metadata including columns, types, indexes * Sync coordination: Marks tables as "in-flight" via SyncTracker * Replication messaging: Emits TABLE\_SYNC messages via `pg_logical_emit_message()` **Snapshot Capture:** ```cpp theme={null} // Captured for each table copy: struct PgCopyResult { uint64_t target_xid; // Springtail XID assigned to this copy uint32_t pg_xid; // PostgreSQL XID at snapshot time uint32_t xmin, xmax; // Snapshot boundaries uint32_t xmin_epoch, xmax_epoch; std::vector xips; // In-progress transactions std::string pg_xids; // Format: "xmin:xmax:xid,xid,..." } ``` **Copy Process:** 1. Lock table in ACCESS SHARE MODE (prevents schema changes) 2. Capture snapshot via `pg_current_snapshot()` 3. Mark table as in-flight in SyncTracker 4. Execute `COPY table TO STDOUT WITH (FORMAT binary)` 5. Parse binary data and insert into snapshot table 6. Emit `pg_logical_emit_message()` with TABLE\_SYNC\_MSG ### SyncTracker **Location:** `src/pg_log_mgr/sync_tracker.{cc,hh}` Tracks table synchronization state and determines when mutations should be skipped during log replay. Acts as the bridge between PgCopyTable and the replication pipeline. **Five Primary Data Structures:** 1. **`_resync_map`** - Tables where resync was issued but not yet picked up by copy thread 2. **`_resync_picked_map`** - Tables picked for resync at specific XID 3. **`_inflight_map`** - Tables whose COPY is currently in-flight (snapshot metadata stored here) 4. **`_table_map`** - Completed syncs indexed by table\_id (persists for skip logic) 5. **`_sync_map`** - Completed syncs indexed by snapshot PG\_XID (used for commit detection) **State Transition Flow:** ``` issue_resync_and_wait() ↓ (Redis queue) pick_table_for_sync() ↓ (copy thread begins) mark_inflight() → _inflight_map ↓ (TABLE_SYNC_MSG arrives in log) add_sync() → _sync_map + _table_map ↓ (check at commits) check_commit() → SwapRequest when snapshot xid done ↓ clear_tables() → cleanup ``` **Snapshot-based Skip Logic:** The core visibility algorithm in `Snapshot::should_skip()`: ```cpp theme={null} bool should_skip(uint32_t pg_xid) const { // Handle XID wraparound (32-bit XID wraps at 2^32) if (pg_xid < (1 << 26) && _xmax > (1 << 30)) { return false; // pg_xid wrapped, is ahead of xmax } if (_xmax < (1 << 26) && pg_xid > (1 << 30)) { if (_inflight.contains(pg_xid)) { return false; // Was in-flight, don't skip } return true; // xmax wrapped ahead } // No wrapping case if (pg_xid >= _xmax) { return false; // Started after snapshot } if (_inflight.contains(pg_xid)) { return false; // Was in-flight during snapshot } return true; // Committed before snapshot } ``` **Key Principle:** Skip mutations from transactions that committed **before** the table snapshot was taken, since those rows are already in the copied table data. ### PgLogReader **Location:** `src/pg_log_mgr/pg_log_reader.{cc,hh}` Reads replication logs and applies mutations to the write cache. Coordinates with SyncTracker to skip mutations during table syncs. **Skip Logic Integration:** Every mutation (INSERT/UPDATE/DELETE) checks: ```cpp theme={null} auto sync_skip = SyncTracker::get_instance()->should_skip(_db, tid, pg_xid); if (sync_skip.should_skip()) { return; // Don't apply mutation } ``` Lines where skip logic is applied: * Line 229-235: INSERT/UPDATE/DELETE mutations * Line 313-317: TRUNCATE operations * Line 487-491: CREATE\_TABLE/ALTER\_RESYNC DDL * Line 501-505: CREATE\_INDEX/ALTER\_TABLE/DROP\_TABLE DDL **Check-Sync-Commit Logic:** When a `COPY_SYNC` message arrives (line 1193-1197) or during transaction commits (line 1351), `_check_sync_commit()` is called: ```cpp theme={null} void PgLogReader::_check_sync_commit(uint64_t db_id, int32_t pg_xid) { auto swap = SyncTracker::get_instance()->check_commit(db_id, pg_xid); if (swap != nullptr) { // Swap/commit is ready SyncTracker::get_instance()->clear_tables(swap); uint64_t xid = get_next_xid(); // Update system tables and swap table into production for (auto &entry : swap->table_info()) { server->swap_sync_table(...); } // Notify committer _committer_queue->push(xid_msg); } } ``` ## Synchronization Flow ### Full Table Sync Process #### 1. Sync Request Initiation **Trigger Points:** * Startup sync (STATE\_STARTUP\_SYNC) * ALTER TABLE requiring resync * Explicit resync via Redis queue * Table validation changes (ALTER\_RESYNC) **Entry Point:** `PgLogMgr::_copy_thread()` blocks on Redis queue for sync requests ```cpp theme={null} // In pg_log_mgr.cc:_copy_thread() while (!_shutdown) { auto request = _redis_sync_queue->pop_with_timeout(timeout); if (request) { table_oids.insert(request->table_oids.begin(), request->table_oids.end()); } // Batch multiple requests... } ``` #### 2. Pipeline Stall Initiation **Critical Section:** `PgLogMgr::_do_table_copies()` ```cpp theme={null} // Wait for pipeline to enter stall-able state _internal_state.wait_and_set( {STATE_RUNNING, STATE_STARTUP_SYNC, STATE_REPLAYING}, STATE_SYNC_STALL ); // Block commits to prevent GC from removing data we're copying SyncTracker::get_instance()->block_commits(_db_id, _committer_queue); // Push STALL message to logger queue _notify_xact_start_sync(); // Pushes PgLogQueueEntry::Type::STALL // Wait for log reader to acknowledge stall _internal_state.wait_for_state(STATE_SYNCING); ``` **Logger Queue STALL Message:** The logger queue (`_logger_queue`) bridges the writer and reader threads. When a STALL message is pushed: 1. **Writer thread** continues writing replication data but queues it 2. **Reader thread** processes the STALL message in its main loop 3. Reader sets state to `STATE_SYNCING` and blocks 4. Writer receives acknowledgment and begins table copy **Stall Handling in Log Reader:** In the reader thread's processing loop, when a STALL entry is detected: ```cpp theme={null} if (log_entry->type == PgLogQueueEntry::Type::STALL) { assert(_internal_state.is(STATE_SYNC_STALL)); _internal_state.set(STATE_SYNCING); // Acknowledge stall // Wait for table sync to complete while (!_shutdown && !_internal_state.wait_for_state( {STATE_REPLAYING, STATE_RUNNING}, timeout)) { // Blocked until copy completes } _internal_state.set(STATE_RUNNING); // Resume processing } ``` #### 3. Table Copy Execution **Assign Target XID:** ```cpp theme={null} auto xid = _pg_log_reader->get_next_xid(); LOG_INFO("Copying tables; target xid={}", xid); ``` This XID becomes the `target_xid` for all tables in this sync batch. **Execute Copy:** ```cpp theme={null} std::vector res; if (table_oids.empty()) { res = PgCopyTable::copy_db(_db_id, xid); } else { res = PgCopyTable::copy_tables(_db_id, xid, table_oids); } ``` **Worker Thread Process (PgCopyTable):** Each of 4 worker threads: 1. **Pop table from queue** 2. **Connect to PostgreSQL** 3. **Lock table:** `LOCK TABLE schema.table IN ACCESS SHARE MODE` 4. **Capture snapshot:** ```sql theme={null} SELECT pg_current_xact_id(), pg_current_snapshot() ``` Returns: `(pg_xid, "xmin:xmax:xid,xid,...")` 5. **Mark in-flight in SyncTracker:** ```cpp theme={null} SyncTracker::get_instance()->mark_inflight( db_id, table_oid, xid, snapshot_details, schema ); ``` 6. **Create snapshot table:** ```cpp theme={null} auto table = TableMgr::get_instance()->get_snapshot_table( db_id, table_oid, xid.xid, schema, secondary_keys, ... ); ``` 7. **Execute COPY:** ```sql theme={null} COPY schema.table TO STDOUT WITH (FORMAT binary, ENCODING 'UTF-8') -- Or for ordered copy: COPY (SELECT * FROM schema.table ORDER BY pk_col1, pk_col2) TO STDOUT WITH (FORMAT binary, ENCODING 'UTF-8') ``` 8. **Parse binary data:** * Verify header: `"PGCOPY\n\377\r\n\0"` * Read tuples in PostgreSQL binary format * Insert into snapshot table 9. **Emit sync message:** ```cpp theme={null} _send_sync_msg(result); // Generates: std::string query = fmt::format( "SELECT pg_logical_emit_message(false, '{}', '{}');", MSG_PREFIX_COPY_SYNC, R"({"target_xid":, "pg_xid":})" ); ``` This message enters the replication stream and will be processed by PgLogReader. #### 4. Resume Pipeline After all table copies complete: ```cpp theme={null} _internal_state.set(STATE_REPLAYING); _internal_state.wait_for_state(STATE_RUNNING); ``` The log reader thread unblocks and resumes processing queued replication messages. ### How the System is Stalled #### Stall Mechanism Architecture The stall mechanism uses **inter-thread coordination** via state synchronizer and queue messages: **Components:** 1. **StateSynchronizer** - Thread-safe state machine with atomic test-and-set 2. **Logger Queue** - Passes STALL message from writer to reader 3. **Committer Queue** - Receives TABLE\_SYNC\_START to block commits **Detailed Stall Sequence:** ```mermaid theme={null} sequenceDiagram participant Copy as PgLogMgr Copy Thread participant Reader as PgLogMgr Reader Thread participant Committer Note over Copy: 1. Detect sync request (from Redis queue) Note over Copy: 2. State: RUNNING → SYNC_STALL Copy->>Committer: 3. block_commits(): push TABLE_SYNC_START to committer queue Note over Committer: Blocks commits Note over Copy: block_commits() returns immediately Copy->>Reader: 4. Push STALL to logger queue Note over Reader: 5. Pop STALL message Note over Reader: 6. Set state: SYNCING Note over Reader: 7. Block in wait loop:
wait_for_state({REPLAYING, RUNNING}) Note over Copy: 8. Detect state == SYNCING (stall acknowledged) Note over Copy: 9. Execute table copies:
copy_tables(), capture snapshots,
mark_inflight(), emit sync messages Copy->>Reader: 10. Set state: REPLAYING Note over Reader: 11. Detect state change Note over Reader: 12. Exit wait loop Note over Reader: 13. Set state: RUNNING Note over Reader: 14. Resume processing logs ``` **Why This Works:** * **Writer continues writing** - Replication stream isn't disconnected, just queued * **Reader blocks safely** - No partial transaction application * **Snapshots are consistent** - Captured while mutations are queued * **No race conditions** - State transitions are atomic #### Commit Blocking Details When `SyncTracker::block_commits()` is called: ```cpp theme={null} void SyncTracker::block_commits(uint64_t db_id, CommitterQueuePtr committer_queue) { _block_commits(db_id, committer_queue); } void SyncTracker::_block_commits(uint64_t db_id, CommitterQueuePtr committer_queue) { auto msg = std::make_shared( committer::XidReady::Type::TABLE_SYNC_START, db_id ); committer_queue->push(msg); } ``` The committer receives this message and: 1. Stops advancing the committed XID 2. Prevents garbage collection from removing data being copied 3. Resumes when table swap completes ### How PostgreSQL XIDs are Tracked and Used #### XID Architecture Overview Springtail maintains **two parallel XID spaces**: 1. **PostgreSQL XIDs (pg\_xid)** - 32-bit transaction IDs from PostgreSQL * Subject to wraparound at 2^32 * Used for snapshot visibility * Tracked with epoch for 64-bit uniqueness 2. **Springtail XIDs (xid)** - 64-bit global transaction IDs * Monotonically increasing * Never wrap around * Used for internal MVCC and garbage collection **Mapping:** PgLogReader maintains `_xid_ts_tracker` (WalProgressTracker) that maps pg\_xid → Springtail xid. #### XID Assignment Flow **During Normal Operation:** ```cpp theme={null} // In PgLogReader::_process_commit() uint64_t xid = this->get_next_xid(); // Atomic fetch-and-add // get_next_xid() implementation: uint64_t get_next_xid() { return _next_xid.fetch_add(1, std::memory_order_relaxed); } ``` Each committed PostgreSQL transaction receives a Springtail XID sequentially. **During Table Copy:** ```cpp theme={null} // In PgLogMgr::_do_table_copies() auto xid = _pg_log_reader->get_next_xid(); std::vector res = PgCopyTable::copy_db(_db_id, xid); ``` All tables in a sync batch share the **same Springtail XID** (`target_xid`), ensuring atomic visibility. **Result Structure:** ```cpp theme={null} PgCopyResult { target_xid: // Same for all tables in sync pg_xid: // Different per table (snapshot time) xmin: <32-bit> xmax: <32-bit> xips: [<32-bit>, ...] } ``` #### XID Tracking During Sync **Three Critical XIDs:** 1. **Target XID** - Springtail XID assigned before copy starts * Used for snapshot table creation * Used for system table updates * Used for swap operation 2. **Copy PG XID** - PostgreSQL XID when snapshot was taken * Captured via `pg_current_xact_id()` * Stored in `PgCopyResult::pg_xid` * Used in TABLE\_SYNC message 3. **Snapshot XIDs** - xmin/xmax/xips defining visibility * Captured via `pg_current_snapshot()` * Used by `Snapshot::should_skip()` for mutations **XID Flow Through System:** ```mermaid theme={null} sequenceDiagram participant PG as PostgreSQL participant Copy as PgCopyTable participant ST as SyncTracker participant Reader as PgLogReader PG->>Copy: pg_xid = X Note over Copy: Capture snapshot
(target_xid = T, pg_xid = X) Copy->>ST: mark_inflight() Note over ST: Store xmin/xmax, xips, schema Copy->>Reader: Emit sync message {target_xid: T, pg_xid: X} Note over Reader: Process mutations:
if (pg_xid < xmax && !in xips): skip mutation Reader->>ST: add_sync() Note over ST: Move to _sync_map[X] Note over Reader: Check if X committed Reader->>ST: check_commit(X) ST-->>Reader: SwapRequest (if commit seen at X) Note over Reader: Assign new XID S
swap_sync_table(S) Reader->>ST: clear_tables() ``` #### Snapshot Visibility Rules **PostgreSQL Snapshot Format:** `"xmin:xmax:xid1,xid2,..."` Example: `"1000:1010:1002,1005,1008"` * **xmin = 1000** - Oldest transaction still running * **xmax = 1010** - One past highest completed XID * **xips = \[1002, 1005, 1008]** - Transactions in progress between xmin and xmax **Visibility Decision for Mutation with pg\_xid:** ``` if pg_xid >= xmax: # Transaction started after snapshot → Apply mutation (not in table copy) elif pg_xid in xips: # Transaction was in-progress during snapshot → Apply mutation (outcome unknown) else: # Transaction committed before snapshot → Skip mutation (already in table copy) ``` **Wraparound Handling:** PostgreSQL XIDs wrap at 2^32. SyncTracker detects wraparound using threshold detection: ```cpp theme={null} // Detect if pg_xid wrapped ahead of xmax if (pg_xid < (1 << 26) && _xmax > (1 << 30)) { return false; // Assume pg_xid is ahead } // Detect if xmax wrapped ahead of pg_xid if (_xmax < (1 << 26) && pg_xid > (1 << 30)) { if (_inflight.contains(pg_xid)) { return false; // Was in-flight } return true; // Before snapshot } ``` Threshold of 2^26 (≈67 million) provides safe margin for detecting wraps. ### Table Swap and Commit #### When Swap Occurs The swap happens when **all in-progress transactions at snapshot time have committed**. **Check Logic in SyncTracker::check\_commit():** ```cpp theme={null} std::shared_ptr check_commit(uint64_t db_id, uint32_t pg_xid) { std::lock_guard lock(_mutex); // Look up sync records at this pg_xid auto sync_i = _sync_map[db_id].find(pg_xid); if (sync_i == _sync_map[db_id].end()) { return nullptr; // No sync at this pg_xid } // Found a sync that completed at this pg_xid // Collect all tables from this sync std::vector table_info; for (auto &tid_entry : _table_map[db_id]) { if (tid_entry.second->pg_xid() == pg_xid) { table_info.insert(table_info.end(), tid_entry.second->tids().begin(), tid_entry.second->tids().end()); } } return std::make_shared( XidReady::Type::TABLE_SYNC_SWAP, db_id, std::move(table_info) ); } ``` **Key Point:** When PgLogReader processes a commit for transaction X, it calls `check_commit(X)`. If X matches the `pg_xid` from a table sync snapshot, all transactions visible to that snapshot have now committed, so it's safe to swap. #### Swap Process **In PgLogReader::\_check\_sync\_commit():** ```cpp theme={null} auto swap = SyncTracker::get_instance()->check_commit(db_id, pg_xid); if (swap != nullptr) { // Clear from tracker SyncTracker::get_instance()->clear_tables(swap); // Assign new Springtail XID for swap operation uint64_t xid = get_next_xid(); // For each table in the swap for (auto &entry : swap->table_info()) { _exists_cache->insert(db_id, entry->table_id, true); auto copy_info = entry->info; // Set XIDs for system table updates copy_info->mutable_namespace_req()->set_xid(xid); copy_info->mutable_namespace_req()->set_lsn(RESYNC_NAMESPACE_LSN); copy_info->mutable_table_req()->set_xid(xid); copy_info->mutable_table_req()->set_lsn(RESYNC_CREATE_LSN); // Perform atomic swap auto ddl_str = server->swap_sync_table( *namespace_req, *create_req, indexes_vec, *roots_req ); // Queue for FDW notification RedisDDL::get_instance()->add_ddl(_db_id, xid, ddl_str); } // Notify committer auto xid_msg = std::make_shared( swap->type(), swap->db(), _pg_log_timestamp, committer::XidReady::SwapMsg(xid, ddls, table_ids) ); _committer_queue->push(xid_msg); } ``` **System Table Updates:** The `swap_sync_table()` operation: 1. Updates `springtail.tables` with new table root pointer 2. Updates `springtail.namespaces` if needed 3. Creates index entries in `springtail.indexes` 4. Invalidates client caches 5. Returns DDL statements for FDW propagation **Atomicity:** The swap is atomic from the perspective of readers - they either see the old table or the new table, never partial state. ## Recovery and Error Handling ### Log Recovery on Startup **Entry Point:** `PgLogMgr::startup()` ```cpp theme={null} if (do_init) { // Fresh initialization _startup_init(); _wal_buffer_flag = true; _start_threads(do_init, lsn); } else { // Recovery mode PgLogRecovery recovery(_db_id, _repl_log_path, _pg_log_reader, _committer_queue, _index_requests_mgr); lsn = recovery.repair_logs(); _start_threads(do_init, lsn); recovery.replay_logs(); // Signal committer for index recovery _wal_buffer_flag = false; } ``` #### Recovery Phases **Phase 1: Repair Logs** Scans replication logs to find last valid committed LSN: ```cpp theme={null} uint64_t PgLogRecovery::repair_logs() { // Scan log files backward // Find last committed transaction // Return LSN to resume from } ``` **Phase 2: Replay Logs** Four-step replay process: ```cpp theme={null} void PgLogRecovery::replay_logs() { _revert_system_tables(); // Step 1: Revert to last committed XID _skip_committed(); // Step 2: Skip already-processed records _replay_active(); // Step 3: Replay incomplete transactions _replay_uncommitted(); // Step 4: Replay post-commit records } ``` **Step 1 - Revert System Tables:** * Query `xid_mgr` for last committed XID * Revert `springtail.tables`, `springtail.namespaces`, etc. to that XID * Ensures system catalog consistency **Step 2 - Skip Committed:** * Read log from last committed LSN * Skip transactions already committed before crash * Prevents duplicate application **Step 3 - Replay Active:** * Replay transactions that were in-progress at crash time * Use snapshot visibility to skip invalid mutations **Step 4 - Replay Uncommitted:** * Process messages after last committed transaction * Re-apply schema changes and mutations * Rebuild write cache state ### Handling Copy Failures **Worker Thread Error Handling:** ```cpp theme={null} // In PgCopyTable::_worker() try { auto info = copy_table._copy_table(db_id, xid, request->table_oid, ...); copy_result->tids.push_back(info); } catch (PgRetryError &e) { // Transient error - re-queue table copy_queue->push(request); } catch (PgTableNotFoundError &e) { // Table dropped between discovery and copy LOG_ERROR("Table not found: oid {}", request->table_oid); // Continue with other tables } catch (PgQueryError &e) { e.log_backtrace(); CHECK(false); // Fatal error } ``` **Retry Logic:** * Transient errors (connection loss, deadlock) → Re-queue table * Table not found → Log error, mark as dropped, continue * Other errors → Fatal, stop process **Table Dropped During Sync:** If a table is dropped after copy starts but before swap: ```cpp theme={null} // In PgLogReader::_check_sync_commit() if (copy_info->is_table_dropped()) { // Generate DROP TABLE DDL instead of swap PgMsgDropTable drop_msg; drop_msg.oid = table_info.id(); drop_msg.namespace_name = table_info.namespace_name(); drop_msg.table = table_info.name(); std::string ddl_stmt = server->drop_table(_db_id, XidLsn{xid}, drop_msg); ddls.emplace_back(ddl_stmt); } ``` ### XID Consistency Across Restarts **XID Manager Persistence:** The XidMgr tracks committed XIDs to persistent storage. On restart: ```cpp theme={null} // In PgLogReader constructor _committed_xid = xid_mgr::XidMgrServer::get_instance()->get_committed_xid(db_id, 0); // In PgLogMgr::startup() uint64_t committed_xid = xid_mgr->get_committed_xid(_db_id, 0); uint64_t next_xid = committed_xid + 2; // Skip one for cleanup _pg_log_reader->set_next_xid(next_xid); ``` **Duplicate Prevention:** In commit processing: ```cpp theme={null} if (xid <= _committed_xid) { // Already processed - abort this batch _current_batch->abort(md.pg_commit_ts); _xid_ts_tracker->remove_pg_xid(_current_xact->xid); } else { // New transaction - process normally _current_batch->commit(xid, md); } ``` This prevents double-application of transactions during recovery. ## Message Flow Diagram ### Full Sync Lifecycle ```mermaid theme={null} sequenceDiagram participant Copy as Copy Thread participant LQ as Logger Queue participant Reader as Reader Thread participant ST as SyncTracker participant PG as PostgreSQL Note over Copy: T0 Wait on Redis sync request Note over Copy: T1 Receive request, table_oids=[100] Note over Copy: T2 State: RUNNING → SYNC_STALL Copy->>ST: T3 block_commits() Note over ST: _table_map[100] = resync (lock table) Copy->>LQ: T4 Push STALL LQ->>Reader: T5 Pop STALL Note over Reader: State: SYNCING, enter wait loop Note over Copy: T6 Detect SYNCING Copy->>PG: T7 copy_tables(xid=5000) PG-->>Copy: pg_current_xact_id()=2000
snapshot "1990:2000:1995" Copy->>ST: T8 mark_inflight(table=100, xid=5000,
xmin=1990, xmax=2000, xips=[1995]) Note over ST: _inflight_map[100] = Inflight{2000, 2000, [1995]} Copy->>PG: T9 COPY table TO STDOUT PG-->>Copy: returns 1M rows Note over Copy: T10 Worker inserts to snapshot_table Copy->>PG: T11 pg_logical_emit_message('table_sync',
{target_xid:5000, pg_xid:2000}) Note over Copy: T12 All workers complete Note over Copy: T13 State: SYNCING → REPLAYING Note over Reader: T14 Detect REPLAYING, exit wait loop,
State: RUNNING, resume processing Note over Reader: T15 Process queued replication logs LQ->>Reader: T16 [BEGIN 1996] Note over Reader: _process_begin(1996), create batch LQ->>Reader: T17 [INSERT t100] pg_xid=1996 Note over Reader: should_skip? 1996 < 2000 ✓,
1996 in [1995]? ✗ → SKIP LQ->>Reader: T18 [COMMIT 1996] Note over Reader: check_commit(1996)? → No sync at 1996 LQ->>Reader: T19 [BEGIN 1995] LQ->>Reader: T20 [INSERT t100] pg_xid=1995 Note over Reader: should_skip? 1995 < 2000 ✓,
1995 in [1995]? ✓ → APPLY LQ->>Reader: T21 [COMMIT 1995] Note over Reader: check_commit(1995)? → No sync at 1995 LQ->>Reader: T22 [BEGIN 2001] LQ->>Reader: T23 [INSERT t100] pg_xid=2001 Note over Reader: should_skip? 2001 >= 2000 ✓ → APPLY LQ->>Reader: T24 [TABLE_SYNC] (msg from T11) Reader->>ST: add_sync(table=100, pg_xid=2000,
xmin=1990, xmax=2000, xips=[1995]) Note over ST: _inflight_map.erase(100)
_sync_map[2000] = XidRecord
_table_map[100] = XidRecord LQ->>Reader: T25 [COMMIT 2000] Reader->>ST: check_commit(2000)? ST-->>Reader: YES! Found sync at pg_xid=2000 Reader->>ST: clear_tables() Note over ST: _sync_map.erase(2000) Note over Reader: xid=5001, swap_sync_table(5001) Reader->>PG: Notify committer → SWAP table 100@5001 Note over PG: T26 Table 100 now visible to queries ``` ## Performance Considerations ### Parallel Table Copying * **4 worker threads** copy tables concurrently * Each worker maintains independent PostgreSQL connection * Results aggregated when all workers complete * Thread-safe queue coordinates work distribution ### Binary COPY Protocol * Efficient binary data transfer (vs. text COPY) * No serialization overhead * Direct field parsing into internal format * Typical throughput: 100K+ rows/second per table ### Memory Management * **Write cache batching:** 4MB extent size before flush * **Snapshot tables:** Stored in mutable B-trees, not memory * **Queue flow control:** Memory/file hybrid mode based on watermarks * **Log archiving:** Old logs cleaned based on min active timestamp ### Stall Duration Minimization The pipeline stall only occurs during: 1. Snapshot capture (\~milliseconds) 2. Table metadata extraction (\~seconds) Bulk data copying happens **after** pipeline resumes, so replication lag is minimal. ## Configuration ### Key Constants **From pg\_log\_mgr.hh:** ```cpp theme={null} LOG_ROLLOVER_SIZE_BYTES = 128 * 1024 * 1024 // 128MB log files QUEUE_SIZE = 256 * 1024 // Logger queue size FSYNC_LOOP_INTERVAL_MS = 500 // Log fsync frequency ``` **From pg\_copy\_table.hh:** ```cpp theme={null} WORKER_THREADS = 4 // Worker thread count ``` **From pg\_log\_reader.hh:** ```cpp theme={null} MAX_BATCH_SIZE = 4 * 1024 * 1024 // 4MB write-cache batch flush threshold ``` ### Tuning Recommendations 1. **Increase worker threads** for databases with many small tables 2. **Decrease batch size** if memory pressure is high 3. **Increase queue size** if replication lag spikes during sync 4. **Enable log archiving** for regulatory compliance *** ## Summary Springtail's table synchronization architecture achieves **zero downtime** and **consistency** through: 1. **Snapshot isolation** - Captures PostgreSQL snapshots to establish visibility boundaries 2. **Pipeline coordination** - Stalls log processing during snapshot capture only 3. **Skip-based replay** - Uses PostgreSQL XID visibility rules to skip redundant mutations 4. **Atomic swap** - Swaps tables when all in-flight transactions commit 5. **Recovery support** - Replays logs correctly after crashes 6. **Parallel execution** - Copies multiple tables concurrently for performance The system ensures that: * No mutations are lost during table sync * No mutations are double-applied * Table data is consistent with a specific point in the replication stream * The system can recover from failures at any stage This architecture enables Springtail to maintain real-time replication while performing bulk table synchronization operations. # System Metadata Source: https://docs.springtail.io/system-metadata # Schema Management Architecture ## Overview The Springtail schema management system provides versioned metadata tracking for all database objects including tables, schemas, indexes, and user-defined types. The architecture is designed to support: * **Multi-Version Concurrency Control (MVCC)**: Every metadata entry is versioned by transaction ID (XID) and Log Sequence Number (LSN) * **Multi-Process Caching**: Shared memory caches enable efficient metadata sharing across processes * **DDL Evolution Tracking**: Complete history of schema changes over time * **Client-Server Architecture**: Centralized management via gRPC with distributed caching The system consists of three primary components: 1. **System Tables**: Persistent storage of metadata on disk 2. **Caching Layers**: Multi-level caches (in-process and shared memory) 3. **System Table Manager**: Service layer managing reads, writes, and synchronization *** ## System Tables System tables are special metadata tables that track information about user tables, schemas, and indexes. They are stored on disk and managed by the storage engine like regular tables, but contain metadata rather than user data. ### The Eight System Tables Defined in `include/sys_tbl_mgr/system_tables.hh`: #### 1. TableNames (ID: 1) Tracks all tables in the database. **Data Columns:** * `namespace_id` - Schema/namespace ID * `name` - Table name * `table_id` - Unique table identifier * `xid`, `lsn` - Transaction version * `exists` - Deletion marker (soft delete) * `parent_table_id` - For partitioned tables * `partition_key` - Partition key expression * `partition_bound` - Partition bounds * `rls_enabled` - Row-level security flag * `rls_forced` - Force RLS for table owner * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, xid, lsn)` **Secondary Index:** `(namespace_id, name, xid, lsn)` - For lookup by qualified name #### 2. TableRoots (ID: 2) Stores B-tree root extent IDs and table statistics at each XID. **Data Columns:** * `table_id` - Table identifier * `index_id` - Index identifier (0 = primary) * `xid` - Transaction version * `extent_id` - Root extent ID of the B-tree * `snapshot_xid` - Snapshot identifier for consistency * `end_offset` - Data file offset after sync * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, index_id, xid)` **Purpose:** Enables accessing a table's B-tree root at any historical XID. #### 3. Indexes (ID: 3) Maps which columns participate in which indexes. **Data Columns:** * `table_id` - Table identifier * `index_id` - Index identifier * `xid`, `lsn` - Transaction version * `position` - Position in index (0-based) * `column_id` - Column position in table * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, index_id, xid, lsn, position)` **Purpose:** Multiple rows per index define the column ordering. #### 4. Schemas (ID: 4) Column definitions for all tables. **Data Columns:** * `table_id` - Table identifier * `position` - Column position (can have gaps) * `xid`, `lsn` - Transaction version * `exists` - Column active/dropped flag * `name` - Column name * `type` - Springtail internal type (SchemaType) * `pg_type` - PostgreSQL type OID * `nullable` - NULL constraint * `default` - Default value expression * `update_type` - Type of change (ADD, DROP, MODIFY) * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, position, xid, lsn)` **Purpose:** Tracks column history, allowing schema evolution tracking. #### 5. TableStats (ID: 5) Table statistics at each XID. **Data Columns:** * `table_id` - Table identifier * `xid` - Transaction version * `row_count` - Number of rows * `last_internal_row_id` - Last assigned row ID * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, xid)` **Purpose:** Used for query planning and optimization. #### 6. IndexNames (ID: 6) Secondary index metadata and lifecycle state. **Data Columns:** * `table_id` - Table identifier * `index_id` - Index identifier * `xid`, `lsn` - Transaction version * `namespace_id` - Schema/namespace ID * `name` - Index name * `state` - Lifecycle state (NOT\_READY, READY, DELETED, BEING\_DELETED) * `is_unique` - Uniqueness constraint * `internal_row_id` - Internal identifier **Primary Index:** `(table_id, index_id, xid, lsn)` **Purpose:** Tracks index build status and metadata. #### 7. NamespaceNames (ID: 7) Database schemas/namespaces from PostgreSQL. **Data Columns:** * `namespace_id` - Unique namespace identifier * `name` - Namespace name (e.g., "public") * `xid`, `lsn` - Transaction version * `exists` - Deletion marker * `internal_row_id` - Internal identifier **Primary Index:** `(namespace_id, xid, lsn)` **Secondary Index:** `(name, xid, lsn)` - For lookup by name #### 8. UserTypes (ID: 8) User-defined types (primarily enums). **Data Columns:** * `type_id` - Type identifier * `namespace_id` - Schema containing the type * `name` - Type name * `value` - JSON-encoded type values * `xid`, `lsn` - Transaction version * `type` - Type category ('E' for enum) * `exists` - Deletion marker * `internal_row_id` - Internal identifier **Primary Index:** `(type_id, xid, lsn)` **Purpose:** Stores enum values and other user-defined type information. ### Key Concepts **XID/LSN Versioning:** * Every metadata entry is stamped with `(xid, lsn)` pair * Enables querying metadata at any historical transaction * Critical for MVCC and consistency **Soft Deletes:** * The `exists` flag marks objects as deleted without physical removal * Allows historical queries to see deleted objects at past XIDs * Physical cleanup can happen during maintenance **Snapshot XID:** * Stored in TableRoots and TableNames for partitioned tables * Represents the XID at which table data was synchronized * Used for schema version matching during data reads *** ## Schema Information Management Schema information describes the structure of a table (columns, types, indexes) at a specific point in time. ### SchemaMetadata Structure Defined in `include/storage/schema.hh`: ```cpp theme={null} struct SchemaMetadata { XidRange access_range; // XID range where this schema is valid XidRange target_range; // XID range for change history std::vector columns; // Current column definitions std::vector history; // Historical changes std::vector indexes; // Index definitions }; ``` ### SchemaColumn Each column is described by: * **Identity**: `name`, `position`, `xid`, `lsn` * **Type Information**: `type` (internal), `pg_type` (PostgreSQL OID) * **Constraints**: `nullable`, `pkey_position`, `default_value` * **Type Metadata**: `type_name`, `type_namespace`, `collation`, `type_category` * **Lifecycle**: `exists`, `update_type` ### Index Structure ```cpp theme={null} struct Index { uint64_t id; // Index identifier std::string schema; // Schema name std::string name; // Index name uint64_t table_id; // Table identifier bool is_unique; // Uniqueness constraint uint8_t state; // Lifecycle state std::vector columns; // Index column mappings }; ``` Each `Index::Column` contains: * `idx_position` - Position in index (0-based) * `position` - Column position in table ### Schema Change Tracking The `update_type` field tracks the nature of each change: * `NEW_COLUMN` - Column added * `REMOVE_COLUMN` - Column dropped * `NAME_CHANGE` - Column renamed * `NULLABLE_CHANGE` - NULL constraint changed * `RESYNC` - Table resynchronized * `NEW_INDEX` - Index added * `DROP_INDEX` - Index dropped * `NO_CHANGE` - No modification **History Tracking:** * The `history` vector in `SchemaMetadata` tracks all changes in XID order * Each entry represents a schema modification event * Enables replaying schema evolution for synchronization *** ## Caching Architecture The system employs a multi-level caching strategy to minimize disk I/O and RPC overhead. ### 1. Shared Memory Cache (ShmCache) **Location:** `include/sys_tbl_mgr/shm_cache.hh`, `src/sys_tbl_mgr/shm_cache.cc` **Purpose:** Cross-process caching of serialized metadata using Boost interprocess shared memory. #### Architecture **Technology Stack:** * `boost::interprocess::managed_shared_memory` - Shared memory segment * `boost::interprocess::named_sharable_mutex` - Cross-process locking * `boost::multi_index_container` - LRU eviction tracking **Five Global Cache Instances:** 1. `SHM_CACHE_ROOTS` (`"springtail.roots"`) - Table roots and statistics 2. `SHM_CACHE_SCHEMAS` (`"springtail.schemas"`) - Schema metadata 3. `SHM_CACHE_USERTYPES` (`"springtail.usertypes"`) - User-defined types 4. `SHM_CACHE_TABLE_IDS` (`"springtail.table_ids"`) - Table ID lookups 5. `SHM_CACHE_EXTENTS` (`"springtail.extents"`) - Extent metadata #### Cache Data Structure ``` Key: (DbId, ObjId) Value: vector sorted by XID/LSN Message { XidLsn xid; vector serialized_data; // Protobuf message bool dropped; // Deletion marker } ``` **Key Features:** 1. **Serialized Storage**: Stores protobuf-serialized messages for portability 2. **XID Versioning**: Multiple versions per object, sorted by XID/LSN 3. **LRU Eviction**: Automatically evicts least-recently-used entries when memory fills 4. **Dropped Markers**: Can mark objects as dropped without removal 5. **Memory Management**: Auto-evicts when free memory falls below 30% (target: 50%) #### XID Lifecycle Management **Committed XID Tracking:** ```cpp theme={null} void update_committed_xid(DbId db, Xid xid, bool has_schema_changes, bool real_commit); std::optional get_committed_xid(DbId db, Xid schema_xid); ``` * Tracks the last committed XID per database * Records whether schema changes occurred * Maintains XID history to prevent accessing stale schemas **Keep-Alive Mechanism:** ```cpp theme={null} static constexpr std::chrono::duration XID_KEEP_ALIVE_PERIOD = 60ms; void keep_alive(); bool is_alive(); ``` * Must call `keep_alive()` or `update_committed_xid()` every 60ms * Prevents using stale committed XIDs from crashed processes * Timestamp-based freshness checking **Pending XID Tracking:** ```cpp theme={null} std::vector get_pending_xids(DbId db, Xid last_committed_xid); void reset_pending_xids(DbId db); ``` * Tracks XIDs that have modified metadata but haven't yet committed * Used during crash recovery and consistency checks #### XID History Optionally tracks schema change history: ```cpp theme={null} struct XidHistoryEntry { Xid schema_xid; // XID where schema changed Xid latest_real_commit_xid; // Last real commit before this change }; ``` * Survives `finalize()` to track schema evolution * Used to find appropriate committed XID for a given schema XID * Cleaned up via `cleanup_xid_history()` ### 2. In-Process Schema Cache (SchemaCache) **Location:** `include/sys_tbl_mgr/schema_cache.hh`, `src/sys_tbl_mgr/schema_cache.cc` **Purpose:** In-process LRU cache of constructed `SchemaMetadata` objects (not serialized). #### Architecture **Cache Entry:** ```cpp theme={null} struct SchemaEntry { XidLsn start_xid; // When this schema version became valid SchemaMetadataPtr schema; // Constructed schema object bool fetching; // Currently being populated bool invalidated; // Marked invalid by DDL std::condition_variable cond; // Coordination for concurrent fetches }; ``` **Key:** `(db_id, table_id)` **Capacity:** Configured via the `cache_size` setting (4096 in the production configuration) #### Features 1. **LRU Eviction**: Removes least-recently-used schemas when capacity exceeded 2. **Lazy Loading**: Populates on demand via callback function 3. **Invalidation**: Marks schemas invalid when DDL detected 4. **Index Mapping**: Tracks `(db_id, index_id) -> table_id` for drop-index invalidation 5. **Concurrent Fetch Coordination**: Uses condition variables to prevent duplicate fetches #### Invalidation Strategies **Table Invalidation:** ```cpp theme={null} void invalidate_table(uint64_t db, uint64_t tid, const XidLsn &xid); ``` * Marks the schema entry as ending at the provided XID * Future accesses beyond this XID will trigger re-fetch **Index Invalidation:** ```cpp theme={null} void invalidate_by_index(uint64_t db, uint64_t index_id, const XidLsn &xid); ``` * Uses index-to-table mapping to find affected table * Invalidates table schema **Database Invalidation:** ```cpp theme={null} void invalidate_db(uint64_t db, const XidLsn &xid); ``` * Invalidates all tables in the database * Called when DDL changes detected at FDW level ### 3. Generic Message Cache (MsgCache) **Location:** `include/sys_tbl_mgr/msg_cache.hh` **Purpose:** Template-based foundation for ShmCache, providing generic serialized message caching. **Design Pattern:** * Uses traits-based design for customization * Supports any allocator (regular or shared memory) * LRU eviction via Boost multi-index container * Thread-safe via template mutex parameter ### Caching Strategy Summary | Cache Level | Scope | Data Format | Eviction | Use Case | | ----------------------------- | -------------- | ------------------- | ---------------------- | ---------------------------------- | | **ShmCache** | Cross-process | Protobuf serialized | LRU + memory threshold | Share metadata between FDW workers | | **SchemaCache** | Single process | Constructed objects | LRU (128 entries) | Fast in-memory access to schemas | | **Server Uncommitted Caches** | Server only | Native structures | Manual (on finalize) | Track pending DDL changes | **Data Flow:** 1. Client checks in-process SchemaCache 2. On miss, checks ShmCache 3. On miss, RPCs to server 4. Server checks uncommitted caches, then disk 5. Response propagates back: Server → SHM → SchemaCache → Client *** ## System Table Manager The `sys_tbl_mgr` is a gRPC service that manages reading and writing system tables. ### Architecture - Two Modes #### Server-Side (`sys_tbl_mgr::Server`) **Location:** `include/sys_tbl_mgr/server.hh`, `src/sys_tbl_mgr/server.cc` **Purpose:** Manages read/write access to system tables within the server process. **Key Responsibilities:** 1. Handle CREATE/ALTER/DROP table/index/namespace/type operations 2. Maintain uncommitted caches for pending transactions 3. Persist system tables to disk on finalize 4. Manage XID progression and synchronization 5. Serve gRPC requests from clients #### Client-Side (`sys_tbl_mgr::Client`) **Location:** `include/sys_tbl_mgr/client.hh`, `src/sys_tbl_mgr/client.cc` **Purpose:** Remote access to system tables via gRPC from FDW processes. **Key Responsibilities:** 1. Proxy read requests to server 2. Maintain local caches (SchemaCache + SHM caches) 3. Handle invalidation notifications 4. Coordinate with multiple worker processes ### gRPC Service Definition **Location:** `src/proto/sys_tbl_mgr.proto` ```protobuf theme={null} service SysTblMgr { rpc Ping() returns (Empty); rpc GetRoots(GetRootsRequest) returns (GetRootsResponse); rpc GetSchema(GetSchemaRequest) returns (GetSchemaResponse); rpc GetTargetSchema(GetTargetSchemaRequest) returns (GetSchemaResponse); rpc Exists(ExistsRequest) returns (Empty); rpc GetUserType(GetUserTypeRequest) returns (GetUserTypeResponse); } ``` **RPC Methods:** * `Ping`: Health check * `GetRoots`: Fetch table roots and stats at XID * `GetSchema`: Fetch table schema at XID * `GetTargetSchema`: Fetch schema with change history between XIDs * `Exists`: Check if table exists at XID * `GetUserType`: Fetch user-defined type at XID ### Server-Side Uncommitted Caches The server maintains multiple in-memory caches for uncommitted DDL changes: #### 1. Table Cache (`_table_cache`) ```cpp theme={null} struct TableCacheRecord { uint64_t id, xid, lsn, namespace_id; std::string name; bool rls_enabled, rls_forced, exists; std::optional parent_table_id; std::optional partition_key, partition_bound; }; ``` **Map:** `DB → Table ID → XID/LSN → TableCacheRecord` **Purpose:** Track table metadata during CREATE/ALTER TABLE before commit. #### 2. Roots Cache (`_roots_cache`) ```cpp theme={null} using RootsCacheRecord = std::shared_ptr; ``` **Map:** `DB → Table ID → XID/LSN → RootsCacheRecord` **Purpose:** Track table roots and statistics during data synchronization. #### 3. Schema Cache (`_schema_cache`) ```cpp theme={null} using ColumnIdToInfoMap = std::map>; ``` **Map:** `DB → Table ID → Column ID → vector` **Purpose:** Track column additions, drops, and modifications during ALTER TABLE. #### 4. Index Cache (`_index_cache`) ```cpp theme={null} struct IndexCacheItem { XidLsn xid; proto::IndexInfo info; }; ``` **Map:** `DB → Table ID → Index ID → vector` **Purpose:** Track index definitions during CREATE/DROP INDEX. #### 5. Namespace Caches * `_namespace_id_cache`: Map `DB → Namespace ID → XID/LSN → NamespaceRecord` * `_namespace_name_cache`: Map `DB → Namespace Name → XID/LSN → NamespaceRecord` **Purpose:** Track namespace changes during CREATE/ALTER/DROP SCHEMA. #### 6. User Type Cache (`_usertype_id_cache`) **Map:** `DB → Type ID → XID/LSN → UserTypeRecord` **Purpose:** Track user-defined type changes during CREATE/ALTER/DROP TYPE. #### 7. Table Existence Cache (`_table_existence_cache`) **Special Property:** **Persists across `finalize()` calls** ```cpp theme={null} struct TableExistenceRange { XidLsn start_xid_lsn; // First XID where table exists (inclusive) XidLsn end_xid_lsn; // First XID where table no longer exists (exclusive) }; ``` **Map:** `DB → Table ID → vector` **Purpose:** * Track table lifecycle across drops and recreates (resync operations) * Survives finalize to support historical queries * Protected by dedicated `_table_existence_cache_mutex` ### Server Lifecycle Operations #### DDL Operations ```cpp theme={null} std::string create_table(uint64_t db_id, const XidLsn &xid, const PgMsgTable &msg); std::string alter_table(uint64_t db_id, const XidLsn &xid, const PgMsgTable &msg); std::string drop_table(uint64_t db_id, const XidLsn &xid, const PgMsgDropTable &msg); proto::IndexProcessRequest create_index(...); proto::IndexProcessRequest drop_index(...); void set_index_state(...); std::string create_namespace(...); std::string create_usertype(...); ``` **Behavior:** * Populate uncommitted caches * Do NOT write to disk * Return DDL JSON for DDL manager #### Synchronization Operations ```cpp theme={null} void update_roots(uint64_t db_id, uint64_t table_id, uint64_t xid, const TableMetadata &metadata); ``` **Purpose:** Record table roots and stats after data sync. #### Transaction Finalization ```cpp theme={null} void finalize(uint64_t db_id, uint64_t xid, bool call_sync); void sync(uint64_t db_id, uint64_t xid); ``` **Actions:** 1. Write all uncommitted caches to system tables on disk 2. Flush system tables to disk (if `call_sync=true` or via separate `sync()`) 3. Update committed XID in SHM caches with schema change flags 4. Clear uncommitted caches (except `_table_existence_cache`) **Note:** `finalize()` can be called without `sync()` for async persistence. #### Transaction Abort ```cpp theme={null} void revert(uint64_t db_id, uint64_t xid); ``` **Actions:** 1. Discard all uncommitted changes for the given XID 2. Clear entries from all uncommitted caches 3. No disk writes #### Cache Invalidation ```cpp theme={null} void invalidate_table(uint64_t db_id, uint64_t table_id, const XidLsn &xid); void invalidate_db(uint64_t db_id, const XidLsn &xid); ``` **Purpose:** Propagate DDL changes to in-process SchemaCache. ### Client-Side Operations The `Client` singleton provides a simplified interface: ```cpp theme={null} TableMetadataPtr get_roots(uint64_t db_id, uint64_t table_id, uint64_t xid); std::shared_ptr get_schema(uint64_t db_id, uint64_t table_id, const XidLsn &xid); SchemaMetadataPtr get_target_schema(uint64_t db_id, uint64_t table_id, const XidLsn &access_xid, const XidLsn &target_xid); bool exists(uint64_t db_id, uint64_t table_id, const XidLsn &xid); std::shared_ptr get_usertype(uint64_t db_id, uint64_t type_id, const XidLsn &xid); ``` **Client Caching Strategy:** 1. Check in-process SchemaCache 2. On miss, check SHM cache 3. On miss, issue gRPC call to server 4. Cache response in both SHM and SchemaCache 5. Return to caller **Cache Registration:** ```cpp theme={null} void use_roots_cache(std::shared_ptr c); void use_schema_cache(std::shared_ptr c); void use_usertype_cache(std::shared_ptr c); ``` Allows client to opt-in to specific SHM caches. *** ## Data Flow ### Schema Read Flow (FDW → Server) ```mermaid theme={null} flowchart TD FDW["FDW (PostgreSQL Foreign Data Wrapper)"] Client1["TableMgrClient.get_schema(db_id, table_id, xid)"] SC{"SchemaCache.get()
(in-process)"} SHM{"Client.get_schema()
check SHM cache"} GRPC["gRPC: GetSchemaRequest"] Svc["sys_tbl_mgr::Service
(server-side gRPC handler)"] Srv["Server.get_schema()
read lock, check uncommitted caches"] Disk["Read system tables
Schemas / Indexes / IndexNames"] Build["Construct SchemaMetadata
(columns, indexes, access_range)"] Ser["Serialize to proto::GetSchemaResponse"] Ret["Client stores in SHM + SchemaCache,
returns SchemaMetadataPtr"] FDW --> Client1 --> SC SC -->|hit| Ret SC -->|miss| SHM SHM -->|"hit (deserialize)"| Ret SHM -->|miss| GRPC --> Svc --> Srv Srv -->|hit in uncommitted| Ser Srv -->|miss| Disk --> Build --> Ser --> Ret ``` ### Table Roots Read Flow ```mermaid theme={null} flowchart TD FDW["FDW"] C["TableMgrClient.get_roots(db_id, table_id, xid)"] SHM{"Check SHM cache
(springtail.roots)"} GRPC["gRPC: GetRootsRequest"] Srv["Server.get_roots()"] UC{"Check _roots_cache
(uncommitted)"} RR["Read TableRoots system table"] RS["Read TableStats system table"] Build["Construct TableMetadata"] Ser["Serialize to proto::GetRootsResponse"] Ret["Client stores in SHM cache,
returns TableMetadataPtr"] FDW --> C --> SHM SHM -->|hit| Ret SHM -->|miss| GRPC --> Srv --> UC UC -->|hit| Ser UC -->|miss| RR --> RS --> Build --> Ser --> Ret ``` ### DDL Write Flow (CREATE TABLE) ```mermaid theme={null} flowchart TD ET["PostgreSQL Event Trigger
captures CREATE TABLE, sends PgMsgTable"] CT["Server.create_table(db_id, xid, msg)
write lock, assign table_id"] subgraph UnCache [Populate uncommitted caches] direction TB TC["_table_cache[db][table_id][xid]"] SCH["_schema_cache[db][table_id][col_id]"] IDX["_index_cache[db][table_id][PRIMARY_INDEX]"] end DDL["Return DDL JSON for DDL manager"] FIN["Server.finalize(db_id, xid, call_sync=true)
unique write lock"] subgraph Persist [Write to system tables] direction TB TN["TableNames"] SchT["Schemas"] IdxT["Indexes"] InT["IndexNames"] end Sync["SystemTable.sync() per modified table"] Upd["ShmCache.update_committed_xid(has_schema_changes=true)"] Clr["Clear uncommitted caches
(except _table_existence_cache)"] ET --> CT --> UnCache --> DDL --> FIN --> Persist --> Sync --> Upd --> Clr ``` ### Cache Invalidation Flow When DDL changes occur, caches must be invalidated: ```mermaid theme={null} flowchart TD DDL["DDL operation committed
(e.g. ALTER TABLE ADD COLUMN)"] Notify["DDL manager notifies FDW"] Detect["FDW detects schema change at new XID"] Inv["Client.invalidate_table(db_id, table_id, xid)"] SCInv["SchemaCache.invalidate_table(db, tid, xid)
marks entry ending at xid; next access refetches"] SHM["SHM cache already updated by Server during finalize
next get_schema(xid > old) hits SHM or server"] DDL --> Notify --> Detect --> Inv --> SCInv --> SHM ``` **Key Points:** * Server updates SHM cache during `finalize()` with new schema * Clients invalidate local SchemaCache when notified * Next access automatically fetches new schema from SHM or server * Multi-process coordination via SHM ensures consistency *** ## Key Design Patterns ### 1. Two-Level Client Caching **Pattern:** Client maintains both SHM cache (cross-process, serialized) and SchemaCache (in-process, deserialized). **Benefits:** * SHM cache enables sharing between FDW worker processes * SchemaCache provides fast in-memory access without deserialization * Reduces RPC overhead significantly **Trade-off:** Memory overhead for duplicate storage, mitigated by LRU eviction. ### 2. Uncommitted vs. Committed Data Separation **Pattern:** Server maintains separate caches for uncommitted DDL changes. **Benefits:** * Allows querying pending changes without disk I/O * Clean separation between in-flight and committed metadata * Enables atomic commit via `finalize()` **Implementation:** * Uncommitted: `_table_cache`, `_schema_cache`, `_index_cache`, etc. * Committed: System tables on disk, SHM caches * Cleared on `finalize()` or `revert()` ### 3. XID/LSN Versioning **Pattern:** Every metadata entry tagged with `(xid, lsn)`. **Benefits:** * Enables querying schema at any historical point (MVCC) * Supports concurrent transactions without locking * Critical for consistency in distributed system **Example:** Reading table at XID 100 returns schema as of XID 100, even if XID 150 has altered it. ### 4. LRU Eviction with XID History Preservation **Pattern:** SHM cache uses LRU for memory management but preserves XID history. **Benefits:** * Prevents memory exhaustion * Retains critical XID commit information * Allows detecting dropped objects without full history **Implementation:** * Message data evicted via LRU * XID history retained via `_xid_history_map` * `keep_alive()` ensures timestamp freshness ### 5. Lazy Evaluation **Pattern:** SchemaCache populates on-demand via callback. **Benefits:** * Defers expensive construction until needed * Allows server to control fetch logic * Reduces memory footprint for unused schemas **Implementation:** ```cpp theme={null} SchemaMetadataPtr get(uint64_t db, uint64_t tid, const XidLsn &xid, PopulateFn populate); ``` Callback invoked only on cache miss. ### 6. Persistent Table Existence Cache **Pattern:** `_table_existence_cache` survives `finalize()` calls. **Benefits:** * Supports resync operations (drop and recreate table) * Enables fast existence checks without disk I/O * Tracks complete table lifecycle across multiple creation/deletion cycles **Implementation:** ```cpp theme={null} std::map>> _table_existence_cache; ``` Each range represents one lifecycle: `[start_xid, end_xid)`. ### 7. Index-to-Table Mapping **Pattern:** SchemaCache maintains `(db, index_id) -> table_id` mapping. **Benefits:** * Enables invalidation during DROP INDEX when table\_id not provided * PostgreSQL event trigger doesn't provide table\_id for index drops * Efficient schema cache invalidation **Implementation:** ```cpp theme={null} std::map, uint64_t> _index_map; // (db, index_id) -> table_id ``` ### 8. Soft Deletes with `exists` Flag **Pattern:** Mark objects as deleted with `exists=false` rather than physical deletion. **Benefits:** * Historical queries can see deleted objects at past XIDs * Simplifies MVCC implementation * Avoids complex deletion cascade logic **Trade-off:** Requires periodic cleanup (garbage collection). ### 9. Write-Through Caching **Pattern:** Server writes to both uncommitted cache AND system tables simultaneously. **Benefits:** * Ensures consistency between cache and disk * Simplifies finalize logic (just flush to disk) * Enables fast queries during transaction **Implementation:** ```cpp theme={null} void _set_table_info(uint64_t db_id, TableCacheRecordPtr table_info) { // Write to _table_cache _table_cache[db_id][table_id][xid] = table_info; // Write to TableNames system table _write_table_names_entry(...); } ``` ### 10. XID Heartbeat Mechanism **Pattern:** Require periodic `keep_alive()` calls to validate committed XIDs. **Benefits:** * Prevents using stale XIDs from crashed processes * Simple liveness detection * No complex distributed consensus required **Implementation:** * `XID_KEEP_ALIVE_PERIOD = 60ms` * `is_alive()` checks timestamp freshness * `get_committed_xid()` fails if not alive *** ## Appendix: Key File Reference ### Core Headers * `include/sys_tbl_mgr/system_tables.hh` - System table schemas and helper classes * `include/sys_tbl_mgr/shm_cache.hh` - Shared memory cache interface * `include/sys_tbl_mgr/schema_cache.hh` - In-process schema cache * `include/sys_tbl_mgr/server.hh` - Server-side management * `include/sys_tbl_mgr/client.hh` - Client-side RPC interface * `include/sys_tbl_mgr/table.hh` - Table interface * `include/sys_tbl_mgr/msg_cache.hh` - Generic message cache template * `include/storage/schema.hh` - SchemaColumn, Index, SchemaMetadata definitions ### Implementations * `src/sys_tbl_mgr/system_tables.cc` - System table schema definitions * `src/sys_tbl_mgr/shm_cache.cc` - Shared memory cache implementation * `src/sys_tbl_mgr/schema_cache.cc` - In-process cache implementation * `src/sys_tbl_mgr/server.cc` - Server logic and DDL handling * `src/sys_tbl_mgr/client.cc` - Client logic and RPC implementation ### Protocol Definitions * `src/proto/sys_tbl_mgr.proto` - gRPC service and message definitions *** ## Summary The Springtail schema management architecture provides a robust, versioned metadata system supporting MVCC, DDL evolution tracking, and efficient multi-process access. Key strengths include: 1. **Multi-Version Concurrency Control**: Every metadata entry versioned by XID/LSN 2. **Three-Tier Caching**: SHM (cross-process) → SchemaCache (in-process) → Disk 3. **Uncommitted Change Tracking**: Server-side caches for pending transactions 4. **Historical Queries**: Access schema at any past XID via version tracking 5. **Efficient Invalidation**: Targeted cache invalidation on DDL changes 6. **Cross-Process Coordination**: Shared memory enables worker process efficiency The system balances performance (multi-level caching, lazy evaluation) with correctness (MVCC versioning, atomic commits) to provide a scalable foundation for PostgreSQL FDW schema management. # System Properties Source: https://docs.springtail.io/system-properties # Properties Component The Properties class is shared by all backend processes and is responsible for accessing and managing configuration and system properties in the Springtail system. ## Setup This component can be initialized in two ways: * through properties file override * through Redis For properties file override it is required to configure `SPRINGTAIL_PROPERTIES_FILE` environment variable that contains properties file like `system.json.settings`. This file is designed for running a minimalistic setup in the development environment. This file will be loaded into Redis and all the information for accessing Redis will come from this file. To run the services through Redis without properties file, it is required to setup a number of environment variables: ```C++ theme={null} REDIS_HOSTNAME REDIS_USER REDIS_PASSWORD REDIS_USER_DATABASE_ID REDIS_CONFIG_DATABASE_ID REDIS_PORT REDIS_SSL ORGANIZATION_ID ACCOUNT_ID DATABASE_INSTANCE_ID INSTANCE_KEY LUSTRE_DNS_NAME LUSTRE_MOUNT_NAME MOUNT_POINT FDW_ID ``` With the exception of `INSTANCE_KEY` and `FDW_ID`, these environment variables will be all setup in the same way, while `INSTANCE_KEY` and `FDW_ID` will be different per server process and FDW instance. The information in these environment variables will be used to access Redis and read the rest of the configuration from there. ## Configuration The configuration consists of multiple components and is stored in JSON format in Redis. All top level keys created in Redis are prefixed with the value of `DATABASE_INSTANCE_ID` followed by ":". It will be stored in `instance_config` hash under `system_settings` key. The value of the system settings is a JSON string that contains configuration for different system components that belong to one or more processes. Here is a list of various configuration components and what they belong to: 1. `logging` - common for all processes 2. `iopool` - only for processes accessing data storage 3. `write_cache` - only for log manager that implements ingestion pipelin 4. `sys_tbl_mgr` - only for processes accessing data storage 5. `storage` - only for processes accessing data storage 6. `log_mgr` - only for log manager process 7. `fs` - only for processes accessing data storage 8. `extension_config` - only for FDW processes 9. `proxy` - only for proxy 10. `aws_users_override` - common for all process, for testing in local development environment only, it prevents **Properties** component from accessing AWS 11. `otel` - Open Telemetry configuration used by all processes ## Functionality **Properties** class provides a variety of get and set API calls for accessing and manipulating the data stored in Redis configuration database. Upon startup, it will read all the data from Redis and store them internally. The data that it is going to manage are all stored with `DATABASE_INSTANCE_ID` prefix in the key. ## Redis Cache A very important part of **Properties** is **RedisCache**. This component actually stores all the data read from Redis and **Properties** class accesses all the data through this cache. To keep up to date with the changes in Redis, the cache registers for generic notifications from Redis and updates the internal storage as notifications are delivered. Also, when the data are changed by **Properties**, the change will be applied to the cache first and then propagated by the cache to Redis. **RedisCache** also delivers an important Redis data update notification mechanism. It allows system services to register with RedisCache for notifications and be notified when updates happen. # Unit Testing Source: https://docs.springtail.io/unit-testing Unit tests are intended to test individual class interfaces and functionality. They should be narrow in scope and not involve complex integrations between components. The unit tests are written using the [Google Test framework](https://github.com/google/googletest). They should be run from the build directory via: ```bash theme={null} make check ``` This will build and run all of the unit tests. To run individual tests, you can run `ctest` with the appropriate filters. If the tests are already built, you can also run `make test` both at the top level to run the entire test suite or within individual subdirectories in order to run a portion of the tests. # User Management Source: https://docs.springtail.io/user-management # User Management ## Overview This document describes how user credentials are sourced and managed, the roles expected in the secrets store, and how authorization is performed for client and server logins. It focuses on behavioral and operational aspects rather than implementation details. ## Retrieving Users from Secrets Mgr * Secrets are centrally stored and periodically fetched by the user-management component. The component queries the secrets provider and constructs an in-memory set of users and their associated data. Currently the AWS Secrets Mgr is supported. * For each secret entry the system extracts: username, password (or secret blob), role, and password-type metadata. Only entries with an approved role are incorporated into the active user set. * After secret retrieval, the system periodically queries the primary database to determine the set of databases each user is allowed to connect to, producing a mapping of username → allowed databases. * Changes in secrets are applied incrementally: existing users are updated if credentials change, new users are added, and removed users are discarded. ## Roles in the Secrets Manager The secrets entries include a role attribute that determines how the credential is used by the proxy. Typical roles are: * **Primary DB login (replication role)**: Credentials used as the canonical login for the primary database. These credentials are used by the system to perform privileged queries (such as discovering user-to-database grants) and to validate global configuration. This role represents the primary authoritative login for administrative queries. * **Database user (database role)**: Credentials for application or end-user database accounts. These entries are mapped to application users and are associated with the set of databases the user may connect to. These are the credentials used for authenticating client sessions that request access to a particular database. * **FDW proxy (proxy-to-fdw role)**: A special credential used when the proxy needs to authenticate to replica on behalf of a client session. The FDW proxy password is used as the server-side password when establishing connections to replica nodes, allowing controlled access to read-only replicas. Notes on role behavior: * The FDW proxy entry is a shared credential that the system can attach to per-user server-side logins when interacting with replicas; it is not the client-visible password for end users. * The database user entries include metadata listing which databases the user is permitted to access. That mapping is authoritative for request-time authorization checks and is updated periodically by querying the Primary for databases to which the user has **connect** privileges. ## Password Types and Handling * Secrets may indicate different storage formats or authentication schemes (plain text, MD5, SCRAM, etc.). The system preserves the indicated type and uses it to drive the authentication exchange with clients or servers. * For SCRAM-style credentials, the system maintains the derived keys necessary to complete the SCRAM exchange without exposing raw passwords. ## Authorization Flow (Client and Server) * When a client initiates a session, the proxy examines the requested username and database and resolves them against the active user set. * If the username exists and is allowed for the requested database, the proxy proceeds with the authentication exchange using the stored credential material and the appropriate authentication mechanism. * The client-facing authentication verifies the client’s presented proof (for SCRAM) or password (for MD5/TEXT) using stored keys or password hashes from the user record. * Once the client is authenticated, the proxy may perform a server-side login to a database instance on behalf of the client. The proxy selects the correct credential material depending on the destination instance and role: * For replicas, the proxy may reuse the FDW-proxy credential where applicable. * For primary writes or actions requiring the user's identity, the proxy uses the database-user credential mapped to that username. * The proxy always authenticates against the Primary database first to ensure the ability of the user to connect. * Throughout authentication, the system logs and records high-level events (auth success/failure) and enforces policy such as database access restrictions. ## Updates and Invalidation * When secrets change, the user set is reconciled: passwords and permitted database lists are updated. If a user's credential changes, any pooled server sessions that would reuse the old credential are prevented from being reused and are evicted as necessary. * If a user entry is removed or a database becomes unavailable, the system invalidates affected pooled sessions and ensures future allocations do not use stale credential material. ## Summary User management centralizes credentials in a secrets provider, classifies secret entries by role, and applies these credentials to client- and server-side authentication flows. The system ensures safety via validation, eviction on invalidation, and synchronized updates so authorization remains correct and robust during secret changes. # Vacuumer Source: https://docs.springtail.io/vacuumer ## Overview The **Vacuumer** is a storage management component in the Springtail database system that reclaims disk space from expired data. Springtail follows an **append-only storage model** — mutations (inserts, updates, deletes) create new extents rather than modifying existing data in place. Once a new extent is written, the previous extent becomes "expired" and eligible for vacuum. **XID-based safety** is central to the Vacuumer's operation: it only reclaims extents that have expired at an XID and all active transactions have moved past that XID point. The vacuum cutoff XID is computed as `min(min_fdw_xid, last_committed_xid, min_index_xid)`, ensuring that foreign data wrapper queries, uncommitted transactions, and ongoing index operations can still access the data they need. The Vacuumer operates as a singleton background service that: * Tracks expired extents (superseded by new extents) and dropped table snapshots * Performs **hole punching** via `fallocate()` to return unused disk blocks to the filesystem * Cleans up dropped table directories and old snapshot/roots files *** ## Key Components | Component | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **`Vacuumer`** | Main singleton class managing vacuum operations (`vacuumer.hh:97`) | | **`VacuumerUtils`** | Utility class for querying vacuum state without instantiating the full Vacuumer (`vacuumer.hh:42`) | | **`VacuumConfig`** | Namespace with configuration defaults: block size for hole punching (4KB), global vacuum file size threshold to trigger vacuum run (20KB), max expired extent entries held in memory before flushing to disk (10K) | | **`HoleInfo`** | Struct holding an expired extent's location: `{ offset, size }` (`vacuumer.hh:207-210`) | | **`ExtentMap`** | Tracks expired extents: `file → xid → vector` (`vacuumer.hh:248`) | | **`SnapshotMap`** | Tracks expired snapshots: `db_id → xid → list` (`vacuumer.hh:255`) | | **Global vacuum file** | Persistent log of pending vacuum work (`.global.vcm`) | | **Partial files** | Track unaligned leftover regions that couldn't be hole-punched (`_partials.vcm`) | *** ## Data Flow ``` 1. EXTENT EXPIRATION (triggered when append-only writes create new extents) StorageCache -> expire_extent() -> _extent_map[file][xid].push_back(offset, size) 2. SNAPSHOT EXPIRATION (triggered by DROP TABLE/INDEX or schema changes) DDL operations -> expire_snapshot() -> _snapshot_map[db_id][xid].push_back(table_dir) 3. COMMIT (on transaction commit) commit_expired_extents() -> writes entries to global vacuum file (.global.vcm) 4. VACUUM RUN (background thread, every 1 second) ``` ```mermaid theme={null} flowchart TD Start["_do_vacuum_run()"] Flush["Flush in-memory expired entries to global
vacuum file if count exceeds threshold"] ReadE["Read expired extents from
global vacuum file"] PerFile["For each file with expired extents:
1. Merge current expired extents with leftover partials
2. Align extent boundaries to filesystem block size (4KB)
3. _hole_punch_file() → fallocate() to reclaim aligned blocks
4. Save unaligned remainders as partials for future coalescing"] DelSnap["Delete expired snapshot directories
(dropped tables/indexes)"] Rotate["Rotate/truncate global vacuum file
(clear processed, keep unprocessed)"] Start --> Flush --> ReadE --> PerFile --> DelSnap --> Rotate ``` *** ## Implementation Details **Extent Expiration Tracking** (`vacuumer.cc:370-396`) * `expire_extent()` is called via a callback registered with `StorageCache` (`vacuumer.cc:92-95`) * Each expired extent is recorded as a `HoleInfo` struct containing offset and size within the file, along with the XID at which it was superseded by a new extent * Entries are held in memory (`_extent_map`) until committed, then persisted to the global vacuum file * Memory threshold (`_max_entries_in_memory`, default 10K) triggers flush to disk if exceeded **Hole Punching Mechanics** (`vacuumer.cc:398-421`, `vacuumer.cc:916-998`) * Uses Linux `fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, offset, len)` to deallocate blocks * **Block alignment requirement**: Filesystem hole punching only works on block-aligned regions * `_align_up()` / `_align_down()` align to `_hole_punch_block_size` (default 4KB) * If an extent spans `[100, 5000]`, only `[4096, 4096]` can be punched; `[100, 4096]` and `[4096, 5000]` become partials * **Interval merging**: Uses `IntervalTree` to coalesce adjacent/overlapping expired regions before punching (`vacuumer.cc:928-972`) * **Partial handling**: Unaligned remainders are saved to per-file partial files (`_partials.vcm`) and merged in subsequent runs **XID-based Vacuum Safety** (`vacuumer.cc:430-439`) * **Cutoff XID** = `min(min_fdw_xid, last_committed_xid, min_index_xid)` * `min_fdw_xid`: Minimum XID in use by foreign data wrappers (active queries from remote) * `last_committed_xid`: Latest committed transaction (protects uncommitted data) * `min_index_xid`: Minimum XID for ongoing index builds/drops * Only extents with `XID < cutoff` are vacuumed, ensuring no active transaction can reference the data * Cutoff XIDs are persisted to Redis per-database for monitoring (`_save_last_seen_cutoff_xid`) **Persistence & Schema** (`vacuumer.cc:62-76`) * **Global vacuum schema**: `(file TEXT, offset UINT64, size UINT64, file_dropped BOOLEAN)` * `file_dropped=true` indicates a snapshot/directory deletion rather than hole punch * **Partial file schema**: `(offset UINT64, size UINT64)` — simpler, no file path needed (one file per source) * Atomic writes via runfiles: write to `.vcm.run`, then `rename()` to `.vcm` **Snapshot & Directory Cleanup** (`vacuumer.cc:1000-1055`) * Dropped tables/indexes are tracked in `_snapshot_map` * Uses `std::filesystem::remove_all()` to recursively delete table directories * Also cleans up associated partial files via `_cleanup_partial_files()` **Roots File Cleanup** (`vacuumer.cc:783-851`) * System tables maintain the roots in the files of the format (`roots.{xid}`) * Vacuum removes roots files with `XID < cutoff`, preserving the current symlinked version * Iterates all system tables defined in `sys_tbl::TABLE_IDS` **Recovery Protocol** (`vacuumer.cc:710-781`) Handles 4 crash states based on file presence: | State | Global File | Runfile | Partials Runfile | Recovery Action | | ----- | ----------- | ------- | ---------------- | --------------------------------------------------------- | | A | Empty | — | — | None | | B | Present | Present | — | Rename runfile → global, truncate to committed XID | | C | Present | — | Present | Remove partials runfile, truncate global to committed XID | | D | Present | — | — | Truncate global to committed XID | **Threading Model** (`vacuumer.cc:1118-1135`) * Background thread wakes every 1 second via `condition_variable::wait_until()` * All public methods acquire `_mutex` before accessing shared state * Graceful shutdown: `_internal_thread_shutdown()` signals CV, thread exits loop **Configuration** (loaded from `storage_config.vacuum_config` JSON) * `enabled`: Enable/disable vacuum service * `hole_punch_block_size`: Alignment for hole punching (default 4KB) * `global_file_size_threshold`: Minimum global file size to trigger vacuum run (default 20KB) * `max_entries_in_memory`: Memory threshold before forced flush (default 10K entries) * `vacuum_dir`: Base directory for vacuum metadata files # Write Cache Source: https://docs.springtail.io/write-cache # Write Cache ## Overview The Write Cache is a transactional in-memory (with disk spillover) cache that temporarily stores uncommitted database changes from PostgreSQL replication before they are committed and indexed into Springtail's storage layer. It serves as a critical component in Springtail's replication pipeline, bridging the gap between PostgreSQL's transaction model and Springtail's versioned storage system. ## Purpose The Write Cache serves several key purposes: 1. **Transaction Buffering**: Stores uncommitted data (extents) from PostgreSQL replication streams while transactions are in-flight 2. **XID Translation**: Maps PostgreSQL transaction IDs (pg\_xid) to Springtail transaction IDs (xid), handling subtransactions 3. **Query Acceleration**: Enables query nodes to access recently committed data before it has been fully indexed ("hurry-ups") 4. **Memory Management**: Automatically spills to disk when memory pressure exceeds configurable watermarks 5. **Transaction Isolation**: Maintains proper visibility semantics by organizing data by transaction and table ## Architecture ### Data Structure Hierarchy The Write Cache uses a multi-level tree structure for efficient lookup and organization: ``` ROOT (per database) └─> XID Nodes (Postgres transaction IDs) └─> TABLE Nodes (table IDs) └─> EXTENT Nodes (LSN-ordered extents) └─> Data (in-memory or on-disk) ``` ### Key Components #### WriteCacheServer (`write_cache_server.hh/cc`) * **Singleton** server managing the entire write cache system * Maintains a map of `WriteCacheIndex` instances, one per database * Handles memory watermark management (high/low thresholds) * Decides when to spill extents to disk based on memory pressure * Exposes gRPC service via `WriteCacheService` * **Location**: `include/write_cache/write_cache_server.hh`, `src/write_cache/write_cache_server.cc` Key responsibilities: * Adding extents: `add_extent(db_id, tid, pg_xid, lsn, data)` * Committing transactions: `commit(db_id, xid, pg_xids, metadata)` * Aborting transactions: `abort(db_id, pg_xid)` or `abort(db_id, pg_xids)` * Memory tracking: monitors `_current_memory_bytes` against watermarks #### WriteCacheIndex (`write_cache_index.hh/cc`) * **Per-database** index containing partitioned table data * Uses **8 partitions** by default (configurable) to enable parallel access * Each partition is a `WriteCacheTableSet` managing a subset of tables * Partitioning strategy: `table_id % num_partitions` * Tracks total memory usage across all partitions * **Location**: `include/write_cache/write_cache_index.hh`, `src/write_cache/write_cache_index.cc` #### WriteCacheTableSet (`write_cache_table_set.hh/cc`) * **Per-partition** implementation of the core index logic * Maintains the tree structure: ROOT → XID → TABLE → EXTENT * Manages two critical mappings: * `_xid_map`: Maps Springtail XID → Postgres XID(s) (multimap for subtransactions) * `_xid_ts_map`: Maps Springtail XID → Metadata (commit timestamps) * Thread-safe with shared mutexes for concurrent reads * **Location**: `include/write_cache/write_cache_table_set.hh`, `src/write_cache/write_cache_table_set.cc` #### WriteCacheIndexNode (`write_cache_index_node.hh/cc`) * **Generic tree node** used at all levels of the hierarchy * Types: `ROOT`, `XID`, `TABLE`, `EXTENT`, `EXTENT_ON_DISK` * Contains: * `id`: XID, table ID, or LSN depending on level * `data`: ExtentPtr for in-memory extents * `data_offset`, `data_size`: For disk-based extents * `children`: Ordered set of child nodes (sorted by ID) * Thread-safe with shared mutex for concurrent access * **Location**: `include/write_cache/write_cache_index_node.hh`, `src/write_cache/write_cache_index_node.cc` #### WriteCacheService (`write_cache_service.hh/cc`) * **gRPC service** implementation exposing write cache functionality * Implements the `proto::WriteCache` service interface * RPC methods: * `Ping()`: Health check * `GetExtents()`: Retrieve extents for a table at a given XID * `ListTables()`: Get list of tables modified in a transaction * `EvictTable()`: Remove specific table data from cache * `EvictXid()`: Remove all data for a transaction * **Location**: `include/write_cache/write_cache_service.hh`, `src/write_cache/write_cache_service.cc` #### WriteCacheClient (`write_cache_client.hh/cc`) * **Singleton** client for query nodes to access the write cache * Communicates with `WriteCacheServer` via gRPC * Used by pg\_fdw (Foreign Data Wrapper) for "hurry-up" queries * Provides extent caching via shared memory (`ShmCache`) * **Location**: `include/write_cache/write_cache_client.hh`, `src/write_cache/write_cache_client.cc` ### Protocol Definition The gRPC API is defined in `src/proto/write_cache.proto`: * **Extent**: Contains xid, xid\_seq (LSN), and data * **GetExtentsRequest/Response**: Paginated extent retrieval * **ListTablesRequest/Response**: Paginated table list retrieval * **EvictTableRequest**: Remove table from cache * **EvictXidRequest**: Remove transaction from cache ## Transaction Lifecycle ### 1. Data Ingestion (Write Path) **Component**: `PgLogReader` (`src/pg_log_mgr/pg_log_reader.cc`) 1. PostgreSQL replication stream is parsed by `PgLogReader` 2. For each INSERT/UPDATE/DELETE operation: * Data is accumulated into extents per table * `WriteCacheServer::add_extent(db_id, tid, pg_xid, lsn, extent)` is called * Extent is added to the tree under the appropriate pg\_xid and table 3. Memory is tracked; if high watermark is exceeded, `_store_to_disk` flag is set 4. Subsequent extents are written to disk files named by pg\_xid ### 2. Transaction Commit **Component**: `PgLogReader::Batch` (`src/pg_log_mgr/pg_log_reader.cc`) 1. When a COMMIT record is received: * All accumulated extents for the transaction are already in the write cache * `WriteCacheServer::commit(db_id, xid, pg_xids, metadata)` is called * Maps Springtail XID to all associated Postgres XIDs (handling subtransactions) * Stores metadata including: * `pg_commit_ts`: PostgreSQL commit timestamp * `local_begin_ts`: Local transaction start time * `local_commit_ts`: Local transaction commit time 2. Mapping is replicated across all partitions for efficient lookup ### 3. Transaction Abort **Component**: `PgLogReader::Batch` (`src/pg_log_mgr/pg_log_reader.cc`) 1. When an ABORT record is received: * `WriteCacheServer::abort(db_id, pg_xid)` is called * All extents for the pg\_xid are removed from the tree * Memory is freed and tracked * Associated disk files are deleted ### 4. Data Consumption (Read Path) **Component**: `PgFdwMgr` (`src/pg_fdw/pg_fdw_mgr.cc`) 1. Query nodes use `WriteCacheClient` to fetch recent data 2. "Hurry-up" queries check the write cache for data not yet in the main index 3. `WriteCacheClient::get_extents(db_id, tid, xid, count, cursor, commit_ts)`: * Sends gRPC request to `WriteCacheService` * Retrieves up to `count` extents starting from `cursor` * Returns extents and associated commit timestamp 4. Extents may be served from memory or read from disk transparently ### 5. Data Eviction **Component**: `Committer` (`src/pg_log_mgr/committer.cc`) 1. After data has been committed to the main storage system: * `WriteCacheServer::evict_xid(db_id, xid)` is called * Removes all data for the transaction from cache * Frees memory and deletes disk files * Prevents cache from growing unbounded ## Memory Management ### Watermark System The write cache uses a two-level watermark system to manage memory: * **High Watermark** (`memory_high_watermark_bytes`): When exceeded, new extents are written to disk * **Low Watermark** (`memory_low_watermark_bytes`): When memory drops below this, writing to memory resumes * **Current Memory** (`_current_memory_bytes`): Tracked atomically across all operations Configuration is specified in `Properties::WRITE_CACHE_CONFIG`: ```json theme={null} { "disk_storage_dir": "/path/to/storage", "memory_high_watermark_bytes": 10737418240, // 10GB "memory_low_watermark_bytes": 8589934592, // 8GB "rpc_config": { ... } } ``` ### Disk Spillover When `_store_to_disk` is true: 1. Extents are written to files in `_disk_storage_dir/db_id/pg_xid` 2. Tree nodes store `data_offset` and `data_size` instead of extent data 3. `EXTENT_ON_DISK` node type marks disk-based extents 4. On read, extents are transparently loaded from disk via `IOMgr` ### Partitioning for Concurrency * Default: **8 partitions** per database * Tables are hashed by `table_id % num_partitions` * Enables parallel access to different tables * Each partition has independent locking * Memory accounting is aggregated across partitions ## Integration Points ### Components That Write to Write Cache 1. **PgLogReader** (`src/pg_log_mgr/pg_log_reader.cc`) * Primary writer: adds extents during replication * Commits/aborts transactions based on WAL records * Main entry point: `Batch::commit()` and `Batch::abort()` ### Components That Read from Write Cache 1. **PgFdwMgr** (`src/pg_fdw/pg_fdw_mgr.cc`) * Query execution: retrieves recent uncommitted/recently-committed data * Uses `WriteCacheClient::get_extents()` for hurry-up queries * Maximum fetch size: `MAX_WRITE_CACHE_EXTENTS = 10` 2. **Committer** (`src/pg_log_mgr/committer.cc`) * Manages write cache evictions after data is persisted * Maintains `_write_cache_evictions` map per database * Calls `evict_xid()` during cleanup ### Thread Safety * All data structures use `std::shared_mutex` for concurrent access * Read operations (get\_extents, list\_tables) use shared locks * Write operations (add\_extent, commit, abort) use unique locks * Memory tracking uses `std::atomic` ## Testing Test files are located in `src/write_cache/test/`: 1. **test\_wc\_index.cc**: Unit tests for WriteCacheIndex and WriteCacheTableSet 2. **test\_wc\_server.cc**: Integration tests for WriteCacheServer and gRPC service ## Performance Considerations 1. **Partitioning**: 8-way partitioning reduces lock contention for multi-table transactions 2. **Ordered Sets**: Extents are stored in LSN order for efficient range queries 3. **Memory Tracking**: Atomic counters avoid lock overhead for memory accounting 4. **Disk I/O**: Extents are written/read asynchronously via `IOMgr` 5. **Pagination**: Get operations support cursor-based pagination to limit memory overhead ## Key Invariants 1. Every pg\_xid maps to at most one Springtail xid 2. A Springtail xid may map to multiple pg\_xids (subtransactions) 3. Extents within a table are ordered by LSN 4. Memory accounting must be exact for watermark enforcement 5. Disk files are named by pg\_xid and deleted on abort/evict 6. All partitions must be updated on commit for correct lookups # XID Management Source: https://docs.springtail.io/xid-management # XID Management XID management is how Springtail maintains a single, consistent notion of "what has committed" as transactions flow through the ingest pipeline. It assigns every replicated change a monotonic Springtail transaction id (XID), records the durable commit order, and publishes a single **committed-XID watermark** that every reader in the system uses to decide what data is visible. This page covers the XID model and the central XID manager service. The mechanics that build on top of it are documented elsewhere: how XIDs are tracked while a table is being copied lives in [Syncing Tables](/syncing-tables), how schema versions are tied to XIDs lives in [System Metadata](/system-metadata), how the committed XID anchors restart lives in [Recovery](/recovery), and how query nodes consume commit notifications lives in [XID Subscriber](/xid-subscriber). ## Two transaction-id spaces Springtail does not reuse the primary's transaction ids as its consistency clock. Two distinct id spaces are involved: * **Postgres XID (`pg_xid`)** — the 32-bit transaction id assigned by the primary database. It is not gap-free across the changes Springtail sees, and it wraps around, so it cannot serve as a global ordering key. * **Springtail XID (`xid`)** — a 64-bit, monotonically increasing id that Springtail assigns itself. It defines a total order over every replicated change and never wraps. The `PgLogReader` assigns the Springtail XID as it reads changes off the replication log, by atomically incrementing a counter (`include/pg_log_mgr/pg_log_reader.hh`). Each XID carries the originating `pg_xid` alongside it, so the mapping back to the primary's transaction is preserved. Where an LSN is also needed, the two are paired in an `XidLsn` (`include/storage/xid.hh`). A single primary transaction may produce several replicated changes, so multiple Springtail XIDs can map to the same `pg_xid`. The XID that represents the actual transaction commit is flagged as a **real commit**; the others are intermediate records. ## The XID manager service The XID manager (`XidMgrServer`, `src/xid_mgr/xid_mgr_server.cc`) is a singleton gRPC service that owns the committed-XID state for every database on the node. Per database it maintains: * a durable **transaction log** of XID records, written through `PgXactLogWriter` (`include/xid_mgr/pg_xact_log_writer.hh`), * the **last committed XID** — the highest real-commit XID, which is the watermark returned to readers, and * a **schema-change history**, used to answer schema-consistent reads (see below). Ingest components call the server directly; everything else reaches it over gRPC through `XidMgrClient` (`include/xid_mgr/xid_mgr_client.hh`). The service definition is in `src/proto/xid_manager.proto`. ### The committed-XID watermark `get_committed_xid(db_id, schema_xid)` is the consistency boundary for the whole system: a change is visible only once its XID is at or below the committed watermark. Readers throughout Springtail gate on it — the FDW and DDL manager on query nodes, the [vacuumer](/vacuumer), and the replication log readers all ask the XID manager (or a cached copy of its value) for the committed XID before deciding what they may act on. When `schema_xid` is `0`, or the database has no recorded schema changes, the call returns the most recent committed XID. When a non-zero `schema_xid` is supplied, the manager consults its schema-change history and returns the latest commit that is still consistent with that schema version, so a query node sees a stable schema for the duration of a read even as later DDL commits arrive. The history itself — how schema versions map to XIDs — is described in [System Metadata](/system-metadata). ## How commits are recorded The [committer](/message-processing) in the ingest pipeline drives the XID manager through a small set of entry points, chosen according to where the transaction's data already is. The distinction exists to keep the durable watermark from ever moving ahead of durable data. | Call | Used when | Effect | | -------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | | `commit_xid` | A real commit whose data is already durable (or has no table mutations) | Advances the watermark and writes the record to the transaction log immediately. | | `commit_xid_no_xlog` | A real commit whose table data has not yet been fsynced | Advances the in-memory watermark and notifies subscribers, but **defers** the log write until the data is persisted. | | `commit_xlog` | The deferred data has since been persisted | Flushes the pending log entries to the transaction log, making the commit durable. | | `record_mapping` | An intermediate XID, or a commit recorded while commits are blocked for a table sync | Records the XID-to-`pg_xid` mapping and any schema change, without advancing the committed watermark. | The deferred path is what guarantees crash safety: the durable committed XID is only written once the corresponding table data is on disk, so after a restart the watermark can never point past data that was actually persisted. The reason commits are blocked during a table copy, and how they are released afterward, is covered in [Syncing Tables](/syncing-tables). ## Commit notifications When a real commit is recorded — and for intermediate records that touched tables — the XID manager pushes a notification to its subscribers over a gRPC server-side stream. Each notification carries the database id, the committed XID, whether the transaction included schema changes, whether it is a real commit, and the list of modified table ids. Query nodes subscribe to this stream to keep their caches current and to learn when newly committed data is available; that consumer is documented in [XID Subscriber](/xid-subscriber). ## Durability, recovery, and cleanup The transaction log is memory-mapped and persisted per database. Each record packs the Springtail XID, the originating `pg_xid`, and the real-commit flag. On startup, the log is sanitized before the watermark is established: any trailing records that are not real commits are dropped, so the log always ends on a durable commit, and the recovered XID becomes the starting watermark. The ingest pipeline then seeds the `PgLogReader`'s next-XID counter from that value (`src/pg_log_mgr/pg_log_mgr.cc`), so XID assignment resumes exactly where it left off. How this committed XID anchors the broader recovery scan is described in [Recovery](/recovery). A background thread runs on a fixed interval (`XIG_MGR_MIN_SYNC_MS`, 500 ms) to flush the log and trim the schema-change history. History entries below the minimum schema XID still in use — obtained from Redis — are discarded, and older log files are cleaned up by timestamp once they are no longer needed. ## Key files | File | Role | | -------------------------------------------------------------------- | -------------------------------------------------------------- | | `src/xid_mgr/xid_mgr_server.cc`, `include/xid_mgr/xid_mgr_server.hh` | The XID manager service and per-database commit state. | | `include/xid_mgr/xid_mgr_client.hh` | Client used by off-node components to query the committed XID. | | `include/xid_mgr/pg_xact_log_writer.hh` | Durable, memory-mapped transaction log. | | `src/proto/xid_manager.proto` | gRPC service and message definitions. | | `include/storage/xid.hh` | `XidLsn` / `XidRange` value types. | | `include/pg_log_mgr/pg_log_reader.hh` | Assigns monotonic Springtail XIDs during ingest. | # XID Subscriber Source: https://docs.springtail.io/xid-subscriber # PgXidSubscriberMgr ## Overview `PgXidSubscriberMgr` is a singleton service that manages subscriptions to transaction ID (XID) commit push notifications from the XID manager. It acts as a bridge between the XID manager and the PostgreSQL Foreign Data Wrapper (pg\_fdw) by maintaining shared memory caches that enable efficient inter-process communication. ## Architecture * **Design Pattern**: Singleton (`include/pg_fdw/pg_xid_subscriber_mgr.hh:22`) * **Execution Model**: Runs as a dedicated daemon process with its own main thread and a pool of worker threads * **Location**: `src/pg_fdw/pg_xid_subscriber_daemon.cc` ## gRPC Streaming Subscription ### Streaming RPC Protocol The manager uses **gRPC server-side streaming** to receive real-time XID commit notifications from the XID manager: * **Service**: `XidManager::Subscribe(SubscribeRequest) returns (stream XidPushResponse)` (`src/proto/xid_manager.proto:40`) * **Pattern**: Single request initiates a long-lived stream that continuously pushes commit events * **Message Format**: Each `XidPushResponse` contains: * `db_id`: Database identifier * `xid`: Transaction ID * `has_schema_changes`: Whether the transaction modified schema * `real_commit`: Distinguishes full commits from recorded transactions * `table_ids`: List of tables modified (for recorded commits only) **Key characteristics:** 1. **Asynchronous Reactor**: Inherits from `ClientReadReactor` for non-blocking, event-driven streaming 2. **Lifecycle Management**: gRPC manages object lifetime from `StartCall()` until `OnDone()` callback 3. **Thread Safety**: Callbacks execute on gRPC's internal thread pool, requiring careful synchronization **Stream Termination** (`src/xid_mgr/xid_mgr_subscriber.cc:75-86`): * `OnDone(const grpc::Status&)` called when stream ends (graceful or error) * Signals completion to allow safe destruction * Disconnect callback invoked to notify subscriber ## Core Responsibilities ### 1. Transaction Notification Handling The manager subscribes to the XID manager to receive push notifications when transactions commit. It handles two types of commits: * **Real Commits** (`real_commit = true`): Full transaction commits that trigger proactive cache population * **Recorded Commits** (`real_commit = false`): Transactions with pending mutations that need tracking for write cache lookups ### 2. Shared Memory Cache Management Maintains five IPC caches populated via shared memory for use by pg\_fdw processes: | Cache | Purpose | | ------------------- | ---------------------------------------------- | | **Roots Cache** | Table root metadata | | **Schema Cache** | Table schema definitions | | **User Type Cache** | User-defined type information | | **Table IDs Cache** | Maps (DbId, Xid) → modified table IDs | | **Extents Cache** | Stores table mutation extents from write cache | ### 3. Asynchronous Cache Population * When a real commit notification arrives, the manager enqueues a populate job (`src/pg_fdw/pg_xid_subscriber_mgr.cc:96`) * Worker threads (`src/pg_fdw/pg_xid_subscriber_mgr.cc:185-234`) process the queue and make RPC calls to fetch: * Table roots via `get_roots()` * Table schemas via `get_schema()` * User type definitions via `get_usertype()` * The main subscriber thread remains non-blocking while workers populate caches in the background ## Integration Points * **XID Manager**: Receives push notifications via `XidMgrSubscriber` using gRPC server-side streaming * **System Table Manager**: Uses `sys_tbl_mgr::Client` to fetch metadata * **PostgreSQL FDW**: Provides cached data to pg\_fdw processes running in separate address spaces