Documentation Index

Fetch the complete documentation index at: https://docs.springtail.io/llms.txt

Use this file to discover all available pages before exploring further.

​

Configuration in Redis

• Database 0: This is the configuration database used to hold configuration data that is fairly static.
• Database 1: This is the data database used to hold runtime data and queues.

​

Database 0 (config DB)

• Database Instance Config (HSET)
  • key = <springtail_db_instance.id>:instance_config
  • value = Hash

    field key	value
    id	<springtail_db_instance.id>
    primary_db	JSON: {host: “”, port: <int>}
    database_ids	JSON: <springtail_db_config.id> “[…]” (array of bigints)
    system_settings	JSON: See example below,
    system_settings_gitsha	String: The 7-byte short GitSHA of the latest commit of the prod.system.settings.json , used for version checking
    hostname:proxy	(either missing or set to the complete value, when rebooting, need to delete first, so coordinator won’t grab the wrong values)
    hostname:ingestion	(either missing or set to the complete value, when rebooting, need to delete first, so coordinator won’t grab the wrong values)

    system_settings JSON example (NOTE: this is an example see prod.system.settings.json for all settings)
    {
      "logging": {
        "log_level": "debug",
        "log_path": "/opt/springtail/logs",
        "log_file_size": 104857600,
        "log_file_count": 10,
        "log_pattern": "[%Y-%m-%d %T.%e %z] [%^%l%$] [%s:%#:%!] [thread %t] %v",
        "log_modules": [
          "all"
        ],
        "pid_path": "/opt/springtail/pids",
        "log_rotation_enabled": true
      },
      "io_pool": {
        "threads": 10,
        "filehandles": 30
      },
      "redis": {
        "keep_alive_secs": 30,
        "pool": {
          "connections": 10,
          "max_idle_secs": 300,
          "max_connection_lifetime_secs": 0
        }
      },
      "write_cache": {
        "rpc_config": {
          "server_cert": "/opt/springtail/certs/server_cert.pem",
          "server_key": "/opt/springtail/certs/server_key.pem",
          "server_trusted": "/opt/springtail/certs/ca_cert.pem",
          "client_cert": "/opt/springtail/certs/client_cert.pem",
          "client_key": "/opt/springtail/certs/client_key.pem",
          "client_trusted": "/opt/springtail/certs/ca_cert.pem",
          "server_port": 55051,
          "server_worker_threads": 8,
          "ssl": true,
          "client_connections": 8
        },
        "io_threads": 8
      },
      "sys_tbl_mgr": {
        "rpc_config": {
          "server_cert": "/opt/springtail/certs/server_cert.pem",
          "server_key": "/opt/springtail/certs/server_key.pem",
          "server_trusted": "/opt/springtail/certs/ca_cert.pem",
          "client_cert": "/opt/springtail/certs/client_cert.pem",
          "client_key": "/opt/springtail/certs/client_key.pem",
          "client_trusted": "/opt/springtail/certs/ca_cert.pem",
          "server_port": 55053,
          "server_worker_threads": 16,
          "ssl": true,
          "client_connections": 8
        },
        "cache_size": 4096,
        "roots_shm_cache_size": 10485760
      },
      "storage": {
        "table_dir": "table",
        "data_cache_size": 16384,
        "page_cache_size": 16384,
        "btree_cache_size": 512,
        "max_extent_per_page": 16
      },
      "log_mgr": {
        "replication_log_path": "repl_logs",
        "transaction_log_path": "xact_logs",
        "log_size_rollover_threshold": 134217728,
        "archive_logs": true,
        "rpc_config": {
          "server_cert": "/opt/springtail/certs/server_cert.pem",
          "server_key": "/opt/springtail/certs/server_key.pem",
          "server_trusted": "/opt/springtail/certs/ca_cert.pem",
          "client_cert": "/opt/springtail/certs/client_cert.pem",
          "client_key": "/opt/springtail/certs/client_key.pem",
          "client_trusted": "/opt/springtail/certs/ca_cert.pem",
          "server_port": 55052,
          "server_worker_threads": 32,
          "ssl": true,
          "client_connections": 8
        }
      },
      "proxy": {
        "enable_ssl": true,
        "port": 5432,
        "threads": 4,
        "cert_path": "/opt/springtail/certs/server_cert.pem",
        "key_path": "/opt/springtail/certs/server_key.pem",
        "mode": "primary",
        "shadow_log_path": "/shadow_logs/proxy_shadow.log",
        "log_level": 2,
        "keep_alive_port": 10080,
        "use_pg_shadow": false,
        "pool_size_limit": 10,
        "pool_timeout_limit": 5,
        "pool_expiration_interval_secs": 300
      },
      "otel": {
        "enabled": true,
        "host": "localhost",
        "port": 4318,
        "metrics_host": "localhost",
        "metrics_port": 4318
      }
    }
    
• Database Config (HSET)
  • key = <springtail_db_instance.id>:db_config
  • field_key = <springtail_db_postgres_config.id>
  • field_value = JSON blob.
    • For the “include” section:
      • if it is empty or "*" in "schemas" section, we can assume the equivalence of Postgres FOR ALL TABLES; all tables will be replicated, including all future tables
      • for each entry in “schemas” we assume equivalence of Postgres FOR TABLES IN SCHEMA; all tables in that schema will be replicated, including all future tables
      • for each entry in "tables" we assume equivalence of Postgres FOR TABLE; only that table and its descendants will be replicated (no wildcards allowed). A "tables" section when "schemas": ["*"] is not valid.
    JSON:
    {
    		"id": "<springtail_db_postgres_config.id>",
    		"name": "<springtail_db_postgres_config.name>",
    		"replication_slot": "<springtail_db_postgres_config.replication_slot>",
    		"publication_name": "<springtail_db_postgres_config.publication_name>",
        
        // example include sections
        // example 1: schemas only
        "include": { 
    	    "schemas": ["*"],  // * = all schemas (default); otherwise list specific schemas
        }
        
        // example 2: schemas with tables
        "include": {
    	    "schemas": ["schemaA", "schemaB", ...], // all current and future tables in schema 'schemaA' and 'schemaB'
    	    "tables": [{"schema": "schemaC", "table": "tableA"}, ...],  // also include table 'tableA' from schema 'schemaC'
    	  }
    }
    
• FDW IDs (Set)
  • key = <springtail_db_instance.id>:fdw_ids
  • values = <fdw_ids>
  E.g.,
  SADD fdw-ids:1234 "fdw-01" "fdw-02"
  SMEMBERS fdw-ids:1234
  SREM fdw-ids:1234 "fdw-01"
  
• FDW Config (HSET)
  • key = <springtail_db_instance.id>:fdw
  • field_key = <fdw_opaque_id>
  • field_value = JSON: {”host”: ”ip”, “port”: <int>, “state”: “”, “sync_seconds” }, state = initialize, running, draining, stopped
  • State transitions:
    • The www service will set the FDW state to initialize upon startup
    • The Springtail service (the ddlmgr) will set FDW state to running after it has completed initialization
    • The www service will set FDW state to draining upon scale-down
    • The Springtail service (the coordinator) will set FDW state to stopped once all connections completely drained from FDW
  • Note:
    • For each FDW instance, make sure the delete the field_key then set to the right value, so the coordinator won’t grab the wrong values.
    • The Springtail service does not interpret fdw_ip_as_opaque_id . It should ignore the initialize state, which is essentially a state when the FDW scaling is requested but not yet available.
    • If state is stopped safe to remove from Redis
    • FDW user and password are set in AWS secrets mgr with role fdw_superuser
• DB State (HSET)
  • Contains state for each DB within the instance
  • key = <springtail_db_instance.id>:instance_state
  • field_key = <springtail_db_postgres_config.id>
  • field_value = one of: initialize startup synchronizing running failed stopped removed
    • Set to initialize for each db upon Launching, the very very first launching, after copy tables has been run the system will set the state to running.
    • If a table copy is required the state will change to synchronizing while the copy table is in progress and then will change back to running
    • If the db is restarted (or instance is restarted) then the state should be set to startup
• Admin Console (HSET)
  • Contains ip and port pair for each daemon for admin console access
  • key = <springtail_db_instance.id>:admin_console
  • field_key = <instance_key>:<daemon_name>
  • field_value = <ip>:<port>
• Coordinator State (HSET)
  • Contains state of coordinator on each instance
  • key = <springtail_db_instance.id>:coordinator_state
  • field_key = <service_name>:<instance_key>
  • field_value = one of: startup, running, reload, reloading, shutdown, dead
    • running set after startup is complete or after reload is successful; set by coordinator
    • dead set after shutdown is complete; set by coordinator
    • startup, reload, shutdown set by external API to trigger coordinator action to move to running (startup) or dead (shutdown); reload causes a reload of system daemons
      • If the state is set to shutdown the coordinator will be shutdown immediately, so this shutdown flag is set in the FDW Clean up cronjob, before tearing down the FDWs.
      • reload should cause a reinstall of a running system.  I think the main difference is that startup requires the coordinator to be restarted manually, while reload is picked up while the coordinator is running and should do the same thing, but appears broken.
• Include Schemas Change (HSET)
  • Include schemas changes after a DB Instance is live
  • key = <springtail_db_instance.id>:include_changes
  • field_key = <springtail_db_postgres_config.id>
  • field_value = New list of schemas, can be either ["*"] or ["schema_1", "schema_2", ..., "schemaN"]

​

Database 1 (data)

• FDW → DB list (HSET)
  • Mapping of FDW id → list of database ids the FDW supports.
  • key = <springtail_db_instance.id>:fdw_dbs
  • field_key = FDW id
  • field_value = comma/JSON list of database ids
• Namespace include-changes (HSET)
  • Per-database namespace restriction lists used by change filters.
  • key = <springtail_db_instance.id>:include_changes
  • field_key = <database id>
  • field_value = list of namespaces (wildcard [”-”] or explicit list)
• Pending include-changes (HSET)
  • Pending schema include/exclude changes staged for the log manager.
  • key = <springtail_db_instance.id>:pending_include_changes
  • field_key = <database id>
  • field_value = JSON object with add/remove lists
• DDL queue by XID (QUEUE)
  • Queue of DDL operations for a specific XID emitted by the log parser.
  • key = <springtail_db_instance.id>:queue:ddl:xid:<db_id>:<xid>
  • queue entries = serialized DDL operations for the given <springtail_db_instance.id>, <db_id>, <xid>
• Index DDL queue by XID (QUEUE)
  • Queue of index-related DDL operations for a specific XID.
  • key = <springtail_db_instance.id>:queue:index:ddl:xid:<db_id>:<xid>
  • queue entries = serialized index DDL ops
• Pre-commit DDL hash (HSET)
  • Hash storing pre-commit DDL operations (indexed by db/xid).
  • key = <springtail_db_instance.id>:hash:ddl:pc
  • field_key = db_id:xid (or similar)
  • field_value = serialized pre-commit DDL ops
• DDL queue for FDW (QUEUE)
  • Queue of DDL changes for FDW processing.
  • key = <springtail_db_instance.id>:queue:ddl:fdw:<fdw_id>
  • queue entries = tasks for FDW id
• DDL FDW schema_xids (HSET)
  • Hash of schema_xids per FDW (tracking per fdw:db).
  • key = <springtail_db_instance.id>:hash:ddl:fdw
  • field_key = <db_id>:<fdw_id>
  • field_value = schema_xid
• Daemon liveness (HSET)
  • Liveness timestamps for daemons (used by coordinator).
  • key = <springtail_db_instance.id>:hash:liveness
  • field_key = <daemon_type>:<thread_id>
  • field_value = timestamp
• Coordinator liveness pub/sub (PUBSUB)
  • Pub/sub channel used to notify coordinator of dead daemons.
  • key = <springtail_db_instance.id>:pubsub:liveness_notify
  • message = <daemon_type>:<thread_id>
• DB table set (SET)
  • Set of schema.table names for a given database id.
  • key = <springtail_db_instance.id>:set:db_tables:<db_id>
  • members = quoted(schema).quoted(table)
• DB table changes pub/sub (PUBSUB)
  • Pub/sub channel notifying the proxy of table changes.
  • key = <springtail_db_instance.id>:pubsub:db_table_changes
  • message = <db_id>:<add|remove>:<schema>:<table>
• Index precommit DDL hash (HSET)
  • Hash for pre-commit index DDL operations.
  • key = <springtail_db_instance.id>:hash:idx:ddl:pc
  • field_key = db_id:xid (or similar)
  • field_value = serialized index DDL ops
• Invalid tables hash (HSET)
  • Hash of excluded/invalid tables for a db_instance.
  • key = <springtail_db_instance.id>:hash:invalid_tables
  • field_key = <table_oid>
  • field_value = JSON describing exclusion
• FDW min XID (HSET)
  • Minimum XIDs per fdw:db_id for an instance (tracking progress).
  • key = <springtail_db_instance.id>:fdw_min_xids
  • field_key = <fdw_id>:<db_id>
  • field_value = min xid
• FDW PIDs (SET)
  • Set of new FDW process ids that have not started querying yet.
  • key = <springtail_db_instance.id>:fdw_pids
  • members = <fdw_id>:<db_id>:<pid>
• DB index XIDs (SET)
  • Set holding index XIDs for each db id.
  • key = <springtail_db_instance.id>:set:db_index_xids:<db_id>
  • members = xid values
• Table sync state (HSET)
  • Set of Table IDs (TID/OID) and the corresponding xmin:xmax and in progress pg xids as determined by pg_current_snapshot()
  • key = set:sync_table_state:<springtail_db_instance.id>:<springtail_db_postgres_config.id>
  • field_key = OID (TID)
  • field_value = xmin:xmax:xid1,xid2,…. xids=list of in progress xids
• Table sync queue (QUEUE); for GC only
  • Queue of table IDs that require resyncing
  • key = queue:sync_tables:<db_instance.id><springtail_db_postgres_config.id>
  • value = list of <TID> (OIDs)
• Log Mgr resync point (STRING)
  • Resync point for the log mgr, for it to start reprocessing replication log messages after it was stalled
  • key = string:log_resync:<db_instance.id><springtail_db_postgres_config.id>
  • value = filename:offset
• Vacuum cutoff XIDs (HSET)
  • Hash of vacuum cutoff XIDs per db_id for an instance. Set of XIDs at which index is ddl is received and is being processed by the Indexer
  • key = <db_instance.id>:vacuum_cutoff_xids
  • field_key = <db_id>
  • field_value = set of XIDs