Redis

Springtail has two main purposes for Redis. First is to store configuration in it and second, as an IPC mechanism for sharing the states between server processes and queuing DDL updates that need to be processed. All storage keys in Redis are prefixed with the database instance id that the data structure belongs to.

Redis configuration database

The configuration database has the following data structures:

<instance id>:instance_state — this is a hash table that has Postgres database id as a key and Springtail state of that database as a value. During normal operation the state of the database is set to running. When a new database is first introduced for replication, its state will be set to initialize.
<instance id>:db_config — this is a hash table that has Postgres database id a a key and Springtail configuration data for this database as a value. Here are the data stored per database:

{
   "name": "<database name>",
   "replication_slot": "<replication slot name>",
   "publication_name": "<publication name>",
   "include": {
       "schemas": [<list of replicated schemas or "*">]
   }
}

<instance id>:fdw — this is a hash table that has an FDW id as a key and an FDW configuration as a value. Here are the data stored in this hash table per FDW:

{
    "db_prefix": "<database name prefix>",
    "fdw_user": "<name of FDW user>",
    "host": "<host name>",
    "password": "<user password>",
    "port": <database port>,
    "proxy_password": "<proxy password>",
    "state": "<FDW state>",
    "sync_seconds": <sync time>
}

During normal operation the state of FDW will be set to running.

<instance id>:instance_config — this is a hash table that has several predefined keys that determine springtail instance configuration. Here is the list of all the keys together with the JSON structures associated with each key:

"primary_db": "{
    "host": "<primary Postgres database host name>",
    "port": <primary Postgres database port>
}"
"system_settings": "{
    "logging": {
        "log_level": "debug",
        "log_level_debug": 3,
        "log_path": "/opt/springtail/logs/",
        "log_file_size": 1000000000,
        "log_file_count": 100,
        "log_pattern": "[%Y-%m-%d %T.%e %z] [%^%l%$] [%s:%#:%!] [thread %t] %v",
        "log_modules": ["all"],
        "log_rotation_enabled": true,
        "pid_path": "/opt/springtail/pids"
    },
    "iopool": {
        "threads": 10,
        "filehandles": 30
    },
    "write_cache": {
        "rpc_config": {
            "server_port": 5051,
            "server_worker_threads": 8,
            "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem",
            "server_key": "/home/dev/springtail/ca_certs/server_key.pem",
            "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem",
            "ssl": false,
            "client_connections": 8,
            "client_cert": "/home/dev/springtail/ca_certs/client_cert.pem",
            "client_key": "/home/dev/springtail/ca_certs/client_key.pem",
            "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem"
        },
        "disk_storage_dir": "/opt/springtail/data/write_cache",
        "NOTE": "memory for write cache 640MB and 512MB",
        "memory_high_watermark_bytes": 671088640,
        "memory_low_watermark_bytes": 536870912
    },
    "storage": {
        "table_dir": "table",
        "data_cache_size": 32768,
        "page_cache_size": 16384,
        "btree_cache_size": 512,
        "max_extent_per_page": 16,
        "vacuum_config": {
            "enabled": true,
            "vacuum_dir": "vacuum",
            "global_file_size_threshold": 102400,
            "hole_punch_block_size": 4096,
            "max_entries_in_memory": 100000
        },
        "metrics_update_freq_sec": 10,
        "io_request_queue_size": 16
    },
    "redis": {
        "host": "localhost",
        "port": 6379,
        "user": "default",
        "password": null,
        "ssl": false,
        "db": 1,
        "config_db": 0,
        "keep_alive_secs": 30,
        "pool": {
            "connections": 10,
            "max_idle_secs": 300,
            "max_connection_lifetime_secs": 0
        }
    },
    "log_mgr": {
        "replication_log_path": "repl_logs",
        "transaction_log_path": "xact_logs",
        "log_size_rollover_threshold": 134217728,
        "archive_logs": true,
        "indexer_worker_threads": 1,
        "reader_queue_mem_high_watermark": 134217728,
        "reader_queue_mem_low_watermark": 100663296,
        "committer_fsync_interval_ms": 500,
        "committer_fsync_worker_threads": 1,
        "rpc_config": {
            "server_port": 5052,
            "server_worker_threads": 32,
            "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem",
            "server_key": "/home/dev/springtail/ca_certs/server_key.pem",
            "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem",
            "ssl": false,
            "client_connections": 8,
            "client_cert":
            "/home/dev/springtail/ca_certs/client_cert.pem",
            "client_key": "/home/dev/springtail/ca_certs/client_key.pem",
            "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem"
        }
    },
    "sys_tbl_mgr": {
        "rpc_config": {
            "server_port": 5053,
            "server_worker_threads": 16,
            "server_cert": "/home/dev/springtail/ca_certs/server_cert.pem",
            "server_key": "/home/dev/springtail/ca_certs/server_key.pem",
            "server_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem",
            "ssl": false,
            "client_connections": 8,
            "client_cert": "/home/dev/springtail/ca_certs/client_cert.pem",
            "client_key": "/home/dev/springtail/ca_certs/client_key.pem",
            "client_trusted": "/home/dev/springtail/ca_certs/ca_cert.pem"
        },
        "cache_size": 4096,
        "roots_shm_cache_size": 10485760,
        "schema_shm_cache_size": 10485760,
        "usertype_shm_cache_size": 4194304
    },
    "proxy": {
        "NOTE": "cert and key required for ssl; modes = [primary|normal|shadow]; pg_shadow for testing",
        "enable_ssl": false,
        "port": 8888,
        "keep_alive_port": 0,
        "threads": 4,
        "cert_path": null,
        "key_path": null,
        "mode": "normal",
        "shadow_log_path": null,
        "pool_size_limit": 0,
        "pool_timeout_limit": 0,
        "pool_expiration_interval_secs": 0,
        "user_mgr_sleep_interval_secs": 15
    },
    "otel": {
        "enabled": false,
        "remote": false,
        "host": "127.0.0.1",
        "port": 4318,
        "metrics_export_interval_millis": 60000,
        "metrics_export_timeout_millis": 30000,
        "remote_log_level": "info"
    },
    "extension_config": {
        "lib_path": "/usr/lib/postgresql/16/lib/",
        "1": {
            "cube": {
                "version": "1.5"
            },
            "hstore": {
                "version": "1.8"
            },
            "pg_trgm": {
                "version": "1.6"
            }
        }
    },
    "aws_users_override": [
        {
            "role": "replication",
            "username": "springtail",
            "password": "springtail",
            "type": "text"
        }, {
            "role": "fdw_superuser",
            "username": "springtail",
            "password": "springtail",
            "type": "text"
        }, {
            "role": "proxy_to_fdw",
            "username": "springtail",
            "password": "springtail",
            "type": "text"
        }, {
            "role": "database",
            "username": "springtail_proxy",
            "password": "springtail_proxy",
            "type": "text"
        }
    ]
}"
"id": "<springtail instance ids>"
"database_ids": "[<list of database ids>]"
"hostname:proxy": "<proxy host name>"
"hostname:ingestion": "<ingestion host name>"

system_settings contains an example configuration.

<instance id>:fdw_ids - this is a set that contains a list of ids for FDWs configured for this database instance.
<instance id>:fdw_dbs - this is a hash table that maps FDW id to a JSON array of database ids supported by this FDW instance:

[<db id 1>, <db id 2>, ..., <db id n>]

<instance id>:admin_console - this is a hash table that daemon processes use to publish their admin IP address and port for HTTP access. The key of this table is in the format <instance key>:<process name> and the value is an IP address and port number that admin process listens on. Here is an example:

"test:pg_log_mgr_daemon": "0.0.0.0:47057"
"test:pg_xid_subscriber_daemon": "0.0.0.0:50361"
"test:pg_ddl_daemon": "0.0.0.0:42641"
"test:proxy": "0.0.0.0:58435"

Here instance key is needed to differentiate between FDW instances because both pg_xid_subscriber_daemon and pg_ddl_daemon exist in each FDW.

<instance id>:coordinator_state - this is a hash table that contains the state of all coordinators. Coordinator is a process that is used to monitor that Springtail processes are alive in each container. It is also responsible for loading up to date libraries and executables into the container. The key is a combination of instance type and instance key, <instance type>:<instance key>, and the value is the state of that specific instance. Here is an example of the data stored in this hash table:

"proxy:default": "running"
"ingestion:default": "running"
"fdw:fdw1": "running"
"fdw:fdw2": "running"

Redis data database

This database contains data structures used by Springtail system at runtime. It contains the following data structures:

<instance id>:queue:ddl:xid:<db id>:<xid> — Queue of DDL operations for a given XID coming out of the LogParser.
<instance id>:queue:index:ddl:xid:<db id>:<xid> — Queue of DDL index operations for a given XID coming out of the GC1 LogParser.
<instance id>:hash:ddl:pc — HASH of pre-commit DDL operations. Stored with a key of <db id>:<xid>.
<instance id>:queue:ddl:fdw:<FDW id> — Queue of DDL changes for the FDW to process coming out of the Committer.
<instance id>:hash:ddl:fdw — Hash set of schema XIDs per FDW. Hash key: <db id>:<fdw id>, value: <schema xid>.
<instance id>:queue:sync_tables:<db id> — Queue for table sync requests; value is the table OID/TID.
<instance id>:string:log_resync:<db id> — Key / value for log mgr resync point; value is <filename>:<offset>.
<instance id>:hash:sync_table_ops:<db id> — Hash table holding the table operations (drop, create and update_roots) for each table sync. Each entry in the hash is stored as a JSON array of JSON objects.
<instance id>:hash:liveness — Hash set for tracking liveness of a daemon. This hash is used by coordinator.
<instance id>:set:db_tables:<db id> — Set holding schema.table names for each database. Value is a list of <quoted(schema)>.<quoted(table)>.
<instance id>:hash:idx:ddl:pc — Hash set of pre-commit DDL operations for index mutations. Stored with a key of <db id>:<xid>.
<instance id>:hash:invalid_tables — Hash set of excluded items for a given <instance id>. Key: <table_oid>, value: <json>.
<instance id>:fdw_min_xids — Hash set of min XIDs per <fdw id>:<db id> for a given <instance id>. Key: <fdw id>:<db id>, value: <min xid>.
<instance id>:fdw_pids — Set of new FDW processes that have not started querying the database yet. Value: <fdw id>:<db id>:<pid>.
<instance id>:set:db_index_xids:<db id> — Set holding index XIDs for each db id. Value: <xid>.
**<instance id>:vacuum_cutoff_xids — Hash of vacuum cutoff XIDs per db_id for a given <instance id>. Key: <db id>, Value: <cutoff xid>.

​Redis

​Redis configuration database

​Redis data database

Redis

Redis configuration database

Redis data database