Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.springtail.io/llms.txt

Use this file to discover all available pages before exploring further.

​
Local Cluster Deployment

This section provides a step-by-step guide to deploying the system in a local cluster environment using Docker. This setup is ideal for development, testing, and small-scale production environments.

​
Prerequisites

  • Docker
  • Docker Compose
  • Python3
  • AWS CLI
  • Base container image for Springtail Service *
    • More on this in the “Building a Base Image” section below.
  • Check out the springtail repository to your local machine.

​
Cluster Building Flow

​
Building a Base Image

The Springtail service need a set of dependencies to run. These dependencies are packaged into a base container image, which is then used to build the service containers. The local cluster will assume the existence of such base image. All the steps below assume you are at the root of springtail repository. In the local-cluster/env directory, there is a env.setup file that defines the name of the base image to use. 
IMAGE=local-cluster-img:latest
The base image (along with the controller and custom PostgreSQL images) is built automatically by ./cluster up if it is not already present, so you normally do not need to build it by hand. To build or rebuild it manually, run the following from the repository root: 
$ docker build -t local-cluster-img:latest -f docker/Dockerfile.local-cluster .

​
Working with Local Cluster

​
Launching the Cluster

  1. First build a springtail package. You can reuse an existing package if you have one though.
$ ./cluster build-package <out-dir>
  1. Start the local cluster. You can specify the full package path on your local machine (the tarball file generated from the previous step). This step also builds any missing images (base, controller, and custom PostgreSQL).
$ ./cluster up <full-package-path>
To disable SSL for inter-service connections, append the --disable-ssl flag: 
$ ./cluster up <full-package-path> --disable-ssl
  1. It takes a few minutes (usually less than 2) to start the cluster. You can check the status by:
$ ./cluster status

​
Managing / Interacting with the Cluster

Once all containers are up and running, you can shell into any container. For example: 
$ ./cluster sh <service>
The service names are:
  • proxy
  • ingestion
  • fdw1
  • fdw2
  • controller
Other useful commands: 
$ ./cluster logs <service>      # show logs for a service
$ ./cluster restart <service>   # restart a service container
$ ./cluster ls                  # list all services
By default, the springtail-coordinator service is not started. You can start it by (inside a container): 
$ systemctl start springtail-coordinator

​
Host Ports

The following services are exposed on the host:
ServiceHost Port
Primary DB15432
Redis16379
Proxy55432
FDW 145432
FDW 245433
AWS Mock (Moto)29999
Controller API19824
Connect to the proxy from the host with any PostgreSQL client: 
$ psql -h localhost -p 55432 -U postgres

​
Tearing Down the Cluster

To tear down the Springtail containers, 
$ ./cluster down
To stop a single service and its dependencies, pass its name: 
$ ./cluster down proxy
To tear down everything including the supporting services (Redis, local AWS mock, local DB), 
$ ./cluster down all

​
Internals

This section explains what happens under the hood when you run the above commands.

​
Building the Package

The ./cluster build-package <out-dir> command does the following: Start a temporary build container from the image identified by BASE_BUILDER_IMAGE_TAG in the cluster script, which resides in the DevSupport ECR repo. If the BASE_BUILDER_IMAGE_TAG is not available locally it will try to pull from the remote repo, thus requiring AWS SSO configured. It saves the package into the <out-dir> directory on the host machine. The package is a tarball file named springtail-<date-version>-<system-settings-gitsha>.tar.gz.

​
Starting the Cluster

The ./cluster up <full-package-path> command does the following: It sets the <full-package-path> to the PACKAGE_FILE_NAME environment variable, making it available to the build process of all the containers. Then it starts the supporting local services, like the AWS mock, Redis, and a local PostgreSQL database serving as primary DB. Then it uploads the package to the local mock S3 service. Then it launches a bootstrap container that sets up shared environment files, secrets, and Redis configuration. Specifically, the bootstrap process will add new env vars into ./env/.env file. Finally, it launches the main service containers: proxy, ingestion, and fdw, picking up all the env files in the ./env directory.

​
Service Initializations

​
local-cluster-bootstrap

This runs before any service container is started. It does the following:
  1. Set up shared environment files in the ./env directory, which will be picked up by all service containers;
  2. Set up Redis configuration, creating a default user and password, and saving the config to ./env/redis.env;
  3. Set up secrets, creating a self-signed certificate for HTTPS, and saving the secrets to ./env/secrets.env;
In summary, this is setting up the supporting environment for the service containers to run.

​
”CloudInit” Script

For each service container, upon startup, it runs a init-services script, managed by supervisord, that does the following:
  1. Download S3 package and extract it, moving the coordinator to the /opt/springtail directory as the bootstrap coordinator;
  2. Make sure all necessary directories exist, creating them if needed;
  3. For fdw specifically, it triggers an Ansible script to customize the PostgreSQL; This is akin to the EC2’s userdata script (CloudInit), which is used to initialize a container.