Local Cluster Deployment

This section provides a step-by-step guide to deploying the system in a local cluster environment using Docker. This setup is ideal for development, testing, and small-scale production environments.

Prerequisites

Docker
Docker Compose
Python3
AWS CLI
Base container image for Springtail Service *
- More on this in the “Building a Base Image” section below.
Check out the springtail repository to your local machine.

Cluster Building Flow

Building a Base Image

The Springtail service need a set of dependencies to run. These dependencies are packaged into a base container image, which is then used to build the service containers. The local cluster will assume the existence of such base image. All the steps below assume you are at the root of springtail repository. In the local-cluster/env directory, there is a env.setup file that defines the name of the base image to use.

IMAGE=local-cluster-img:latest

The base image (along with the controller and custom PostgreSQL images) is built automatically by ./cluster up if it is not already present, so you normally do not need to build it by hand. To build or rebuild it manually, run the following from the repository root:

$ docker build -t local-cluster-img:latest -f docker/Dockerfile.local-cluster .

Working with Local Cluster

Launching the Cluster

First build a springtail package. You can reuse an existing package if you have one though.

$ ./cluster build-package <out-dir>

Start the local cluster. You can specify the full package path on your local machine (the tarball file generated from the previous step). This step also builds any missing images (base, controller, and custom PostgreSQL).

$ ./cluster up <full-package-path>

To disable SSL for inter-service connections, append the --disable-ssl flag:

$ ./cluster up <full-package-path> --disable-ssl

It takes a few minutes (usually less than 2) to start the cluster. You can check the status by:

$ ./cluster status

Managing / Interacting with the Cluster

Once all containers are up and running, you can shell into any container. For example:

$ ./cluster sh <service>

The service names are:

proxy
ingestion
fdw1
fdw2
controller

Other useful commands:

$ ./cluster logs <service>      # show logs for a service
$ ./cluster restart <service>   # restart a service container
$ ./cluster ls                  # list all services

By default, the springtail-coordinator service is not started. You can start it by (inside a container):

$ systemctl start springtail-coordinator

Host Ports

The following services are exposed on the host:

Service	Host Port
Primary DB	15432
Redis	16379
Proxy	55432
FDW 1	45432
FDW 2	45433
AWS Mock (Moto)	29999
Controller API	19824

Connect to the proxy from the host with any PostgreSQL client:

$ psql -h localhost -p 55432 -U postgres

Tearing Down the Cluster

To tear down the Springtail containers,

$ ./cluster down

To stop a single service and its dependencies, pass its name:

$ ./cluster down proxy

To tear down everything including the supporting services (Redis, local AWS mock, local DB),

$ ./cluster down all

Internals

This section explains what happens under the hood when you run the above commands.

Building the Package

The ./cluster build-package <out-dir> command does the following: Start a temporary build container from the image identified by BASE_BUILDER_IMAGE_TAG in the cluster script, which resides in the DevSupport ECR repo. If the BASE_BUILDER_IMAGE_TAG is not available locally it will try to pull from the remote repo, thus requiring AWS SSO configured. It saves the package into the <out-dir> directory on the host machine. The package is a tarball file named springtail-<date-version>-<system-settings-gitsha>.tar.gz.

Starting the Cluster

The ./cluster up <full-package-path> command does the following: It sets the <full-package-path> to the PACKAGE_FILE_NAME environment variable, making it available to the build process of all the containers. Then it starts the supporting local services, like the AWS mock, Redis, and a local PostgreSQL database serving as primary DB. Then it uploads the package to the local mock S3 service. Then it launches a bootstrap container that sets up shared environment files, secrets, and Redis configuration. Specifically, the bootstrap process will add new env vars into ./env/.env file. Finally, it launches the main service containers: proxy, ingestion, and fdw, picking up all the env files in the ./env directory.

Service Initializations

local-cluster-bootstrap

This runs before any service container is started. It does the following:

Set up shared environment files in the ./env directory, which will be picked up by all service containers;
Set up Redis configuration, creating a default user and password, and saving the config to ./env/redis.env;
Set up secrets, creating a self-signed certificate for HTTPS, and saving the secrets to ./env/secrets.env;

In summary, this is setting up the supporting environment for the service containers to run.

”CloudInit” Script

For each service container, upon startup, it runs a init-services script, managed by supervisord, that does the following:

Download S3 package and extract it, moving the coordinator to the /opt/springtail directory as the bootstrap coordinator;
Make sure all necessary directories exist, creating them if needed;
For fdw specifically, it triggers an Ansible script to customize the PostgreSQL; This is akin to the EC2’s userdata script (CloudInit), which is used to initialize a container.

​Local Cluster Deployment

​Prerequisites

​Cluster Building Flow

​Building a Base Image

​Working with Local Cluster

​Launching the Cluster

​Managing / Interacting with the Cluster

​Host Ports

​Tearing Down the Cluster

​Internals

​Building the Package

​Starting the Cluster

​Service Initializations

​local-cluster-bootstrap

​”CloudInit” Script