pg_ext: PostgreSQL Compatibility Library

Overview

The pg_ext library provides a minimal reimplementation of PostgreSQL internal APIs, enabling Springtail to execute functions from PostgreSQL extensions without requiring a full PostgreSQL backend. It creates a compatible runtime environment for extension code that expects PostgreSQL’s data types, memory management, and function call interface. Location: Source files in src/pg_ext/, Headers in include/pg_ext/ Purpose:

Provide PostgreSQL-compatible data structures (Datum, text, arrays, etc.)
Implement function call interface (DirectFunctionCall*())
Emulate PostgreSQL memory context system
Support extension-defined types and operators

Architecture

Core Components

Source files (src/pg_ext/):           Header files (include/pg_ext/):
├── fmgr.cc                           ├── fmgr.hh
├── memory.cc                         ├── memory.hh
├── string.cc                         ├── string.hh
├── array.cc                          ├── array.hh
├── numeric.cc                        ├── numeric.hh
├── date.cc                           ├── date.hh
├── jsonb.cc                          ├── jsonb.hh
├── error.cc                          ├── error.hh
├── node.cc                           ├── node.hh
├── hash.cc                           ├── hash.hh
├── list.cc                           ├── list.hh
├── parser.cc                         ├── parser.hh
├── heaptuple.cc                      ├── heaptuple.hh
├── pqformat.cc                       ├── pqformat.hh
├── bit.cc                            ├── bit.hh
├── float.cc                          ├── float.hh
├── guc.cc                            ├── guc.hh
├── extn_registry.cc                  ├── extn_registry.hh
└── extn_parser.cc                    └── extn_parser.hh

Components:

fmgr: Function manager (DirectFunctionCall* wrappers)
memory: Memory contexts (palloc, pfree, TopMemoryContext)
string: Text type and string utilities
array: PostgreSQL array handling
numeric: Numeric type support
date: Date/time types
jsonb: JSONB type support
error: Error reporting (ereport, elog)
node: PostgreSQL node types
hash: Hash functions
list: PostgreSQL list structures
parser: libpg_query integration
heaptuple: Heap tuple representation
pqformat: Wire protocol formatting
bit: Bit string operations
float: Float type utilities
guc: GUC (configuration) stubs
extn_registry: Extension registry (covered separately)
extn_parser: Extension SQL parsing

Function Manager (fmgr)

Overview

Provides macros and functions for calling PostgreSQL functions with a compatible calling convention. Files: src/pg_ext/fmgr.cc, include/pg_ext/fmgr.hh

Key Macros

LOCAL_FCINFO()

Allocates stack-local FunctionCallInfo structure:

#define LOCAL_FCINFO(name, nargs) \
    FunctionCallInfoBaseData name##data; \
    FunctionCallInfo name = &name##data

Usage:

LOCAL_FCINFO(fcinfo, 2);  // Allocate fcinfo for 2 arguments

InitFunctionCallInfoData()

Initialize FunctionCallInfo structure:

void InitFunctionCallInfoData(FunctionCallInfo fcinfo,
                              FmgrInfo* flinfo,
                              int nargs,
                              Oid collation,
                              void* context,
                              void* resultinfo);

DirectFunctionCall Family

Convenience functions for calling PostgreSQL functions:

DirectFunctionCall1()

Datum DirectFunctionCall1(PGFunction func, Datum arg1);

Implementation:

Datum DirectFunctionCall1(PGFunction func, Datum arg1) {
    LOCAL_FCINFO(fcinfo, 1);
    InitFunctionCallInfoData(*fcinfo, nullptr, 1, 0, nullptr, nullptr);

    fcinfo->args[0].value = arg1;
    fcinfo->args[0].isnull = false;

    return func(fcinfo);
}

Example:

// Call int4in("12345")
PGFunction int4in_func = (PGFunction)get_type_func("int4in");
Datum result = DirectFunctionCall1(int4in_func, CStringGetDatum("12345"));
int32_t value = DatumGetInt32(result);  // value = 12345

DirectFunctionCall2()

Datum DirectFunctionCall2(PGFunction func, Datum arg1, Datum arg2);

Example:

// Call int4add(42, 10)
Datum result = DirectFunctionCall2(int4add_func,
                                  Int32GetDatum(42),
                                  Int32GetDatum(10));
// result = 52

DirectFunctionCall3()

Datum DirectFunctionCall3(PGFunction func, Datum arg1, Datum arg2, Datum arg3);

Collation Variants

For functions that require collation (string operations):

Datum DirectFunctionCall1Coll(PGFunction func, Oid collation, Datum arg1);
Datum DirectFunctionCall2Coll(PGFunction func, Oid collation, Datum arg1, Datum arg2);
Datum DirectFunctionCall3Coll(PGFunction func, Oid collation, Datum arg1, Datum arg2, Datum arg3);
Datum DirectFunctionCall7Coll(PGFunction func, Oid collation, ...);  // For GIN extractQuery

Example:

// Case-insensitive text comparison
Datum result = DirectFunctionCall2Coll(texteq_func,
                                      DEFAULT_COLLATION_OID,
                                      text1_datum,
                                      text2_datum);

Datum Type Conversions

Macros for converting between C types and Datum:

// Integer conversions
#define Int32GetDatum(x)     ((Datum)(x))
#define DatumGetInt32(x)     ((int32_t)(x))
#define Int64GetDatum(x)     ((Datum)(x))
#define DatumGetInt64(x)     ((int64_t)(x))

// Pointer conversions
#define PointerGetDatum(x)   ((Datum)(x))
#define DatumGetPointer(x)   ((void*)(x))

// Boolean conversions
#define BoolGetDatum(x)      ((Datum)((x) ? 1 : 0))
#define DatumGetBool(x)      ((bool)(x))

// Float conversions
#define Float4GetDatum(x)    /* implementation */
#define DatumGetFloat4(x)    /* implementation */
#define Float8GetDatum(x)    /* implementation */
#define DatumGetFloat8(x)    /* implementation */

// String conversions
#define CStringGetDatum(x)   PointerGetDatum(x)
#define DatumGetCString(x)   ((char*)DatumGetPointer(x))

// Object ID
#define ObjectIdGetDatum(x)  ((Datum)(x))
#define DatumGetObjectId(x)  ((Oid)(x))

Memory Management

Overview

PostgreSQL uses a memory context system for allocation tracking and bulk deallocation. pg_ext provides a simplified implementation. Files: src/pg_ext/memory.cc, include/pg_ext/memory.hh

MemoryContext Class

class MemoryContext {
public:
    MemoryContext(MemoryContext* parent,
                  std::string_view name,
                  size_t init_size,
                  size_t max_size);

    void* alloc(size_t size);
    void free(void* ptr);
    void reset();  // Free all allocations
    void delete_context();

private:
    std::string _name;
    size_t _init_size;
    size_t _max_size;
    MemoryContext* _parent;

    std::map<size_t, std::unique_ptr<MemoryBlock>> _blocks;
    std::unordered_map<char*, std::unique_ptr<MemoryBlock>> _large_allocs;
};

Global Memory Context

extern MemoryContext TopMemoryContext;

The global TopMemoryContext is the root of all memory contexts:

// In memory.cc
MemoryContext TopMemoryContext(nullptr, "TopMemoryContext", 8192, 1048576);

Allocation Functions

palloc()

Allocate memory from a memory context:

void* palloc(size_t size);
void* palloc0(size_t size);  // Zero-initialized

Implementation:

void* palloc(size_t size) {
    return TopMemoryContext.alloc(size);
}

void* palloc0(size_t size) {
    void* ptr = palloc(size);
    memset(ptr, 0, size);
    return ptr;
}

Example:

// Allocate 1024 bytes
char* buffer = (char*)palloc(1024);
// Use buffer...
pfree(buffer);

pfree()

Free memory allocated by palloc:

void pfree(void* ptr);

repalloc()

Reallocate memory:

void* repalloc(void* ptr, size_t size);

Memory Blocks

Internal structure for memory management:

struct MemoryBlock {
    size_t size;     // Total size of block
    size_t pos;      // Current position
    char* memory;    // Allocated memory

    MemoryBlock(size_t size);
    size_t remaining() const { return size - pos; }
};

Usage Notes

Extension functions expect palloc() for allocations
Memory contexts are simplified compared to PostgreSQL
No support for memory context switching or hierarchies
All allocations go to TopMemoryContext

String and Text Types

Overview

PostgreSQL’s text type is a variable-length binary string with a 4-byte header. Files: src/pg_ext/string.cc, include/pg_ext/string.hh

Text Structure

struct varlena {
    int32_t vl_len_;  // Length including header (bit 0 = compressed flag)
    char vl_dat[1];   // Flexible array member
};

typedef struct varlena text;

String Utilities

cstring_to_text()

Convert C string to PostgreSQL text:

text* cstring_to_text(const char* str);
text* cstring_to_text_with_len(const char* str, int len);

Implementation:

text* cstring_to_text(const char* str) {
    int len = strlen(str);
    return cstring_to_text_with_len(str, len);
}

text* cstring_to_text_with_len(const char* str, int len) {
    text* result = (text*)palloc(len + VARHDRSZ);
    SET_VARSIZE(result, len + VARHDRSZ);
    memcpy(VARDATA(result), str, len);
    return result;
}

Example:

text* my_text = cstring_to_text("Hello, World!");

cstring_to_text_auto()

Wrapper that handles const correctness:

text* cstring_to_text_auto(const char* str);

text_to_cstring()

Convert text to C string:

char* text_to_cstring(const text* t);

Implementation:

char* text_to_cstring(const text* t) {
    int len = VARSIZE_ANY_EXHDR(t);
    char* result = (char*)palloc(len + 1);
    memcpy(result, VARDATA_ANY(t), len);
    result[len] = '\0';
    return result;
}

Example:

char* cstr = text_to_cstring(my_text);
printf("%s\n", cstr);
pfree(cstr);

Macros

// Get size of varlena (excluding header)
#define VARSIZE_ANY_EXHDR(ptr) (VARSIZE_ANY(ptr) - VARHDRSZ)

// Get data pointer
#define VARDATA_ANY(ptr)       ((char*)(ptr) + VARHDRSZ)
#define VARDATA(ptr)           VARDATA_ANY(ptr)

// Set size (including header)
#define SET_VARSIZE(ptr, len)  (((varlena*)(ptr))->vl_len_ = (len))

// Variable header size
#define VARHDRSZ  sizeof(int32_t)

Array Support

Overview

Support for PostgreSQL’s array type. Files: src/pg_ext/array.cc, include/pg_ext/array.hh

ArrayType Structure

struct ArrayType {
    int32_t vl_len_;        // varlena header
    int ndim;               // Number of dimensions
    int32_t dataoffset;     // Offset to data
    Oid elemtype;           // Element type OID
    int* dim;               // Dimension sizes
    int* lbound;            // Lower bounds
    char* data;             // Element data
};

Array Functions

// Get array element
Datum array_get_element(ArrayType* array, int n);

// Get number of elements
int array_length(ArrayType* array);

// Iterate array
void array_foreach(ArrayType* array, void (*callback)(Datum, void*), void* context);

Numeric Type

Overview

PostgreSQL’s arbitrary-precision numeric type. Files: src/pg_ext/numeric.cc, include/pg_ext/numeric.hh

Numeric Structure

struct NumericData {
    int ndigits;        // Number of digits
    int weight;         // Weight of first digit
    int sign;           // Sign (positive/negative/NaN)
    int dscale;         // Display scale
    int16_t* digits;    // Digit array
};

typedef NumericData* Numeric;

Numeric Functions

// Convert numeric to int64
int64_t numeric_to_int64(Numeric num);

// Convert numeric to double
double numeric_to_double(Numeric num);

// Convert int64 to numeric
Numeric int64_to_numeric(int64_t value);

Date and Time Types

Overview

PostgreSQL date/time types and operations. Files: src/pg_ext/date.cc, include/pg_ext/date.hh

Types

typedef int32_t DateADT;        // Days since 2000-01-01
typedef int64_t Timestamp;      // Microseconds since 2000-01-01 00:00:00
typedef int64_t TimestampTz;    // Timestamp with timezone
typedef int64_t TimeADT;        // Microseconds since midnight
typedef Interval IntervalADT;   // Time interval

struct Interval {
    int64_t time;    // Microseconds
    int32_t day;     // Days
    int32_t month;   // Months
};

Date/Time Functions

// Current timestamp
Timestamp GetCurrentTimestamp();

// Date arithmetic
Timestamp timestamp_add_interval(Timestamp ts, Interval* interval);

// Conversions
DateADT timestamp_to_date(Timestamp ts);
Timestamp date_to_timestamp(DateADT date);

Error Handling

Overview

PostgreSQL error reporting system. Files: src/pg_ext/error.cc, include/pg_ext/error.hh

Error Levels

#define DEBUG5    10
#define DEBUG4    11
#define DEBUG3    12
#define DEBUG2    13
#define DEBUG1    14
#define LOG       15
#define NOTICE    18
#define WARNING   19
#define ERROR     20
#define FATAL     21
#define PANIC     22

ereport Macro

#define ereport(level, rest) \
    do { \
        if (level >= ERROR) { \
            springtail_error_handler rest; \
        } else { \
            springtail_log_handler rest; \
        } \
    } while(0)

Usage:

if (invalid_input) {
    ereport(ERROR,
            (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
             errmsg("Invalid input value: %d", value)));
}

elog Macro

#define elog(level, ...) \
    do { \
        if (level >= ERROR) { \
            throw std::runtime_error(fmt::format(__VA_ARGS__)); \
        } else { \
            LOG_INFO(__VA_ARGS__); \
        } \
    } while(0)

Usage:

elog(ERROR, "Failed to process request: %s", error_msg);

StringInfo (pqformat)

Overview

Dynamic string buffer for building messages. Files: src/pg_ext/pqformat.cc, include/pg_ext/pqformat.hh

StringInfoData Structure

struct StringInfoData {
    char* data;      // Buffer
    int len;         // Current length
    int maxlen;      // Allocated size
    int cursor;      // Read cursor
};

typedef StringInfoData* StringInfo;

StringInfo Functions

// Initialize
void initStringInfo(StringInfoData* str);
void resetStringInfo(StringInfoData* str);

// Append operations
void appendStringInfo(StringInfoData* str, const char* fmt, ...);
void appendStringInfoChar(StringInfoData* str, char ch);
void appendBinaryStringInfo(StringInfoData* str, const char* data, int datalen);
void appendBinaryStringInfoNT(StringInfoData* str, const char* data, int datalen);

// Enlarge buffer
void enlargeStringInfo(StringInfoData* str, int needed);

Example:

StringInfoData buf;
initStringInfo(&buf);

appendStringInfo(&buf, "Value: %d, Name: %s", 42, "test");
appendStringInfoChar(&buf, '\n');

// Use buf.data
pfree(buf.data);

Node Types

Overview

PostgreSQL’s node type system for parse trees and plans. Files: src/pg_ext/node.cc, include/pg_ext/node.hh

Node Structure

struct Node {
    NodeTag type;  // Node type identifier
};

enum NodeTag {
    T_Invalid = 0,
    T_List,
    T_Integer,
    T_String,
    T_SelectStmt,
    // ... many more
};

Node Functions

Node* copyObject(const Node* node);
bool equal(const Node* a, const Node* b);
char* nodeToString(const Node* node);

List Support

Overview

PostgreSQL’s doubly-linked list implementation. Files: src/pg_ext/list.cc, include/pg_ext/list.hh

List Structure

struct ListCell {
    union {
        void* ptr_value;
        int int_value;
        Oid oid_value;
    } data;
    ListCell* next;
};

struct List {
    NodeTag type;       // T_List
    int length;
    ListCell* head;
    ListCell* tail;
};

List Functions

// Create
List* list_make1(void* datum1);
List* list_make2(void* datum1, void* datum2);
List* lappend(List* list, void* datum);
List* lcons(void* datum, List* list);

// Access
void* linitial(const List* list);
void* lsecond(const List* list);
void* llast(const List* list);

// Iteration
#define foreach(cell, list) \
    for ((cell) = list_head(list); (cell) != NULL; (cell) = lnext(list, cell))

Example:

List* mylist = NIL;
mylist = lappend(mylist, some_pointer);
mylist = lappend(mylist, another_pointer);

ListCell* cell;
foreach(cell, mylist) {
    void* item = lfirst(cell);
    // Process item...
}

Integration with Springtail

Type System Integration

Extension types are stored in Springtail’s type system through create_usertype():

// In pg_copy_table.cc
PgMsgUserType msg;
msg.oid = enum_type_oid;
msg.name = extn_type_name;
msg.type = constant::USER_TYPE_EXTENSION;

server->create_usertype(db_id, {xid, 0}, msg);

Comparator Integration

Extension operators are used in Springtail comparisons:

// In field.cc
ExtensionContext ctx;
ctx.type_oid = field->type_oid;
ctx.op_str = operator_symbol;

return PgExtnRegistry::comparator_func(&ctx, lhs_binary, rhs_binary);

Limitations

Not Implemented

PostgreSQL features NOT implemented in pg_ext:

Transaction Management: No MVCC, snapshot isolation
Catalog Access: No pg_catalog queries
SPI (Server Programming Interface): No SQL execution from C functions
Triggers: No trigger mechanism
Full Memory Context Tree: Simplified single-level context
Signal Handling: No PostgreSQL signal handlers
Shared Memory: No PostgreSQL shared memory segments
TOAST: No out-of-line storage for large values
Vacuum: No MVCC cleanup
Write-Ahead Log: No WAL integration

Compatibility Notes

Extension functions must not rely on PostgreSQL backend state
SPI-using extensions will NOT work
Trigger-based extensions will NOT work
Extensions accessing pg_catalog may fail
Extension functions assuming PostgreSQL memory semantics may leak memory

Adding New pg_ext Support

To add support for a new PostgreSQL feature:

Identify Required APIs: Determine what PostgreSQL functions the extension uses
Create Stub Implementation: Add stubs in appropriate src/pg_ext/*.cc file
Add Type Definitions: Add necessary structs/types in include/pg_ext/*.hh
Implement Core Logic: Implement minimal functionality needed by extensions
Test with Extension: Verify extension functions work correctly

Example: Adding Support for Range Types

Create header file include/pg_ext/range.hh:

// include/pg_ext/range.hh
struct RangeType {
    Oid rangetypid;
    char flags;
    Datum lower;
    Datum upper;
};

Datum range_in(const char* str, Oid rangetypid);
char* range_out(RangeType* range);
bool range_contains(RangeType* range, Datum element);

Create source file src/pg_ext/range.cc:

// src/pg_ext/range.cc
#include <pg_ext/range.hh>

Datum range_in(const char* str, Oid rangetypid) {
    // Parse range string "[lower,upper)"
    // Create RangeType structure
    // Return as Datum
}

// ... implement other functions

​pg_ext: PostgreSQL Compatibility Library

​Overview

​Architecture

​Core Components

​Function Manager (fmgr)

​Overview

​Key Macros

​LOCAL_FCINFO()

​InitFunctionCallInfoData()

​DirectFunctionCall Family

​DirectFunctionCall1()

​DirectFunctionCall2()

​DirectFunctionCall3()

​Collation Variants

​Datum Type Conversions

​Memory Management

​Overview

​MemoryContext Class

​Global Memory Context

​Allocation Functions

​palloc()

​pfree()

​repalloc()

​Memory Blocks

​Usage Notes

​String and Text Types

​Overview

​Text Structure

​String Utilities

​cstring_to_text()

​cstring_to_text_auto()

​text_to_cstring()

​Macros

​Array Support

​Overview

​ArrayType Structure

​Array Functions

​Numeric Type

​Overview

​Numeric Structure

​Numeric Functions

​Date and Time Types

​Overview

​Types

​Date/Time Functions

​Error Handling

​Overview

​Error Levels

​ereport Macro

​elog Macro

​StringInfo (pqformat)

​Overview

​StringInfoData Structure

​StringInfo Functions

​Node Types

​Overview

​Node Structure

​Node Functions

​List Support

​Overview

​List Structure

​List Functions

​Integration with Springtail

​Type System Integration

​Comparator Integration

​Limitations

​Not Implemented

​Compatibility Notes

​Adding New pg_ext Support

pg_ext: PostgreSQL Compatibility Library

Overview

Architecture

Core Components

Function Manager (fmgr)

Overview

Key Macros

LOCAL_FCINFO()

InitFunctionCallInfoData()

DirectFunctionCall Family

DirectFunctionCall1()

DirectFunctionCall2()

DirectFunctionCall3()

Collation Variants

Datum Type Conversions

Memory Management

Overview

MemoryContext Class

Global Memory Context

Allocation Functions

palloc()

pfree()

repalloc()

Memory Blocks

Usage Notes

String and Text Types

Overview

Text Structure

String Utilities

cstring_to_text()

cstring_to_text_auto()

text_to_cstring()

Macros

Array Support

Overview

ArrayType Structure

Array Functions

Numeric Type

Overview

Numeric Structure

Numeric Functions

Date and Time Types

Overview

Types

Date/Time Functions

Error Handling

Overview

Error Levels

ereport Macro

elog Macro

StringInfo (pqformat)

Overview

StringInfoData Structure

StringInfo Functions

Node Types

Overview

Node Structure

Node Functions

List Support

Overview

List Structure

List Functions

Integration with Springtail

Type System Integration

Comparator Integration

Limitations

Not Implemented

Compatibility Notes

Adding New pg_ext Support