ChristianLempa_Boilerplates/AGENTS.md

AGENTS.md

Guidance for AI Agents working with this repository.

Project Overview

A sophisticated collection of infrastructure templates (boilerplates) with a Python CLI for management. Supports Terraform, Docker, Ansible, Kubernetes, etc. Built with Typer (CLI) and Jinja2 (templating).

Development Setup

Running and Testing

# Run the CLI application
python3 -m cli
# Debugging and Testing commands
python3 -m cli --log-level DEBUG compose list

Linting and Formatting

Should always happen before pushing anything to the repository.

• Use yamllint for YAML files and ruff for Python code.

Project Management and Git

The project is stored in a public GitHub Repository, use issues, and branches for features/bugfixes and open PRs for merging.

Naming Conventions and Best-Practices:

• Branches, PRs: feature/2314-add-feature, problem/1249-fix-bug
• Issues should have clear titles and descriptions, link related issues/PRs, and have appropriate labels like (problem, feature, discussion, question).
• Commit messages should be clear and concise, following the format: type(scope): subject (e.g., fix(compose): correct variable parsing).

Architecture

File Structure

• cli/ - Python CLI application source code
  • cli/core/ - Core Components of the CLI application
  • cli/modules/ - Modules implementing variable specs and technology-specific functions
  • cli/__main__.py - CLI entry point, auto-discovers modules and registers commands
• library/ - Template collections organized by module
  • library/ansible/ - Ansible playbooks and configurations
  • library/compose/ - Docker Compose configurations
  • library/docker/ - Docker templates
  • library/kubernetes/ - Kubernetes deployments
  • library/packer/ - Packer templates
  • library/terraform/ - OpenTofu/Terraform templates and examples
• archetypes/ - Testing tool for template snippets (archetype development)
  • archetypes/__init__.py - Package initialization
  • archetypes/__main__.py - CLI tool entry point
  • archetypes/<module>/ - Module-specific archetype snippets (e.g., archetypes/compose/)

Core Components

• cli/core/collection.py - VariableCollection class (manages sections and variables)
  • Key Attributes: _sections (dict of VariableSection objects), _variable_map (flat lookup dict)
  • Key Methods: get_satisfied_values() (returns enabled variables), apply_defaults(), sort_sections()
• cli/core/config.py - Configuration management (loading, saving, validation)
• cli/core/display.py - Centralized CLI output rendering (Always use this to display output - never print directly)
• cli/core/exceptions.py - Custom exceptions for error handling (Always use this for raising errors)
• cli/core/library.py - LibraryManager for template discovery from git-synced libraries and static file paths
• cli/core/module.py - Abstract base class for modules (defines standard commands)
• cli/core/prompt.py - Interactive CLI prompts using rich library
• cli/core/registry.py - Central registry for module classes (auto-discovers modules)
• cli/core/repo.py - Repository management for syncing git-based template libraries
• cli/core/section.py - VariableSection class (stores section metadata and variables)
  • Key Attributes: key, title, toggle, required, needs, variables (dict of Variable objects)
• cli/core/template.py - Template Class for parsing, managing and rendering templates
• cli/core/variable.py - Variable class (stores variable metadata and values)
  • Key Attributes: name, type, value (stores default or current value), description, sensitive, needs
  • Note: Default values are stored in value attribute, NOT in a separate default attribute
• cli/core/validators.py - Semantic validators for template content (Docker Compose, YAML, etc.)
• cli/core/version.py - Version comparison utilities for semantic versioning

Modules

Module Structure: Modules can be either single files or packages:

• Single file: cli/modules/modulename.py (for simple modules)
• Package: cli/modules/modulename/ with __init__.py (for multi-schema modules)

Creating Modules:

• Subclass Module from cli/core/module.py
• Define name, description, and schema_version class attributes
• For multi-schema modules: organize specs in separate files (e.g., spec_v1_0.py, spec_v1_1.py)
• Call registry.register(YourModule) at module bottom
• Auto-discovered and registered at CLI startup

Module Spec: Module-wide variable specification defining defaults for all templates of that kind.

Important: The spec variable is an OrderedDict (or regular dict), NOT a VariableCollection object. It's converted to VariableCollection when needed.

Example:

from collections import OrderedDict

# Spec is a dict/OrderedDict, not a VariableCollection
spec = OrderedDict({
  "general": {
    "title": "General",
    "vars": {
      "common_var": {
        "type": "str",
        "default": "value",
        "description": "A common variable"
      }
    }
  },
  "networking": {
    "title": "Network",
    "toggle": "net_enabled",
    "vars": {...}
  }
})

# To use the spec, convert it to VariableCollection:
from cli.core.collection import VariableCollection
variable_collection = VariableCollection(spec)

Multi-Schema Modules: For modules supporting multiple schema versions, use package structure:

cli/modules/compose/
  __init__.py          # Module class, loads appropriate spec
  spec_v1_0.py         # Schema 1.0 specification
  spec_v1_1.py         # Schema 1.1 specification

Existing Modules:

• cli/modules/compose/ - Docker Compose package with schema 1.0 and 1.1 support
  • spec_v1_0.py - Basic compose spec
  • spec_v1_1.py - Extended with network_mode, swarm support

(Work in Progress): terraform, docker, ansible, kubernetes, packer modules

LibraryManager

• Loads libraries from config file
• Stores Git Libraries under: ~/.config/boilerplates/libraries/{name}/
• Uses sparse-checkout to clone only template directories for git-based libraries (avoiding unnecessary files)
• Supports two library types: git (synced from repos) and static (local directories)
• Priority determined by config order (first = highest)

Library Types:

• git: Requires url, branch, directory fields
• static: Requires path field (absolute or relative to config)

Duplicate Handling:

• Within same library: Raises DuplicateTemplateError
• Across libraries: Uses qualified IDs (e.g., alloy.default, alloy.local)
• Simple IDs use priority: compose show alloy loads from first library
• Qualified IDs target specific library: compose show alloy.local

Config Example:

libraries:
  - name: default       # Highest priority (checked first)
    type: git
    url: https://github.com/user/templates.git
    branch: main
    directory: library
  - name: local         # Lower priority
    type: static
    path: ~/my-templates
    url: ''             # Backward compatibility fields
    branch: main
    directory: .

Note: Static libraries include dummy url/branch/directory fields for backward compatibility with older CLI versions.

ConfigManager

• User Config stored in ~/.config/boilerplates/config.yaml

DisplayManager and IconManager

External code should NEVER directly call IconManager or console.print, instead always use DisplayManager methods.

• DisplayManager provides a centralized interface for ALL CLI output rendering (Use display_*** methods from DisplayManager for ALL output)
• IconManager provides Nerd Font icons internally for DisplayManager, don't use Emojis or direct console access

Templates

Templates are directory-based. Each template is a directory containing all the necessary files and subdirectories for the boilerplate.

Requires template.yaml or template.yml with metadata and variables:

---
kind: compose
schema: "1.0"  # Optional: Defaults to 1.0 if not specified
metadata:
  name: My Nginx Template
  description: >
    A template for a simple Nginx service.


    Project: https://...

    Source: https://

    Documentation: https://
  version: 0.1.0
  author: Christian Lempa
  date: '2024-10-01'
spec:
  general:
    vars:
      nginx_version:
        type: string
        description: The Nginx version to use.
        default: latest

Template Schema Versioning

Templates and modules use schema versioning to ensure compatibility. Each module defines a supported schema version, and templates declare which schema version they use.

---
kind: compose
schema: "1.0"  # Defaults to 1.0 if not specified
metadata:
  name: My Template
  version: 1.0.0
  # ... other metadata fields
spec:
  # ... variable specifications

How It Works:

• Module Schema Version: Each module defines schema_version (e.g., "1.1")
• Module Spec Loading: Modules load appropriate spec based on template's schema version
• Template Schema Version: Each template declares schema at the top level (defaults to "1.0")
• Compatibility Check: Template schema ≤ Module schema → Compatible
• Incompatibility: Template schema > Module schema → IncompatibleSchemaVersionError

Behavior:

• Templates without schema field default to "1.0" (backward compatible)
• Old templates (schema 1.0) work with newer modules (schema 1.1)
• New templates (schema 1.2) fail on older modules (schema 1.1) with clear error
• Version comparison uses 2-level versioning (major.minor format)

When to Use:

• Increment module schema version when adding new features (new variable types, sections, etc.)
• Set template schema when using features from a specific schema
• Example: Template using new variable type added in schema 1.1 should set schema: "1.1"

Single-File Module Example:

class SimpleModule(Module):
  name = "simple"
  description = "Simple module"
  schema_version = "1.0"
  spec = VariableCollection.from_dict({...})  # Single spec

Multi-Schema Module Example:

# cli/modules/compose/__init__.py
class ComposeModule(Module):
  name = "compose"
  description = "Manage Docker Compose configurations"
  schema_version = "1.1"  # Highest schema version supported
  
  def get_spec(self, template_schema: str) -> VariableCollection:
    """Load spec based on template schema version."""
    if template_schema == "1.0":
      from .spec_v1_0 import get_spec
    elif template_schema == "1.1":
      from .spec_v1_1 import get_spec
    return get_spec()

Version Management:

• CLI version is defined in cli/__init__.py as __version__
• pyproject.toml version must match __version__ for releases
• GitHub release workflow validates version consistency

Template Files

• Jinja2 Templates (.j2): Rendered by Jinja2, .j2 extension removed in output. Support {% include %} and {% import %}.
• Static Files: Non-.j2 files copied as-is.
• Sanitization: Auto-sanitized (single blank lines, no leading blanks, trimmed whitespace, single trailing newline).

Variables

Precedence (lowest to highest):

1. Module spec (defaults for all templates of that kind)
2. Template spec (overrides module defaults)
3. User config.yaml (overrides template and module defaults)
4. CLI --var (highest priority)

Variable Types:

• str (default), int, float, bool
• email - Email validation with regex
• url - URL validation (requires scheme and host)
• hostname - Hostname/domain validation
• enum - Choice from options list

Variable Properties:

• sensitive: true - Masked in prompts/display (e.g., passwords)
• autogenerated: true - Auto-generates value if empty (shows *auto placeholder)
• default - Default value
• description - Variable description
• prompt - Custom prompt text (overrides description)
• extra - Additional help text
• options - List of valid values (for enum type)

Section Features:

• Required Sections: Mark with required: true (general is implicit). Users must provide all values.
• Toggle Settings: Conditional sections via toggle: "bool_var_name". If false, section is skipped.
• Dependencies: Use needs: "section_name" or needs: ["sec1", "sec2"]. Dependent sections only shown when dependencies are enabled. Auto-validated (detects circular/missing/self dependencies). Topologically sorted.

Example Section with Dependencies:

spec:
  traefik:
    title: Traefik
    required: false
    toggle: traefik_enabled
    vars:
      traefik_enabled:
        type: bool
        default: false
      traefik_host:
        type: hostname
  
  traefik_tls:
    title: Traefik TLS/SSL
    needs: traefik
    toggle: traefik_tls_enabled
    vars:
      traefik_tls_enabled:
        type: bool
        default: true
      traefik_tls_certresolver:
        type: str
        sensitive: false
        default: myresolver

Validation

Jinja2 Validation:

• Templates validated for Jinja2 syntax errors during load
• Checks for undefined variables (variables used but not declared in spec)
• Built into Template class

Semantic Validation:

• Validator registry system in cli/core/validators.py
• Extensible: ContentValidator abstract base class
• Built-in validators: DockerComposeValidator, YAMLValidator
• Validates rendered output (YAML structure, Docker Compose schema, etc.)
• Triggered via compose validate command with --semantic flag (enabled by default)

Prompt

Uses rich library for interactive prompts. Supports:

• Text input
• Password input (masked, for sensitive: true variables)
• Selection from list (single/multiple)
• Confirmation (yes/no)
• Default values
• Autogenerated variables (show *auto placeholder, generate on render)

To skip the prompt use the --no-interactive flag, which will use defaults or empty values.

Commands

Standard Module Commands (auto-registered for all modules):

• list - List all templates
• search <query> - Search templates by ID
• show <id> - Show template details
• generate <id> [directory] - Generate from template (supports --dry-run, --var, --no-interactive)
• validate [id] - Validate templates (Jinja2 + semantic)
• defaults - Manage config defaults (get, set, rm, clear, list)

Core Commands:

• repo sync - Sync git-based libraries
• repo list - List configured libraries

Archetypes Testing Tool

The archetypes package provides a testing tool for developing and testing individual template snippets (Jinja2 files) without needing a full template directory structure.

Purpose

Archetypes are template "snippets" or "parts" that can be tested in isolation. This is useful for:

• Developing specific sections of templates (e.g., network configurations, volume mounts)
• Testing Jinja2 logic with different variable combinations
• Validating template rendering before integrating into full templates

Usage

# Run the archetypes tool
python3 -m archetypes

# List all archetypes for a module
python3 -m archetypes compose list

# Show details of an archetype (displays variables and content)
python3 -m archetypes compose show network-v1

# Preview generated output (always in preview mode - never writes files)
python3 -m archetypes compose generate network-v1

# Preview with variable overrides
python3 -m archetypes compose generate network-v1 \
  --var network_mode=macvlan \
  --var network_macvlan_ipv4_address=192.168.1.100

# Preview with reference directory (for context only - no files written)
python3 -m archetypes compose generate network-v1 /tmp/output --var network_mode=host

Structure

archetypes/
  __init__.py           # Package initialization
  __main__.py           # CLI tool (auto-discovers modules)
  compose/              # Module-specific archetypes
    network-v1.j2       # Archetype snippet (just a .j2 file)
    volumes-v1.j2       # Another archetype
  terraform/            # Another module's archetypes
    vpc.j2

Key Features

• Auto-discovers modules: Scans archetypes/ for subdirectories (module names)
• Reuses CLI components: Imports actual CLI classes (Template, VariableCollection, DisplayManager) for identical behavior
• Loads module specs: Pulls variable specifications from cli/modules/<module>/spec_v*.py for defaults
• Full variable context: Provides ALL variables with defaults (not just satisfied ones) for complete rendering
• Three commands: list, show, generate
• Testing only: The generate command NEVER writes files - it always shows preview output only

Implementation Details

How it works:

1. Module discovery: Finds subdirectories in archetypes/ (e.g., compose)
2. For each module, creates a Typer sub-app with list/show/generate commands
3. Archetype files are simple .j2 files (no template.yaml needed)
4. Variable defaults come from module spec: cli/modules/<module>/spec_v*.py
5. Rendering uses Jinja2 with full variable context from spec

ArchetypeTemplate class:

• Simplified template wrapper for single .j2 files
• Loads module spec and converts to VariableCollection
• Extracts ALL variables (not just satisfied) from spec sections
• Merges user overrides (--var) on top of spec defaults
• Renders using Jinja2 FileSystemLoader

Variable defaults source:

# Defaults come from module spec files
from cli.modules.compose import spec  # OrderedDict with variable definitions
vc = VariableCollection(spec)         # Convert to VariableCollection

# Extract all variables with their default values
for section_name, section in vc._sections.items():
    for var_name, var in section.variables.items():
        if var.value is not None:  # var.value stores the default
            render_context[var_name] = var.value