ChristianLempa_Boilerplates/AGENTS.md

AGENTS.md

Guidance for AI Agents working with this repository.

Project Overview

A sophisticated collection of infrastructure templates (boilerplates) with a Python CLI for management. Supports Terraform, Docker, Ansible, Kubernetes, etc. Built with Typer (CLI) and Jinja2 (templating).

Development Setup

Running and Testing

# Run the CLI application
python3 -m cli
# Debugging and Testing commands
python3 -m cli --log-level DEBUG compose list

Linting and Formatting

Should always happen before pushing anything to the repository.

• Use yamllint for YAML files
• Use ruff for Python code:
  • ruff check --fix . - Check and auto-fix linting errors (including unused imports)
  • ruff format . - Format code according to style guidelines
  • Both commands must be run before committing

Project Management and Git

The project is stored in a public GitHub Repository, use issues, and branches for features/bugfixes and open PRs for merging.

Naming Conventions and Best-Practices:

• Branches, PRs: feature/2314-add-feature, problem/1249-fix-bug
• Issues should have clear titles and descriptions, link related issues/PRs, and have appropriate labels like (problem, feature, discussion, question).
• Commit messages should be clear and concise, following the format: type(scope): subject (e.g., fix(compose): correct variable parsing).

Architecture

File Structure

• cli/ - Python CLI application source code
  • cli/core/ - Core Components of the CLI application
  • cli/modules/ - Modules implementing variable specs and technology-specific functions
  • cli/__main__.py - CLI entry point, auto-discovers modules and registers commands
• library/ - Template collections organized by module
  • library/ansible/ - Ansible playbooks and configurations
  • library/compose/ - Docker Compose configurations
  • library/docker/ - Docker templates
  • library/kubernetes/ - Kubernetes deployments
  • library/packer/ - Packer templates
  • library/terraform/ - OpenTofu/Terraform templates and examples
• archetypes/ - Testing tool for template snippets (archetype development)
  • archetypes/__init__.py - Package initialization
  • archetypes/__main__.py - CLI tool entry point
  • archetypes/<module>/ - Module-specific archetype snippets (e.g., archetypes/compose/)

Core Components

• cli/core/collection.py - VariableCollection class (manages sections and variables)
  • Key Attributes: _sections (dict of VariableSection objects), _variable_map (flat lookup dict)
  • Key Methods: get_satisfied_values() (returns enabled variables), apply_defaults(), sort_sections()
• cli/core/config.py - Configuration management (loading, saving, validation)
• cli/core/display/ - Centralized CLI output rendering package (Always use DisplayManager - never print directly)
  • __init__.py - Package exports (DisplayManager, DisplaySettings, IconManager)
  • display_manager.py - Main DisplayManager facade
  • display_settings.py - DisplaySettings configuration class
  • icon_manager.py - IconManager for Nerd Font icons
  • variable_display.py - VariableDisplayManager for variable rendering
  • template_display.py - TemplateDisplayManager for template display
  • status_display.py - StatusDisplayManager for status messages
  • table_display.py - TableDisplayManager for table rendering
• cli/core/exceptions.py - Custom exceptions for error handling (Always use this for raising errors)
• cli/core/library.py - LibraryManager for template discovery from git-synced libraries and static file paths
• cli/core/module.py - Abstract base class for modules (defines standard commands)
• cli/core/prompt.py - Interactive CLI prompts using rich library
• cli/core/registry.py - Central registry for module classes (auto-discovers modules)
• cli/core/repo.py - Repository management for syncing git-based template libraries
• cli/core/section.py - VariableSection class (stores section metadata and variables)
  • Key Attributes: key, title, toggle, required, needs, variables (dict of Variable objects)
• cli/core/template.py - Template Class for parsing, managing and rendering templates
• cli/core/variable.py - Variable class (stores variable metadata and values)
  • Key Attributes: name, type, value (stores default or current value), description, sensitive, needs
  • Note: Default values are stored in value attribute, NOT in a separate default attribute
• cli/core/validators.py - Semantic validators for template content (Docker Compose, YAML, etc.)
• cli/core/version.py - Version comparison utilities for semantic versioning

Modules

Module Structure: Modules can be either single files or packages:

• Single file: cli/modules/modulename.py (for simple modules)
• Package: cli/modules/modulename/ with __init__.py (for multi-schema modules)

Creating Modules:

• Subclass Module from cli/core/module.py
• Define name, description, and schema_version class attributes
• For multi-schema modules: organize specs in separate files (e.g., spec_v1_0.py, spec_v1_1.py)
• Call registry.register(YourModule) at module bottom
• Auto-discovered and registered at CLI startup

Module Spec: Module-wide variable specification defining defaults for all templates of that kind.

Important: The spec variable is an OrderedDict (or regular dict), NOT a VariableCollection object. It's converted to VariableCollection when needed.

Example:

from collections import OrderedDict

# Spec is a dict/OrderedDict, not a VariableCollection
spec = OrderedDict({
  "general": {
    "title": "General",
    "vars": {
      "common_var": {
        "type": "str",
        "default": "value",
        "description": "A common variable"
      }
    }
  },
  "networking": {
    "title": "Network",
    "toggle": "net_enabled",
    "vars": {...}
  }
})

# To use the spec, convert it to VariableCollection:
from cli.core.collection import VariableCollection
variable_collection = VariableCollection(spec)

Multi-Schema Modules: For modules supporting multiple schema versions, use package structure:

cli/modules/compose/
  __init__.py          # Module class, loads appropriate spec
  spec_v1_0.py         # Schema 1.0 specification
  spec_v1_1.py         # Schema 1.1 specification

Existing Modules:

• cli/modules/compose/ - Docker Compose package with schema 1.0 and 1.1 support
  • spec_v1_0.py - Basic compose spec
  • spec_v1_1.py - Extended with network_mode, swarm support

Compose Module Schema Differences:

Schema 1.0:

• network section: Has toggle: "network_enabled" with explicit network_enabled boolean variable (default: False)
• ports section: Has toggle: "ports_enabled" with explicit ports_enabled boolean variable (default: True)
• Templates check {% if network_enabled %} and {% if ports_enabled %}

Schema 1.1:

• network section: NO toggle - always available, controlled by network_mode enum (bridge/host/macvlan)
• ports section: Has toggle: "ports_enabled" but the variable is AUTO-CREATED and NOT available in templates
• Templates check {% if network_mode == 'bridge' %} for networks and {% if network_mode == 'bridge' and not traefik_enabled %} for ports
• Key changes:
  • network_enabled doesn't exist - use network_mode conditionals instead
  • ports_enabled is not usable in templates - use network_mode and traefik_enabled conditionals instead
  • Port visibility is controlled by: network mode (must be bridge) + Traefik (ports not needed when using Traefik)

(Work in Progress): terraform, docker, ansible, kubernetes, packer modules

LibraryManager

• Loads libraries from config file
• Stores Git Libraries under: ~/.config/boilerplates/libraries/{name}/
• Uses sparse-checkout to clone only template directories for git-based libraries (avoiding unnecessary files)
• Supports two library types: git (synced from repos) and static (local directories)
• Priority determined by config order (first = highest)

Library Types:

• git: Requires url, branch, directory fields
• static: Requires path field (absolute or relative to config)

Duplicate Handling:

• Within same library: Raises DuplicateTemplateError
• Across libraries: Uses qualified IDs (e.g., alloy.default, alloy.local)
• Simple IDs use priority: compose show alloy loads from first library
• Qualified IDs target specific library: compose show alloy.local

Config Example:

libraries:
  - name: default       # Highest priority (checked first)
    type: git
    url: https://github.com/user/templates.git
    branch: main
    directory: library
  - name: local         # Lower priority
    type: static
    path: ~/my-templates
    url: ''             # Backward compatibility fields
    branch: main
    directory: .

Note: Static libraries include dummy url/branch/directory fields for backward compatibility with older CLI versions.

ConfigManager

• User Config stored in ~/.config/boilerplates/config.yaml

DisplayManager and IconManager

CRITICAL RULE - NEVER violate this:

• NEVER use console.print() outside of display manager classes (cli/core/display/ directory)
• NEVER import Console from rich.console except in display manager classes or cli/__main__.py
• ALWAYS use module_instance.display.display_*() or display.display_*() methods for ALL output
• Display managers (cli/core/display/*.py) are the ONLY exception - they implement console output

Rationale:

• DisplayManager provides a centralized interface for ALL CLI output rendering
• Direct console usage bypasses formatting standards, icon management, and output consistency
• IconManager provides Nerd Font icons internally for DisplayManager - never use emojis or direct icons

DisplayManager Architecture (Refactored for Single Responsibility Principle):

DisplayManager acts as a facade that delegates to specialized manager classes:

1. VariableDisplayManager - Handles all variable-related rendering

   • render_variable_value() - Variable value formatting with context awareness
   • render_section() - Section header display
   • render_variables_table() - Complete variables table with dependencies

2. TemplateDisplayManager - Handles all template-related rendering

   • render_template() - Main template display coordinator
   • render_template_header() - Template metadata display
   • render_file_tree() - Template file structure visualization
   • render_file_generation_confirmation() - Files preview before generation

3. StatusDisplayManager - Handles status messages and error display

   • display_message() - Core message formatting with level-based routing
   • display_error(), display_warning(), display_success(), display_info() - Convenience methods
   • display_template_render_error() - Detailed render error display
   • display_warning_with_confirmation() - Interactive warning prompts

4. TableDisplayManager - Handles table rendering

   • render_templates_table() - Templates list with library indicators
   • render_status_table() - Status tables with success/error indicators
   • render_config_tree() - Configuration tree visualization

Usage Pattern:

# External code uses DisplayManager methods (backward compatible)
display = DisplayManager()
display.display_template(template, template_id)

# Internally, DisplayManager delegates to specialized managers
# display.templates.render_template(template, template_id)

Design Principles:

• External code calls DisplayManager methods only
• DisplayManager delegates to specialized managers internally
• Each specialized manager has a single, focused responsibility
• Backward compatibility maintained through delegation methods
• All managers can access parent DisplayManager via self.parent

Templates

Templates are directory-based. Each template is a directory containing all the necessary files and subdirectories for the boilerplate.

Requires template.yaml or template.yml with metadata and variables:

---
kind: compose
schema: "1.0"  # Optional: Defaults to 1.0 if not specified
metadata:
  name: My Nginx Template
  description: >
    A template for a simple Nginx service.


    Project: https://...

    Source: https://

    Documentation: https://
  version: 0.1.0
  author: Christian Lempa
  date: '2024-10-01'
spec:
  general:
    vars:
      nginx_version:
        type: string
        description: The Nginx version to use.
        default: latest

Template Metadata Versioning

Template Version Field: The metadata.version field in template.yaml should reflect the version of the underlying application or resource:

• Compose templates: Match the Docker image version (e.g., nginx:1.25.3 → version: 1.25.3)
• Terraform templates: Match the provider version (e.g., AWS provider 5.23.0 → version: 5.23.0)
• Other templates: Match the primary application/tool version being deployed
• Use latest or increment template-specific version (e.g., 0.1.0, 0.2.0) only when no specific application version applies

Rationale: This helps users identify which version of the application/provider the template is designed for and ensures template versions track upstream changes.

Application Version Variables:

• IMPORTANT: Application/image versions should be hardcoded in template files (e.g., image: nginx:1.25.3)
• Do NOT create template variables for application versions (e.g., no nginx_version variable)
• Users should update the template file directly when they need a different version
• This prevents version mismatches and ensures templates are tested with specific, known versions
• Exception: Only create version variables if there's a strong technical reason (e.g., multi-component version pinning)

Template Schema Versioning

Templates and modules use schema versioning to ensure compatibility. Each module defines a supported schema version, and templates declare which schema version they use.

---
kind: compose
schema: "1.0"  # Defaults to 1.0 if not specified
metadata:
  name: My Template
  version: 1.0.0
  # ... other metadata fields
spec:
  # ... variable specifications

How It Works:

• Module Schema Version: Each module defines schema_version (e.g., "1.1")
• Module Spec Loading: Modules load appropriate spec based on template's schema version
• Template Schema Version: Each template declares schema at the top level (defaults to "1.0")
• Compatibility Check: Template schema ≤ Module schema → Compatible
• Incompatibility: Template schema > Module schema → IncompatibleSchemaVersionError

Behavior:

• Templates without schema field default to "1.0" (backward compatible)
• Old templates (schema 1.0) work with newer modules (schema 1.1)
• New templates (schema 1.2) fail on older modules (schema 1.1) with clear error
• Version comparison uses 2-level versioning (major.minor format)

When to Use:

• Increment module schema version when adding new features (new variable types, sections, etc.)
• Set template schema when using features from a specific schema
• Example: Template using new variable type added in schema 1.1 should set schema: "1.1"

Single-File Module Example:

class SimpleModule(Module):
  name = "simple"
  description = "Simple module"
  schema_version = "1.0"
  spec = VariableCollection.from_dict({...})  # Single spec

Multi-Schema Module Example:

# cli/modules/compose/__init__.py
class ComposeModule(Module):
  name = "compose"
  description = "Manage Docker Compose configurations"
  schema_version = "1.1"  # Highest schema version supported
  
  def get_spec(self, template_schema: str) -> VariableCollection:
    """Load spec based on template schema version."""
    if template_schema == "1.0":
      from .spec_v1_0 import get_spec
    elif template_schema == "1.1":
      from .spec_v1_1 import get_spec
    return get_spec()

Version Management:

• CLI version is defined in cli/__init__.py as __version__
• pyproject.toml version must match __version__ for releases
• GitHub release workflow validates version consistency

Template Files

• Jinja2 Templates (.j2): Rendered by Jinja2, .j2 extension removed in output. Support {% include %} and {% import %}.
• Static Files: Non-.j2 files copied as-is.
• Sanitization: Auto-sanitized (single blank lines, no leading blanks, trimmed whitespace, single trailing newline).

Docker Compose Best Practices

Traefik Integration:

When using Traefik with Docker Compose, the traefik.docker.network label is CRITICAL for stacks with multiple networks. When containers are connected to multiple networks, Traefik must know which network to use for routing.

Implementation:

• ALL templates using Traefik MUST follow the patterns in archetypes/compose/traefik-v1.j2 (standard mode) and archetypes/compose/swarm-v1.j2 (swarm mode)
• These archetypes are the authoritative reference for correct Traefik label configuration
• The traefik.docker.network={{ traefik_network }} label must be present in both standard labels: and deploy.labels: sections

Variables

Precedence (lowest to highest):

1. Module spec (defaults for all templates of that kind)
2. Template spec (overrides module defaults)
3. User config.yaml (overrides template and module defaults)
4. CLI --var (highest priority)

Template Variable Overrides: Template spec variables can:

• Override module defaults: Only specify properties that differ from module spec (e.g., change default value)
• Create new variables: Define template-specific variables not in module spec
• Minimize duplication: Do NOT re-specify type, description, or other properties if they remain unchanged from module spec

Example:

# Module spec defines: service_name (type: str, no default)
# Template spec overrides:
spec:
  general:
    vars:
      service_name:
        default: whoami  # Only override the default, type already defined in module

Variable Types:

• str (default), int, float, bool
• email - Email validation with regex
• url - URL validation (requires scheme and host)
• hostname - Hostname/domain validation
• enum - Choice from options list

Variable Properties:

• sensitive: true - Masked in prompts/display (e.g., passwords)
• autogenerated: true - Auto-generates value if empty (shows *auto placeholder)
• default - Default value
• description - Variable description
• prompt - Custom prompt text (overrides description)
• extra - Additional help text
• options - List of valid values (for enum type)

Section Features:

• Required Sections: Mark with required: true (general is implicit). Users must provide all values.
• Toggle Settings: Conditional sections via toggle: "bool_var_name". If false, section is skipped.
  • IMPORTANT: When a section has toggle: "var_name", that boolean variable is AUTO-CREATED by the system
  • In schema 1.0: Toggle variables are explicitly defined in the section's vars
  • In schema 1.1: Toggle variables are auto-created and don't need explicit definition (unless customizing defaults)
  • Example: ports section with toggle: "ports_enabled" automatically provides ports_enabled boolean
• Dependencies: Use needs: "section_name" or needs: ["sec1", "sec2"]. Dependent sections only shown when dependencies are enabled. Auto-validated (detects circular/missing/self dependencies). Topologically sorted.

Example Section with Dependencies:

spec:
  traefik:
    title: Traefik
    required: false
    toggle: traefik_enabled
    vars:
      traefik_enabled:
        type: bool
        default: false
      traefik_host:
        type: hostname
  
  traefik_tls:
    title: Traefik TLS/SSL
    needs: traefik
    toggle: traefik_tls_enabled
    vars:
      traefik_tls_enabled:
        type: bool
        default: true
      traefik_tls_certresolver:
        type: str
        sensitive: false
        default: myresolver

Validation

Jinja2 Validation:

• Templates validated for Jinja2 syntax errors during load
• Checks for undefined variables (variables used but not declared in spec)
• Built into Template class

Semantic Validation:

• Validator registry system in cli/core/validators.py
• Extensible: ContentValidator abstract base class
• Built-in validators: DockerComposeValidator, YAMLValidator
• Validates rendered output (YAML structure, Docker Compose schema, etc.)
• Triggered via compose validate command with --semantic flag (enabled by default)

Prompt

Uses rich library for interactive prompts. Supports:

• Text input
• Password input (masked, for sensitive: true variables)
• Selection from list (single/multiple)
• Confirmation (yes/no)
• Default values
• Autogenerated variables (show *auto placeholder, generate on render)

To skip the prompt use the --no-interactive flag, which will use defaults or empty values.

Commands

Standard Module Commands (auto-registered for all modules):

• list - List all templates
• search <query> - Search templates by ID
• show <id> - Show template details
• generate <id> [directory] - Generate from template (supports --dry-run, --var, --no-interactive)
• validate [id] - Validate templates (Jinja2 + semantic)
• defaults - Manage config defaults (get, set, rm, clear, list)

Core Commands:

• repo sync - Sync git-based libraries
• repo list - List configured libraries

Archetypes

The archetypes package provides reusable, standardized template building blocks for creating boilerplates. Archetypes are modular Jinja2 snippets that represent specific configuration sections (e.g., networks, volumes, service labels).

Purpose

1. Template Development: Provide standardized, tested building blocks for creating new templates
2. Testing & Validation: Enable testing of specific configuration sections in isolation with different variable combinations

Structure

archetypes/
  __init__.py              # Package initialization
  __main__.py              # CLI tool (auto-discovers modules)
  compose/                 # Module-specific archetypes
    archetypes.yaml        # Configuration: schema version + variable overrides
    compose.yaml.j2        # Main composition file (includes all components)
    service-*.j2           # Service-level components (networks, ports, volumes, labels, etc.)
    networks-*.j2          # Top-level network definitions
    volumes-*.j2           # Top-level volume definitions
    configs-*.j2           # Config definitions
    secrets-*.j2           # Secret definitions

Key Files:

• archetypes.yaml: Configures schema version and variable overrides for testing
• compose.yaml.j2: Main composition file that includes all archetype components to test complete configurations
• Individual *.j2 files: Modular components for specific configuration sections

Usage

# List available archetypes
python3 -m archetypes compose list

# Preview complete composition (all components together)
python3 -m archetypes compose generate compose.yaml

# Preview individual component
python3 -m archetypes compose generate networks-v1

# Test with variable overrides
python3 -m archetypes compose generate compose.yaml \
  --var traefik_enabled=true \
  --var swarm_enabled=true

Template Development Workflow

1. Start with archetypes: Copy relevant archetype components to your template directory
2. Customize: Modify components as needed (hardcode image, add custom labels, etc.)
3. Test: Validate using python3 -m archetypes compose generate
4. Validate: Use compose validate to check Jinja2 syntax and semantic correctness

Implementation Details

How it works:

• Loads module spec based on schema version from archetypes.yaml
• Merges variable sources: module spec → archetypes.yaml → CLI --var
• Renders using Jinja2 with support for {% include %} directives
• Testing only: The generate command NEVER writes files - always shows preview output