ChristianLempa_Boilerplates/AGENTS.md

AGENTS.md

Guidance for AI Agents working with this repository.

Project Overview

A Python CLI for managing infrastructure boilerplates, plus archetypes and template-loading logic. Built with Typer (CLI) and Jinja2 (templating).

Important library location note:

• The canonical template library does not live in this repository for modern development.
• The checked-in library/ directory here is legacy/backward-compatibility content for versions older than 0.2.0.
• For 0.2.0+, the active template library lives in the separate GitHub repository boilerplates-library.

Development Setup

Running and Testing

# Create and activate a local virtual environment first
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e ".[test]"

# Run the CLI application
python3 -m cli
# Debugging and Testing commands
python3 -m cli --log-level DEBUG compose list
# Run tests
python3 -m pytest

Production-Ready Testing

For detailed information about testing boilerplates in a production-like environment before release, see WARP-LOCAL.md (local file, not in git). This document covers:

• Test server infrastructure and Docker contexts
• Step-by-step testing procedures for Docker Compose, Swarm, and Kubernetes
• Comprehensive testing checklists
• Production release criteria

Linting and Formatting

Should always happen before pushing anything to the repository and before making a release.

• Use ruff for Python code:
  • ruff check --fix . - Check and auto-fix linting errors (including unused imports)
  • ruff format . - Format code according to style guidelines
  • Both commands must be run before committing
• Minimum pre-release verification:
  • ruff check --fix .
  • ruff format .
  • python3 -m pytest
• Agents should treat these as release gates, not optional cleanup.

Project Management and Git

The project is stored in a public GitHub Repository, use issues, and branches for features/bugfixes and open PRs for merging.

Naming Conventions and Best-Practices:

• Branches, PRs: feature/2314-add-feature, problem/1249-fix-bug
• Issues should have clear titles and descriptions, link related issues/PRs, and have appropriate labels like (problem, feature, discussion, question).
• Commit messages should be clear and concise, following the format: type(scope): subject (e.g., fix(compose): correct variable parsing).

Architecture

File Structure

• cli/ - Python CLI application source code
  • cli/core/ - Core Components of the CLI application
  • cli/modules/ - Modules implementing technology-specific functions
  • cli/__main__.py - CLI entry point, auto-discovers modules and registers commands
• library/ - Legacy in-repo template content kept only for backward compatibility with versions older than 0.2.0
  • Do not treat this directory as the canonical source for new template work
  • The active 0.2.0+ library is maintained in the separate GitHub repository boilerplates-library
• archetypes/ - Testing tool for template snippets (archetype development)
  • archetypes/__init__.py - Package initialization
  • archetypes/__main__.py - CLI tool entry point
  • archetypes/<module>/ - Module-specific archetype snippets (e.g., archetypes/compose/)

Core Components

• cli/core/collection.py - VariableCollection class (manages sections and variables)
  • Key Attributes: _sections (dict of VariableSection objects), _variable_map (flat lookup dict)
  • Key Methods: get_satisfied_values() (returns enabled variables), apply_defaults(), sort_sections()
• cli/core/config.py - Configuration management (loading, saving, validation)
• cli/core/display/ - Centralized CLI output rendering package (Always use DisplayManager - never print directly)
  • __init__.py - Package exports (DisplayManager, DisplaySettings, IconManager)
  • display_manager.py - Main DisplayManager facade
  • display_settings.py - DisplaySettings configuration class
  • icon_manager.py - IconManager for Nerd Font icons
  • variable_display.py - VariableDisplayManager for variable rendering
  • template_display.py - TemplateDisplayManager for template display
  • status_display.py - StatusDisplayManager for status messages
  • table_display.py - TableDisplayManager for table rendering
• cli/core/exceptions.py - Custom exceptions for error handling (Always use this for raising errors)
• cli/core/library.py - LibraryManager for template discovery from git-synced libraries and static file paths
• cli/core/module.py - Abstract base class for modules (defines standard commands)
• cli/core/prompt.py - Interactive CLI prompts using rich library
• cli/core/registry.py - Central registry for module classes (auto-discovers modules)
• cli/core/repo.py - Repository management for syncing git-based template libraries
• cli/core/section.py - VariableSection class (stores section metadata and variables)
  • Key Attributes: key, title, toggle, needs, variables (dict of Variable objects)
• cli/core/template.py - Template Class for parsing, managing and rendering templates
• cli/core/variable.py - Variable class (stores variable metadata and values)
  • Key Attributes: name, type, value (stores default or current value), description, sensitive, needs
  • Note: Default values are stored in value attribute, NOT in a separate default attribute
• cli/core/validators.py - Semantic validators for template content (Docker Compose, YAML, etc.)
• cli/core/version.py - Version comparison utilities for semantic versioning

Modules

Currently supported kinds in code:

• ansible
• compose
• helm
• kubernetes
• packer
• swarm
• terraform

These are the kinds supported by the registered CLI modules in cli/modules/. When answering "what kinds do we support?", prefer this code-level list over whatever legacy content happens to exist under library/.

Module Structure: Modules can be either single files or packages:

• Single file: cli/modules/modulename.py (for simple modules)

Creating Modules:

• Subclass Module from cli/core/module.py
• Call registry.register(YourModule) at module bottom
• Auto-discovered and registered at CLI startup

Module Discovery and Registration:

The system automatically discovers and registers modules at startup:

1. Discovery: CLI __main__.py imports all Python files in cli/modules/ directory
2. Registration: Each module file calls registry.register(ModuleClass) at module level
3. Storage: Registry stores module classes in a central dictionary by module name
4. Command Generation: CLI framework auto-generates subcommands for each registered module
5. Instantiation: Modules are instantiated on-demand when commands are invoked

Benefits:

• No manual registration needed - just add a file to cli/modules/
• Modules are self-contained - can be added/removed without modifying core code

• Type-safe - registry validates module interfaces at registration time

  compose/
  v1.0.json
  v1.1.json
  v1.2.json
  terraform/
  v1.0.json
  ansible/
  v1.0.json
  ...other modules...
  

  [
  {
  "key": "section_key",
  "title": "Section Title",
  "description": "Optional section description",
  "toggle": "optional_toggle_variable_name",
  "needs": "optional_dependency",
  "required": true,
  "vars": [
    {
      "name": "variable_name",
      "type": "str",
      "description": "Variable description",
      "default": "default_value",
      "required": true,
      "sensitive": false,
      "autogenerated": false,
      "options": ["option1", "option2"],
      "needs": "other_var=value",
      "extra": "Additional help text"
    }
  ]
  }
  ]
  

  
  class MyModule(Module):
  name = "mymodule"
      
      # Convert JSON array to OrderedDict format
      return self._convert_json_to_dict(json_spec)
  

• Auto-created toggle variables: Sections with toggle automatically create boolean variables

• Conditional visibility: Variables use needs constraints to show/hide based on other variable values

• Mode-based organization: Related settings grouped by operational mode (e.g., network_mode, volume_mode)

Older Python-based spec_v*.py files have been migrated to JSON. The module __init__.py now:

1. Converts JSON array format to OrderedDict for backward compatibility

Existing Modules:

• Other modules (ansible, terraform, kubernetes, helm, packer) - Work in Progress

(Work in Progress): terraform, docker, ansible, kubernetes, packer modules

LibraryManager

• Loads libraries from config file
• Stores Git Libraries under: ~/.config/boilerplates/libraries/{name}/
• Uses sparse-checkout to clone only template directories for git-based libraries (avoiding unnecessary files)
• Supports two library types: git (synced from repos) and static (local directories)
• Priority determined by config order (first = highest)
• In this repository, the checked-in library/ directory is legacy compatibility data only and should not be treated as the primary source for current template development
• For current 0.2.0+ template work, use the external GitHub repository boilerplates-library

Library Types:

• git: Requires url, branch, directory fields
• static: Requires path field (absolute or relative to config)

Duplicate Handling:

• Within same library: Raises DuplicateTemplateError
• Across libraries: Uses qualified IDs (e.g., alloy.default, alloy.local)
• Simple IDs use priority: compose show alloy loads from first library
• Qualified IDs target specific library: compose show alloy.local

Config Example:

libraries:
  - name: default       # Highest priority (checked first)
    type: git
    url: https://github.com/user/templates.git
    branch: main
    directory: library
  - name: local         # Lower priority
    type: static
    path: ~/my-templates
    url: ''             # Backward compatibility fields
    branch: main
    directory: .

Note: Static libraries include dummy url/branch/directory fields for backward compatibility with older CLI versions.

ConfigManager

• User Config stored in ~/.config/boilerplates/config.yaml

DisplayManager and IconManager

CRITICAL RULE - NEVER violate this:

• NEVER use console.print() outside of display manager classes (cli/core/display/ directory)
• NEVER import Console from rich.console except in display manager classes or cli/__main__.py
• ALWAYS use module_instance.display.display_*() or display.display_*() methods for ALL output
• Display managers (cli/core/display/*.py) are the ONLY exception - they implement console output

Rationale:

• DisplayManager provides a centralized interface for ALL CLI output rendering
• Direct console usage bypasses formatting standards, icon management, and output consistency
• IconManager provides Nerd Font icons internally for DisplayManager - never use emojis or direct icons

DisplayManager Architecture (Refactored for Single Responsibility Principle):

DisplayManager acts as a facade that delegates to specialized manager classes:

1. VariableDisplayManager - Handles all variable-related rendering

   • render_variable_value() - Variable value formatting with context awareness
   • render_section() - Section header display
   • render_variables_table() - Complete variables table with dependencies

2. TemplateDisplayManager - Handles all template-related rendering

   • render_template() - Main template display coordinator
   • render_template_header() - Template metadata display
   • render_file_tree() - Template file structure visualization
   • render_file_generation_confirmation() - Files preview before generation

3. StatusDisplayManager - Handles status messages and error display

   • display_message() - Core message formatting with level-based routing
   • display_error(), display_warning(), display_success(), display_info() - Convenience methods
   • display_template_render_error() - Detailed render error display
   • display_warning_with_confirmation() - Interactive warning prompts

4. TableDisplayManager - Handles table rendering

   • render_templates_table() - Templates list with library indicators
   • render_status_table() - Status tables with success/error indicators
   • render_config_tree() - Configuration tree visualization

Usage Pattern:

# External code uses DisplayManager methods (backward compatible)
display = DisplayManager()
display.display_template(template, template_id)

# Internally, DisplayManager delegates to specialized managers
# display.templates.render_template(template, template_id)

Design Principles:

• External code calls DisplayManager methods only
• DisplayManager delegates to specialized managers internally
• Each specialized manager has a single, focused responsibility
• Backward compatibility maintained through delegation methods
• All managers can access parent DisplayManager via self.parent

Templates

Templates are directory-based. Each template is a directory containing all the necessary files and subdirectories for the boilerplate.

Template Rendering Flow

How templates are loaded and rendered:

1. Discovery: LibraryManager finds template directories containing template.json
2. Parsing: Template class loads and parses the template metadata and spec
3. Variable Merging: Template spec overrides are merged with module spec (precedence: module < template < user config < CLI)
4. Collection Building: VariableCollection is constructed with merged variables and sections
5. Dependency Resolution: Sections are topologically sorted based on needs constraints
6. Variable Resolution: Variables with needs constraints are evaluated for visibility
7. Jinja2 Rendering: Template files (.j2) are rendered with final variable values
8. Sanitization: Rendered output is cleaned (whitespace, blank lines, trailing newline)
9. Validation: Optional semantic validation (YAML structure, Docker Compose config, etc.)

Key Architecture Points:

• Variable visibility is dynamic based on needs constraints (evaluated at prompt/render time)
• Jinja2 templates support {% include %} and {% import %} for composition
• Template loading can read configured libraries from disk, but the checked-in library/ tree in this repo is legacy compatibility data, not the canonical 0.2.0+ library source

Template Structure

Requires template.json with metadata and variables:

{
  "slug": "my-service",
  "kind": "compose",
  "metadata": {
    "name": "My Service Template",
    "description": "A template for a service.",
    "author": "Your Name",
    "date": "2024-01-01",
    "version": {
      "name": "v1.0.0"
    }
  },
  "variables": [
    {
      "name": "general",
      "title": "General",
      "items": [
        {
          "name": "service_name",
          "type": "str",
          "title": "Service name"
        }
      ]
    }
  ]
}

Template Metadata Versioning

Template Version Field: The metadata.version field in template.json is a structured object and may be partial:

• name
• source_dep_name
• source_dep_version
• source_dep_digest
• upstream_ref
• notes

Rationale: This separates the user-facing template version label from the tracked upstream dependency metadata used by the template snapshot.

Application Version Variables:

• IMPORTANT: Application/image versions should be hardcoded in template files (e.g., image: nginx:1.25.3)
• Do NOT create template variables for application versions (e.g., no nginx_version variable)
• Users should update the template file directly when they need a different version
• This prevents version mismatches and ensures templates are tested with specific, known versions

• Exception: Only create version variables if there's a strong technical reason (e.g., multi-component version pinning)

  {
  "slug": "my-template",
  "kind": "compose",
  "metadata": {
  "name": "My Template",
  "version": {
    "name": "v1.0.0"
  }
  },
  "variables": [
  {
    "name": "general",
    "title": "General",
    "items": []
  }
  ]
  }
  

How It Works:

Behavior:

• Older templates work with newer module versions (backward compatibility)
• Version comparison uses MAJOR.MINOR format (e.g., "1.0" < "1.2" < "2.0")

When to Use:

Single-File Module Example:

class SimpleModule(Module):
  name = "simple"
  description = "Simple module"
  spec = VariableCollection.from_dict({...})  # Single spec

# cli/modules/modulename/__init__.py
class ExampleModule(Module):
  name = "modulename"
  description = "Module description"
  
    # Dynamically load the appropriate spec version
    spec_module = importlib.import_module(f".{version_file}", package=__package__)
    return spec_module.get_spec()

Version Management:

• CLI version is defined in cli/__init__.py as __version__
• pyproject.toml version must match __version__ for releases
• GitHub release workflow validates version consistency

Template Files

• Jinja2 Templates (.j2): Rendered by Jinja2, .j2 extension removed in output. Support {% include %} and {% import %}.
• Static Files: Non-.j2 files copied as-is.
• Sanitization: Auto-sanitized (single blank lines, no leading blanks, trimmed whitespace, single trailing newline).
• Shortcodes: Template descriptions support emoji-style shortcodes (e.g., :warning:, :info:, :docker:) which are automatically replaced with Nerd Font icons during display. Add new shortcodes to IconManager.SHORTCODES dict.

Docker Compose Best Practices

Traefik Integration:

When using Traefik with Docker Compose, the traefik.docker.network label is CRITICAL for stacks with multiple networks. When containers are connected to multiple networks, Traefik must know which network to use for routing.

Implementation:

• Review archetypes/compose/ directory for reference implementations of Traefik integration patterns
• The traefik.docker.network={{ traefik_network }} label must be present in both standard labels: and deploy.labels: sections
• Standard mode and Swarm mode require different label configurations - check archetypes for examples

Variables

How Templates Inherit Variables:

When to Define Template Variables:

You only need to define variables in your template's spec section when:

1. Overriding defaults: Change default values for module variables (e.g., hardcode service_name for your specific app)

Variable Precedence (lowest to highest):

1. Module spec (defaults for all templates of that kind)
2. Template spec (overrides module defaults)
3. User config.yaml (overrides template and module defaults)
4. CLI --var (highest priority)

Template Variable Override Rules:

• Override module defaults: Only specify properties that differ from module spec (e.g., change default value)
• Create new variables: Define template-specific variables not in module spec
• Minimize duplication: Do NOT re-specify type, description, or other properties if they remain unchanged from module spec

Example:

# Template spec ONLY needs to override specific defaults:
spec:
  general:
    vars:
      service_name:
        default: whoami  # Only override the default, type already defined in module

Variable Types:

• str (default), int, float, bool
• email - Email validation with regex
• url - URL validation (requires scheme and host)
• hostname - Hostname/domain validation
• enum - Choice from options list

Variable Properties:

• sensitive: true - Masked in prompts/display (e.g., passwords)
• autogenerated: true - Auto-generates value if empty (shows *auto placeholder)
• default - Default value
• description - Variable description
• prompt - Custom prompt text (overrides description)
• extra - Additional help text
• options - List of valid values (for enum type)

Section Features:

• Toggle Settings: Conditional sections via toggle: "bool_var_name". If false, section is skipped.
  • IMPORTANT: When a section has toggle: "var_name", that boolean variable is AUTO-CREATED by the system
  • Example: ports section with toggle: "ports_enabled" automatically provides ports_enabled boolean
• Dependencies: Use needs: "section_name" or needs: ["sec1", "sec2"]. Dependent sections only shown when dependencies are enabled.

Dependency Resolution Architecture:

Sections and variables support needs constraints to control visibility based on other variables.

Section-Level Dependencies:

• Format: needs: "section_name" or needs: ["sec1", "sec2"]
• Section only appears when all required sections are enabled (their toggle variables are true)
• Automatically validated: detects circular, missing, and self-dependencies
• Topologically sorted: ensures dependencies are prompted/processed before dependents

Variable-Level Dependencies:

• Format: needs: "var_name=value" or needs: "var1=val1;var2=val2" (semicolon-separated)
• Variable only visible when constraint is satisfied (e.g., needs: "network_mode=bridge")
• Supports multiple values: needs: "network_mode=bridge,macvlan" (comma = OR)
• Evaluated dynamically at prompt and render time

Validation:

• Circular dependencies: Raises error if A needs B and B needs A
• Missing dependencies: Raises error if referencing non-existent sections/variables
• Self-dependencies: Raises error if section depends on itself

Example Section with Dependencies:

spec:
  traefik:
    title: Traefik
    required: false
    toggle: traefik_enabled
    vars:
      traefik_enabled:
        type: bool
        default: false
      traefik_host:
        type: hostname
  
  traefik_tls:
    title: Traefik TLS/SSL
    needs: traefik
    toggle: traefik_tls_enabled
    vars:
      traefik_tls_enabled:
        type: bool
        default: true
      traefik_tls_certresolver:
        type: str
        sensitive: false
        default: myresolver

Validation

Jinja2 Validation:

• Templates validated for Jinja2 syntax errors during load
• Checks for undefined variables (variables used but not declared in spec)
• Built into Template class

Semantic Validation:

• Validator registry system in cli/core/validators.py
• Extensible: ContentValidator abstract base class
• Built-in validators: DockerComposeValidator, YAMLValidator
• Validates rendered output (YAML structure, Docker Compose config, etc.)
• Triggered via compose validate command with --semantic flag (enabled by default)

Prompt

Uses rich library for interactive prompts. Supports:

• Text input
• Password input (masked, for sensitive: true variables)
• Selection from list (single/multiple)
• Confirmation (yes/no)
• Default values
• Autogenerated variables (show *auto placeholder, generate on render)

To skip the prompt use the --no-interactive flag, which will use defaults or empty values.

Commands

Standard Module Commands (auto-registered for all modules):

• list - List all templates
• search <query> - Search templates by ID
• show <id> - Show template details
• generate <id> -o <directory> - Generate from template (supports --dry-run, --var, --no-interactive)
• validate [template_id] - Validate template(s) (Jinja2 + semantic). Omit template_id to validate all templates
• defaults - Manage config defaults (get, set, rm, clear, list)

Core Commands:

• repo sync - Sync git-based libraries
• repo list - List configured libraries

Archetypes

The archetypes package provides reusable, standardized template building blocks for creating boilerplates. Archetypes are modular Jinja2 snippets that represent specific configuration sections.

Purpose

1. Template Development: Provide standardized, tested building blocks for creating new templates
2. Testing & Validation: Enable testing of specific configuration sections in isolation with different variable combinations

Usage

# List available archetypes for a module
python3 -m archetypes compose list

# Preview an archetype component
python3 -m archetypes compose generate <archetype-name>

# Test with variable overrides
python3 -m archetypes compose generate <archetype-name> \
  --var traefik_enabled=true \
  --var swarm_enabled=true

# Validate templates against archetypes
python3 -m archetypes compose validate            # All templates
python3 -m archetypes compose validate <template> # Single template

Archetype Validation

The validate command compares templates against archetypes to measure coverage and identify which archetype patterns are being used.

What it does:

• Compares each template file against all available archetypes using structural pattern matching
• Abstracts away specific values to focus on:
  • Jinja2 control flow: {% if %}, {% elif %}, {% else %}, {% for %} structures
  • YAML structure: Key names, indentation, and nesting patterns
  • Variable usage patterns: Presence of {{ }} placeholders (not specific names)
  • Wildcard placeholders: __ANY__, __ANYSTR__, __ANYINT__, __ANYBOOL__
  • Repeat markers: {# @repeat-start #} / {# @repeat-end #}
  • Optional markers: {# @optional-start #} / {# @optional-end #}
• This allows detection of archetypes even when specific values differ (e.g., grafana_data vs alloy_data)
• Calculates containment ratio: what percentage of each archetype structure is found within the template
• Reports usage status: exact (≥95%), high (≥70%), partial (≥30%), or none (<30%)
• Provides coverage metrics: (exact + high matches) / total archetypes

Advanced Pattern Matching in Archetypes

Archetypes support special annotations for flexible pattern matching:

Wildcard Placeholders (match any value):

• __ANY__ - Matches anything
• __ANYSTR__ - Matches any string
• __ANYINT__ - Matches any integer
• __ANYBOOL__ - Matches any boolean

Repeat Markers (pattern can appear 1+ times):

{# @repeat-start #}
  pattern
{# @repeat-end #}

Optional Markers (section may or may not exist):

{# @optional-start #}
  pattern
{# @optional-end #}

Example:

volumes:
  {# @repeat-start #}
  __ANY__:
    driver: local
  {# @repeat-end #}

Matches any number of volumes with driver: local

Usage:

# Validate all templates in library - shows summary table
python3 -m archetypes compose validate

# Validate specific template - shows detailed archetype breakdown
python3 -m archetypes compose validate whoami

# Validate templates in custom location
python3 -m archetypes compose validate --library /path/to/templates

Output:

• Summary mode (all templates): Table showing exact/high/partial/none counts and coverage % per template
• Detail mode (single template): Table showing each archetype's status, similarity %, and matching file

Use cases:

• Quality assurance: Ensure templates follow established patterns
• Refactoring: Identify templates that could benefit from archetype alignment
• Documentation: Track which archetypes are most/least used across templates

Template Development Workflow

1. Discover: Use list command to see available archetype components for your module
2. Review: Preview archetypes to understand implementation patterns
3. Copy: Copy relevant archetype components to your template directory
4. Customize: Modify as needed (hardcode image, add custom labels, etc.)
5. Validate: Use compose validate to check Jinja2 syntax and semantic correctness

Architecture

Key Concepts:

• Each module can have its own archetypes/<module>/ directory with reusable components
• Components are modular Jinja2 files that can be tested in isolation or composition
• Testing only: The generate command NEVER writes files - always shows preview output

How it works:

• Merges variable sources: module spec → archetypes.yaml → CLI --var
• Renders using Jinja2 with support for {% include %} directives