diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..da5bf40 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,306 @@ +# Copilot Custom Instructions — Azure Local Network Switch Config Generator + +> **Role context:** You are a world-class network architecture and automation engineer +> specialising in data centre fabric design, switch configuration automation, and +> infrastructure-as-code for hyperscale deployments. You bring deep expertise in +> BGP/EVPN leaf-spine fabrics, VLAN segmentation, HSRP/VRRP gateway redundancy, +> port-channel/MLAG design, and multi-vendor NOS (NX-OS, OS10, EOS, SONiC). +> Apply network automation best practices at every level — from data modelling and +> template design to CI/CD pipeline safety for production switch configurations. + +--- + +## Project Overview + +Python + Jinja2 CLI tool that generates **production network switch configurations** +for **Azure Local** deployments. It takes a lab-format JSON +definition, converts it to a standardised intermediate JSON schema, then renders +vendor-specific `.cfg` files using Jinja2 templates. + +These configs are deployed to **real production switches** in customer environments. +Correctness, consistency, and safety are paramount — a bad config can take down an +entire Azure Local cluster. + +**Supported vendors:** Cisco NX-OS, Dell EMC OS10 (Arista EOS, SONiC planned). +**Supported roles:** TOR1, TOR2 (production top-of-rack switches), BMC (internal +lab-only baseboard management switch — optional). + +--- + +## Architecture + +The pipeline follows a strict **separation of concerns** pattern common in network +automation — input normalisation, data transformation, and config rendering are +completely decoupled: + +``` +Lab JSON → Converter → Standard JSON → Generator + Jinja2 Templates → .cfg files + ↑ ↑ ↑ + Input data Vendor-agnostic Vendor-specific + (site-specific) intermediate schema template rendering +``` + +| Stage | Module | Purpose | +|-------|--------|---------| +| Entry | `src/main.py` | CLI parser, logging setup, orchestration | +| Load | `src/loader.py` | JSON/file loading, `get_real_path()` for PyInstaller | +| Convert | `src/convertors/convertors_lab_switch_json.py` | Lab JSON → Standard JSON (TOR switches via `StandardJSONBuilder`) | +| Convert | `src/convertors/convertors_bmc_switch_json.py` | Lab JSON → Standard JSON (BMC switches via `BMCSwitchConverter`, optional) | +| Generate | `src/generator.py` | Standard JSON + Jinja2 → `.cfg` output files | +| Constants | `src/constants.py` | All magic strings, defaults, lookup tables | +| Utilities | `src/utils.py` | Shared helpers (IP math, VLAN classification, firmware inference) | + +### Why This Architecture Matters + +- **Standard JSON is the contract** — converters can be swapped or extended without + touching the generator. The generator only cares about the schema, not the source. +- **Generator is 100% vendor-agnostic** — it discovers templates dynamically via + `glob("//*.j2")`. Zero vendor names in generator code. +- **Adding a new vendor** requires only: new constants (2–3 lines), new Jinja2 + templates, and new switch interface model JSONs. No converter or generator changes. + +--- + +## Design Principles + +> These are non-negotiable. Every code change, template edit, and PR must respect them. + +1. **Unified, generic implementation** — templates and converter logic are parameterized + and data-driven. No sample-specific or model-specific code branches. A single code + path handles any valid input regardless of vendor, model, or topology. + +2. **Data-driven configuration** — port counts, VLAN IDs, ASNs, IP addresses, hostnames, + and deployment patterns all come from input JSON. Only structural constants (fixed port + roles like uplink/downlink positions) are hardcoded. If a value could change between + deployments, it belongs in the input data — not in code. + +3. **TOR = production-quality, BMC = internal-minimal** — TOR switch configs are + customer-facing deliverables used in real production environments. TOR templates and + converter code must be fully data-driven, parameterized, and follow network engineering + best practices — no shortcuts. BMC switch configs are internal-only lab tooling. BMC + code may hardcode settings (VLANs, QoS, port-channel config) to keep implementation + simple, but every hardcoded value **must have a clear comment** explaining what it is + and why, so future engineers can troubleshoot without reverse-engineering. + +4. **BMC is optional — TOR-only is the default** — not all deployment patterns have a + BMC switch. BMC configs are only generated when a `Type: "BMC"` entry exists in the + lab input JSON. No BMC entry → only TOR configs. The converter skips gracefully with + an info log. Never assume BMC presence. + +5. **Multi-vendor extensible by design** — the framework uses convention-based template + discovery (`//*.j2`), shared interface template schemas across + vendors, and table-driven vendor mappings in `constants.py`. Adding a new vendor + (e.g., Arista EOS) should require only constant additions and new template files — + zero changes to converter or generator logic. + +6. **One folder per OS family** — template directories are `//` + (e.g., `cisco/nxos/`), not per-version. Minor syntax differences between OS versions + are handled via Jinja2 conditionals using the `version` field from standard JSON. + Only create a separate folder when the CLI syntax is fundamentally different (i.e., + it's a different NOS, not a different version). + +7. **Refactor freely, test at every turn** — we're on a dev branch; breaking changes are + acceptable. Upgrade and refine the codebase with best practices. Every critical change + or bug fix gets a unit test. Existing tests can be rewritten if their structure is poor. + +--- + +## Code Conventions + +### Python Standards + +- **Python 3.12** is the target version for CI, dev container, and PyInstaller builds. +- **Type hints**: Use `from __future__ import annotations` and modern syntax + (`dict[str, str]` not `Dict[str, str]`, `str | None` not `Optional[str]`). +- **Logging**: Use `logging` module exclusively, never `print()`. + Get logger via `logging.getLogger(__name__)` at module level. +- **Path handling**: Use `pathlib.Path` everywhere, never `os.path`. + Use `get_real_path()` from `loader.py` for PyInstaller compatibility (`sys._MEIPASS`). + +### Package & Import Structure + +- `src/` is a proper Python package with `__init__.py`. +- Use **relative imports** within `src/` (e.g., `from .constants import CISCO`). +- Top-level entry via `python -m src.main` or PyInstaller binary. + +### Constants Discipline + +- **ALL** magic strings, VLAN maps, vendor mappings, template dicts, and lookup tables + live in `src/constants.py`. Never hardcode values in converter or generator code. +- In-code JSON templates (`SWITCH_TEMPLATE`, `SVI_TEMPLATE`, `VLAN_TEMPLATE`) must be + **deep-copied** via `copy.deepcopy()` before mutation — never modify the module-level + constant. This is a common source of bugs in multi-switch generation. +- Vendor-specific behaviour (e.g., redundancy protocol) uses **lookup tables**, not + `if/else` on vendor names. If you find yourself writing `if vendor == "cisco":` + in converter logic, stop — add a mapping to `constants.py` instead. + +### Network Engineering Standards + +- **MTU**: Use `JUMBO_MTU = 9216` constant, never hardcode `9216` directly. +- **VLAN IDs**: Resolved from input Supernets via symbolic mapping (`VLAN_GROUP_MAP`), + never hardcoded in converter logic (exception: BMC hardcoded VLANs with clear comments). +- **BGP ASN**: Passed as integer, supports both 16-bit and 32-bit (4-byte) ASNs. + Templates render in asplain notation. +- **IP addressing**: Point-to-point links use /31 pairs (standard RFC 3021 practice). + SVIs use the CIDR from Supernet definitions. +- **Redundancy**: TOR pairs use HSRP (Cisco) or VRRP (Dell/others) for SVI gateway + redundancy. TOR1 = active (priority 150), TOR2 = standby (priority 140). + +--- + +## File Layout + +``` +src/ +├── __init__.py +├── main.py # CLI entry point, argument parsing, orchestration +├── generator.py # Jinja2 template rendering (vendor-agnostic) +├── loader.py # File loading, path resolution, PyInstaller support +├── constants.py # ALL constants, lookup tables, JSON templates +├── utils.py # Shared utilities (IP math, VLAN classification) +└── convertors/ + ├── __init__.py + ├── convertors_lab_switch_json.py # TOR converter (StandardJSONBuilder, 684 lines) + └── convertors_bmc_switch_json.py # BMC converter (BMCSwitchConverter) + +input/ +├── standard_input.json # Example standard-format input +├── jinja2_templates/ # Jinja2 config templates (per vendor/firmware) +│ ├── cisco/nxos/ # 10 templates: bgp, interface, login, etc. +│ └── dellemc/os10/ # 11 templates: same + vlt.j2 +└── switch_interface_templates/ # Hardware port-layout definitions (per vendor/model) + ├── cisco/ # 93180YC-FX.json, 93180YC-FX3.json, etc. + └── dellemc/ # S5248F-ON.json, N3248TE-ON.json, etc. + +tests/ +├── conftest.py # Shared fixtures, helpers, pytest hooks +├── test_unit.py # Unit tests for StandardJSONBuilder methods (85 tests) +├── test_convertors.py # Golden-file integration tests (converter pipeline, 12 tests) +├── test_generator.py # Golden-file integration tests (generator pipeline, 6 tests) +└── test_cases/ # Test data: input JSON + expected outputs per scenario +``` + +--- + +## Testing Rules + +### Framework & Execution + +- **Framework**: `pytest` — run with `python -m pytest tests/ -v`. +- **103 tests total**: 85 unit + 12 converter integration + 6 generator integration. +- **CI gate**: All tests must pass before build. Build must pass before release. + +### Test Architecture (Hybrid Strategy) + +| Layer | Strategy | Purpose | +|-------|----------|---------| +| **Unit** (`test_unit.py`) | Synthetic inputs, one method at a time | Fast, pinpointed failure diagnosis for `StandardJSONBuilder` methods | +| **Converter integration** (`test_convertors.py`) | Golden-file comparison | End-to-end converter pipeline smoke test | +| **Generator integration** (`test_generator.py`) | Golden-file comparison | End-to-end Jinja2 rendering smoke test | + +### Golden-File Pattern + +- Converter tests: Lab JSON → converter → compare full standard JSON against expected + files in `tests/test_cases//expected_outputs/`. +- Generator tests: Standard JSON → generator + Jinja2 → compare `.cfg` output against + expected files. +- Test cases auto-discovered by folder naming convention (`convert_*`, `std_*`). + +### Unit Test Standards + +- Use small synthetic inputs — test one method at a time. +- Use `tmp_path` fixture for any file I/O. No global state, no side effects at import time. +- Shared fixtures and helpers live in `conftest.py`: `load_json()`, + `find_json_differences()`, `find_text_differences()`. +- **Every new feature, converter method, template change, or bug fix must have + corresponding test coverage.** No exceptions. + +--- + +## Template Conventions + +### Jinja2 Config Templates + +- Live in `input/jinja2_templates///`. +- Template names match config sections: `bgp.j2`, `vlan.j2`, `interface.j2`, + `port_channel.j2`, `login.j2`, `system.j2`, `qos.j2`, `prefix_list.j2`, + `static_route.j2`, `full_config.j2`. +- Vendor-specific templates are allowed (e.g., Dell `vlt.j2` for VLT peer-link + config, Cisco has no equivalent). +- **All templates must use guards** (e.g., `{% if port_channels %}`, + `{% if bgp %}`) so they render safely for any switch type — a BMC switch with + no BGP section produces empty output, not an error. +- The generator renders **every** `.j2` file in the vendor directory — no hardcoded + list. Dropping a new `.j2` file into the folder automatically includes it. + +### Switch Interface Templates (Hardware Port Layouts) + +- Live in `input/switch_interface_templates//.json`. +- Define which physical ports get which roles (host, uplink, BMC, MLAG, unused). +- **Schema is shared across all vendors** — Cisco and Dell use identical JSON structure: + - Top-level keys: `make`, `model`, `type`, `interface_templates`, `port_channels` + - `interface_templates` sub-keys: `common`, `fully_converged`, `switched`, `switchless` + (Dell adds `fully_converged1`/`fully_converged2` for trunk/access mode variants) + - Interface entries: `name`, `type`, `intf_type`, `intf`/`start_intf`/`end_intf`, + `access_vlan`/`native_vlan`/`tagged_vlans`, `shutdown`, `service_policy` +- **New vendors must follow the same schema** so the converter processes them + generically. Only interface naming conventions differ (Cisco `1/48` vs Dell + `1/1/48:1`). +- Adding a new switch model = adding a JSON file. No code changes needed. + +### Template Design Best Practices + +- **Idempotent output**: Generated configs should produce the same result if applied + twice. Avoid commands that toggle state. +- **Deterministic ordering**: VLANs, interfaces, and BGP neighbors render in + consistent order (sorted by ID/name) to avoid false diffs in version control. +- **No credentials in templates**: Login templates use placeholder patterns. + Real credentials are injected at deployment time, never in generated configs. + +--- + +## Multi-Vendor Extensibility + +The framework is designed so that adding a new vendor requires **zero changes** to the +generator, loader, main entry point, or BMC converter. The work is: + +| Task | Where | Effort | +|------|-------|--------| +| Add vendor/firmware constants | `src/constants.py` | 2–3 lines | +| Add `VENDOR_FIRMWARE_MAP` entry | `src/constants.py` | 1 line | +| Create Jinja2 templates | `input/jinja2_templates///` | Main effort | +| Create switch model JSONs | `input/switch_interface_templates//` | Moderate | +| Add golden-file test cases | `tests/test_cases/` | Moderate | + +The generator discovers templates dynamically — no vendor names anywhere in its code. +The converter uses table-driven mappings for vendor-specific behaviour (firmware +inference, redundancy protocol). This is intentional — treat it as an architectural +invariant and don't introduce vendor-specific `if/else` branches. + +--- + +## Build & Release + +- **PyInstaller** builds cross-platform executables (Windows + Linux). +- **CI/CD pipeline**: `test` (pytest, 103 tests) → `build` (PyInstaller matrix) → + `release` (GitHub Release, tag-based with environment approval gate). +- **Python 3.12** is the target version for CI and dev container. +- **Tag-based release**: `v*` tags trigger the full pipeline. GitHub Environment + `production` requires human approval before release publishes. +- **Concurrency**: In-flight CI runs are cancelled when a newer commit arrives + (except tag pushes, which always run to completion). + +--- + +## Quick Reference — What NOT To Do + +| Anti-Pattern | Why | Do This Instead | +|-------------|-----|-----------------| +| `if vendor == "cisco":` in converter | Breaks multi-vendor extensibility | Use lookup table in `constants.py` | +| `print("debug info")` | Lost in PyInstaller, no log levels | `logger.debug("info")` | +| `os.path.join(...)` | Inconsistent, no PyInstaller support | `Path(...) / "sub"`, `get_real_path()` | +| Hardcoded VLAN ID in converter | Site-specific, breaks other deployments | Read from Supernet input data | +| `Dict[str, str]` (typing module) | Legacy syntax | `dict[str, str]` with `from __future__ import annotations` | +| Modify `SWITCH_TEMPLATE` directly | Corrupts shared state across switches | `deepcopy(SWITCH_TEMPLATE)` | +| Test with real file paths | Leaks state between tests | Use `tmp_path` fixture | +| Vendor-specific template filename list | Breaks when new vendor adds templates | `glob("*.j2")` (already done) | diff --git a/.github/workflows/build-switchgentool.yml b/.github/workflows/build-switchgentool.yml index c608da9..b3b5704 100644 --- a/.github/workflows/build-switchgentool.yml +++ b/.github/workflows/build-switchgentool.yml @@ -2,7 +2,7 @@ name: Build Switch Config Generator Tool on: push: - branches: [ main, dev/nl/newDesignv1 ] + branches: [ main ] tags: [ 'v*' ] paths: - 'src/**' @@ -20,16 +20,50 @@ on: - '.github/workflows/build-switchgentool.yml' workflow_dispatch: +# ── Test job needs read; release job needs write (scoped per-job) ───────── permissions: contents: read actions: read +# Cancel in-flight runs for the same branch/PR (tags always run to completion) +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ !startsWith(github.ref, 'refs/tags/') }} + jobs: + # ════════════════════════════════════════════════════════════════════════ + # Job 1: Test — run full pytest suite + # ════════════════════════════════════════════════════════════════════════ + test: + name: Test (pytest) + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + cache: 'pip' + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + + - name: Run tests + run: python -m pytest tests/ -v --tb=short + + # ════════════════════════════════════════════════════════════════════════ + # Job 2: Build — PyInstaller cross-platform executables + # ════════════════════════════════════════════════════════════════════════ build: name: Build on ${{ matrix.os }} + needs: test runs-on: ${{ matrix.os }} strategy: - fail-fast: false + fail-fast: true matrix: include: - os: windows-latest @@ -46,7 +80,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v5 with: - python-version: '3.11' + python-version: '3.12' cache: 'pip' - name: Install build tools (Linux only) @@ -55,22 +89,13 @@ jobs: sudo apt-get update sudo apt-get install -y upx - - name: Ensure requirements + - name: Install dependencies shell: bash run: | python -m pip install --upgrade pip - if [ ! -f requirements.txt ]; then - echo "jinja2>=3.0.0" > requirements.txt - fi pip install -r requirements.txt pip install "pyinstaller==6.11.*" - - name: Quick source test (run as script) - shell: bash - run: | - python -c "import sys; print(sys.version)" - python src/main.py --help - - name: Build (Windows) if: matrix.os == 'windows-latest' shell: powershell @@ -83,7 +108,7 @@ jobs: --exclude-module matplotlib ` --exclude-module PIL ` --paths=src ` - src/main.py + pyinstaller_entry.py if (!(Test-Path "dist\\${{ matrix.exe_name }}")) { Write-Error "Build failed"; exit 1 } - name: Build (Linux) @@ -98,7 +123,7 @@ jobs: --exclude-module matplotlib \ --exclude-module PIL \ --paths=src \ - src/main.py + pyinstaller_entry.py test -f "dist/${{ matrix.exe_name }}" || (echo "Build failed" && exit 1) upx --best --lzma "dist/${{ matrix.exe_name }}" || true @@ -107,19 +132,13 @@ jobs: run: | chmod +x "dist/${{ matrix.exe_name }}" || true "dist/${{ matrix.exe_name }}" --help || (echo "Executable failed to run" && exit 1) - - - name: Test BMC converter module inclusion + + - name: Verify BMC converter module inclusion shell: bash run: | python -c " - import sys - sys.path.insert(0, 'src') - try: - from convertors.convertors_bmc_switch_json import convert_bmc_switches - print('[OK] BMC converter module can be imported') - except ImportError as e: - print('[ERROR] BMC converter module import failed:', e) - sys.exit(1) + from src.convertors.convertors_bmc_switch_json import convert_bmc_switches + print('[OK] BMC converter module can be imported') " - name: Stage artifact @@ -135,47 +154,57 @@ jobs: name: ${{ matrix.asset_name }} path: artifact_staging/ - # release: - # name: Create Release - # needs: build - # runs-on: ubuntu-latest - # if: startsWith(github.ref, 'refs/tags/v') - # steps: - # - name: Checkout - # uses: actions/checkout@v4 - - # - name: Download artifacts - # uses: actions/download-artifact@v4 - # with: - # path: ./artifacts - - # - name: Show downloaded files - # run: ls -Rla ./artifacts - - # - name: Create GitHub Release - # uses: softprops/action-gh-release@v2 - # with: - # files: | - # ./artifacts/network-config-generator-windows-amd64/network_config_generator.exe - # ./artifacts/network-config-generator-linux-amd64/network_config_generator - # body: | - # ## Network Switch Config Generator ${{ github.ref_name }} - - # ### Download - # 1) Grab the file for your OS - # 2) (Linux) Make it executable: `chmod +x network_config_generator` - # 3) Run: `./network_config_generator --help` - - # ### Usage - # ```bash - # ./network_config_generator --input_json lab_input.json --output_folder output/ - # ./network_config_generator --input_json standard_input.json --output_folder output/ - # ./network_config_generator --input_json lab_input.json --convertor my.custom.convertor - # ``` - - # ### Features - # - Auto-detects input format (lab vs standard) - # - Converts lab format to standard automatically - # - Generates network switch configurations - # - Custom convertors & multi-switch support - # - Cross-platform executables + # ════════════════════════════════════════════════════════════════════════ + # Job 3: Release — create GitHub Release (tag pushes only) + # ════════════════════════════════════════════════════════════════════════ + release: + name: Create Release + needs: build + runs-on: ubuntu-latest + if: startsWith(github.ref, 'refs/tags/v') + + # Require approval via GitHub Environment before publishing + environment: production + + permissions: + contents: write + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Download artifacts + uses: actions/download-artifact@v4 + with: + path: ./artifacts + + - name: Show downloaded files + run: ls -Rla ./artifacts + + - name: Create GitHub Release + uses: softprops/action-gh-release@v2 + with: + files: | + ./artifacts/network-config-generator-windows-amd64/network_config_generator.exe + ./artifacts/network-config-generator-linux-amd64/network_config_generator + body: | + ## Network Switch Config Generator ${{ github.ref_name }} + + ### Download + 1) Grab the file for your OS + 2) (Linux) Make it executable: `chmod +x network_config_generator` + 3) Run: `./network_config_generator --help` + + ### Usage + ```bash + ./network_config_generator --input_json lab_input.json --output_folder output/ + ./network_config_generator --input_json standard_input.json --output_folder output/ + ./network_config_generator --input_json lab_input.json --convertor my.custom.convertor + ``` + + ### Features + - Auto-detects input format (lab vs standard) + - Converts lab format to standard automatically + - Generates network switch configurations + - Custom convertors & multi-switch support + - Cross-platform executables diff --git a/.github/workflows/triage-submissions.yml b/.github/workflows/triage-submissions.yml index 3214c44..1880d2b 100644 --- a/.github/workflows/triage-submissions.yml +++ b/.github/workflows/triage-submissions.yml @@ -7,6 +7,9 @@ on: issues: types: [opened, edited] +permissions: + issues: write + jobs: validate-and-assign: # Only run for config submissions @@ -71,16 +74,23 @@ jobs: const spamPatterns = [ /