TG-197: Add distributed deployment guide and automated PDF generation

docs/architecture.pdf

@@ -0,0 +1,51 @@
+%PDF-1.7
+%µ¶
+% Written by MuPDF 1.27.2
+
+1 0 obj
+<</Type/Catalog/Pages 2 0 R/Info<</Producer(MuPDF 1.27.2)>>>>
+endobj
+
+2 0 obj
+<</Type/Pages/Count 1/Kids[5 0 R]>>
+endobj
+
+3 0 obj
+<<>>
+endobj
+
+4 0 obj
+<</Length 44>>
+stream
+1 0 0 -1 -0 842 cm
+q
+36 36 523 770 re
+W n
+Q
+
+endstream
+endobj
+
+5 0 obj
+<</Type/Page/MediaBox[0 0 595 842]/Rotate 0/Resources 3 0 R/Contents 4 0 R/Parent 2 0 R>>
+endobj
+
+6 0 obj
+<</CreationDate(D:20260519072827+02'00')/ModDate(D:20260519072827+02'00')/Creator(PyMuPDF library: https://pypi.org/project/PyMuPDF)/Producer null/Title null/Author null/Subject null/Keywords null>>
+endobj
+
+xref
+0 7
+0000000000 65535 f 
+0000000042 00000 n 
+0000000120 00000 n 
+0000000172 00000 n 
+0000000193 00000 n 
+0000000286 00000 n 
+0000000392 00000 n 
+
+trailer
+<</Size 7/Info 6 0 R/Root 1 0 R/ID[<C2A7056BC39EC299C287C39AC38B13C2><FC0669A25B8EDADAD3A21B018C9E5C1B>]>>
+startxref
+607
+%%EOF

docs/distributed_deployment.md

@@ -1,95 +1,64 @@
-# Multi-Hypervisor Distributed Deployment (Proof of Concept)
+# Distributed Deployment Guide
 
-This document provides the exact procedure to decouple the monolithic `docker-compose.yml` into a fully distributed, cross-hypervisor microservice architecture.
+This document outlines the procedure to deploy the Local Food AI stack across a mixed topology of Windows 11 subsystems and hypervisors on the same local network.
 
-## 1. Architectural Topology
-To demonstrate cross-platform interoperability, the application stack is split across three distinct virtualized environments on the host machine.
+## Supported Hypervisor Topologies
+You can distribute the services across any combination of:
+- **Windows Subsystem for Linux (WSL 2)**: Ideal for the frontend and LLM nodes.
+- **Hyper-V**: Ideal for the Database node.
+- **VirtualBox**: Ideal for isolated Monitoring nodes.
 
-- **VM 1: Hyper-V (Ubuntu Server)**
-  - **Container**: `mysql` (Database Engine)
-  - **IP Subnet Allocation**: `192.168.130.170` (Bridged)
-- **VM 2: VirtualBox (Debian/Ubuntu)**
-  - **Container**: `ollama` (Local LLM) + `searxng` (Web Search)
-  - **IP Subnet Allocation**: `192.168.130.171` (Bridged)
-- **VM 3: WSL2 (Windows Subsystem for Linux)**
-  - **Container**: `app` (Streamlit Web Interface)
-  - **IP Subnet Allocation**: `192.168.130.172` (NAT/Bridged via Hyper-V switch)
+## Port Conflict Matrix
+When deploying nodes on the same IP subnet or host machine, ensure the following ports are open on your host firewall (e.g., Windows Defender Firewall) and not conflicting with existing services:
 
-## 2. Networking Configuration
-To ensure these isolated VMs can communicate, you must configure a **Bridged Virtual Switch**:
-1. Open Hyper-V Virtual Switch Manager.
-2. Create an "External" switch mapped to your physical network adapter.
-3. Attach VM 1 (Hyper-V) and VM 3 (WSL2) to this switch.
-4. In VirtualBox, set the Network Adapter for VM 2 to "Bridged Adapter" pointing to the same physical interface.
-5. Disable `ufw` or Windows Firewall for the `192.168.130.0/24` subnet on all hosts.
+| Service Name | Default Port | Protocol | Purpose |
+|--------------|--------------|----------|---------|
+| Nginx (App)  | `80`         | HTTP     | Main Application User Interface |
+| Streamlit    | `8502`       | HTTP     | Direct Application Interface |
+| SearXNG API  | `8080`       | HTTP     | AI web searching endpoint |
+| MySQL DB     | `3307`       | TCP      | Relational database port |
+| Zabbix Web   | `8081`       | HTTP     | Zabbix monitoring dashboard |
+| Zabbix HTTPS | `8444`       | HTTPS    | Zabbix monitoring dashboard secure |
+| Zabbix Agent | `10050`      | TCP      | Node metric scraping |
+| Zabbix Trap  | `10051`      | TCP      | Active monitoring trap receiver |
 
-## 3. Deployment Steps
+## Distributed Setup Procedure
 
-### Step 1: Deploy Database on Hyper-V
-On VM 1, create a `docker-compose.yml` containing *only* the MySQL service.
-```yaml
-services:
-  mysql:
-    build:
-      context: ./docker/mysql
-    ports:
-      - "3306:3306"
-      - "161:161/udp" # Expose SNMP
-    volumes:
-      - mysql_data:/var/lib/mysql
-```
+### 1. Network Bridging
+If you are using VirtualBox or Hyper-V, you **must** configure the VM network adapter to use a **Bridged Adapter** or **External Virtual Switch**. This ensures that the VMs receive an IP address on the same physical subnet as your host machine (e.g., `192.168.x.x`). 
 
-### Step 2: Deploy AI Engines on VirtualBox
-On VM 2, create a `docker-compose.yml` containing the AI services.
-```yaml
-services:
-  ollama:
-    image: ollama/ollama:latest
-    ports:
-      - "11434:11434"
-      - "161:161/udp" # Requires sidecar or custom image for SNMP
-  searxng:
-    image: searxng/searxng:latest
-    ports:
-      - "8080:8080"
-```
+For WSL 2, use `wsl --set-version <Distro> 2` and ensure `localhost` forwarding is enabled, or use a tool like `wsl-vpnkit` if you need a dedicated IP.
+
+### 2. Configure the Node via Python
+On each designated node, clone the repository and execute the interactive setup script.
 
-### Step 3: Deploy Frontend on WSL2
-On VM 3, configure the App container to point to the external IP addresses rather than Docker DNS hostnames.
-Update your `.env` file on WSL2:
-```ini
-DB_HOST=192.168.130.170
-OLLAMA_HOST=http://192.168.130.171:11434
-SEARXNG_HOST=http://192.168.130.171:8080
+```bash
+python scripts/setup_deploy.py
 ```
 
-## 4. SNMP Telemetry within Containers
-By default, Docker containers run a single process (PID 1). To run `snmpd` alongside the application in *every* container, we use `supervisord`.
+The script will ask you for:
+1. **Node Role**: Choose whether this node is the Database, the Application Frontend, or the Monitoring hub.
+2. **Network IPs**: If you are setting up the Application node, it will ask you for the IP address of the Database node (e.g., the Hyper-V VM IP).
+3. **Credentials**: It will securely generate a local `.env` file containing your passwords so they are not committed to Git.
 
-**Example Dockerfile Modification for App Container:**
-```dockerfile
-RUN apt-get update && apt-get install -y supervisor snmpd
-COPY snmpd.conf /etc/snmp/snmpd.conf
-COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
-EXPOSE 8501 161/udp
-CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
+### 3. Deploy Docker
+Once the script generates the role-specific `docker-compose.yml`, run:
+```bash
+docker compose up -d
 ```
 
-**supervisord.conf:**
-```ini
-[supervisord]
-nodaemon=true
+## Moving Docker Images Offline
+If your Hyper-V or VirtualBox nodes do not have internet access, you can transfer the Docker images directly from a machine that does.
 
-[program:app]
-command=streamlit run app.py
-autorestart=true
-
-[program:snmpd]
-command=/usr/sbin/snmpd -f
-autorestart=true
+**On the Internet-connected machine (Export):**
+```bash
+docker save -o local_food_app.tar local_food_ai-app:latest nginx:latest
+docker save -o local_food_db.tar mysql:8.0
+docker save -o local_food_monitoring.tar zabbix/zabbix-server-mysql:ubuntu-7.0-latest
 ```
 
-## 5. Zabbix Monitoring Integration
-1. On the Zabbix Server (`192.168.130.170:8081`), navigate to **Configuration > Hosts**.
-2. Add three separate Hosts corresponding to the VM IPs (`192.168.130.170`, `192.168.130.171`, `192.168.130.172`).
-3. Attach the "Linux SNMP" template to each host. Zabbix will now automatically poll CPU, RAM, and Disk I/O natively from within each Docker container across the distributed environment.
+**On the Offline Node (Import):**
+Copy the `.tar` files via USB or SCP, then run:
+```bash
+docker load -i local_food_app.tar
+```

scripts/generate_pdfs.py

@@ -0,0 +1,27 @@
+import os
+import glob
+from markdown_pdf import MarkdownPdf
+from markdown_pdf import Section
+
+def main():
+    docs_dir = os.path.join(os.path.dirname(__file__), '..', 'docs')
+    md_files = glob.glob(os.path.join(docs_dir, '*.md'))
+    
+    if not md_files:
+        print("No markdown files found in docs/")
+        return
+        
+    for md_file in md_files:
+        pdf_file = md_file.replace('.md', '.pdf')
+        print(f"Converting {os.path.basename(md_file)} to PDF...")
+        
+        with open(md_file, 'r', encoding='utf-8') as f:
+            md_content = f.read()
+            
+        pdf = MarkdownPdf(toc_level=2)
+        pdf.add_section(Section(md_content))
+        pdf.save(pdf_file)
+        print(f"Saved {os.path.basename(pdf_file)}")
+
+if __name__ == "__main__":
+    main()

scripts/setup_deploy.py

@@ -0,0 +1,195 @@
+import os
+import sys
+import getpass
+
+def clear_screen():
+    os.system('cls' if os.name == 'nt' else 'clear')
+
+print("="*60)
+print(" Local Food AI - Distributed Deployment Configuration Tool")
+print("="*60)
+
+print("Select the role for this specific node in the network:")
+print("1. All-in-One (Runs everything, default)")
+print("2. Application Frontend (Runs Streamlit, Nginx, AI Services)")
+print("3. Database Node (Runs MySQL & Ingestion)")
+print("4. Monitoring Node (Runs Zabbix Server & UI)")
+
+choice = input("\nEnter choice (1-4) [1]: ").strip() or "1"
+
+# Environment Variables
+env_vars = {}
+
+if choice != "1":
+    print("\n--- Network Configuration ---")
+    if choice != "3":
+        env_vars['DB_HOST'] = input("Enter the IP address of the Database Node: ").strip()
+    else:
+        env_vars['DB_HOST'] = "mysql"
+        
+    if choice != "4":
+        env_vars['ZBX_SERVER_HOST'] = input("Enter the IP address of the Monitoring Node: ").strip()
+    else:
+        env_vars['ZBX_SERVER_HOST'] = "zabbix-server"
+else:
+    env_vars['DB_HOST'] = "mysql"
+    env_vars['ZBX_SERVER_HOST'] = "zabbix-server"
+
+print("\n--- Security Configuration ---")
+env_vars['MYSQL_ROOT_PASSWORD'] = getpass.getpass("Enter MySQL Root Password (will not echo): ")
+env_vars['DB_READER_PASS'] = getpass.getpass("Enter DB Reader Password: ")
+env_vars['DB_LOADER_PASS'] = getpass.getpass("Enter DB Loader Password: ")
+env_vars['DB_APP_AUTH_PASS'] = getpass.getpass("Enter App Auth Password: ")
+env_vars['MYSQL_ZABBIX_PASSWORD'] = getpass.getpass("Enter Zabbix DB Password: ")
+
+# Generate .env
+print("\n[+] Generating .env file...")
+with open(".env", "w") as f:
+    for k, v in env_vars.items():
+        f.write(f"{k}={v}\n")
+
+# Base compose dictionaries
+compose_services = {}
+
+mysql_service = """
+  mysql:
+    build:
+      context: ./docker/mysql
+    ports:
+      - "3307:3306"
+    volumes:
+      - mysql_data:/var/lib/mysql
+      - ./my.cnf:/etc/mysql/conf.d/custom_ai_app.cnf
+      - ./init.sql:/docker-entrypoint-initdb.d/1-init.sql
+    environment:
+      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+    healthcheck:
+      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
+      interval: 10s
+      timeout: 5s
+      retries: 20
+    restart: always
+"""
+
+ingest_service = """
+  ingest:
+    build:
+      context: .
+      dockerfile: docker/ingest/Dockerfile
+    environment:
+      - DB_HOST=${DB_HOST}
+      - DB_USER=food_loader
+      - DB_PASS=${DB_LOADER_PASS}
+    volumes:
+      - ./:/app
+    profiles:
+      - manual
+"""
+
+ai_services = """
+  ollama:
+    image: ollama/ollama:latest
+    volumes:
+      - ollama_data:/root/.ollama
+    restart: always
+
+  searxng:
+    image: searxng/searxng:latest
+    ports:
+      - "8080:8080"
+    volumes:
+      - ./searxng:/etc/searxng
+    environment:
+      - SEARXNG_BASE_URL=http://localhost:8080/
+    restart: always
+"""
+
+app_service = """
+  app:
+    build:
+      context: .
+      dockerfile: docker/app/Dockerfile
+    ports:
+      - "8502:8501"
+    environment:
+      - DB_HOST=${DB_HOST}
+      - DB_USER=food_reader
+      - DB_PASS=${DB_READER_PASS}
+      - APP_AUTH_USER=food_app_auth
+      - APP_AUTH_PASS=${DB_APP_AUTH_PASS}
+      - OLLAMA_HOST=http://ollama:11434
+      - SEARXNG_HOST=http://searxng:8080
+    restart: always
+
+  nginx:
+    image: nginx:latest
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+    restart: always
+"""
+
+monitoring_services = """
+  zabbix-server:
+    image: zabbix/zabbix-server-mysql:ubuntu-7.0-latest
+    environment:
+      - DB_SERVER_HOST=${DB_HOST}
+      - MYSQL_USER=zabbix
+      - MYSQL_PASSWORD=${MYSQL_ZABBIX_PASSWORD}
+      - ZBX_SNMPTRAPPER=1
+    restart: always
+    ports:
+      - "10051:10051"
+
+  zabbix-web:
+    image: zabbix/zabbix-web-nginx-mysql:ubuntu-7.0-latest
+    ports:
+      - "8081:8080"
+      - "8444:8443"
+    environment:
+      - DB_SERVER_HOST=${DB_HOST}
+      - MYSQL_USER=zabbix
+      - MYSQL_PASSWORD=${MYSQL_ZABBIX_PASSWORD}
+      - ZBX_SERVER_HOST=zabbix-server
+      - PHP_TZ=Europe/Paris
+    restart: always
+
+  zabbix-agent:
+    image: zabbix/zabbix-agent:ubuntu-7.0-latest
+    environment:
+      - ZBX_HOSTNAME=DistributedNode
+      - ZBX_SERVER_HOST=${ZBX_SERVER_HOST}
+    privileged: true
+    pid: "host"
+    volumes:
+      - /var/run:/var/run
+    restart: always
+"""
+
+header = "services:\n"
+footer = """
+volumes:
+  mysql_data:
+  ollama_data:
+"""
+
+compose_content = header
+
+if choice == "1":
+    compose_content += mysql_service + ingest_service + ai_services + app_service + monitoring_services
+elif choice == "2":
+    compose_content += ai_services + app_service
+    footer = "volumes:\n  ollama_data:\n"
+elif choice == "3":
+    compose_content += mysql_service + ingest_service
+    footer = "volumes:\n  mysql_data:\n"
+elif choice == "4":
+    compose_content += monitoring_services
+    footer = ""
+
+print("[+] Generating docker-compose.yml for selected role...")
+with open("docker-compose.yml", "w") as f:
+    f.write(compose_content + footer)
+
+print("\nDone! You can now run `docker compose up -d`.")