Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between Hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus model portfolio approaches
- Comparative analysis of cost per decision
Hardware Sizing and Capacity Planning
- Memory footprint assessment based on parameter count
- CPU-only versus GPU-accelerated performance
- Concurrency and throughput capacity planning
- Planning headroom for increasing task complexity
Serving Infrastructure
- vLLM and SGLang framework options
- llama.cpp and GGUF execution capabilities
- Ollama for local development workflows
- Selecting stacks based on decision workload characteristics
Quantization while Preserving Calibration
- Precision formats and associated memory efficiencies
- Assessing quality and calibration degradation
- Per-tensor and importance-aware quantization variants
- Post-quantization acceptance testing protocols
Latency Optimization
- Distinguishing median versus tail latency
- Latency behavior relative to context length
- Warm start techniques and model caching strategies
- Batching multiple decision queries for efficiency
Edge and Constrained Environments
- Running small decision models on CPU resources
- Deployment on Apple Silicon and embedded systems
- Offline operation and software update strategies
- Distribution of weights in air-gapped settings
Containerization and Scalability
- Designing container images for model serving
- Managing volumes and cache data
- Horizontal scaling and traffic routing mechanisms
- Resource constraints and GPU sharing protocols
Observability and Monitoring
- Tracking request, latency, and error metrics
- Monitoring confidence distribution patterns
- Recording decision outcomes and escalation events
- Configuring alerts for calibration drift
Versioning and Deployment Rollouts
- Versioning models and adapters
- Implementing shadow and canary deployment strategies
- Maintaining pinned model files for audit compliance
- Defining safe rollback procedures
Security and Multi-Tenancy
- Authentication mechanisms and rate limiting
- Tenant isolation for prompts and logging data
- Network segmentation and egress control
- Audit logging of all decision events
Requirements
- Experience with Linux administration and containerization.
- Familiarity with model inference servers and GPU management.
- Foundational knowledge of REST APIs, load balancing, and monitoring systems.
Target Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams supporting AI-driven services.
- AI engineers implementing decision models on-premises.
- Architects designing air-gapped or regulated AI deployment strategies.
