LLMOps Guide

Model Deployment

Ray Serve Configuration

apiVersion: ray.io/v1alpha1
kind: RayService
metadata:
  name: llm-inference
spec:
  serviceUnhealthySecondThreshold: 300
  deploymentUnhealthySecondThreshold: 300
  serveDeployments:
    - name: llm-deployment
      numReplicas: 2
      rayStartParams:
        num-cpus: "16"
        num-gpus: "1"
      containerConfig:
        image: llm-server:latest
        env:
          - name: MODEL_NAME
            value: "llama2-7b"
          - name: BATCH_SIZE
            value: "4"

Model Monitoring

Prometheus Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-monitoring
spec:
  groups:
  - name: LLMMetrics
    rules:
    - alert: HighLatency
      expr: |
        rate(llm_inference_duration_seconds_sum[5m])
        /
        rate(llm_inference_duration_seconds_count[5m])
        > 1.0
      for: 5m
      labels:
        severity: warning
    - alert: HighErrorRate
      expr: |
        rate(llm_inference_errors_total[5m])
        /
        rate(llm_inference_requests_total[5m])
        > 0.01
      for: 5m
      labels:
        severity: critical

Performance Optimization

Triton Inference Server

apiVersion: serving.kubeflow.org/v1beta1
kind: InferenceService
metadata:
  name: llm-triton
spec:
  predictor:
    triton:
      storageUri: s3://models/llm
      resources:
        limits:
          cpu: "8"
          memory: "16Gi"
          nvidia.com/gpu: "1"
      containerConcurrency: 4
      env:
        - name: TRITON_CACHE_CONFIG
          value: |
            {
              "cache_size": "8GB",
              "cache_policy": "LRU"
            }

Best Practices

1. Model Management
   • Version control
   • A/B testing
   • Canary deployment
   • Model registry
2. Observability
   • Performance metrics
   • Token usage
   • Response quality
   • Cost tracking
3. Optimization
   • Quantization
   • Batching
   • Caching
   • Load balancing
4. Security
   • Input validation
   • Output filtering
   • Rate limiting
   • Access control