Skip to main content

Kiali Service Mesh Observability

Kiali provides real-time visualization and validation of the Istio service mesh in Rossoctl. It is used both as a developer dashboard and as an automated E2E validation tool in CI.

Architecture​

How Kiali Fits in Rossoctl​

Rossoctl Traffic Flow​

All inter-service traffic within the mesh is encrypted via mTLS (mutual TLS) through Istio's ZTunnel in Ambient mode.

What We Check​

Our Kiali E2E validation tests run after all other tests complete (two-phase approach) and verify:

1. Istio Configuration Validation​

Queries Kiali for misconfigurations in Istio resources across all Rossoctl namespaces:

ResourceWhat's Checked
VirtualServicesRoutes reference valid destinations
DestinationRulesHosts exist in service registry
GatewaysMatching VirtualService exists
PeerAuthenticationValid selector labels
AuthorizationPoliciesValid principal/namespace references

2. Traffic Error Detection (2h window)​

Analyzes the service mesh traffic graph from the last 2 hours:

  • Detects HTTP 5xx errors above threshold (default: 1%)
  • Reports every service-to-service communication edge with pass/fail status
  • Groups results by namespace for readability

3. mTLS Compliance​

Verifies every observed traffic edge uses mutual TLS:

  • Fails with a detailed list of unencrypted communications
  • Critical for zero-trust security validation
  • Reports all edges with their encryption status

Test Execution Flow​

Configuration​

VariableDefaultDescription
KIALI_URLAuto-detectOverride Kiali endpoint
KIALI_NAMESPACESRossoctl namespacesComma-separated list
KIALI_IGNORE_NAMESPACESOpenShift AI, GPUNamespaces to skip
KIALI_SKIP_VALIDATION_CODESNoneValidation codes to ignore
KIALI_FAIL_ON_WARNINGStrueFail on config warnings
KIALI_TRAFFIC_DURATION2hTime window for traffic analysis
KIALI_ERROR_RATE_THRESHOLD0.01Max allowed error rate (1%)

Standalone Usage​

The test file doubles as a CLI tool for ad-hoc scanning:

# Basic validation scan
python rossoctl/tests/e2e/test_kiali_validations.py

# With traffic analysis over 2 hours
python rossoctl/tests/e2e/test_kiali_validations.py --check-traffic --duration 2h

# JSON output for automation
python rossoctl/tests/e2e/test_kiali_validations.py --json --check-traffic

# Scan specific namespaces
python rossoctl/tests/e2e/test_kiali_validations.py --namespaces team1,team2

Sample CI Output​

Traffic Health Report (all edges)​

==========================================================================================
 Traffic Health Report
==========================================================================================

  STATUS   SOURCE                                   DEST                                     PROTO  REQS     ERR%     mTLS
  --------------------------------------------------------------------------------------------------------
  PASS     rossoctl-system/otel-collector             rossoctl-system/phoenix                   http   342      0.0%     yes
  PASS     team1/weather-service                     team1/weather-tool                       http   156      0.0%     yes
  PASS     team1/weather-service                     keycloak/keycloak                        http   48       0.0%     yes
  FAIL     team1/weather-service                     external/api-service                     http   12       8.3%     NO

  SUMMARY
  --------------------------------------------------------------------------------------------------------
  Total edges:         4
  Total requests:      558
  Edges with errors:   1
  Edges without mTLS:  1

  1 edge(s) have issues - see FAIL entries above
==========================================================================================

Future Roadmap​

Phase 1: Expected Traffic Topology (Priority: High)​

Define expected service-to-service connections and verify they exist in the graph. Alert on unexpected traffic patterns (e.g., agent directly calling external APIs without going through the gateway).

Options:

  • A) YAML manifest defining expected edges -- simple, version-controlled
  • B) Auto-generate baseline from healthy cluster -- less manual, risk of codifying bugs

Phase 2: Baseline/Delta Mode (Priority: Medium)​

Store a validation baseline in git and only fail on NEW issues. Useful for gradually improving mesh hygiene without blocking CI on pre-existing issues.

Options:

  • A) JSON baseline file committed to repo -- transparent, easy to review
  • B) Annotation-based suppression on Istio resources -- closer to source of truth

Phase 3: Response Time SLO Validation (Priority: Medium)​

Check p99 response times from Kiali graph against defined SLOs. Catch performance regressions in CI.

Options:

  • A) Per-service SLO definitions in YAML -- flexible, per-service thresholds
  • B) Global p99 threshold -- simpler, single threshold for all services

Phase 4: AuthorizationPolicy Audit (Priority: Low)​

Use Kiali to verify AuthorizationPolicies are correctly applied and no services have overly permissive access.

Phase 5: CI Dashboard Screenshots (Priority: Low)​

Capture Kiali graph screenshots in CI using headless browser, attach to PR as artifacts for visual review.

Kiali Validation Codes Reference​

CodeSeverityDescription
KIA0101WarningNo matching workload for service
KIA0201WarningHost not in service registry
KIA0301WarningNo matching Gateway for VirtualService
KIA1001ErrorDestinationRule with no matching workloads
KIA1106ErrorVirtualService references undefined route
KIA1201ErrorPeerAuthentication with invalid selector

See Kiali Validation Documentation for the complete list.

References​