AWS EKS Troubleshooting

Run from the extracted package folder using the affected environment's configuration. Confirm the account and Kubernetes context first:

NAMESPACE="aicc-staging"
RELEASE="aicc"
VALUES="values-staging.yaml"
kubectl config current-context
./scripts/aicc-eks.sh status --namespace "$NAMESPACE" --release "$RELEASE"
./scripts/aicc-preflight.sh --values "$VALUES" --namespace "$NAMESPACE" \
  --chart-version YOUR_CHART_VERSION
kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp

Resolve [FAIL] entries and review warnings, including checks skipped for missing AWS credentials. Preflight is not a complete readiness check. Its optional --db test creates a temporary Pod and tests only TCP reachability.

Images and chart

Symptom	Investigation
ImagePullBackOff or ErrImagePull	Verify the repository/tag in ECR, node pull permissions, and network access to image-pull dependencies. Mirror Redis as well as the six application images.
Chart cannot be pulled	Verify the OCI repository, chart version, and ECR Helm authentication. Default repository is aicc/aicc; a four-part image tag maps to a hyphenated chart version.
Unexpected tag or ECR check failure	Preserve one-line aicc-*.image.tag entries in this script revision and keep all six tags consistent.
Unsupported executable format or scheduling	Check node architecture. Supplied application images require linux/amd64.

The mirror script's --dry-run does not test registry credentials or the availability of source artifacts. An actual successful import and image approval are separate requirements.

Storage

If the PVC or Pods remain Pending, inspect the claim and a representative affected Pod:

kubectl -n "$NAMESPACE" get pvc
kubectl -n "$NAMESPACE" describe pvc YOUR_PVC_NAME
kubectl -n "$NAMESPACE" describe pod YOUR_POD_NAME
kubectl get storageclass efs-aicc -o yaml
kubectl -n kube-system get pods

Confirm EFS CSI availability and IAM permissions, the file system ID, access-point provisioning, RWX access, and UID/GID 10001. Check mount targets in worker-node Availability Zones and TCP 2049 access. Healthy CSI Pods alone do not prove controller authorization: CreateAccessPoint, Failed to create Access point, or role-discovery errors can indicate missing IAM configuration.

For permission failures, verify application write access to /aicc and access-point ownership. Do not remove the PVC or recreate the namespace as a routine troubleshooting step; configuration and Secrets are required for recovery.

Ingress, DNS and TLS

kubectl -n "$NAMESPACE" describe ingress "$RELEASE"
kubectl -n "$NAMESPACE" get service "$RELEASE-frontend"

• No ALB address: inspect controller events, IngressClass alb, IAM permissions, subnet discovery, and security groups.
• Certificate warning: verify Issued status, matching hostname, correct ARN, and ALB region.
• No response: check DNS against the ALB address and whether an internal ALB is reachable from the caller's network.
• HTTP 502/503: inspect frontend readiness, Service endpoints/target ports, ALB target health, and network access.
• Chat/API returns 404: keep ingress routed through the frontend service; do not add a separate /api ingress rule.
• Dropped WebSockets: verify the 3600-second idle timeout and any additional proxy timeouts.
• Upload failures: check the complete ingress path and application limit; NGINX annotations do not configure ALB.

For an authorized diagnostic session before ingress is ready:

kubectl -n "$NAMESPACE" port-forward service/"$RELEASE-frontend" 8080:80

Open http://localhost:8080 on that workstation. This is a diagnostic path, not the published production endpoint. Stop forwarding with Ctrl+C.

Database and setup

Use a cluster-reachable endpoint, not localhost, and an account with the required database/schema permissions. Check security groups, DNS, and the matching master-key Secret. The primary connection is configured in the wizard, not Helm values.

./scripts/aicc-preflight.sh --values "$VALUES" --namespace "$NAMESPACE" \
  --chart-version YOUR_CHART_VERSION --db your-db-host:5432
./scripts/aicc-eks.sh logs api --namespace "$NAMESPACE" --release "$RELEASE"

Database names are normalized to lowercase with underscores. Initial HTTP 400 from /api/setup/status is not sufficient evidence of failure. If work stays at Initiated after setup, inspect API, Worker, Scheduler, and Redis logs and readiness before scheduling a restart. The restart utility targets all release deployments and can interrupt messaging and calls.

If encrypted configuration cannot be read after a redeployment, compare the key and retained storage with the known-good backup. Do not generate a replacement master key for existing encrypted data.

Reset saved database setup

Use this procedure only when a replaced, restored, or emptied database requires the setup wizard to run again. Investigate network, credentials, and master-key errors first. Schedule a maintenance window, verify configuration and database backups, and confirm the account, cluster context, namespace, and release.

./scripts/aicc-eks.sh reset-setup --namespace "$NAMESPACE" --release "$RELEASE"

The command prompts for confirmation, copies /aicc/data/appsettings.json to appsettings.json.bak-<date> alongside the original, clears saved connection fields (DefaultConnection, SecretKey, ConnectionString), resets numeric DatabaseType to zero, and restarts all deployments bearing the release label. It requires a running API deployment, permission to execute commands in its Pod, and write access to the configuration file. Restarting can interrupt Redis messaging and voice sessions.

The database itself is not changed or deleted. Keep the existing master key and select the intended database when completing setup again. Protect the dated backup as sensitive configuration because it can contain credentials or encrypted connection settings; apply your access and retention policy. After setup, repeat the deployment verification checks.

Browser voice

1. Verify HTTPS, browser microphone permission, and an appropriately configured conversational agent/AI provider.
2. Verify the complete Helm global.extraEnv ICE configuration and referenced aicc-turn credential Secret without printing its contents.
3. Start a test call, then check the effective ICE servers:

./scripts/aicc-eks.sh logs voice --namespace "$NAMESPACE" --release "$RELEASE" \
  | grep 'ICE servers in use'

4. Confirm the expected TURN hostname, transports, credentials, relay-port range, and connectivity from browsers and Voice Pods.
5. Verify outbound connectivity to the configured speech and language providers.

After changing Helm values, apply an upgrade. After changing Secret-backed environment credentials, restart applicable workloads during the agreed maintenance window. Administration settings should not be assumed to override deployment environment values.

Support information

Provide the environment, release/chart versions, time range and timezone, recent changes, sanitized status/preflight output, relevant events, and service logs. Review diagnostics before sharing them: Helm output, application logs, and incorrectly populated values can contain sensitive data. Do not send master keys, database passwords, TURN credentials, or registry tokens in ordinary tickets.

For a failed upgrade, coordinate database compatibility and recovery before rolling back application resources. See Enterprise Operations.