AWS EKS Enterprise Operations
Use this guide to prepare and operate an enterprise AI Command Center deployment on AWS EKS. Establish operating procedures with Customer Infrastructure, DevOps, Security, application owners, and IntelliBuddies before making Production available. Review the prerequisites and deployment verification checklist alongside these requirements.
The guidance below defines operational expectations. Product-specific scaling, recovery, and upgrade procedures must match the deployed IB-X release and the configuration agreed with IntelliBuddies.
Operational ownership
| Area | Customer responsibility | IntelliBuddies responsibility |
|---|---|---|
| Infrastructure | Operate compute, networking, storage, databases, and supporting services | Provide application infrastructure requirements |
| Security | Manage access, certificates, secrets, infrastructure patching, and security monitoring | Provide application security and release guidance |
| Availability | Define service objectives, capacity, maintenance windows, and incident response | Confirm supported application deployment and scaling behavior |
| Recovery | Implement backups, protect recovery credentials, and conduct restoration exercises | Provide application recovery requirements and guidance |
| Releases | Approve, test, deploy, and record releases | Supply release artifacts, compatibility information, and upgrade guidance |
| Support | Collect sanitized diagnostics and coordinate incidents | Investigate application issues through the agreed support process |
Record named owners, escalation contacts, and support coverage for each environment. Agree on availability targets, acceptable maintenance downtime, and incident severity definitions.
Supported configuration and release inventory
Maintain an inventory of the IB-X release, Helm chart version, container image references, EKS version, node operating system and architecture, Helm client, ingress controller, storage drivers, PostgreSQL, and applicable ingestion or supporting services. Confirm compatibility with IntelliBuddies before introducing a version change.
Record the release name, namespace, environment configuration, storage mappings, and external endpoints. Store configuration in controlled version management without plaintext credentials. Prefer immutable image references and retain the approved artifacts needed for recovery.
Where the deployment package uses Redis or an object store, confirm their purpose, ownership, availability, security, persistence, and recovery requirements for the selected release. Include external services in monitoring and capacity planning.
Security and access
Apply least privilege to AWS identities, Kubernetes access, deployment automation, and application administration. Separate routine deployment access from administrative access, and review permissions periodically. Give application workloads only the AWS permissions they require.
Restrict access to the cluster API, application endpoints, databases, and supporting services according to customer policy. Configure security groups and supported network controls for necessary traffic. Separate Staging and Production access; namespaces alone do not define the complete security boundary.
Use HTTPS for external application access and monitor certificate expiry. Define encryption requirements for databases, persistent volumes, backups, and internal connections according to customer policy and supported application configuration.
Protect database credentials, registry credentials, application secrets, and the application master key. Restrict access to configuration files containing sensitive values. Use an approved secrets-management process and validate credential rotation with the deployed application before applying it in Production. Preserve the master key required to recover existing application data; coordinate any key change with IntelliBuddies.
Scan approved images, define vulnerability remediation ownership, and maintain node and cluster components. Collect appropriate infrastructure, Kubernetes, and application audit records with controlled access and retention. Exclude credentials and sensitive customer data from routine diagnostic exports.
For infrastructure controls, follow the Amazon EKS security guidance.
Availability and capacity
The default single replica per application service does not establish application high availability. Confirm supported replica counts and coordination behavior for each service before scaling. In particular, confirm scheduler coordination, worker execution behavior, shared configuration access, and voice-session affinity and draining.
Design node placement and database availability to meet the agreed failure scenarios. Verify that persistent storage supports the intended placement across nodes and Availability Zones. Multiple replicas do not remove a shared storage or database dependency.
Set resource requests and limits using representative Staging measurements. Measure concurrent executions, scheduling delay, ingestion throughput, voice concurrency, database connections, and storage growth. Maintain capacity headroom and define thresholds for review. Introduce autoscaling only for services whose scaling behavior has been confirmed.
Plan node maintenance and Pod disruption around active executions and voice conversations. Validate drain behavior and service recovery before relying on disruption policies or rolling updates for continuity. Test failure scenarios and define whether affected work resumes, fails, or requires operator action; do not assume automatic replay is safe.
Use the Amazon EKS reliability guidance when designing infrastructure availability.
Monitoring and incident response
Centralize operational logs and monitoring using customer-approved tools. Define retention, access controls, alert thresholds, and an owner for each alert. Infrastructure monitoring should be accompanied by application checks that establish usability and execution progress.
| Area | Monitor and investigate |
|---|---|
| Kubernetes | Unavailable workloads, restart loops, failed scheduling, image pulls, node capacity, and resource exhaustion |
| Application | Endpoint availability, error rates, latency, failed or stalled executions, and scheduling delays |
| Databases and supporting services | Connectivity, capacity, storage growth, service health, and backup failures |
| Storage | Mount failures, permissions, free capacity, and persistence of configuration and runtime data |
| Networking | DNS, ingress errors, TLS expiry, and required external connectivity |
| Optional capabilities | Ingestion failures, voice connectivity and session errors, and archival processing |
Where an IB-X release does not expose a required metric directly, agree on a supported log-based or functional check. Monitor license status and required licensing connectivity under the applicable licensing configuration.
For incidents, record the environment, release, time range and timezone, affected functions, workload impact, and recent changes. Collect relevant Pod status, Kubernetes events, sanitized logs, and dependency health. Assign an incident owner and preserve evidence before making changes. Escalate application issues through the agreed IntelliBuddies support channel; provide credentials only through an approved secure mechanism when required.
Backup and disaster recovery
Define a recovery point objective (RPO), the acceptable amount of data loss, and a recovery time objective (RTO), the acceptable restoration time, for each environment. Set backup frequency, retention, protection, and recovery location to meet those objectives. Document which failures the recovery plan addresses, including database loss, volume loss, and loss of the deployment environment.
| Asset | Recovery requirements |
|---|---|
| Primary and archival databases | Consistent backups, retention, database-native recovery procedures, and restored connectivity |
| Ingestion data stores | Coordinated recovery of PostgreSQL, Qdrant, and Neo4j when used; confirm any supported re-ingestion procedure |
| Persistent application data | Back up writable configuration, documents, and applicable runtime files |
| Secrets and master key | Protect recoverable copies or references and retain access to required encryption keys |
| Deployment configuration | Preserve release-specific Helm values, infrastructure configuration, namespaces, and storage mappings |
| Release artifacts | Retain approved image references and access to compatible images and Helm packages |
| Supporting services | Back up persistent data or document supported reconstruction according to their role |
Confirm application consistency requirements before selecting backup timing or restoring databases and files from different recovery points. A volume snapshot alone does not establish application-consistent recovery. Monitor backup failures and protect backups from unauthorized modification or deletion.
Recovery exercise
- Select a recovery point and compatible release using the agreed recovery procedure.
- Provision an isolated recovery environment and restrict outbound integrations to avoid duplicate executions or unintended external actions.
- Restore databases, persistent data, required secrets and keys, and deployment configuration according to the agreed consistency and startup requirements.
- Deploy the matching application release and verify database access, configuration, licensing, documents, and applicable integrations.
- Check execution state and determine how interrupted work should be handled before enabling workers, scheduling, or external traffic.
- Measure achieved RPO and RTO, record discrepancies, and obtain approval before switching Production traffic or execution ownership.
Perform restoration exercises periodically and after material architecture changes. Successful backup jobs are insufficient evidence of recoverability.
Upgrades and rollback
Review release notes and compatibility information before changing IB-X, its chart, cluster components, or dependencies. Identify database migrations, configuration changes, expected downtime, and recovery requirements with IntelliBuddies.
Validate the upgrade in Staging with representative data and workloads. Confirm application setup, execution, scheduling, optional capabilities, persistence, and master-key continuity. Take the required backups, record the previous configuration and artifact references, and agree on a maintenance and rollback decision window.
During Production maintenance, control new work and active sessions according to the supported application procedure. Deploy the approved release, perform the acceptance checks, and monitor application and dependency health before closing the change.
A Helm rollback does not automatically reverse database migrations or restore persistent data. Confirm whether the previous application release can use the modified schema and configuration. If it cannot, use the agreed database and data recovery procedure. Test rollback or recovery in Staging before making it part of the Production change plan.
Production acceptance
Record evidence and approval for the following criteria before opening Production:
- Supported release and infrastructure versions are documented, with approved images in ECR.
- Security controls, secrets ownership, access reviews, and certificate renewal procedures are established.
- Representative workloads meet agreed performance targets with sufficient capacity headroom.
- Node or Pod failure and maintenance scenarios demonstrate the agreed application recovery behavior.
- Configuration and runtime data survive restarts and the tested upgrade procedure.
- Backups are monitored and a restoration exercise meets the agreed RPO and RTO.
- Upgrade and rollback or recovery procedures are tested and assigned to named owners.
- Monitoring, actionable alerts, escalation contacts, and support diagnostics are available.
- The deployment verification checklist is completed, including licensing and optional capabilities in use.
Retain the acceptance record, configuration inventory, recovery results, and change history with the customer's operational documentation.