Operations handbook

This page describes how the DQC Platform is operated in a private cloud or on-premises installation: the recurring procedures, what to do after a failure, and the settings that harden an installation.

Division of responsibilities: you operate the cluster, the database and the network; DQC provides the container images, the deployment templates and support. Everything below runs in your infrastructure and under your change management.

Standard operating procedures

Procedure	Frequency	What happens
Initial installation	Once, per environment	Check tools and configuration, render the manifests, review the dry-run output, then deploy. The deployment order is NATS, Kratos, Oathkeeper, Keto, the platform manifests and - in self-hosted mode - the AI gateway.
Configuration change	On demand	Change values.yaml, render again, compare the rendered manifests, then apply. Configuration and secrets are never edited in the cluster directly.
Version update	Per release	Set the new image tags in values.yaml and deploy again. Stateless components are replaced as a rolling update; the database schema is migrated automatically when the API starts.
Backup	Daily, automated	Back up the platform database with the means of your PostgreSQL installation (point-in-time recovery, snapshots) and the persistent volumes or object storage with the mechanisms of your platform. Before every change to the deployment, take a release snapshot as well.
Health check	Continuous plus after every change	Every component answers on a health endpoint that Kubernetes uses for its liveness and readiness probes. After a change, check that the workloads qserviceapi, qserviceworker, qui, kratos, oathkeeper, keto, nats - and bifrost in self-hosted mode - are running.
User administration	On demand	Invitations, roles and groups are managed in the platform's settings, not in the cluster. Where SSO is in use, the identity provider remains the leading system.
Incident and support	On occurrence	Collect the symptom, the affected component and the relevant container logs, then contact DQC support. For a defective change, roll back first and analyse afterwards.

The deployment templates automate these procedures. The commands are run from the directory of your installation:

make check-config    # validate values.yaml and the secrets
make render          # render the manifests from the templates
make dry-run         # assemble complete-manifests.yaml, deploy nothing
make deploy          # roll out (charts first, then the platform manifests)
make update          # apply changed platform manifests only
make status          # state of the deployed components
make backup          # snapshot of the current release state
make rollback COMPONENT=<name>   # roll a single component back

In a GitOps setup the render step happens once per change at authoring time; the cluster only ever syncs the rendered manifests.

Restart and recovery plan

Components start in a fixed order, because each one builds on the previous: message bus, identity, access proxy, authorization, platform services, and finally the AI gateway. A full restart follows the same order.

Scenario	Procedure
Single pod or node fails	Kubernetes restarts the pod and reschedules it. The stateless components need no manual step; running checks are picked up again from the message bus.
Defective update	Roll the affected component back to the previous release, then re-render and re-apply from the last known-good configuration.
Platform database unavailable	The API and the workers cannot serve requests. Restore the database from your backup or fail over to the standby, then restart the API and the workers. The schema is checked and migrated automatically on start.
Identity or authorization service unavailable	Sign-in fails while the rest of the platform keeps running. Restart Kratos, Oathkeeper and Keto in that order; Keto reloads its namespace configuration on start.
AI gateway unavailable	Only the AI functions fail; checks, profiling, exports and alerts are unaffected. Verify the endpoint, the key and - in restricted networks - proxy and CA bundle.
Complete loss of the environment	Provision the cluster again, restore the database and the persistent volumes from backup, then deploy from the versioned configuration. Recovery time and the acceptable data loss follow from your backup regime, not from the platform.

Before every intervention: take a snapshot of the current state, so you can return to it.

make backup                      # snapshot the current release state
make rollback COMPONENT=keto     # roll a single component back
make render && make update-all   # re-apply from the known-good configuration

Hardening guide

Area	Recommendation
Transport	Terminate TLS 1.2 or 1.3 at the ingress and redirect plain HTTP. Certificates come either from cert-manager or from your own certificate management.
Network	Internal traffic stays inside the cluster; a single ingress endpoint is exposed. Reach cloud databases over private endpoints, and route outbound LLM traffic through your proxy where one is mandated.
Secrets	Credentials belong in Kubernetes secrets or an external secret store, never in the configuration file and never in version control. The configuration references them by environment variable.
Containers	All components run without root privileges and can be pinned to a fixed user id, which is what a restricted security context - OpenShift restricted-v2, for instance - requires.
Images	Pull the images from the access-protected registry provided by DQC, or mirror them into your own registry and pin them to a concrete release instead of a floating tag.
Database accounts	Give the platform its own database user, and give each connected source an account with only the read rights the checks need. Require TLS on database connections.
Identities	Where possible, use single sign-on and enforce multi-factor authentication - the platform supports TOTP as well as WebAuthn. Review API tokens regularly.
Authorizations	Grant tenant capabilities sparingly and control access to connectors, tables and rulesets over object-level rights. Tenants are separated at database schema level.
Telemetry	Error tracking and product telemetry are optional and can be switched off entirely; logs and traces then stay within your own observability stack.

Monitoring

All components write structured logs to standard output and can export traces and metrics over OpenTelemetry to your own collector.
Liveness and readiness probes are part of every workload, so Kubernetes restarts a component that stops answering.
Worth alerting on: pods not ready, database connections and disk usage, failed scheduled checks, and error rates on the AI gateway.