High availability

This guide covers how to deploy Infrahub in a high availability configuration that eliminates single points of failure.

How HA works in Infrahub

A highly available Infrahub deployment runs redundant instances of every stateful and stateless component, so the system continues to operate even if individual components or availability zones fail:

• Neo4j cluster — 3+ nodes for graph database redundancy with automatic failover
• Redis Sentinel — automatic detection of master failures and promotion of replicas
• RabbitMQ cluster — message bus redundancy with quorum queues replicated across the cluster nodes
• PostgreSQL replication — task manager database with primary-replica setup
• Multiple API and worker instances — stateless services distributed across availability zones
• Redundant load balancers — fronting the API layer for traffic distribution

For the full architecture diagram and deeper explanation, see Scalability and High Availability.

The following sections show how to deploy this architecture using Terraform or Helm.

Deployment examples

The following examples demonstrate how to deploy Infrahub in a highly available configuration using Kubernetes. These deployments provide resilience, scalability, and are suitable for production environments.

warning

These examples are for reference purposes and may require customization for your specific environment. Ensure you understand the requirements and dependencies before deploying to production.

Version requirements

The examples connect Infrahub and Prefect directly to the Redis Sentinel nodes, which requires the following versions:

• Infrahub 1.11.10 or later. Earlier versions do not have the INFRAHUB_CACHE_URL setting, so they ignore it and connect to INFRAHUB_CACHE_ADDRESS, which defaults to localhost. Their task manager image also includes a version of Docket, the component that coordinates Prefect's background services, that does not accept redis+sentinel:// URLs. The task manager runs the Infrahub image, so the examples set the task manager image tag, prefectTag, to the same version as infrahubTag.
• Infrahub Enterprise chart 4.22.0 or later. Earlier charts include a prefect-server chart that rejects the backgroundServices.messaging.redis.url value, so the installation fails. Chart 4.22.0 is also the first release in which the task manager takes its startup lock in Redis through that messaging URL.
• Redis 6.0 or later, for the Redis nodes and for Sentinel. Infrahub opens each connection with the HELLO command, which Redis 5.0 and earlier reject.

Using Terraform

Example HA deployment using Terraform on a 3-node Kubernetes cluster

deploy.tf

terraform {
  required_providers {
    kubectl = {
      source  = "alekc/kubectl"
      version = "2.1.3"
    }
  }
}

provider "helm" {
  kubernetes = {
    config_path = "~/.kube/config"
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "kubectl" {
  config_path = "~/.kube/config"
}

locals {
  target_namespace = "infrahub"
  infrahub_version = "1.11.10"
}

### Infrahub

resource "helm_release" "infrahub_ha" {
  depends_on = [helm_release.cache_ha, helm_release.messagequeue_ha, helm_release.database_ha, helm_release.objectstore_ha, kubectl_manifest.taskmanagerdb_ha]

  name    = "infrahub"
  chart   = "oci://registry.opsmill.io/opsmill/chart/infrahub-enterprise"
  version = "4.22.0-small"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
infrahub:
  global:
    infrahubTag: ${local.infrahub_version}
  infrahubServer:
    podLabels:
      infrahub/service: server
    replicas: 3
    persistence:
      enabled: false
    infrahubServer:
      env:
        INFRAHUB_DB_ADDRESS: infrahub-headless.${local.target_namespace}.svc.cluster.local # use FQDN to match neo4j's Helm cluster domain so that client-side routing is used
        INFRAHUB_DB_PROTOCOL: neo4j # required for client-side routing
        INFRAHUB_BROKER_ADDRESS: messagequeue-rabbitmq
        INFRAHUB_CACHE_URL: redis+sentinel://cache-redis:26379/mymaster # connect directly to the Sentinel nodes; Infrahub follows master failover
        INFRAHUB_WORKFLOW_ADDRESS: prefect-server
        INFRAHUB_WORKFLOW_PORT: 4200
        PREFECT_API_URL: "http://prefect-server:4200/api"

        INFRAHUB_STORAGE_DRIVER: s3
        AWS_ACCESS_KEY_ID: admin
        AWS_SECRET_ACCESS_KEY: password
        AWS_S3_BUCKET_NAME: infrahub-data
        AWS_S3_ENDPOINT_URL: objectstore-minio:9000
        AWS_S3_USE_SSL: "false"

        INFRAHUB_ALLOW_ANONYMOUS_ACCESS: "true"
        INFRAHUB_DB_TYPE: neo4j
        INFRAHUB_LOG_LEVEL: INFO
        INFRAHUB_PRODUCTION: "false"
        INFRAHUB_INITIAL_ADMIN_TOKEN: 06438eb2-8019-4776-878c-0941b1f1d1ec
        INFRAHUB_SECURITY_SECRET_KEY: 327f747f-efac-42be-9e73-999f08f86b92
        INFRAHUB_GIT_REPOSITORIES_DIRECTORY: "/opt/infrahub/git"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                service: infrahub-server
            topologyKey: kubernetes.io/hostname
  infrahubTaskWorker:
    podLabels:
      infrahub/service: task-worker
    replicas: 3
    infrahubTaskWorker:
      env:
        INFRAHUB_DB_ADDRESS: infrahub-headless.${local.target_namespace}.svc.cluster.local # use FQDN to match neo4j's Helm cluster domain so that client-side routing is used
        INFRAHUB_DB_PROTOCOL: neo4j # required for client-side routing
        INFRAHUB_BROKER_ADDRESS: messagequeue-rabbitmq
        INFRAHUB_CACHE_URL: redis+sentinel://cache-redis:26379/mymaster # connect directly to the Sentinel nodes; Infrahub follows master failover
        INFRAHUB_WORKFLOW_ADDRESS: prefect-server
        INFRAHUB_WORKFLOW_PORT: 4200
        PREFECT_API_URL: "http://prefect-server:4200/api"

        INFRAHUB_STORAGE_DRIVER: s3
        AWS_ACCESS_KEY_ID: admin
        AWS_SECRET_ACCESS_KEY: password
        AWS_S3_BUCKET_NAME: infrahub-data
        AWS_S3_ENDPOINT_URL: objectstore-minio:9000
        AWS_S3_USE_SSL: "false"

        INFRAHUB_DB_TYPE: neo4j
        INFRAHUB_LOG_LEVEL: DEBUG
        INFRAHUB_PRODUCTION: "false"
        INFRAHUB_API_TOKEN: 06438eb2-8019-4776-878c-0941b1f1d1ec
        INFRAHUB_TIMEOUT: "60"
        INFRAHUB_GIT_REPOSITORIES_DIRECTORY: "/opt/infrahub/git"
        PREFECT_WORKER_QUERY_SECONDS: 3
        PREFECT_AGENT_QUERY_INTERVAL: 3
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                service: infrahub-task-worker
            topologyKey: kubernetes.io/hostname
  redis:
    enabled: false
  neo4j:
    enabled: false
  rabbitmq:
    enabled: false
  prefect-server:
    global:
      prefect:
        image:
          prefectTag: ${local.infrahub_version} # the task manager runs the Infrahub image, so it follows infrahubTag
    server:
      replicaCount: 3
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  infrahub/service: task-manager
              topologyKey: kubernetes.io/hostname
    backgroundServices:
      messaging:
        redis:
          # Passed to Prefect as PREFECT_REDIS_MESSAGING_URL: messaging discovers the master through the
          # Sentinel nodes and follows failover, the same as the cache above. A URL replaces the db
          # setting, so /1 keeps messaging on database 1, apart from the cache on database 0. The chart
          # also passes this URL to the task manager as INFRAHUB_CACHE_URL, for its startup lock.
          url: "redis+sentinel://cache-redis:26379/mymaster/1"
        docket:
          # Passed to Prefect as PREFECT_SERVER_DOCKET_URL, so that Docket also follows failover; /2
          # keeps it on database 2.
          url: "redis+sentinel://cache-redis:26379/mymaster/2"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  infrahub/service: task-manager-background-svc
              topologyKey: kubernetes.io/hostname
    secret:
      create: true
      name: ""
      username: "prefect"
      password: "prefect"
      host: "taskmanagerdb-rw"
      port: "5432"
      database: "prefect"
    serviceAccount:
      create: false
    postgresql:
      enabled: false
EOT
  ]
}

#### Infrahub dependencies

resource "helm_release" "database_ha_service" {
  depends_on = [helm_release.database_ha]

  name       = "database-service"
  chart      = "neo4j-headless-service"
  repository = "https://helm.neo4j.com/neo4j/"
  version    = "2026.5.0"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
neo4j:
  name: "infrahub"
EOT
  ]
}

resource "helm_release" "database_ha" {
  depends_on = [kubernetes_secret_v1.neo4j_secret]

  count = 3

  name       = "database-${count.index}"
  chart      = "neo4j"
  repository = "https://helm.neo4j.com/neo4j/"
  version    = "2026.5.0"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
neo4j:
  name: "infrahub"
  labels:
    infrahub/service: database
  minimumClusterSize: 3
  resources:
    cpu: "4"
    memory: "8Gi"
  passwordFromSecret: "neo4j-user"
  edition: "enterprise"
  acceptLicenseAgreement: "yes"
config:
  dbms.security.auth_minimum_password_length: "4"
  dbms.security.procedures.unrestricted: apoc.*
logInitialPassword: false
volumes:
  data:
    mode: "defaultStorageClass"
    defaultStorageClass:
      accessModes:
        - ReadWriteOnce
      requests:
        storage: 10Gi
services:
  neo4j:
    enabled: false
EOT
  ]
}

resource "helm_release" "messagequeue_ha" {
  name    = "messagequeue"
  chart   = "oci://registry-1.docker.io/bitnamicharts/rabbitmq"
  version = "14.4.1"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
replicaCount: 3
podLabels:
  infrahub/service: message-queue
image:
  repository: bitnamilegacy/rabbitmq
  tag: 4.1.3-debian-12-r1
auth:
  username: infrahub
  password: infrahub
metrics:
  enabled: true
startupProbe:
  enabled: true
podAntiAffinityPreset: hard
EOT
  ]
}

resource "helm_release" "objectstore_ha" {
  name    = "objectstore"
  chart   = "oci://registry-1.docker.io/bitnamicharts/minio"
  version = "15.0.5"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
global:
  security:
    allowInsecureImages: true
image:
  repository: bitnamilegacy/minio
mode: distributed
statefulset:
  replicaCount: 3
  drivesPerNode: 2
auth:
  rootUser: admin
  rootPassword: password
provisioning:
  enabled: true
  buckets:
    - name: infrahub-data
podAntiAffinityPreset: hard
volumePermissions:
  image:
    repository:	bitnamilegacy/os-shell
EOT
  ]
}

#### Task manager database

resource "helm_release" "cache_ha" {
  name       = "cache"
  chart      = "redis"
  repository = "https://charts.bitnami.com/bitnami"
  version    = "19.5.2"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
nameOverride: redis
image:
  repository: bitnamilegacy/redis
  tag: 8.2.1-debian-12-r0
architecture: replication
auth:
  enabled: false
replica:
  replicaCount: 3
  podAntiAffinityPreset: hard
  podLabels:
    infrahub/service: cache
sentinel:
  enabled: true
  image:
    repository: bitnamilegacy/redis-sentinel
    tag: 8.2.1-debian-12-r0
EOT
  ]
}

resource "kubectl_manifest" "taskmanagerdb_ha" {
  depends_on = [helm_release.taskmanagerdb_ha_operator, kubernetes_secret_v1.db_secret]

  yaml_body = <<EOT
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: taskmanagerdb
  namespace: ${local.target_namespace}
  labels:
    infrahub/service: task-manager-db
spec:
  instances: 3
  inheritedMetadata:
    labels:
      infrahub/service: task-manager-db
  storage:
    size: 10Gi
  postgresql:
    pg_hba:
      - host all all 10.0.0.0/8 md5
  bootstrap:
    initdb:
      database: prefect
      owner: prefect
      secret:
        name: prefect-user
EOT
}

resource "kubernetes_secret_v1" "db_secret" {
  depends_on = [helm_release.taskmanagerdb_ha_operator]

  metadata {
    name      = "prefect-user"
    namespace = local.target_namespace
  }
  data = {
    username = "prefect"
    password = "prefect"
  }
}

resource "kubernetes_secret_v1" "neo4j_secret" {
  depends_on = [helm_release.cache_ha] # wait for namespace creation

  metadata {
    name      = "neo4j-user"
    namespace = local.target_namespace
  }
  data = {
    NEO4J_AUTH = "neo4j/admin"
  }
}

resource "helm_release" "taskmanagerdb_ha_operator" {
  name       = "cnpg"
  chart      = "cloudnative-pg"
  repository = "https://cloudnative-pg.github.io/charts"
  version    = "0.23.2"

  create_namespace = true
  namespace        = local.target_namespace

  values = [
    <<EOT
EOT
  ]
}

Using Helm

Example HA deployment using Helm and kubectl

redis-values.yaml

---
nameOverride: redis
image:
  repository: bitnamilegacy/redis
  tag: 8.2.1-debian-12-r0
architecture: replication
auth:
  enabled: false
replica:
  replicaCount: 3
  podAntiAffinityPreset: hard
  podLabels:
    infrahub/service: cache
sentinel:
  enabled: true
  image:
    repository: bitnamilegacy/redis-sentinel
    tag: 8.2.1-debian-12-r0

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install cache bitnami/redis --version 19.5.2 --namespace infrahub --create-namespace --values redis-values.yaml

Infrahub connects to the Sentinel nodes directly through its cache connection URL, so it discovers the current master and follows failover with no proxy in between. infrahub-values.yaml below sets this URL for the Infrahub server and the task workers, which use it for the cache and the distributed locks:

INFRAHUB_CACHE_URL: redis+sentinel://cache-redis:26379/mymaster

The URL lists the Sentinel service (port 26379) and the master group name (mymaster). To list several Sentinel endpoints explicitly, separate them with commas: redis+sentinel://sentinel-a:26379,sentinel-b:26379,sentinel-c:26379/mymaster. When Redis requires a password, see Connecting to a Redis Sentinel that requires a password.

note

Prefect's messaging Redis also follows master failover when it connects through a redis+sentinel:// URL. infrahub-values.yaml below sets that URL with prefect-server.backgroundServices.messaging.redis.url, and the chart passes it to Prefect as PREFECT_REDIS_MESSAGING_URL. A URL replaces the chart's separate host, port, db, username, password and ssl settings, so put the database number in the URL: /mymaster/1 selects database 1, which keeps messaging apart from the cache on database 0.

The task manager takes an Infrahub lock in Redis while it starts. The chart passes the messaging URL to the task manager as INFRAHUB_CACHE_URL for that lock, unless you set INFRAHUB_CACHE_URL on the task manager yourself.

Docket coordinates Prefect's background services, such as the scheduler, through Redis. infrahub-values.yaml sets its URL with prefect-server.backgroundServices.messaging.docket.url, which the chart passes to Prefect as PREFECT_SERVER_DOCKET_URL. The URL uses the same format, and /mymaster/2 keeps Docket on database 2. The sizing presets set this value to an example host, and the chart refuses to install until you replace it.

See Prefect messaging over TLS below for the TLS options.

rabbitmq-values.yaml

---
replicaCount: 3
podLabels:
  infrahub/service: message-queue
image:
  repository: bitnamilegacy/rabbitmq
  tag: 4.1.3-debian-12-r1
auth:
  username: infrahub
  password: infrahub
metrics:
  enabled: true
startupProbe:
  enabled: true
podAntiAffinityPreset: hard

helm install messagequeue oci://registry-1.docker.io/bitnamicharts/rabbitmq --version 14.4.1 --namespace infrahub --create-namespace --values rabbitmq-values.yaml

neo4j-values.yaml

---
neo4j:
  name: "infrahub"
  labels:
    infrahub/service: database
  minimumClusterSize: 3
  resources:
    cpu: "4"
    memory: "8Gi"
  passwordFromSecret: "neo4j-user"
  edition: "enterprise"
  acceptLicenseAgreement: "yes"
config:
  dbms.security.auth_minimum_password_length: "4"
  dbms.security.procedures.unrestricted: apoc.*
logInitialPassword: false
volumes:
  data:
    mode: "defaultStorageClass"
    defaultStorageClass:
      accessModes:
        - ReadWriteOnce
      requests:
        storage: 10Gi
services:
  neo4j:
    enabled: false

helm repo add neo4j https://helm.neo4j.com/neo4j/
helm repo update
kubectl create secret generic neo4j-user --namespace infrahub --from-literal=NEO4J_AUTH=neo4j/admin
helm install database-0 neo4j/neo4j --version 2026.5.0 --namespace infrahub --values neo4j-values.yaml
helm install database-1 neo4j/neo4j --version 2026.5.0 --namespace infrahub --values neo4j-values.yaml
helm install database-2 neo4j/neo4j --version 2026.5.0 --namespace infrahub --values neo4j-values.yaml

Once all instances are running, install the headless service:

service-values.yaml

---
neo4j:
  name: "infrahub"

helm install database-service neo4j/neo4j-headless-service --version 2026.5.0 --namespace infrahub --values service-values.yaml

minio-values.yaml

---
global:
  security:
    allowInsecureImages: true
image:
  repository: bitnamilegacy/minio
mode: distributed
statefulset:
  replicaCount: 3
  drivesPerNode: 2
auth:
  rootUser: admin
  rootPassword: password
provisioning:
  enabled: true
  buckets:
    - name: infrahub-data
podAntiAffinityPreset: hard
volumePermissions:
  image:
    repository: bitnamilegacy/os-shell

helm install objectstore oci://registry-1.docker.io/bitnamicharts/minio --version 15.0.5 --namespace infrahub --create-namespace --values minio-values.yaml

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm install cnpg cnpg/cloudnative-pg --version 0.23.2 --namespace infrahub --create-namespace
kubectl create secret generic prefect-user --namespace infrahub --from-literal=username=prefect --from-literal=password=prefect

taskmanagerdb.yaml

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: taskmanagerdb
  namespace: infrahub
  labels:
    infrahub/service: task-manager-db
spec:
  instances: 3
  inheritedMetadata:
    labels:
      infrahub/service: task-manager-db
  storage:
    size: 10Gi
  postgresql:
    pg_hba:
      - host all all 10.0.0.0/8 md5
  bootstrap:
    initdb:
      database: prefect
      owner: prefect
      secret:
        name: prefect-user

kubectl apply -f taskmanagerdb.yaml --namespace infrahub

infrahub-values.yaml

---
# yamllint disable rule:line-length
infrahub:
  global:
    infrahubTag: 1.11.10
  infrahubServer:
    podLabels:
      infrahub/service: server
    replicas: 3
    persistence:
      enabled: false
    infrahubServer:
      env:
        INFRAHUB_DB_ADDRESS: infrahub-headless.infrahub.svc.cluster.local # use FQDN to match neo4j's Helm cluster domain so that client-side routing is used
        INFRAHUB_DB_PROTOCOL: neo4j # required for client-side routing
        INFRAHUB_BROKER_ADDRESS: messagequeue-rabbitmq
        INFRAHUB_BROKER_QUEUE_TYPE: quorum
        INFRAHUB_CACHE_URL: redis+sentinel://cache-redis:26379/mymaster # connect directly to the Sentinel nodes; Infrahub follows master failover
        INFRAHUB_WORKFLOW_ADDRESS: prefect-server
        INFRAHUB_WORKFLOW_PORT: 4200
        PREFECT_API_URL: "http://prefect-server:4200/api"
        INFRAHUB_STORAGE_DRIVER: s3
        AWS_ACCESS_KEY_ID: admin
        AWS_SECRET_ACCESS_KEY: password
        AWS_S3_BUCKET_NAME: infrahub-data
        AWS_S3_ENDPOINT_URL: objectstore-minio:9000
        AWS_S3_USE_SSL: "false"
        INFRAHUB_ALLOW_ANONYMOUS_ACCESS: "true"
        INFRAHUB_DB_TYPE: neo4j
        INFRAHUB_LOG_LEVEL: INFO
        INFRAHUB_PRODUCTION: "false"
        INFRAHUB_INITIAL_ADMIN_TOKEN: 06438eb2-8019-4776-878c-0941b1f1d1ec
        INFRAHUB_SECURITY_SECRET_KEY: 327f747f-efac-42be-9e73-999f08f86b92
        INFRAHUB_GIT_REPOSITORIES_DIRECTORY: "/opt/infrahub/git"
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                service: infrahub-server
            topologyKey: kubernetes.io/hostname
  infrahubTaskWorker:
    podLabels:
      infrahub/service: task-worker
    replicas: 3
    infrahubTaskWorker:
      env:
        INFRAHUB_DB_ADDRESS: infrahub-headless.infrahub.svc.cluster.local # use FQDN to match neo4j's Helm cluster domain so that client-side routing is used
        INFRAHUB_DB_PROTOCOL: neo4j # required for client-side routing
        INFRAHUB_BROKER_ADDRESS: messagequeue-rabbitmq
        INFRAHUB_BROKER_QUEUE_TYPE: quorum
        INFRAHUB_CACHE_URL: redis+sentinel://cache-redis:26379/mymaster # connect directly to the Sentinel nodes; Infrahub follows master failover
        INFRAHUB_WORKFLOW_ADDRESS: prefect-server
        INFRAHUB_WORKFLOW_PORT: 4200
        PREFECT_API_URL: "http://prefect-server:4200/api"
        INFRAHUB_STORAGE_DRIVER: s3
        AWS_ACCESS_KEY_ID: admin
        AWS_SECRET_ACCESS_KEY: password
        AWS_S3_BUCKET_NAME: infrahub-data
        AWS_S3_ENDPOINT_URL: objectstore-minio:9000
        AWS_S3_USE_SSL: "false"
        INFRAHUB_DB_TYPE: neo4j
        INFRAHUB_LOG_LEVEL: DEBUG
        INFRAHUB_PRODUCTION: "false"
        INFRAHUB_API_TOKEN: 06438eb2-8019-4776-878c-0941b1f1d1ec
        INFRAHUB_TIMEOUT: "60"
        INFRAHUB_GIT_REPOSITORIES_DIRECTORY: "/opt/infrahub/git"
        PREFECT_WORKER_QUERY_SECONDS: 3
        PREFECT_AGENT_QUERY_INTERVAL: 3
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                service: infrahub-task-worker
            topologyKey: kubernetes.io/hostname
  redis:
    enabled: false
  neo4j:
    enabled: false
  rabbitmq:
    enabled: false
  prefect-server:
    global:
      prefect:
        image:
          prefectTag: 1.11.10 # the task manager runs the Infrahub image, so it follows infrahubTag
    server:
      replicaCount: 3
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  infrahub/service: task-manager
              topologyKey: kubernetes.io/hostname
    backgroundServices:
      messaging:
        redis:
          # Passed to Prefect as PREFECT_REDIS_MESSAGING_URL: messaging discovers the master through the
          # Sentinel nodes and follows failover, the same as the cache above. A URL replaces the db
          # setting, so /1 keeps messaging on database 1, apart from the cache on database 0. The chart
          # also passes this URL to the task manager as INFRAHUB_CACHE_URL, for its startup lock.
          url: "redis+sentinel://cache-redis:26379/mymaster/1"
        docket:
          # Passed to Prefect as PREFECT_SERVER_DOCKET_URL, so that Docket also follows failover; /2
          # keeps it on database 2.
          url: "redis+sentinel://cache-redis:26379/mymaster/2"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  infrahub/service: task-manager-background-svc
              topologyKey: kubernetes.io/hostname
    secret:
      create: true
      name: ""
      username: "prefect"
      password: "prefect"
      host: "taskmanagerdb-rw"
      port: "5432"
      database: "prefect"
    serviceAccount:
      create: false
    postgresql:
      enabled: false

helm install infrahub oci://registry.opsmill.io/opsmill/chart/infrahub-enterprise --version 4.22.0-small --namespace infrahub --create-namespace --values infrahub-values.yaml

Connecting Infrahub to a Neo4j cluster

When running Neo4j as a cluster, set INFRAHUB_DB_PROTOCOL=neo4j so the driver discovers the cluster topology and routes reads and writes to the appropriate members. With the default bolt protocol, all traffic pins to a single member without routing awareness.

Infrahub needs at least one reachable cluster member to bootstrap this routing discovery, which can be provided in two ways:

• DNS routing: Point INFRAHUB_DB_ADDRESS at a single DNS name that resolves to all cluster members, such as the Kubernetes headless service used in the Helm example above (infrahub-headless.infrahub.svc.cluster.local).
• Member list: Set INFRAHUB_DB_ADDRESS to a comma-separated list of cluster members in host[:port] format. Members without an explicit port use INFRAHUB_DB_PORT. This is useful when no single DNS name covers all members, for example with virtual machine based deployments.

INFRAHUB_DB_PROTOCOL=neo4j
INFRAHUB_DB_ADDRESS=neo4j-core-1,neo4j-core-2,neo4j-core-3

In both cases the driver tries the members in turn until one answers, so the initial connection succeeds even when some members are down.

Connecting Infrahub to a RabbitMQ cluster

RabbitMQ 4.0 removed classic queue mirroring. A classic queue is stored on one node of the cluster: when that node stops, the queue is unavailable, and Infrahub cannot send or receive the messages that use it until the node is back. A quorum queue stores a copy of its messages on several nodes, and RabbitMQ elects a new leader for the queue when the node of the current leader stops.

Set INFRAHUB_BROKER_QUEUE_TYPE to quorum on the Infrahub server and the task workers, so that Infrahub declares its durable queues as quorum queues:

INFRAHUB_BROKER_QUEUE_TYPE=quorum

• The queues that each Infrahub process creates for its own replies and events stay classic queues. RabbitMQ deletes them when the process disconnects, and the process creates them again when it reconnects.
• A quorum queue stays available while a majority of its copies are on running nodes, so a 3-node cluster continues to operate when one node stops.
• Set INFRAHUB_BROKER_ADDRESS to an address that reaches every node of the cluster, such as the Kubernetes service of the RabbitMQ cluster or a TCP load balancer. When the node of a connection stops, Infrahub reconnects to that address.

warning

Do not set the default queue type of the RabbitMQ virtual host to quorum while INFRAHUB_BROKER_QUEUE_TYPE is classic. RabbitMQ then declares the durable queues of Infrahub as quorum queues with an argument that quorum queues reject, and the Infrahub server fails to start.

Switch an existing deployment to quorum queues

RabbitMQ does not change the type of an existing queue. When the durable queues already exist as classic queues, the Infrahub server fails to start with a PRECONDITION_FAILED - inequivalent arg error until you delete them. Deleting a queue discards the messages it holds.

1. Stop the Infrahub server and the task workers.

2. Delete the durable queues in the virtual host that Infrahub uses (INFRAHUB_BROKER_VIRTUALHOST, / by default). The queue names start with the value of INFRAHUB_BROKER_NAMESPACE, infrahub by default:

   rabbitmqctl delete_queue --vhost / infrahub.rpcs
   rabbitmqctl delete_queue --vhost / infrahub.delay.five_seconds
   rabbitmqctl delete_queue --vhost / infrahub.delay.ten_seconds
   rabbitmqctl delete_queue --vhost / infrahub.delay.twenty_seconds

3. Set INFRAHUB_BROKER_QUEUE_TYPE to quorum on the Infrahub server and the task workers.

4. Start the Infrahub server, then the task workers. The server declares the durable queues again as quorum queues.

To switch back to classic queues, follow the same steps with INFRAHUB_BROKER_QUEUE_TYPE set to classic.

Connecting to a Redis Sentinel that requires a password

When Redis requires a password, put the credentials in the connection URLs. Once INFRAHUB_CACHE_URL is set, Infrahub reads the username and password from the URL and ignores INFRAHUB_CACHE_USERNAME and INFRAHUB_CACHE_PASSWORD.

The following steps add authentication to the Helm example above. In redis-values.yaml, replace auth.enabled: false with the following settings, so that the Redis nodes and Sentinel both require the password stored in the cache-redis-password Secret:

auth:
  enabled: true
  sentinel: true
  existingSecret: cache-redis-password
  existingSecretPasswordKey: redis-password

Create that Secret before you install Redis:

kubectl create namespace infrahub
kubectl create secret generic cache-redis-password --namespace infrahub --from-literal=redis-password='<strong-cache-password>'

Each connection URL then contains the password twice:

• Before the host, for the Redis nodes: redis+sentinel://:<password>@cache-redis:26379/mymaster. The Bitnami chart sets the password of the default user, so the username before the : stays empty. For a Redis ACL user, write <username>:<password>@.
• In the sentinel_password query parameter, for the Sentinel nodes, which authenticate separately: ?sentinel_password=<password>. For a Sentinel ACL user, also set sentinel_username.

Percent-encode the password in both places. Characters such as /, ?, #, @, : and % have a meaning in URLs, and passwords generated in Base64 often contain /. When a /, ? or # that is not percent-encoded moves part of the credentials after the host, Infrahub stops at startup with an error that does not show the password. A % that is not percent-encoded is not detected: it can change the password that Infrahub sends, and Redis then rejects it.

Keep the URLs in Kubernetes Secrets, so that the password is not written in the values file. The following commands read the password from the Secret, percent-encode it, and store the Infrahub cache URL in the infrahub-cache Secret, and the Prefect messaging and Docket URLs in the prefect-redis Secret:

PASSWORD=$(kubectl get secret cache-redis-password --namespace infrahub -o jsonpath='{.data.redis-password}' | base64 -d)
ENCODED=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$PASSWORD")
kubectl create secret generic infrahub-cache --namespace infrahub \
  --from-literal=INFRAHUB_CACHE_URL="redis+sentinel://:${ENCODED}@cache-redis:26379/mymaster?sentinel_password=${ENCODED}"
kubectl create secret generic prefect-redis --namespace infrahub \
  --from-literal=messaging-url="redis+sentinel://:${ENCODED}@cache-redis:26379/mymaster/1?sentinel_password=${ENCODED}" \
  --from-literal=docket-url="redis+sentinel://:${ENCODED}@cache-redis:26379/mymaster/2?sentinel_password=${ENCODED}"

Then change infrahub-values.yaml as follows:

• Remove INFRAHUB_CACHE_URL from the env settings of infrahubServer and infrahubTaskWorker, and load it from the infrahub-cache Secret instead.
• Replace the messaging and Docket url values with references to the prefect-redis Secret. Set the Docket url to an empty value, because the sizing preset sets it to an example host, which the chart refuses.

infrahub:
  infrahubServer:
    infrahubServer:
      envFromExistingSecrets:
        - infrahub-cache
  infrahubTaskWorker:
    infrahubTaskWorker:
      envFromExistingSecrets:
        - infrahub-cache
  prefect-server:
    backgroundServices:
      messaging:
        redis:
          existingSecret: prefect-redis
          existingSecretUrlKey: messaging-url
        docket:
          url: ""
          existingSecret: prefect-redis
          existingSecretKey: docket-url

The task manager takes its startup lock through the messaging URL, so it authenticates with the password in that URL.

Connecting to Redis over TLS

The cache connection URL turns on TLS through its scheme: rediss:// for a single node and rediss+sentinel:// for a Sentinel topology. The standard redis-py query parameters then set how the connection verifies the server.

Query parameter	Description
ssl_cert_reqs	Certificate verification mode: required (default), optional, or none to skip verification. none also turns off the hostname check.
ssl_check_hostname	Whether to check that the certificate matches the hostname or IP address the client connects to (true/false). With false, the client still verifies the certificate against the certificate authority.
ssl_ca_certs	Path to a CA certificate or bundle in PEM format used to verify the server certificate.
ssl_ca_path	Path to a directory of CA certificates used to verify the server certificate.
ssl_ca_data	CA certificates in PEM format, written in the URL, used to verify the server certificate.
sentinel_username	Username to authenticate to the Sentinel daemons, independent of the data-node credentials.
sentinel_password	Password to authenticate to the Sentinel daemons, independent of the data-node credentials.

Any other query parameter is passed through to the Redis client as a standard connection option (for example max_connections, socket_timeout, health_check_interval). Infrahub checks the URL when it loads its configuration, so a misspelled parameter stops startup with an error instead of failing the first cache command. Each connection pool that an Infrahub process opens for the cache and the locks holds up to the number of connections set by INFRAHUB_CACHE_MAX_CONNECTIONS (1000 by default), and so does each of its pools to the Sentinel daemons; a max_connections parameter on the URL takes precedence. Infrahub checks a connection that has been idle for more than 20 seconds before a command uses it, so one whose server has become unreachable is re-established instead of failing the command; set health_check_interval on the URL to change that interval. A master that Sentinel demoted while it stayed reachable still answers that check, so the next write is what discovers the change: Redis replies READONLY, the client drops the connection, and the retry lands on the promoted master.

On a rediss+sentinel:// URL the ssl_* parameters are shared with the Sentinel daemon connections, so a single setting covers both the data nodes and the daemons. Verify against a private certificate authority by supplying a CA bundle:

INFRAHUB_CACHE_URL: rediss+sentinel://cache-redis:26379/mymaster?ssl_ca_certs=/etc/ssl/redis-ca.pem

Without ssl_ca_certs, ssl_ca_path or ssl_ca_data on the URL, a TLS connection verifies against INFRAHUB_CACHE_TLS_CA_FILE, or against the global INFRAHUB_TLS_CA_BUNDLE that fills it, so a private certificate authority configured once for the whole deployment covers the cache too. INFRAHUB_CACHE_TLS_INSECURE does not apply to a URL connection: only ssl_cert_reqs=none on the URL turns verification off.

Verifying the certificates of Redis nodes behind Sentinel

The client checks that each certificate matches the hostname or IP address it connects to. It connects to the Sentinel nodes listed in the URL, and then to the master address that Sentinel returns. By default, Sentinel returns IP addresses, so a certificate that lists only hostnames fails with an IP address mismatch error.

The following options keep certificate verification. They are listed from the strictest to the least strict:

1. Configure Sentinel to announce hostnames, and issue the Redis node certificates for those hostnames. Sentinel supports hostnames from Redis 6.2, through the sentinel announce-hostnames yes and sentinel resolve-hostnames yes settings, and each Redis node announces its own hostname through replica-announce-ip. The Bitnami Redis chart applies these settings when useHostnames is true, which is its default. In the Helm example above, it announces the nodes as cache-redis-node-<n>.cache-redis-headless.infrahub.svc.cluster.local, so a certificate whose subject alternative names include *.cache-redis-headless.infrahub.svc.cluster.local and the Sentinel service name cache-redis matches every connection.
2. Issue Redis node certificates whose subject alternative names include the IP addresses that Sentinel returns. This works for nodes with fixed IP addresses. It does not work for Kubernetes pods, whose IP addresses change when they restart.
3. Turn off only the hostname check with ssl_check_hostname=false. The client still verifies that your certificate authority signed the certificate, but it accepts any certificate signed by that authority, including a certificate issued for another server.

For option 3, add the parameter to the URL:

INFRAHUB_CACHE_URL: rediss+sentinel://cache-redis:26379/mymaster?ssl_ca_certs=/etc/ssl/redis-ca.pem&ssl_check_hostname=false

warning

ssl_cert_reqs=none turns off certificate verification. The client then accepts any certificate, so a party that can intercept the connection can read and change the traffic, including the Redis password. Use it only in test and development environments, for example with a self-signed certificate:

INFRAHUB_CACHE_URL: rediss+sentinel://cache-redis:26379/mymaster?ssl_cert_reqs=none

Prefect messaging over TLS

Infrahub's INFRAHUB_CACHE_URL configures the cache, distributed-lock and task result storage connections. Prefect's messaging Redis and Docket are configured separately, through the PREFECT_REDIS_MESSAGING_URL and PREFECT_SERVER_DOCKET_URL environment variables. The Helm chart sets them from prefect-server.backgroundServices.messaging.redis.url and prefect-server.backgroundServices.messaging.docket.url. Both accept the same URL grammar: use a rediss+sentinel:// URL so that they discover the master through the Sentinel nodes, follow failover and connect over TLS. Set the CA with ssl_ca_certs to verify against a private certificate authority:

export PREFECT_REDIS_MESSAGING_URL='rediss+sentinel://cache-redis:26379/mymaster/1?ssl_ca_certs=/etc/ssl/redis-ca.pem'
export PREFECT_SERVER_DOCKET_URL='rediss+sentinel://cache-redis:26379/mymaster/2?ssl_ca_certs=/etc/ssl/redis-ca.pem'

The options in Verifying the certificates of Redis nodes behind Sentinel apply to messaging too. To add a password, see Connecting to a Redis Sentinel that requires a password.

Validating Redis Sentinel failover

After deploying, confirm that Infrahub recovers automatically when the Redis master fails:

1. Identify the current master through any Sentinel node:

   kubectl exec -it cache-redis-node-0 --namespace infrahub -- redis-cli -p 26379 sentinel get-master-addr-by-name mymaster

2. Delete the master pod to trigger a Sentinel-driven promotion:

   kubectl delete pod <current-master-pod> --namespace infrahub

3. Once Sentinel has promoted a replica, run any operation that uses the cache or a distributed lock (for example, open the web UI or run a query). Infrahub resolves the new master through Sentinel and the operation succeeds without restarting any Infrahub pod.

Load balancer configuration

For high availability deployments, a load balancer is useful to distribute traffic across multiple Infrahub API server instances. The load balancer must be configured to properly handle both HTTP/HTTPS and WebSocket connections.

HAProxy configuration example

global
    maxconn 4096
    log stdout local0

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httplog

# Frontend for Infrahub API
frontend infrahub_frontend
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/infrahub.pem

    # Redirect HTTP to HTTPS
    redirect scheme https if !{ ssl_fc }

    # ACL for WebSocket connections
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket_path path_beg /ws /graphql-ws

    # Use WebSocket backend for WS connections
    use_backend infrahub_websocket if is_websocket || is_websocket_path

    # Default to HTTP backend
    default_backend infrahub_http

# Backend for regular HTTP/HTTPS traffic
backend infrahub_http
    balance roundrobin
    option httpchk GET /api/health
    http-check expect status 200

    # Sticky sessions for GraphQL subscriptions
    cookie SERVERID insert indirect nocache

    server infrahub-api-1 10.0.1.10:8000 check cookie api1
    server infrahub-api-2 10.0.1.11:8000 check cookie api2
    server infrahub-api-3 10.0.1.12:8000 check cookie api3

# Backend for WebSocket connections
backend infrahub_websocket
    balance source
    option http-server-close
    option forceclose

    # WebSocket specific health check
    option httpchk GET /api/health
    http-check expect status 200

    server infrahub-api-1 10.0.1.10:8000 check
    server infrahub-api-2 10.0.1.11:8000 check
    server infrahub-api-3 10.0.1.12:8000 check

NGINX configuration example

upstream infrahub_backend {
    # IP hash for session affinity
    ip_hash;

    server 10.0.1.10:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8000 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name infrahub.example.com;

    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name infrahub.example.com;

    ssl_certificate /etc/ssl/certs/infrahub.pem;
    ssl_certificate_key /etc/ssl/private/infrahub.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    # Client body size for large GraphQL queries
    client_max_body_size 10M;

    # Timeouts for long-running operations
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;

    # Health check endpoint
    location /health {
        access_log off;
        proxy_pass http://infrahub_backend/api/health;
    }

    # WebSocket support for GraphQL subscriptions
    location ~ ^/(ws|graphql-ws) {
        proxy_pass http://infrahub_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Disable buffering for WebSocket
        proxy_buffering off;
    }

    # Regular API traffic
    location / {
        proxy_pass http://infrahub_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Enable keepalive
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}