Kubernetes Module¶

Terraform module for core Kubernetes configuration including custom CoreDNS settings and the standalone Prometheus Operator CRD release. The preflight module gates this module, and the remaining cluster modules depend on it directly or through the Stage 2 dependency chain.

Architecture¶

flowchart TB
    subgraph terraform [Terraform ordering]
        Preflight["module.preflight<br/>chart compatibility gate"]
        KubernetesModule["module.kubernetes"]
        ExistingCoreDNS["data.kubernetes_config_map_v1.coredns_existing<br/>reads kube-system/coredns"]
        ManagedCoreDNS["kubernetes_config_map_v1.coredns<br/>count=1 only for kubeadm"]
        CRDRelease["helm_release.prometheus_operator_crds"]
    end

    subgraph k8s [Kubernetes cluster]
        subgraph kube_system [Namespace: kube-system]
            CoreDNS[CoreDNS]
            ConfigMap["CoreDNS ConfigMap"]
            HelmMetadata["prometheus-operator-crds<br/>Helm release metadata"]
        end

        subgraph crds [Cluster-scoped Prometheus Operator CRDs]
            AlertmanagerConfig["AlertmanagerConfig"]
            Alertmanager["Alertmanager"]
            PodMonitor["PodMonitor"]
            Probe["Probe"]
            PrometheusAgent["PrometheusAgent"]
            Prometheus["Prometheus"]
            PrometheusRule["PrometheusRule"]
            ScrapeConfig["ScrapeConfig"]
            ServiceMonitor["ServiceMonitor"]
            ThanosRuler["ThanosRuler"]
        end

        subgraph consumers [Downstream custom resources]
            IstioMonitors["module.istio_gateway<br/>PodMonitor"]
            MonitoringObjects["module.monitoring<br/>Prometheus, Alertmanager,<br/>PrometheusRule"]
            LaterMonitors["later modules<br/>ServiceMonitor, PodMonitor,<br/>ScrapeConfig"]
        end
    end

    Preflight -->|must complete first| KubernetesModule
    KubernetesModule -->|always reads current Corefile| ExistingCoreDNS
    KubernetesModule -->|kubeadm only| ManagedCoreDNS
    KubernetesModule --> CRDRelease
    ExistingCoreDNS -->|reads the current Corefile| ManagedCoreDNS
    ManagedCoreDNS -->|updates on kubeadm| ConfigMap
    ConfigMap -->|configures| CoreDNS
    CoreDNS -->|resolves| Domains[Custom Domains]
    CRDRelease -->|stores release state in| HelmMetadata
    CRDRelease -->|installs or adopts| AlertmanagerConfig
    CRDRelease -->|installs or adopts| Alertmanager
    CRDRelease -->|installs or adopts| PodMonitor
    CRDRelease -->|installs or adopts| Probe
    CRDRelease -->|installs or adopts| PrometheusAgent
    CRDRelease -->|installs or adopts| Prometheus
    CRDRelease -->|installs or adopts| PrometheusRule
    CRDRelease -->|installs or adopts| ScrapeConfig
    CRDRelease -->|installs or adopts| ServiceMonitor
    CRDRelease -->|installs or adopts| ThanosRuler
    PodMonitor -->|defines schema for| IstioMonitors
    ServiceMonitor -->|defines schema for| LaterMonitors
    PodMonitor -->|defines schema for| LaterMonitors
    ScrapeConfig -->|defines schema for| LaterMonitors
    Prometheus -->|defines schema for| MonitoringObjects
    Alertmanager -->|defines schema for| MonitoringObjects
    PrometheusRule -->|defines schema for| MonitoringObjects

Components¶

CoreDNS Custom Configuration¶

Resolves hairpin NAT issues by answering listed domains inside the cluster instead of through external DNS.

sequenceDiagram
    participant Pod
    participant CoreDNS
    participant External as External DNS

    Pod->>CoreDNS: Resolve gitlab.chrislee.local
    Note over CoreDNS: rewrite matches a kubernetes_gateway_domains entry
    CoreDNS-->>Pod: Return the gateway Service ClusterIP
    Note over Pod: Connects in-cluster, SNI still gitlab.chrislee.local

Hosts served by the Istio gateway use a rewrite, not a hosts entry. kubernetes_override_domains pins a name to kubernetes_override_ip. A name served by the Istio gateway belongs in kubernetes_gateway_domains instead, which emits a rewrite name <host> <gateway service FQDN> line in the main server block.

The rewrite resolves to the gateway's in-cluster Service rather than a pinned IP, so nothing needs a MetalLB address and the value survives Service recreation. rewrite runs before kubernetes and forward in the plugin chain, so the query is answered inside the cluster. CoreDNS restores the original name in the answer, so TLS SNI still carries the public hostname, which is what selects the gateway listener.

A name belongs to exactly one of the two lists. Moving it across is the DNS half of a cutover; the Cloudflare Tunnel route is the public half.

Only hosts with in-cluster callers need an entry. A gateway host absent from both lists still resolves, by leaving the cluster and returning through Cloudflare, which works but takes the long way round. Nothing reconciles this list against the ListenerSets, so a host that other pods call has to be added here by hand.

Prometheus Operator CRDs¶

The prometheus-operator-crds Helm release installs all ten cluster-scoped Prometheus Operator CRDs before any module that creates ServiceMonitor, PrometheusRule, or related resources.

The release runs in kube-system. take_ownership=true allows it to adopt existing CRDs, and each CRD receives helm.sh/resource-policy=keep so replacing or removing the release does not delete the CRDs and their stored custom resources.

kube-prometheus-stack sets crds.enabled=false, making this standalone release the sole CRD owner.

keep also survives teardown: a Stage 2 terraform destroy leaves the ten CRDs and their stored custom resources on the cluster, and a later chart release that drops a CRD orphans it rather than deleting it. Removing them is a deliberate manual step.

Reverting to stack-managed CRDs is likewise not a plain revert. Setting crds.enabled=true again fails with an ownership metadata error until meta.helm.sh/release-name and meta.helm.sh/release-namespace are rewritten on each of the ten CRDs.

Resources Created¶

kubernetes_config_map_v1.coredns - Modified CoreDNS configuration (kubeadm only)
helm_release.prometheus_operator_crds - Standalone Prometheus Operator CRD chart in kube-system

Variables¶

Name	Description	Default
`kubernetes_cluster_type`	Cluster type (kubeadm, k3s, minikube)	`kubeadm`
`kubernetes_override_domains`	Space-delimited domains pinned to `kubernetes_override_ip` in CoreDNS	`""`
`kubernetes_override_ip`	IP address for custom domain resolution	`192.168.1.100`
`kubernetes_gateway_domains`	Space-delimited hostnames resolved in-cluster to the Istio gateway	`""`
`kubernetes_gateway_service_fqdn`	In-cluster FQDN of the gateway Service, built from the gateway variables in `main.tf`	`public-istio.istio-ingress.svc.cluster.local`

Usage¶

Initial Setup (New Cluster)¶

For a new cluster, the CoreDNS ConfigMap must be imported first:

cd stage2
terraform import 'module.kubernetes.kubernetes_config_map_v1.coredns[0]' kube-system/coredns

Configure Custom Domains¶

Set in Terraform Cloud or .env:

TF_VAR_kubernetes_gateway_domains="gitlab.chrislee.local registry.chrislee.local minio.chrislee.local"

Prometheus CRDs Installed¶

CRD	Purpose
`alertmanagerconfigs.monitoring.coreos.com`	AlertManager configuration
`alertmanagers.monitoring.coreos.com`	AlertManager instances
`podmonitors.monitoring.coreos.com`	Pod metrics scraping
`probes.monitoring.coreos.com`	Blackbox probing
`prometheusagents.monitoring.coreos.com`	Prometheus agents
`prometheuses.monitoring.coreos.com`	Prometheus instances
`prometheusrules.monitoring.coreos.com`	Alerting rules
`scrapeconfigs.monitoring.coreos.com`	Scrape configurations
`servicemonitors.monitoring.coreos.com`	Service metrics scraping
`thanosrulers.monitoring.coreos.com`	Thanos rulers