Infrastructure
One operator = one isolated deployment = one DC. Everything below the app - clusters, databases, networking - is provisioned by the Infrastructure team; app teams own what runs on top.
The DC model
A DC (data center) is one complete deployment environment, named dcNN-N. The staging fleet (dc02-2, dc22-2, the dc82 labs...) is shared by engineering; each production DC belongs to one operator and runs their whole stack - portal, DNO services, databases - isolated from every other operator. The Portal is reachable per DC (portal-xx.lotusflare.com, or an operator domain such as portal-globe-vpn.globetel.com).
What a DC runs for the Portal
- Gateway (Kong / Gateway API HTTPRoute) - the DC's front door, with internal/external/custom endpoint classes.
lf-portal-expresspods - the combined SPA + BFF image, deployed by thelf-portal-expressHelm chart with HPA, PodDisruptionBudget, and Cilium network policies.- Portal MySQL on RDS - sessions, tenants, views, roles, settings. Redis for caching. S3 for documents and assets. Kafka for events. StatsD/Prometheus for metrics.
Kubernetes, portal-sized
The minimum mental model that explains everything above:
- An image is a frozen snapshot of the app, built once by CI.
- A pod is a running copy of an image. Kubernetes keeps N copies alive; a rolling update replaces them one at a time with a new image, so the Portal never fully goes down during a deploy.
- Pods are stateless and disposable - anything written inside a pod dies with it. Everything that must survive lives outside: RDS, Redis, S3.
- The presync migration Job is a one-off pod that upgrades the MySQL schema before new app pods start, so code and schema never disagree.
Who owns what
| Layer | Owner |
|---|---|
| AWS accounts, VPCs, RDS, Redis, clusters | Infrastructure / DevOps team, via Terraform |
| Per-DC desired state | env/<dc>/ trees in ts-polaris (TS services), lfscala (Scala services), and cm (charts + shared config) |
| The portal app, its chart values, its alerts | platform-portal team |
| Feature areas inside the Portal | Their teams (DATAP owns Dataflows, DataHub, Reports) |
Observability and alerts
Metrics land in the management VictoriaMetrics/Prometheus (prometheus.mgmt.lotusflare.info); logs in VictoriaLogs / OpenSearch. Alert rules live in the observability-gitops repo; the portal team owns three: PortalUnhandledError, PortalBackendRequestError, and TMOPortalCPUUsageHigh.
When one fires, 24x7 Platform Support (HDS) triages first - reads the logs, checks downstream DNO health - and only pages the owning team through OpsGenie when it needs code or config changes. Production-impacting incidents become PI tickets, coordinated in #production_issues. There are no formal on-call rotations on the engineering side.
Next: Business - who buys this and who uses it.