# Split the growth_opportunities catalog into dev and prod

Two environments were competing for one Unity Catalog name, so prod apply has failed since v0.67.0. This gives each environment its own catalog and stops the portal API silently reading dev.

| | |
| --- | --- |
| Status | OPEN · PLAN FAILING |
| Source | github:example/platform-infra#339 |
| Workspace | Showcase |
| Tags | terraform, unity-catalog, databricks, api-security |

## What it does

**One catalog name was shared by two environments, and the metastore only allows one owner.**

Unity Catalog names are unique per metastore. Both dev and prod declared growth_opportunities, dev won the race, and every prod apply since has failed on a name that already exists.

The fix introduces a catalog_name variable per environment. Dev resolves to growth_opportunities_dev and prod keeps the clean name, which avoids renaming the catalog production already reads from.

The second half of the change is unrelated to Terraform. The portal API resolved its catalog with a null-coalescing fallback, so an unset variable in prod quietly returned dev data instead of failing. That fallback is now a throw.

**Diagram: One name owned by two environments, and the same topology after the split**

- BEFORE
  - dev deploy (terragrunt apply) [neutral]
  - prod deploy (terragrunt apply) [neutral]
  - growth_opportunities (unique per metastore) [bad]
  - dev deploy -> growth_opportunities : wins
  - prod deploy --> growth_opportunities : fails
- AFTER
  - dev/terragrunt.hcl (catalog_name = …_dev) [neutral]
  - prod/terragrunt.hcl (catalog_name = clean) [neutral]
  - growth_opportunities_dev [accent]
  - growth_opportunities (prevent_destroy) [good]
  - dev/terragrunt.hcl -> growth_opportunities_dev
  - prod/terragrunt.hcl -> growth_opportunities

> **The plan is currently failing**
>
> terraform-plan is red on this branch. The brief describes what the diff intends, not a verified outcome, and nothing here should be read as evidence that the apply will succeed.

## How it works

**One required variable per environment fans out to every consumer.**

The catalog name stops being a constant anywhere. Each consumer takes it as input, and the API edge fails fast when it is missing.

- **Terragrunt declares the name per environment.** dev and prod each set catalog_name. The variable is required, so a new environment cannot forget it.
- **The DAB takes it explicitly.** email ingestion receives uc_catalog and target rather than deriving either from a workspace default.
- **The portal API throws when it is unset.** routes.ts previously fell back to dev. It now raises, so a misconfigured prod boot fails loudly at startup.
- **BDD features read a placeholder.** {CATALOG} is substituted at run time, so the suite runs against whichever environment it was pointed at.

```typescript
-const catalog = process.env.UC_CATALOG ?? "growth_opportunities";
+const catalog = process.env.UC_CATALOG;
+if (!catalog) {
+  throw new Error("UC_CATALOG is not set. Refusing to guess a catalog.");
+}
```

_The removed line is the whole defect. A missing variable in prod resolved to the dev catalog and every read returned plausible, wrong data._

## File map

**Nineteen files, and the one-line config changes are the risky ones.**

Diff size is a poor proxy for blast radius here. A single Terragrunt line decides which catalog production reads.

| File | Role | Change | Risk |
| --- | --- | --- | --- |
| `environments/prod/terragrunt.hcl` | production inputs | Sets catalog_name to the clean name. | high |
| `environments/dev/terragrunt.hcl` | development inputs | Sets catalog_name to the _dev suffix. | medium |
| `modules/catalog/main.tf` | catalog resource | Takes catalog_name as a required variable and adds prevent_destroy. | high |
| `portal/src/api/routes.ts` | request handling | Replaces the dev fallback with a startup throw. | high |
| `features/steps/catalog.py` | test coverage | Substitutes {CATALOG} from the environment. | low |

## Receipts

**Where each claim in this brief comes from.**

Two of these come from CI rather than the diff, and one comes from a review comment.

- **Claim.** Prod apply has failed since v0.67.0.
  - Evidence: Error: catalog 'growth_opportunities' already exists in metastore.
  - Where: CI check: terraform-plan, prod workspace
- **Claim.** The API silently read dev when the variable was unset.
  - Evidence: The removed line is `process.env.UC_CATALOG ?? "growth_opportunities"`.
  - Where: portal/src/api/routes.ts:88
- **Claim.** The catalog is now protected from replacement.
  - Evidence: lifecycle { prevent_destroy = true }
  - Where: modules/catalog/main.tf
- **Claim.** The plan is red at the time of writing.
  - Evidence: terraform-plan: failure
  - Where: CI check: terraform-plan

## Where to attack

**Renaming a catalog is not a code change, it is a data migration.**

The Terraform reads as additive. What it actually does is decide which physical catalog holds production tables.

- **Dev tables live in the old shared catalog** (high)
  - Dev currently owns growth_opportunities. Pointing dev at a new _dev catalog leaves the existing tables behind under a name prod is about to claim.
  - Mitigation: Confirm what is in the shared catalog and where it should end up before applying. This is the question the plan failure is really asking.
- **prevent_destroy makes recovery slower, not faster** (medium)
  - It stops Terraform replacing the catalog, which is the point. It also means a genuinely wrong catalog cannot be removed without editing state.
  - Mitigation: Get the name right before the first apply.
- **The throw turns a silent bug into an outage** (medium)
  - That is the correct trade, but any environment currently relying on the fallback will fail to boot after deploy.
  - Mitigation: Grep every deployment target for UC_CATALOG before merging.

## Merge order

**The config has to land before the code that requires it.**

Merging the API change first takes production down, because the variable it now demands does not exist yet.

1. **Answer the dev-tables question.** Nothing else can proceed safely until it is known what lives in the shared catalog today.
2. **Set UC_CATALOG in every deployment target.** Before the throw ships, or the next prod deploy fails at boot.
3. **Apply the Terraform in dev, then prod.** Confirm the dev catalog exists before prod claims the name.
4. **Merge the API and BDD changes.** Safe once the variable is set everywhere.

## What this brief could not check

- This brief was written by hand as a design fixture. It is modelled on real work, but no model read a diff to produce it, and its claims should not be relied on.
- The plan failure is described from its error message. The full CI log was not read, so the message may not be the only failure.
- Whether the shared catalog holds production data was not determined. That question is unresolved and is the reason the merge order starts where it does.