stg_view
One model per source table. Rename, cast, dedupe, filter soft deletes. No joins, no business logic.
dbt-core on DuckDB. Card authorizations, network settlements and the general ledger, modelled end to end.
What each layer is allowed to do. The rules are the design.
stg_view
One model per source table. Rename, cast, dedupe, filter soft deletes. No joins, no business logic.
int_table · incremental
Joins and business logic. Every model declares its grain in a header comment.
dim_ fct_ mart_table
Wide and denormalised. Derived columns precomputed so downstream queries need no joins.
snap_snapshot
State the source overwrites, captured as it changes. The one thing a full refresh cannot rebuild.
models:
card_spend_warehouse:
staging:
+materialized: view
+schema: staging
intermediate:
+materialized: table
+schema: intermediate
marts:
+materialized: table
+schema: martsmodels/ ├── staging/ stg_*.sql + _sources.yml ├── intermediate/ int_*.sql └── marts/ dim_* fct_* mart_* snapshots/ snap_*.sql seeds/ *.csv macros/ *.sql tests/ assert_*.sql
Authorization ath_52b32444cc46. Highlighted columns are the ones that change at that hop.
Landed twice by a Fivetran re-sync. Cents as integers, local timestamp, soft-delete flag.
| auth_id | auth_code | amount_cents | currency | auth_ts | source_system | is_deleted | _fivetran_synced |
|---|---|---|---|---|---|---|---|
| ath_52b32444cc46 | 9D5DDA | 55,333 | USD | 2025-10-14T04:14:05 | legacy | false | 2026-07-05T06:00:00 |
| ath_52b32444cc46 | 9D5DDA | 55,333 | USD | 2025-10-14T04:14:05 | legacy | false | 2026-07-05T09:00:00 |
qualify row_number() keeps the latest sync. Cents cast to dollars. Timestamp carried forward unresolved — staging does not join.
| auth_id | auth_code | amount_usd | currency | auth_ts_unnormalized | source_system |
|---|---|---|---|---|---|
| ath_52b32444cc46 | 9D5DDA | 553.33 | USD | 2025-10-14T04:14:05 | legacy |
Joins the company to resolve the timezone. source_system = legacy, so the stored time was local and shifts to UTC here.
| auth_id | amount_usd | auth_ts_unnormalized | company_timezone | auth_ts_utc | auth_date |
|---|---|---|---|---|---|
| ath_52b32444cc46 | 553.33 | 2025-10-14T04:14:05 | America/Denver | 2025-10-14T11:14:05 | 2025-10-14 |
The network's side: three partial captures of one hold, each in its own daily file.
| network_txn_id | auth_code | merchant_descriptor | amount_cents | settled_date | file_date |
|---|---|---|---|---|---|
| net_f30f54af60ad | 9D5DDA | DATADOG INC | 12,742 | 2025-10-17 | 2025-10-19 |
| net_9c75d2da9dbd | 9D5DDA | DATADOG INC | 18,664 | 2025-10-18 | 2025-10-20 |
| net_5ca36e449e4f | 9D5DDA | DATADOG INC | 21,534 | 2025-10-19 | 2025-10-23 |
One row per settlement. The hold fans out to three, each matched on auth_code + card_id + window.
| record_type | network_txn_id | auth_amount_usd | settled_amount_usd | settled_date | file_date | match_method | days_auth_to_settle |
|---|---|---|---|---|---|---|---|
| matched | net_f30f54af60ad | 553.33 | 127.42 | 2025-10-17 | 2025-10-19 | auth_code | 3 |
| matched | net_9c75d2da9dbd | 553.33 | 186.64 | 2025-10-18 | 2025-10-20 | auth_code | 4 |
| matched | net_5ca36e449e4f | 553.33 | 215.34 | 2025-10-19 | 2025-10-23 | auth_code | 5 |
Merchant canonicalised, limit joined point-in-time, capture shortfall derived. Query it with a SELECT.
| merchant_name | merchant_category | settled_amount_usd | capture_shortfall_usd | limit_at_transaction_usd | transaction_month | is_money_movement |
|---|---|---|---|---|---|---|
| Datadog | Software & SaaS | 127.42 | 425.91 | 25,000 | 2025-10-01 | true |
| Datadog | Software & SaaS | 186.64 | 366.69 | 25,000 | 2025-10-01 | true |
| Datadog | Software & SaaS | 215.34 | 337.99 | 25,000 | 2025-10-01 | true |
Select one: upstream marks blue, downstream green. The prose is the header comment from the file.
1:1 with source. Rename, cast, dedupe, filter. No joins.
Business logic. Every model declares its grain.
Wide and flat. What a dashboard queries.
History the source overwrites, captured as it happens.
models/intermediate/int_auth_settlement_matched.sql · table · 210,653 rows
The hardest model in the warehouse
An authorization is a HOLD placed at swipe time. A settlement is the money actually moving, delivered by the card network in a daily file, days later, at a different grain. They do not line up one-to-one and nothing guarantees a shared key.
Grain: one row per settlement, PLUS one row per approved authorization that never settled. Nothing is dropped -- a settlement without an authorization is still real money leaving the account, and an authorization without a settlement is still an open hold against the customer's limit. A model that inner-joins these two tables silently loses both, which is how a close ends up off by five figures with no trail.
Matching runs in two tiers:
1. auth_code -- the network echoes the code it was issued. Clean, exact, covers ~96% of rows. 2. fuzzy -- for the ~4% where the network dropped the code, match on card + MCC + a 40-day window, requiring the settled amount not to exceed the authorized amount, then take the nearest amount and date.
Tier 2 is a best-effort ATTRIBUTION, not a fact, and it is labelled as such in `match_method` so downstream consumers can exclude it. Two settlements on the same card, same merchant, same amount, days apart are genuinely ambiguous; the model picks one and stays honest about how it picked.
Reversals arrive as negative rows referencing the same auth. They are kept as their own rows rather than netted here, so the gross/net distinction stays available to finance.
with settlements as (
select * from {{ ref('stg_network_settlements') }}
),
authorizations as (
select *
from {{ ref('int_authorizations') }}
where auth_status = 'approved'
),
cards as (
select card_id, card_token, company_id from {{ ref('stg_cards') }}
),
-- the network only ever sends a token; resolve it back to a card
settlements_with_card as (
select
s.*,
c.card_id,
c.company_id
from settlements s
left join cards c on s.card_token = c.card_token
),
-- ── Tier 1: authorization code, scoped to the card ──────────────────────────
--
-- An auth_code is SIX characters. It is not a globally unique key and was
-- never meant to be one -- the network guarantees uniqueness only within a
-- card over a short window, then recycles. Across 187k authorizations the
-- birthday bound predicts ~1,045 collisions and the data contains 1,015.
--
-- Joining on auth_code alone therefore attributes ~0.6% of settled volume to
-- the wrong company. It looks exact, it passes every not_null and every
-- relationships test, and it is wrong. scripts/validate_matching.py is what
-- caught it; no schema test would have.
tier_1 as (
select
s.network_txn_id,
a.auth_id,
'auth_code' as match_method
from settlements_with_card s
inner join authorizations a
on s.auth_code = a.auth_code
and s.card_id = a.card_id
and s.settled_date between a.auth_date and a.auth_date + 40
qualify row_number() over (
partition by s.network_txn_id
order by a.auth_date desc, a.auth_id
) = 1
),
-- ── Tier 2: fuzzy match for records the network sent with no auth_code ──────
needs_fuzzy as (
select s.*
from settlements_with_card s
left join tier_1 t on s.network_txn_id = t.network_txn_id
where t.network_txn_id is null
),
fuzzy_candidates as (
select
u.network_txn_id,
a.auth_id,
abs(abs(u.amount_usd) - a.amount_usd) as amount_gap,
date_diff('day', a.auth_date, u.settled_date) as day_gap
from needs_fuzzy u
inner join authorizations a
on u.card_id = a.card_id
and u.mcc = a.mcc
-- a settlement never precedes its authorization, and the network
-- gives up on unsettled holds after about 40 days
and u.settled_date between a.auth_date and a.auth_date + 40
-- you cannot capture more than you authorized (1% tolerance for tips)
and abs(u.amount_usd) <= a.amount_usd * 1.01
),
-- best candidate per settlement
fuzzy_best as (
select *
from fuzzy_candidates
qualify row_number() over (
partition by network_txn_id
order by amount_gap asc, day_gap asc, auth_id asc
) = 1
),
-- How much of each authorization has already been captured by tier 1? An
-- authorization for $400 that tier 1 already matched to $380 of settlements
-- has $20 of room left, and a fuzzy candidate for $150 does not belong to it
-- no matter how close the merchant and date look.
capacity as (
select
a.auth_id,
a.amount_usd as auth_amount_usd,
coalesce(sum(case when not s.is_reversal then s.amount_usd else 0 end), 0)
as tier_1_captured_usd
from authorizations a
left join tier_1 t on a.auth_id = t.auth_id
left join settlements_with_card s on t.network_txn_id = s.network_txn_id
group by 1, 2
),
-- Fill the remaining room best-match-first, and stop when it runs out.
fuzzy_within_capacity as (
select
f.network_txn_id,
f.auth_id,
c.auth_amount_usd,
c.tier_1_captured_usd,
sum(case when not s.is_reversal then s.amount_usd else 0 end) over (
partition by f.auth_id
order by f.amount_gap asc, f.day_gap asc, f.network_txn_id asc
rows between unbounded preceding and current row
) as running_fuzzy_usd
from fuzzy_best f
inner join settlements_with_card s on f.network_txn_id = s.network_txn_id
inner join capacity c on f.auth_id = c.auth_id
),
-- A settlement that does not fit is left UNMATCHED rather than forced onto
-- the nearest authorization. That is a deliberate trade: an unattributed
-- settlement is visible and lands in a review queue, while a wrongly
-- attributed one silently bills the wrong company and shows up as a
-- reconciliation break weeks later. Precision over recall, because the two
-- failure modes do not cost the same.
tier_2 as (
select
network_txn_id,
auth_id,
'fuzzy_card_amount_window' as match_method
from fuzzy_within_capacity
where tier_1_captured_usd + running_fuzzy_usd <= auth_amount_usd * 1.01
),
all_matches as (
select * from tier_1
union all
select * from tier_2
),
-- ── Every settlement, matched or not ────────────────────────────────────────
settlement_rows as (
select
case when m.auth_id is null then 'force_post' else 'matched' end as record_type,
s.network_txn_id,
m.auth_id,
coalesce(a.company_id, s.company_id) as company_id,
coalesce(a.card_id, s.card_id) as card_id,
s.merchant_descriptor,
s.mcc,
a.amount_usd as auth_amount_usd,
s.amount_usd as settled_amount_usd,
a.auth_date,
s.settled_date,
s.file_date,
s.file_lag_days,
s.is_reversal,
coalesce(m.match_method, 'unmatched') as match_method,
date_diff('day', a.auth_date, s.settled_date) as days_auth_to_settle
from settlements_with_card s
left join all_matches m on s.network_txn_id = m.network_txn_id
left join authorizations a on m.auth_id = a.auth_id
),
-- ── Approved authorizations that never settled ──────────────────────────────
open_authorization_rows as (
select
'open_auth' as record_type,
cast(null as varchar) as network_txn_id,
a.auth_id,
a.company_id,
a.card_id,
a.merchant_raw_name as merchant_descriptor,
a.mcc,
a.amount_usd as auth_amount_usd,
cast(null as decimal(18, 2)) as settled_amount_usd,
a.auth_date,
cast(null as date) as settled_date,
cast(null as date) as file_date,
cast(null as bigint) as file_lag_days,
false as is_reversal,
'no_settlement' as match_method,
cast(null as bigint) as days_auth_to_settle
from authorizations a
left join all_matches m on a.auth_id = m.auth_id
where m.auth_id is null
)
select * from settlement_rows
union all
select * from open_authorization_rowsThe dbt idiom, and where the project uses it.
from {{ ref('stg_authorizations') }}
join {{ ref('stg_cards') }} using (card_id)The DAG is derived from these, not declared.
every model
qualify row_number() over (
partition by auth_id
order by _fivetran_synced desc
) = 1Re-syncs land the same primary key twice.
stg_authorizations
{{ config(
materialized='incremental',
unique_key=['company_id','spend_date'],
incremental_strategy='delete+insert'
) }}
where file_date > (
select max(max_file_date) from {{ this }}
)Increment on arrival, not event date. Replace whole days, never append.
int_company_daily_spend
lead(effective_date) over (
partition by company_id
order by effective_date
) as valid_toHalf-open windows, so a point-in-time join is a BETWEEN.
int_spend_limit_history
left join limits l
on t.company_id = l.company_id
and t.settled_date >= l.valid_from
and t.settled_date < l.valid_toThe limit that applied that day, not the limit today.
fct_transactions
{{ config(
strategy='check',
check_cols=['plan_tier','industry'],
unique_key='company_id'
) }}The only table here a full refresh cannot rebuild.
snap_company_plan
left join {{ ref('descriptor_map') }} m
on n.normalized like m.pattern
qualify row_number() over (
order by length(m.pattern) desc
) = 1Business owns the mapping; MCC catches what it misses.
int_merchants_normalized
{{ dbt_utils.date_spine('day', ...) }}
cross join companies
avg(net_spend_usd) over (
rows between 6 preceding and current row
)'Last 7 rows' only means 7 days if no day is missing.
fct_company_daily_spend
case when m.auth_id is null
then 'force_post'
else 'matched'
end as record_type
union all -- auths that never settledAn inner join loses real money in both directions.
int_auth_settlement_matched
{% macro cents_to_dollars(col) %}
round(cast({{ col }} as
decimal(18,4)) / 100, 2)
{% endmacro %}Float division on money drifts.
macros/cents_to_dollars.sql
data_tests:
- dbt_utils.unique_combination_of_columns:
arguments:
combination_of_columns:
[company_id, spend_date]The grain stops being a comment and starts failing builds.
int_company_daily_spend
- unique:
config:
severity: warnDocuments a source property; should not fail a build.
stg_authorizations
96 tests. 2 configured to warn rather than fail, because they document a source property instead of a break.
The generator writes a map of which authorization each settlement came from. The warehouse never reads it; two scripts in CI score the models against it.
Same question, same answer: 96 lines against raw sources, 9 against the mart. Both execute at build time and are compared row for row.
-- Monthly settled spend by merchant category, straight off the raw files.
-- Everything the warehouse does has to happen again, here, by hand.
with cards as (
-- soft deletes are still in the source
select card_id, card_token, company_id
from read_csv_auto('data/raw/app_db/cards.csv')
where not is_deleted
),
settlements as (
select
s.network_txn_id,
c.company_id,
s.merchant_descriptor,
s.mcc,
-- amounts are integer cents, and float division loses money
round(cast(s.amount_cents as decimal(18,4)) / 100, 2) as amount_usd,
cast(s.settled_date as date) as settled_date
from read_csv_auto('data/raw/network/settlements.csv') s
inner join cards c on s.card_token = c.card_token
where c.company_id = 'cmp_08d98638c6fc'
),
-- the network sends one merchant under many spellings, so strip punctuation
-- and store numbers before trying to group on anything
normalized as (
select
*,
trim(regexp_replace(
regexp_replace(
regexp_replace(upper(merchant_descriptor), '[*#.,/]', ' ', 'g'),
'[0-9]+', '', 'g'),
'\s+', ' ', 'g')) as descriptor_normalized
from settlements
),
-- the descriptor->merchant mapping, inlined because there is nowhere else to
-- put it. In practice this is the block that gets copy-pasted between
-- notebooks and slowly diverges.
mapped as (
select
n.*,
case
when n.descriptor_normalized like 'AWS%'
or n.descriptor_normalized like 'AMAZON WEB SERVICES%' then 'Software & SaaS'
when n.descriptor_normalized like 'SLACK%'
or n.descriptor_normalized like 'GOOGLE%'
or n.descriptor_normalized like 'DATADOG%'
or n.descriptor_normalized like 'DDOG%'
or n.descriptor_normalized like 'ZOOM%' then 'Software & SaaS'
when n.descriptor_normalized like 'UNITED%'
or n.descriptor_normalized like 'UAL%'
or n.descriptor_normalized like 'DELTA%'
or n.descriptor_normalized like 'DL AIRFARE%'
or n.descriptor_normalized like 'MARRIOTT%'
or n.descriptor_normalized like 'COURTYARD BY MARRIOTT%' then 'Travel'
when n.descriptor_normalized like 'UBER%'
or n.descriptor_normalized like 'LYFT%' then 'Ground Transport'
when n.descriptor_normalized like 'STARBUCKS%'
or n.descriptor_normalized like 'SQ STARBUCKS%' then 'Meals & Entertainment'
when n.descriptor_normalized like 'WEWORK%'
or n.descriptor_normalized like 'WE WORK%' then 'Rent & Facilities'
when n.descriptor_normalized like 'STAPLES%'
or n.descriptor_normalized like 'THE HOME DEPOT%'
or n.descriptor_normalized like 'HOME DEPOT%'
or n.descriptor_normalized like 'HOMEDEPOT%'
or n.descriptor_normalized like 'COSTCO%' then 'Office Supplies'
when n.descriptor_normalized like 'SHELL%' then 'Fuel'
when n.descriptor_normalized like 'FEDEX%' then 'Shipping'
when n.descriptor_normalized like 'APPLE%' then 'Equipment'
-- and every merchant the list above has not caught yet still has
-- to land somewhere, so fall back to the MCC. This list has to
-- mirror seeds/mcc_categories.csv EXACTLY -- add one MCC here that
-- the seed does not have and this query quietly stops agreeing
-- with every other report in the company.
when n.mcc = 7372 then 'Software & SaaS'
when n.mcc in (3000, 3001, 3501) then 'Travel'
when n.mcc = 4121 then 'Ground Transport'
when n.mcc = 5814 then 'Meals & Entertainment'
when n.mcc = 6513 then 'Rent & Facilities'
when n.mcc in (5943, 5200, 5300) then 'Office Supplies'
when n.mcc = 5541 then 'Fuel'
when n.mcc = 4215 then 'Shipping'
when n.mcc = 5732 then 'Equipment'
else 'Uncategorized'
end as merchant_category
from normalized n
)
select
date_trunc('month', settled_date)::date as transaction_month,
merchant_category,
round(sum(amount_usd), 2) as net_spend_usd
from mapped
group by 1, 2
order by 1, 2The dedupe, the soft deletes, the cents cast, the token join, the descriptor normalisation and the MCC fallback all have to happen again, here, by hand. Get any one of them wrong and the number is wrong quietly.
select
transaction_month,
merchant_category,
round(sum(settled_amount_usd), 2) as net_spend_usd
from marts.fct_transactions
where company_id = 'cmp_08d98638c6fc'
and is_money_movement
group by 1, 2
order by 1, 2One table, no joins, no normalisation. Every decision above was made once, in version control, with tests on it.
Real mart data, queried in the browser. SELECT, WHERE, GROUP BY, ORDER BY, JOIN — no CTEs or window functions needed, because the derived columns already exist.