← junxu.meCard Spend WarehouseRepository ↗

Card spend warehouse

dbt-core on DuckDB. Card authorizations, network settlements and the general ledger, modelled end to end.

sourcesstagingintermediatemartsexposuresapp_db.authorizationsapp_db.companiesnetwork.settlementsapp_db.cardsledger.journal_entriessegment.product_eventssalesforce.opportunitiesapp_db.spend_limitsapp_db.usersapp_db.statementsstg_authorizations11stg_companies4stg_network_settlements5stg_cards6stg_journal_entries2stg_product_events2stg_opportunities2stg_spend_limits4stg_users2stg_statements3int_authorizations4int_merchants_normalized4int_company_daily_spend4int_auth_settlement_matched7int_experiment_exposures2int_spend_limit_history5dim_merchant3mart_close_reconciliation4fct_transactions7fct_company_daily_spend6fct_experiment_results4dim_company6fct_statements3finance_month_end_closegrowth_experiment_readoutcompany_spend_overviewrisk_delinquency_monitor
sourceviewtableincrementalexposurenumber = tests on that model · hover to isolate a path
models
23
tests
96
sources
10
seeds
2
exposures
4
rows modelled
403,894
settled volume
$26.4M
full build
~3s

Layer contract

What each layer is allowed to do. The rules are the design.

staging10
stg_

view

One model per source table. Rename, cast, dedupe, filter soft deletes. No joins, no business logic.

intermediate6
int_

table · incremental

Joins and business logic. Every model declares its grain in a header comment.

marts7
dim_ fct_ mart_

table

Wide and denormalised. Derived columns precomputed so downstream queries need no joins.

snapshots1
snap_

snapshot

State the source overwrites, captured as it changes. The one thing a full refresh cannot rebuild.

Enforced in dbt_project.yml

models:
  card_spend_warehouse:
    staging:
      +materialized: view
      +schema: staging
    intermediate:
      +materialized: table
      +schema: intermediate
    marts:
      +materialized: table
      +schema: marts

Directory

models/
├── staging/       stg_*.sql  +  _sources.yml
├── intermediate/  int_*.sql
└── marts/         dim_*  fct_*  mart_*
snapshots/         snap_*.sql
seeds/             *.csv
macros/            *.sql
tests/             assert_*.sql

One record, raw to mart

Authorization ath_52b32444cc46. Highlighted columns are the ones that change at that hop.

sourceapp_db.authorizationscsv2 rows

Landed twice by a Fivetran re-sync. Cents as integers, local timestamp, soft-delete flag.

auth_idauth_codeamount_centscurrencyauth_tssource_systemis_deleted_fivetran_synced
ath_52b32444cc469D5DDA55,333USD2025-10-14T04:14:05legacyfalse2026-07-05T06:00:00
ath_52b32444cc469D5DDA55,333USD2025-10-14T04:14:05legacyfalse2026-07-05T09:00:00
stagingstg_authorizationsview1 row

qualify row_number() keeps the latest sync. Cents cast to dollars. Timestamp carried forward unresolved — staging does not join.

auth_idauth_codeamount_usdcurrencyauth_ts_unnormalizedsource_system
ath_52b32444cc469D5DDA553.33USD2025-10-14T04:14:05legacy
intermediateint_authorizationstable1 row

Joins the company to resolve the timezone. source_system = legacy, so the stored time was local and shifts to UTC here.

auth_idamount_usdauth_ts_unnormalizedcompany_timezoneauth_ts_utcauth_date
ath_52b32444cc46553.332025-10-14T04:14:05America/Denver2025-10-14T11:14:052025-10-14
sourcenetwork.settlementscsv3 rows

The network's side: three partial captures of one hold, each in its own daily file.

network_txn_idauth_codemerchant_descriptoramount_centssettled_datefile_date
net_f30f54af60ad9D5DDADATADOG INC12,7422025-10-172025-10-19
net_9c75d2da9dbd9D5DDADATADOG INC18,6642025-10-182025-10-20
net_5ca36e449e4f9D5DDADATADOG INC21,5342025-10-192025-10-23
intermediateint_auth_settlement_matchedtable3 rows

One row per settlement. The hold fans out to three, each matched on auth_code + card_id + window.

record_typenetwork_txn_idauth_amount_usdsettled_amount_usdsettled_datefile_datematch_methoddays_auth_to_settle
matchednet_f30f54af60ad553.33127.422025-10-172025-10-19auth_code3
matchednet_9c75d2da9dbd553.33186.642025-10-182025-10-20auth_code4
matchednet_5ca36e449e4f553.33215.342025-10-192025-10-23auth_code5
martsfct_transactionstable3 rows

Merchant canonicalised, limit joined point-in-time, capture shortfall derived. Query it with a SELECT.

merchant_namemerchant_categorysettled_amount_usdcapture_shortfall_usdlimit_at_transaction_usdtransaction_monthis_money_movement
DatadogSoftware & SaaS127.42425.9125,0002025-10-01true
DatadogSoftware & SaaS186.64366.6925,0002025-10-01true
DatadogSoftware & SaaS215.34337.9925,0002025-10-01true

Models

Select one: upstream marks blue, downstream green. The prose is the header comment from the file.

Staging10

1:1 with source. Rename, cast, dedupe, filter. No joins.

Intermediate6

Business logic. Every model declares its grain.

Marts7

Wide and flat. What a dashboard queries.

Snapshots1

History the source overwrites, captured as it happens.

int_auth_settlement_matched

models/intermediate/int_auth_settlement_matched.sql · table · 210,653 rows

The hardest model in the warehouse

An authorization is a HOLD placed at swipe time. A settlement is the money actually moving, delivered by the card network in a daily file, days later, at a different grain. They do not line up one-to-one and nothing guarantees a shared key.

Grain: one row per settlement, PLUS one row per approved authorization that never settled. Nothing is dropped -- a settlement without an authorization is still real money leaving the account, and an authorization without a settlement is still an open hold against the customer's limit. A model that inner-joins these two tables silently loses both, which is how a close ends up off by five figures with no trail.

Matching runs in two tiers:

1. auth_code -- the network echoes the code it was issued. Clean, exact, covers ~96% of rows. 2. fuzzy -- for the ~4% where the network dropped the code, match on card + MCC + a 40-day window, requiring the settled amount not to exceed the authorized amount, then take the nearest amount and date.

Tier 2 is a best-effort ATTRIBUTION, not a fact, and it is labelled as such in `match_method` so downstream consumers can exclude it. Two settlements on the same card, same merchant, same amount, days apart are genuinely ambiguous; the model picks one and stays honest about how it picked.

Reversals arrive as negative rows referencing the same auth. They are kept as their own rows rather than netted here, so the gross/net distinction stays available to finance.

with settlements as (
    select * from {{ ref('stg_network_settlements') }}
),

authorizations as (
    select *
    from {{ ref('int_authorizations') }}
    where auth_status = 'approved'
),

cards as (
    select card_id, card_token, company_id from {{ ref('stg_cards') }}
),

-- the network only ever sends a token; resolve it back to a card
settlements_with_card as (
    select
        s.*,
        c.card_id,
        c.company_id
    from settlements s
    left join cards c on s.card_token = c.card_token
),

-- ── Tier 1: authorization code, scoped to the card ──────────────────────────
--
-- An auth_code is SIX characters. It is not a globally unique key and was
-- never meant to be one -- the network guarantees uniqueness only within a
-- card over a short window, then recycles. Across 187k authorizations the
-- birthday bound predicts ~1,045 collisions and the data contains 1,015.
--
-- Joining on auth_code alone therefore attributes ~0.6% of settled volume to
-- the wrong company. It looks exact, it passes every not_null and every
-- relationships test, and it is wrong. scripts/validate_matching.py is what
-- caught it; no schema test would have.
tier_1 as (
    select
        s.network_txn_id,
        a.auth_id,
        'auth_code' as match_method
    from settlements_with_card s
    inner join authorizations a
        on  s.auth_code    = a.auth_code
        and s.card_id      = a.card_id
        and s.settled_date between a.auth_date and a.auth_date + 40
    qualify row_number() over (
        partition by s.network_txn_id
        order by a.auth_date desc, a.auth_id
    ) = 1
),

-- ── Tier 2: fuzzy match for records the network sent with no auth_code ──────
needs_fuzzy as (
    select s.*
    from settlements_with_card s
    left join tier_1 t on s.network_txn_id = t.network_txn_id
    where t.network_txn_id is null
),

fuzzy_candidates as (
    select
        u.network_txn_id,
        a.auth_id,
        abs(abs(u.amount_usd) - a.amount_usd)                as amount_gap,
        date_diff('day', a.auth_date, u.settled_date)        as day_gap
    from needs_fuzzy u
    inner join authorizations a
        on  u.card_id      = a.card_id
        and u.mcc          = a.mcc
        -- a settlement never precedes its authorization, and the network
        -- gives up on unsettled holds after about 40 days
        and u.settled_date between a.auth_date and a.auth_date + 40
        -- you cannot capture more than you authorized (1% tolerance for tips)
        and abs(u.amount_usd) <= a.amount_usd * 1.01
),

-- best candidate per settlement
fuzzy_best as (
    select *
    from fuzzy_candidates
    qualify row_number() over (
        partition by network_txn_id
        order by amount_gap asc, day_gap asc, auth_id asc
    ) = 1
),

-- How much of each authorization has already been captured by tier 1? An
-- authorization for $400 that tier 1 already matched to $380 of settlements
-- has $20 of room left, and a fuzzy candidate for $150 does not belong to it
-- no matter how close the merchant and date look.
capacity as (
    select
        a.auth_id,
        a.amount_usd                                                    as auth_amount_usd,
        coalesce(sum(case when not s.is_reversal then s.amount_usd else 0 end), 0)
                                                                        as tier_1_captured_usd
    from authorizations a
    left join tier_1 t                on a.auth_id = t.auth_id
    left join settlements_with_card s on t.network_txn_id = s.network_txn_id
    group by 1, 2
),

-- Fill the remaining room best-match-first, and stop when it runs out.
fuzzy_within_capacity as (
    select
        f.network_txn_id,
        f.auth_id,
        c.auth_amount_usd,
        c.tier_1_captured_usd,
        sum(case when not s.is_reversal then s.amount_usd else 0 end) over (
            partition by f.auth_id
            order by f.amount_gap asc, f.day_gap asc, f.network_txn_id asc
            rows between unbounded preceding and current row
        )                                                               as running_fuzzy_usd
    from fuzzy_best f
    inner join settlements_with_card s on f.network_txn_id = s.network_txn_id
    inner join capacity c              on f.auth_id = c.auth_id
),

-- A settlement that does not fit is left UNMATCHED rather than forced onto
-- the nearest authorization. That is a deliberate trade: an unattributed
-- settlement is visible and lands in a review queue, while a wrongly
-- attributed one silently bills the wrong company and shows up as a
-- reconciliation break weeks later. Precision over recall, because the two
-- failure modes do not cost the same.
tier_2 as (
    select
        network_txn_id,
        auth_id,
        'fuzzy_card_amount_window' as match_method
    from fuzzy_within_capacity
    where tier_1_captured_usd + running_fuzzy_usd <= auth_amount_usd * 1.01
),

all_matches as (
    select * from tier_1
    union all
    select * from tier_2
),

-- ── Every settlement, matched or not ────────────────────────────────────────
settlement_rows as (
    select
        case when m.auth_id is null then 'force_post' else 'matched' end as record_type,
        s.network_txn_id,
        m.auth_id,
        coalesce(a.company_id, s.company_id)            as company_id,
        coalesce(a.card_id, s.card_id)                  as card_id,
        s.merchant_descriptor,
        s.mcc,
        a.amount_usd                                    as auth_amount_usd,
        s.amount_usd                                    as settled_amount_usd,
        a.auth_date,
        s.settled_date,
        s.file_date,
        s.file_lag_days,
        s.is_reversal,
        coalesce(m.match_method, 'unmatched')           as match_method,
        date_diff('day', a.auth_date, s.settled_date)   as days_auth_to_settle
    from settlements_with_card s
    left join all_matches m    on s.network_txn_id = m.network_txn_id
    left join authorizations a on m.auth_id = a.auth_id
),

-- ── Approved authorizations that never settled ──────────────────────────────
open_authorization_rows as (
    select
        'open_auth'                                     as record_type,
        cast(null as varchar)                           as network_txn_id,
        a.auth_id,
        a.company_id,
        a.card_id,
        a.merchant_raw_name                             as merchant_descriptor,
        a.mcc,
        a.amount_usd                                    as auth_amount_usd,
        cast(null as decimal(18, 2))                    as settled_amount_usd,
        a.auth_date,
        cast(null as date)                              as settled_date,
        cast(null as date)                              as file_date,
        cast(null as bigint)                            as file_lag_days,
        false                                           as is_reversal,
        'no_settlement'                                 as match_method,
        cast(null as bigint)                            as days_auth_to_settle
    from authorizations a
    left join all_matches m on a.auth_id = m.auth_id
    where m.auth_id is null
)

select * from settlement_rows
union all
select * from open_authorization_rows

Techniques

The dbt idiom, and where the project uses it.

ref(), never a hard-coded table

from {{ ref('stg_authorizations') }}
join {{ ref('stg_cards') }} using (card_id)

The DAG is derived from these, not declared.

every model

Deduplicate a replicated source

qualify row_number() over (
  partition by auth_id
  order by _fivetran_synced desc
) = 1

Re-syncs land the same primary key twice.

stg_authorizations

Incremental on late-arriving data

{{ config(
  materialized='incremental',
  unique_key=['company_id','spend_date'],
  incremental_strategy='delete+insert'
) }}

where file_date > (
  select max(max_file_date) from {{ this }}
)

Increment on arrival, not event date. Replace whole days, never append.

int_company_daily_spend

SCD2 from a versioned source

lead(effective_date) over (
  partition by company_id
  order by effective_date
) as valid_to

Half-open windows, so a point-in-time join is a BETWEEN.

int_spend_limit_history

Point-in-time join

left join limits l
  on  t.company_id = l.company_id
  and t.settled_date >= l.valid_from
  and t.settled_date <  l.valid_to

The limit that applied that day, not the limit today.

fct_transactions

Snapshot state the source overwrites

{{ config(
  strategy='check',
  check_cols=['plan_tier','industry'],
  unique_key='company_id'
) }}

The only table here a full refresh cannot rebuild.

snap_company_plan

Seed-driven mapping with a fallback

left join {{ ref('descriptor_map') }} m
  on n.normalized like m.pattern
qualify row_number() over (
  order by length(m.pattern) desc
) = 1

Business owns the mapping; MCC catches what it misses.

int_merchants_normalized

Zero-filled date spine

{{ dbt_utils.date_spine('day', ...) }}
cross join companies

avg(net_spend_usd) over (
  rows between 6 preceding and current row
)

'Last 7 rows' only means 7 days if no day is missing.

fct_company_daily_spend

Preserve unmatched records

case when m.auth_id is null
     then 'force_post'
     else 'matched'
end as record_type

union all  -- auths that never settled

An inner join loses real money in both directions.

int_auth_settlement_matched

Reusable cast in one place

{% macro cents_to_dollars(col) %}
  round(cast({{ col }} as
    decimal(18,4)) / 100, 2)
{% endmacro %}

Float division on money drifts.

macros/cents_to_dollars.sql

Declare the grain as a test

data_tests:
  - dbt_utils.unique_combination_of_columns:
      arguments:
        combination_of_columns:
          [company_id, spend_date]

The grain stops being a comment and starts failing builds.

int_company_daily_spend

Severity tuning

- unique:
    config:
      severity: warn

Documents a source property; should not fail a build.

stg_authorizations

Tests

96 tests. 2 configured to warn rather than fail, because they document a source property instead of a break.

not_null47
unique21
accepted_values10
singular7
unique_combination_of_columns5
accepted_range3
relationships3

Singular tests

  • capture_never_exceeds_authorization
  • close_reconciliation_ties_out
  • daily_spend_spine_has_no_gaps
  • experiment_has_no_sample_ratio_mismatch
  • no_money_lost_between_layers
  • settlement_never_precedes_authorization
  • unmapped_merchant_spend_within_tolerance

Scored against ground truth

The generator writes a map of which authorization each settlement came from. The warehouse never reads it; two scripts in CI score the models against it.

auth ↔ settlement match
0.99994 / 0.99943precision / recall
incremental replay
0rows differ from full refresh
watermark on settled_date
$166,568.190.63% dropped, silently
ledger tie-out
190/190$0.00 unexplained

Raw versus mart

Same question, same answer: 96 lines against raw sources, 9 against the mart. Both execute at build time and are compared row for row.

Against raw sources96 lines
-- Monthly settled spend by merchant category, straight off the raw files.
-- Everything the warehouse does has to happen again, here, by hand.
with cards as (
    -- soft deletes are still in the source
    select card_id, card_token, company_id
    from read_csv_auto('data/raw/app_db/cards.csv')
    where not is_deleted
),

settlements as (
    select
        s.network_txn_id,
        c.company_id,
        s.merchant_descriptor,
        s.mcc,
        -- amounts are integer cents, and float division loses money
        round(cast(s.amount_cents as decimal(18,4)) / 100, 2) as amount_usd,
        cast(s.settled_date as date)                          as settled_date
    from read_csv_auto('data/raw/network/settlements.csv') s
    inner join cards c on s.card_token = c.card_token
    where c.company_id = 'cmp_08d98638c6fc'
),

-- the network sends one merchant under many spellings, so strip punctuation
-- and store numbers before trying to group on anything
normalized as (
    select
        *,
        trim(regexp_replace(
            regexp_replace(
                regexp_replace(upper(merchant_descriptor), '[*#.,/]', ' ', 'g'),
                '[0-9]+', '', 'g'),
            '\s+', ' ', 'g')) as descriptor_normalized
    from settlements
),

-- the descriptor->merchant mapping, inlined because there is nowhere else to
-- put it. In practice this is the block that gets copy-pasted between
-- notebooks and slowly diverges.
mapped as (
    select
        n.*,
        case
            when n.descriptor_normalized like 'AWS%'
              or n.descriptor_normalized like 'AMAZON WEB SERVICES%' then 'Software & SaaS'
            when n.descriptor_normalized like 'SLACK%'
              or n.descriptor_normalized like 'GOOGLE%'
              or n.descriptor_normalized like 'DATADOG%'
              or n.descriptor_normalized like 'DDOG%'
              or n.descriptor_normalized like 'ZOOM%'   then 'Software & SaaS'
            when n.descriptor_normalized like 'UNITED%'
              or n.descriptor_normalized like 'UAL%'
              or n.descriptor_normalized like 'DELTA%'
              or n.descriptor_normalized like 'DL AIRFARE%'
              or n.descriptor_normalized like 'MARRIOTT%'
              or n.descriptor_normalized like 'COURTYARD BY MARRIOTT%' then 'Travel'
            when n.descriptor_normalized like 'UBER%'
              or n.descriptor_normalized like 'LYFT%'    then 'Ground Transport'
            when n.descriptor_normalized like 'STARBUCKS%'
              or n.descriptor_normalized like 'SQ STARBUCKS%' then 'Meals & Entertainment'
            when n.descriptor_normalized like 'WEWORK%'
              or n.descriptor_normalized like 'WE WORK%' then 'Rent & Facilities'
            when n.descriptor_normalized like 'STAPLES%'
              or n.descriptor_normalized like 'THE HOME DEPOT%'
              or n.descriptor_normalized like 'HOME DEPOT%'
              or n.descriptor_normalized like 'HOMEDEPOT%'
              or n.descriptor_normalized like 'COSTCO%'  then 'Office Supplies'
            when n.descriptor_normalized like 'SHELL%'   then 'Fuel'
            when n.descriptor_normalized like 'FEDEX%'   then 'Shipping'
            when n.descriptor_normalized like 'APPLE%'   then 'Equipment'
            -- and every merchant the list above has not caught yet still has
            -- to land somewhere, so fall back to the MCC. This list has to
            -- mirror seeds/mcc_categories.csv EXACTLY -- add one MCC here that
            -- the seed does not have and this query quietly stops agreeing
            -- with every other report in the company.
            when n.mcc = 7372 then 'Software & SaaS'
            when n.mcc in (3000, 3001, 3501) then 'Travel'
            when n.mcc = 4121 then 'Ground Transport'
            when n.mcc = 5814 then 'Meals & Entertainment'
            when n.mcc = 6513 then 'Rent & Facilities'
            when n.mcc in (5943, 5200, 5300) then 'Office Supplies'
            when n.mcc = 5541 then 'Fuel'
            when n.mcc = 4215 then 'Shipping'
            when n.mcc = 5732 then 'Equipment'
            else 'Uncategorized'
        end as merchant_category
    from normalized n
)

select
    date_trunc('month', settled_date)::date as transaction_month,
    merchant_category,
    round(sum(amount_usd), 2)               as net_spend_usd
from mapped
group by 1, 2
order by 1, 2

The dedupe, the soft deletes, the cents cast, the token join, the descriptor normalisation and the MCC fallback all have to happen again, here, by hand. Get any one of them wrong and the number is wrong quietly.

Against the mart9 lines
select
    transaction_month,
    merchant_category,
    round(sum(settled_amount_usd), 2) as net_spend_usd
from marts.fct_transactions
where company_id = 'cmp_08d98638c6fc'
  and is_money_movement
group by 1, 2
order by 1, 2

One table, no joins, no normalisation. Every decision above was made once, in version control, with tests on it.

Run it

Real mart data, queried in the browser. SELECT, WHERE, GROUP BY, ORDER BY, JOIN — no CTEs or window functions needed, because the derived columns already exist.

⌘↵
Everything here — lineage, SQL, every figure — is exported from a real dbt build. 1,726 lines of SQL.Repository ↗