Explainer 12 of 15

Domain aggregates by matrix factorisation

Read first: Exact field classes across programs

An aggregate is a set of fields that make up one concept and are read and written together. A customer's name and address; an account's balance and limits. This page shows the encoding, why NMF suits it, and how rounds handle fields of very different popularity.

The encoding

Rows are paragraphs, (program, paragraph). A paragraph is a unit of behaviour: it does one job, so the fields it touches belong to the job.
Columns are fields, named PROGRAM:RECORD.FIELD. Every member of an exact class is replaced by the class, so one datum is one column across programs. Table occurrences fold into their declared field.
A 1 for every field a read covers, in the read's paragraph and in the paragraph of each of its definers, whichever of the read's bytes that definer wrote.

Nothing is counted in pairs or weighted. An earlier version scored field pairs by pointwise mutual information. It dropped every field seen in fewer than three paragraphs, so rare fields had empty rows.

NMF

Non-negative matrix factorisation approximates the paragraph × field matrix X as a product of two smaller non-negative matrices:

X ≈ W H W ≥ 0 (paragraphs × k), H ≥ 0 (k × fields)
minimise ‖X − W H‖²

Row f of H is factor f's field loadings: what the aggregate is made of.
Column f of W is its paragraph loadings: where it is used.
k is the number of factors, chosen per round.

Why it should work

Model each paragraph as touching the union of a few aggregates: an update paragraph touches the record it updates and a status field. Then each row of X is roughly a sum of a few aggregate rows, which is exactly what W H can express.

Non-negativity is what makes the factors readable. Nothing can be subtracted, so a factor can't be "these fields minus those". Each factor has to be a part that adds up with others. Lee and Seung's 1999 paper showed the same effect on faces: NMF learned parts such as noses and mouths, where principal components learned whole-face templates.

That holds only if paragraphs really do work on a few aggregates each. A paragraph that touches a hundred unrelated fields, such as a commarea copied whole, is a row no small k explains well. Hence the field exclusions, and the diagnostics.

The example

Fourteen paragraphs and fourteen fields in three planted blocks. Every cell inside a block is 1:

block	fields	paragraphs	paragraphs per field
0	F0–F4	P0–P5	6
1	F5–F9	P6–P11	6
2	F10–F13	P12–P13	2

With k = 2 over the fields seen in at least 4 paragraphs, NMF returns blocks 0 and 1 as its two factors: each factor's top fields are one block's fields, and its top paragraphs are that block's paragraphs.

Rounds

Popular fields dominate a single fit, so decompose works in rounds of decreasing threshold. Each round:

keeps the columns of fields seen in at least its threshold of paragraphs;
subtracts what earlier rounds' factors rebuild there, clipping at zero;
factors what is left.

round	threshold	k	columns	finds	already explained
1	4	2	F0–F9	blocks 0, 1	0%
2	2	1	F0–F13	block 2	about 60 of 68 entries

Round 2 sees all fourteen columns, but round 1's factors already rebuild blocks 0 and 1, so almost nothing is left of them. Its single factor goes to block 2.

Keeping it sparse

The residual looks dense: max(0, X − rebuilt), where rebuilt is a full matrix. On a production corpus that is 65,327 paragraphs × 592,104 fields, about 300 GB as doubles. The first version built it, and died before the first fit.

But factors are never negative, so rebuilt is never negative. Where X is 0, the residual is max(0, 0 − something ≥ 0) = 0. So the residual is non-zero only where X already is:

residual(i, j) = max(0, X(i, j) − Σ_rounds W(i, ·) · H(·, j)) only at X's non-zeros
cost per round = non-zeros × k

The answer is the same up to floating-point rounding, at a fraction of the memory. When the change was made, on 2026-10-26, a synthetic matrix of 600,000 fields and 4.5 million non-zeros ran in 50 seconds within 1.2 GB.

What goes wrong

A small k over many programs. If most columns belong to one program each, the matrix is nearly one block per program. With thousands of programs and k = 16, each factor goes to the biggest remaining block, one program's most-touched fields, and structure inside a program never surfaces.
Hub fields. Fields that almost every paragraph of a program touches, such as commarea fields, fill a factor's top fields. Exclude them by pattern, or they are the aggregate every time.
The threshold. A field below every round's threshold is never factored.

On CardDemo

CardDemo's matrix has 455 paragraphs, 1,883 field columns and 5,850 non-zeros. Round 1, fields in at least 4 paragraphs, k = 30, no exclusions. Three of its 30 factors:

Factor 4: the account record. Its 13 heaviest fields are each a class spanning seven programs: CBACT01C, CBACT04C, CBEXPORT, CBTRN02C, COACTUPC, COACTVWC and COBIL00C. The next field loads 0.22.

top fields (loading)	top paragraphs
ACCT-CURR-BAL (1.39) ACCT-CREDIT-LIMIT (1.37) ACCT-ACTIVE-STATUS (1.37) ACCT-GROUP-ID (1.33) ACCT-REISSUE-DATE (1.33) ACCT-OPEN-DATE (1.32) … ACCT-ID (1.17)	CBTRN02C 1500-B-LOOKUP-ACCT CBTRN02C 2800-UPDATE-ACCOUNT-REC CBACT04C 1100-GET-ACCT-DATA CBACT04C 1050-UPDATE-ACCOUNT CBEXPORT 3100-READ-ACCOUNT-RECORD CBEXPORT 3200-CREATE-ACCOUNT-EXP-REC COACTUPC 9300-GETACCTDATA-BYACCT COACTUPC 9500-STORE-FETCHED-DATA CBACT01C 1300-POPUL-ACCT-RECORD …

One aggregate, and the paragraphs that read or update it, across batch and online.

Factor 19: the user security record. SEC-USR-TYPE, SEC-USR-LNAME, SEC-USR-FNAME, SEC-USR-ID, each a class over COSGN00C and COUSR00C to COUSR03C, used mostly by the read and update paragraphs of the four user programs. Its next fields are the programs' WS-MESSAGEs, at under half the loading: messages set alongside the record.

Factor 13: one program's error message. COACTUPC's ERROR-OPNAME, ERROR-FILE, ERROR-RESP, ERROR-RESP2 and the FILLERs between them, all at loading 1.15, then WS-RETURN-MSG at 0.80. Four paragraphs carry nearly all of it: 0000-MAIN and the three that read the account, customer and cross-reference files. It is real, but local to one program.

Hubs. Factors 15, 17, 18, 21 and 24 are each one batch program's IO-STAT1 and IO-STAT2, status fields, each loading above 0.07 on at least 11 paragraphs of that program. Excluding IO-STAT by pattern drops those five; the account, customer and user security records stay factors.