Scoring a factorisation
Read first: Domain aggregates by matrix factorisation
One number hides how a factorisation fails. Here is a headline plus seven sub-scores, each on a different axis, none needing labels.
The headline: held-out AUC
- Hide a random 10% of the round's entries: what is left of the matrix after earlier rounds, so in round 2 they are fractions, and a cell earlier rounds fully explained counts as empty.
- Fit on what remains.
- Score each hidden entry by
W H, and, for each, one empty cell from the same paragraph's own program: up to eight random picks among that program's columns, keeping the first empty one. - AUC is the chance that a hidden entry scores above an empty cell.
0.5 is chance, 1.0 is perfect. It asks whether the factors capture structure that generalises, or just memorise the cells they were shown.
The axes
| score | computed as | good is |
|---|---|---|
| fidelity | ‖residual − W H‖ / ‖residual‖ | low |
| coverage | share of 1s inside some factor's top-20 fields × top-20 paragraphs | high |
| stability | refit on four random 90% samples of the paragraphs; match factors one to one; mean cosine | near 1 |
| distinctness | largest cosine between two factors | low |
| focus | median share of a factor's loading in its top 20 fields | high |
| reach | share of factors whose top fields span 2+ programs (a class counts all its members' programs) | high |
| class share | share of the round's entries in unified-class columns | high |
Reading them together
- High AUC, low coverage: accurate where it speaks, but it explains little.
- Low reconstruction error, low stability: overfitted; resampling changes the factors.
- Largest cosine near 1: near-duplicate factors;
kis too big. - Low reach and low class share: one block per program; cross-program evidence is too thin.
Four matrices
| planted blocks | CardDemo, round 1 | CardDemo, round 2 | random inside each program | |
|---|---|---|---|---|
| held-out AUC | 1.000 | 0.910 | 0.479 | 0.500 |
| fidelity | 0.000 | 0.339 | 0.838 | 0.999 |
| coverage | 100% | 72.8% | 14.3% | 0.1% |
| stability | 1.000 | 0.975 | 0.949 | 0.134 |
| distinctness | 0.000 | 0.557 | 0.101 | 0.000 |
| focus | 1.000 | 0.997 | 0.844 | 0.132 |
| reach | 50% (median 1.5 programs) | 73.3% (median 5.5 programs) | 30% (median 1) | 10% (median 1) |
| class share | 10% | 21.4% | 16.6% | 0% |
Planted blocks: the 14 × 14 matrix from the previous page, threshold 4, k = 2. Its program labels are made up: blocks 0 and 1 are one program's, and F0 is put in a class with a field of an invented second program, which is all its reach and class share come from. CardDemo: threshold 4, k = 30, then threshold 2, k = 10. Random: 3,000 programs of 22 paragraphs, each paragraph making 70 random picks, repeats allowed, among its own program's 200 fields: 66,000 × 600,000 with 3.9 million 1s, threshold 2, k = 10.
- CardDemo round 1 is the result. It predicts which fields a paragraph uses even against its own program's other fields, covers 73% of field uses with its factors' top blocks, survives resampling, and 22 of its 30 factors span programs.
- Round 2 has nothing left to find. Its AUC is below chance and its factors cover 14%: what round 1 leaves is a long tail of fields seen in two or three paragraphs.
- The random matrix sits at the no-structure end on AUC, fidelity, coverage, stability and focus, as a score should. Low distinctness alone means nothing there: random factors are also unlike each other.
Why the empty cells come from the same program
A real corpus matrix is about 99.99% empty, and most of it is one block per program. An empty cell drawn from anywhere almost always pairs a paragraph with another program's field. Ranking a hidden 1 above that is easy, so the AUC would look strong even when each factor is one program's block.
| empty cells drawn from | CardDemo round 1 | CardDemo round 2 |
|---|---|---|
| anywhere in the matrix | 0.981 | 0.731 |
| the paragraph's own program | 0.910 | 0.479 |
About 0.07 of round 1's score, and all of round 2's lead over chance, is the easy call between programs. The scorecard uses only the second row; the first is the same fit and the same hidden entries, scored once for this page against empty cells drawn uniformly from the whole matrix. A class column counts as belonging to every program its members are in. A hidden entry whose eight picks find no empty cell gets no partner; if none finds one, the AUC is nan.