Byte-extent aliasing

In COBOL, memory is bytes, not variables. Treat every field as a byte range and aliasing follows from one test: do two ranges overlap?

Where this shows up in reverse engineering

Any format where more than one name can describe the same bytes needs this test, not just COBOL — the technique is the same wherever the hiding happens.

• A commarea passed between programs is a block of bytes two different programs may lay out two different ways, each matching its own copybook. Following a value across that boundary means tracking the byte range it occupies, not the field name either side happens to give it — the name is local to one program's view, the bytes are not.
• A dataflow result that only tracks declared names is exactly the trap this page's own counter-example sets: a write through one REDEFINES view and a read through another look unrelated by name and identical by byte range. Treating only the name as the identity of a value is the single most direct way to make an otherwise-correct dataflow analysis wrong on real legacy data layouts.
• The same question outside COBOL: a packed binary buffer, a shared-memory structure exposed through a foreign-function boundary, a union type reinterpreting one block of memory as two different shapes — any reverse-engineering task built on top of a byte-oriented format runs into the identical overlap-vs-covers distinction this page's rule answers.

The problem

The reaching definitions page tracked writes by variable name. That works when every name owns its own storage. In COBOL many names share storage. A group like REC is just the bytes of its children laid end to end, so writing REC writes WS-A and WS-B. REDEFINES gives the same bytes a second name with a different shape. A move into one child changes half of the group. Tracking by name gets every one of these wrong.

The intuition

Lay the program's data out as the compiler does: each field is a start offset and a length in some storage region. Forget names. A write paints the bytes it covers. A read looks at the bytes it covers and asks which writes painted any of them. Two fields alias when their ranges overlap — whatever they are called, however they were declared.

The example

REC is eight bytes: WS-A is bytes 0–4 and WS-B is bytes 4–8. WHOLE redefines the same eight bytes as one field. The program writes the group, then one child, then reads each child and the redefinition.

       IDENTIFICATION DIVISION.
       PROGRAM-ID. BYTEEXT.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  REC.
           05  WS-A             PIC X(4).
           05  WS-B             PIC X(4).
       01  WHOLE REDEFINES REC  PIC X(8).
       01  OUT-A                PIC X(4).
       01  OUT-B                PIC X(4).
       01  OUT-ALL              PIC X(8).
       PROCEDURE DIVISION.
       MAIN-PARA.
           MOVE 'AAAABBBB' TO REC.
           MOVE 'XXXX' TO WS-A.
           MOVE WS-A TO OUT-A.
           MOVE WS-B TO OUT-B.
           MOVE WHOLE TO OUT-ALL.
           STOP RUN.

Step by step

The strip is bytes 0 to 8 of working storage. Each byte shows the statement that last wrote it. Below it are the definitions the analysis keeps, one per write, each with the range it wrote.

The rule

Two different tests, on purpose. A definition is a possible source of a read when it overlaps any byte of it. A definition is certainly gone only when a later write covers every byte it wrote. Anything between — a partial overwrite — leaves it alive.

What goes wrong

• The answer can be loose. At MAIN-PARA/2 the strip shows that bytes 0–4 were last written by MAIN-PARA/1 alone. But the analysis keeps one definition per write, and MAIN-PARA/1 covers only half of MAIN-PARA/0's eight bytes, so it cannot kill it. MAIN-PARA/0 still overlaps the read, and is reported as a definer although none of its bytes survive there. That is an over-approximation: it never misses a real source, it sometimes names one too many. Getting it exact would mean splitting a definition into byte ranges every time something writes part of it.
• Kill only on certainty. A write whose position is not known exactly — a table element with a computed subscript, a field after an OCCURS DEPENDING ON — may alias widely but must cover nothing. Treating it as covering would delete real sources: a wrong answer, not a loose one.
• A field that cannot be sized has no extent at all, and the analysis cannot place its reads and writes. Cobble reports such a program as unsizeable rather than guess.

In cobble

Every dataflow stage works on these extents. Each field's extent comes from its declaration alone — level, usage, OCCURS, REDEFINES. The two tests are the ones above: whether two extents may overlap, and whether one covers all of the other. A move of a literal has no source field, so cobble names its source LITERAL.