Corpus protocol¶
The controlled-corpus procedure for closing the open questions in
docs/formats/goodnotes.md and docs/formats/notability.md. Read
docs/reverse-engineering.md first — this is the concrete recipe for that
doc's "known-shape corpus experiments" step, scoped to the Mac App Store
builds of GoodNotes and Notability, to be extended with iPad+Pencil cases
once that hardware is in the loop.
Per-case procedure¶
For every numbered case below:
- Create a new document from the app's default template.
- Perform exactly one action (draw one stroke, change one setting, insert one object — whatever the case specifies). One action per case keeps the diff attributable.
- Export both: the app's native format (
.goodnotes/.note) and the app's own PDF export of the same document. The PDF export is what the rendering-validation gate (docs/reverse-engineering.md) diffs our parse against — always take it in the same pass so it's guaranteed to reflect the exact same document state. - Record the file in
corpus/manifest.tomlwith fields:file,source = "self",case(the number/slug below),app,app_version,os_version,date,sha256.
Naming convention: gn-NNN-slug.goodnotes (GoodNotes) /
nb-NNN-slug.note (Notability), NNN zero-padded to the case number.
Example: gn-02-horizontal-line.goodnotes.
Diff pairs¶
For cases where the isolating signal is "what changed in the file," export twice: once after the case's one action, then perform one more action and export again (both native and PDF). Binary-diffing the two native exports isolates exactly the bytes that second action touched — this is the fastest way to locate an unknown field once its rough location is already known from the typed-section/protobuf structure (e.g. isolating which byte range flips when going from "no highlighter" to "highlighter" on an otherwise identical stroke).
Container snapshots¶
Before and after each save, snapshot the app's container directory
(~/Library/Containers/<bundle-id>/) to find where the app keeps its
working store before a formal export — GoodNotes/Notability may stage
data differently from what an "export" produces, and the live container
can reveal intermediate state an export normalizes away. dump-container.py
(planned, tools/re/) will automate the before/after diff; until it
exists, do this manually with tar or rsync -av --checksum snapshots to
a scratch directory.
The numbered case matrix¶
| # | Case | Isolates |
|---|---|---|
| 00 | Empty doc | Baseline container/index shape with zero strokes |
| 01 | Single dot (tap, no drag) | Minimal single-point stroke encoding |
| 02 | ~100pt horizontal line at a known grid position | Axis mapping, coordinate units, page-space origin |
| 03 | Corner-to-corner diagonal | Origin corner, y-axis direction |
| 04 | Two strokes | Multi-stroke record framing/ordering |
| 05 | One stroke per pen type (every tool the app offers) | Pen-type field |
| 06 | Each preset color + one custom color | Color field encoding, preset vs. custom representation |
| 07 | Highlighter stroke | Highlighter flag/opacity vs. regular pen |
| 08 | Partial erase + whole-stroke erase | Eraser representation (point removal vs. tombstone vs. new geometry) |
| 09 | Shape-tool object (e.g. a drawn rectangle/circle if the app auto-shapes it) | Shape-object encoding vs. freehand stroke |
| 10 | Text box "Hello" | Typed-text object encoding |
| 11 | Inserted image | Image/attachment encoding and page linkage |
| 12 | One-page PDF imported as background + one annotation stroke | PDF-background linkage (attachment ↔ page ↔ ink) |
| 13 | Three-page doc, stroke on page 2 only | Page ordering/indexing with sparse ink |
| 14 | Non-default paper size + landscape | Page-dimension field |
| 15 | Short audio recording (Notability only) | Audio attachment encoding |
| 16 | Pressure ramp stroke (iPad+Pencil only) | Per-point pressure/width channel mapping |
| 17 | Tilt/azimuth stroke (iPad only) | Tilt/azimuth channel presence and mapping |
| 18 | Same doc exported from Mac and iPad | Mac-corpus parity check — confirms cases 00–17's findings hold for the iPad-authored files too, not just Mac-authored ones |
Cases 16–18 require iPad+Pencil hardware and are deferred until that's available; 00–15 (00–14 for GoodNotes, 00–15 for Notability, whose audio support GoodNotes lacks) run on the Mac App Store builds first.
Per-case "resolves" — what each case is for¶
Point every case result back at the specific open question it's meant to
close. From docs/formats/goodnotes.md's open-questions list and
docs/formats/notability.md's open-questions list:
- GoodNotes field 14/15/7 (pen type / highlighter flag, currently unlocated) → cases 05 (one stroke per pen type — diff the candidate fields across pen types) and 07 (highlighter — diff against a same-color, same-width regular pen stroke from case 05).
- GoodNotes geometry-blob section 9 (5 floats/point — x, y, w, +2
unknown columns, suspected pressure/tilt) and Notability
curvesfractionalwidths→ stroke mapping (count mismatch: fractions present don't sum to total points) → case 16 (pressure ramp, iPad+Pencil): for GoodNotes, watch which of section 9's extra two columns moves monotonically with applied pressure; for Notability, watch whethernumfractionalwidthsstarts matchingnumpointsfor a pressure-drawn stroke, which would confirm fractions are per-stroke and only present for pressure-capable input. - GoodNotes page-dimension field (reader currently assumes A4
unconditionally) → case 14 (non-default size + landscape): compare
index.notes.pb/ page metadata across a default-size doc and this one to isolate the field. - GoodNotes eraser representation → case 08.
- GoodNotes images & text boxes (unmapped) → cases 10, 11.
- GoodNotes PDF-background ↔ page linkage → case 12.
- GoodNotes
index.events.pb/index.search.pbcontents → any case with a diff pair (repeated action) showing how those indexes change incrementally. - Notability
eventTokens,InkedSpatialHashinternals → likely derived/cache data; lower priority, revisit if a case's diff pair implicates them directly. - Notability page/paper metadata location + PDF-background alignment → cases 12, 14.
- Notability current-app export shape (legacy zip+plist vs. a newer
"Notability Cloud" format) → case 00, day one: does a fresh
Mac-app export still produce the legacy
Session.plistshape the reader currently expects, or something else? This gates whether the rest of the matrix is even meaningful for the current app version.
Corpus → fixtures promotion¶
Per docs/reverse-engineering.md's sample-ethics section: cases whose
result is small and contains no personal data get promoted from
corpus/scratch/ (gitignored) into core/tests/fixtures/<format>/
(committed, CC0) once a reader exists to exercise them. Third-party
samples (e.g. corpus/third-party/goodparse/samples/) never get promoted
— only self-generated corpus files following this protocol do.