Oracle

An oracle is an independent source of truth that can say whether an output is correct without a human reading it: a reference implementation, a differential comparison against a known-good tool, a property test, a type checker. Software testing has called the difficulty of obtaining one the oracle problem since long before agents existed, and the surveyed answers — specifications, contracts, metamorphic testing — all end at a human where none of them fit. Where an oracle exists, verification is cheap and parallelism is safe. Where none exists, every gate on this page degrades into a model arguing that it is finished. On this catalog's evidence, whether a task has an oracle predicts more about how well agents do on it than any topology or prompt does — an operator's observation from running the work, not a measured result, and offered as one.