Token-Efficient Change Detection in LLM APIs

Résumé

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model’s Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that Border Inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in vivo and in vitro experiments show that Border Inputs are easily found for the majority of tested endpoints, and achieve performance on par with the best available greybox approaches. B3IT reduces costs by 30× compared to existing methods, while operating in a strict black-box setting.