OpenAI has published a new analysis exposing critical flaws in SWE-Bench Pro, a widely used benchmark for assessing AI coding capabilities. These reliability issues cast significant doubt on the accuracy of how top artificial intelligence models are currently being tested and ranked.