How PR Reviews Shaped My Understanding of AI
Author: Brooks Beffa
5 August, 2026
I spent the first 8 years of my career in domains full of sensitive PII, PHI, and HIPAA regulations; primarily healthcare and insurance. Even as the AI Boom overtook the tech world, I was not professionally encouraged to introduce AI tools to the development process. In 2026 I was brought into a Xorbix team that had been deliberate in its AI adoption for years before my arrival. As I’ve worked to catch up with the rest of the team, some of the clearest lessons have come from our Pull Request (PR) review process.
Benefits
A thorough PR review involves multiple steps: reviewing the corresponding work item, maintaining an intimate understanding of the intended architecture and coding standards, and scrutinizing every line for gaps in correctness, completeness, and adherence. We found ourselves facing a classic speed vs. quality dilemma here – do we sacrifice the scrutiny of our review process to keep things moving, or sacrifice development velocity to maximize consistency and minimize tech debt? Xorbix introduced an internal PR review tool to address this dilemma with the help of AI.
All PRs require a linked Azure DevOps (ADO) work item which the tool reads to identify gaps in completeness or correctness. It also identifies violations of our defined architecture, standards, and conventions. After making its full assessment, the tool will point the human reviewer to areas of concern, name the violated standard, suggest remedies, and even offer drafted PR comments which can be edited, ignored, or posted with the click of a button. While human reviewers still hold the sole approval rights, this tool has simultaneously improved the velocity and scrutiny of our review process.
Known Limitations
The tool is of course limited by its input. If the work item is thin, so is the evaluation against it. If there are no acceptance criteria (AC) or detailed spec, we risk someone implementing their own unverified interpretation. Jr developers may surface ambiguities to the team or may implement a “best guess” approach. Coding assistants tend to do the same, but also risk convincing the user they’ve reasoned through the best approach, even if it’s grounded in misconception. Furthermore, an AI reviewer verifying against a thin spec and not identifying any issues serves as a false sense of security. My observation of these limitations drove home the importance of keeping an experienced developer at the helm to shape input and scrutinize output. AI can serve as a powerful assistant throughout design, implementation, and review, but should not be left unchecked.
Observed Challenges
When work is defined clearly, we still face the challenge of ensuring it’s followed consistently. Our repositories each include an Agents.md file with standards, conventions, and architecture relevant to the project. Our PR review tool has done such a good job checking against these standards that it actually surfaced an ongoing issue that we’d like to address upstream. Non-conforming PRs are regularly submitted, despite the standards being explicit and directly available to every agent. We’ve observed that premium models configured for complex tasks do a far better job adhering to standards and identifying violations, as do dedicated agents focused solely on the standards without context bloated by unrelated logic. A lightweight agent reasoning through other gaps and interacting with the user tends to lose sight of the defined standards by the time work is complete.
Conclusions
Investing human time in ensuring work items include clear, testable AC strengthens both the implementation and review steps, regardless of who or what performs them. Implementers face fewer ambiguities to work through, and reviewers have something explicit to verify against. Our PR review tool is effectively flagging named standard violations but can only verify correctness against a well-defined spec. If both the standards and the spec are clear, the remaining limitation becomes how reliably an agent can keep those instructions in context throughout a session. We are still working on ways to improve implementation agents’ alignment with both standards and requirements. Current efforts include a pre-commit check, improved definitions, and balancing cost against model capability and effort level.
The PR review tool is one example that has highlighted the benefits and limitations of AI for me. There’s a massive gap between accessing the tools and using them effectively to optimize the workday. I’ve benefited from the experienced AI users around me, but still have a long way to go in my journey.
