Background

With the adoption of agentic coding tools, the majority of code currently written has been written by AI. Usage varies by company, domain, and language, but Google and Anthropic put the share of new code generated by AI at their companies at 75% and 80% respectively. As more code is generated by LLMs, developing accurate LLM-led reviewing protocols will be essential to developing functional, maintainable code at high velocity.

One approach to developing LLM-led reviewing protocols is to try to increase the diversity of agents that are used in code development. In many code development organizations, having two separate humans review code seems to improve code quality, so having diverse LLM agents working on code seems like it should improve outcomes.

Diversity of perspectives seems to consistently deliver better outcomes across many domains. Theoretical work shows that a diverse set of problem-solvers provides better problem-solving ability than a group of the highest-capability problem-solvers. This theory seems to hold up in reality–when human problem-solvers are put into groups of varying cognitive perspectives, the group’s problem-solving ability increases. In machine learning, ensembling refers to the practice of aggregating results from many smaller independent classifiers in order to create more accurate scores or predictions overall.

It does seem that using ensembles of LLMs can deliver improved performance on many benchmarks over a single model alone. And, LLMs are better able to find security issues within code generated from different models than code from the same model. Additionally, models exhibit self-preference bias and prefer outputs provided by themselves, perhaps due to lower perplexity associated with outputs created from the same model, so models reviewing their own code may struggle to spot issues in the same way that they might when reviewing other code.

However, there are some factors that might lead to models being better reviewers of their own code than of others’ code. The coding styles of different models are unique enough that it is possible to reliably classify code generated by a given model. It may be that a model reviewing code with less familiar code style (because it was not generated in the reviewer’s typical style) may have difficulty identifying an error. In addition, models from different providers tend to show high co-failure rates on benchmarks so it could be that “diversity” gains by using different models may not be sufficient to tackle problems of sufficient difficulty.

In this analysis, I’ll look at the effects of two ways of injecting “diversity” into LLM code reviews in order to ensure that code is functional:

Providing a different “persona” to the system prompt to create a review under different model activation patterns

Within the same LLM model, using different prompts may shift models’ internal activations to produce different “thinking” patterns. “Task vectors” emerge within language models when in-context learning techniques are used to change model behavior. It may be that these internal activation changes could shift the way that models process tokens enough to get “fresh eyes” and give effective feedback. Injecting diversity within the same model by using persona updates seems like it would keep a lot of the benefits of diversity injection while avoiding some operational issues associated with using different models for reviewers/code generators.

Using a different model as the code reviewing model to provide additional diversity

Models from different providers were trained on different data and different training strategies, so it is natural to assume that they might make errors in different ways. At some level, reinforcement learning is probably being used to either generate training data or directly update model weights, and the rewards chosen for that reinforcement learning process will yield models that perform differently. I couldn’t find any publications from Anthropic, Google, or OpenAI about their exact process here but here’s something from DeepSeek.

Prior to running the experiment, I expected that both diversity-improving interventions would be associated with better reviewer performance. I expected that modifying the system prompt would provide a smaller benefit than using a different model.

Setup

To evaluate coding performance under varying review conditions, I used the sanitized subset of the Mostly Basic Python Problems dataset, which includes ~400 problem descriptions as well as corresponding unit tests to evaluate successful code generation. I used a random subset of 200 problems in this dataset.

I chose LLMs from three providers (OpenAI, Google, and Anthropic) that seemed to be at similar price and code performance tiers:

For all models, I turned on some form of thinking (OpenAI supported an “effort” specification which I set to “medium”, Gemini supported a “medium” level in their thinking config, and Anthropic supported setting a reasoning budget for their model).

For each model, I provided a “persona” to the system prompt. One “persona” was meant to be a cautious, thorough coder, and the other “persona” was meant to creatively solve problems with less regard for rules or conventions.

Each model engaged in the following activities:

1. Initial code generation based on the problem specification (this was done only under the cautious persona, for all three model providers).
2. Initial code review for bug identification (this was done for all combinations of persona/model)
3. Post-review bug fixing (this was done only if the persona/model combination identified a bug)

All code can be found in my github.

Metrics

I was primarily concerned with recall and post-reconciliation error rate. In practice, better recall (# of correctly identified bugs / # of total bugs) means that you actually identify issues and are likely to actually fix them whether using the model that identified the error, another model, or human intervention. Post-reconciliation error rate (% of code chunks with errors after the full bug identification and reconciliation process) aligns with real-world utility of a coding strategy; if a strategy yields a low post-reconciliation error rate the reviewer model can fix the error quickly without needing any external intervention or complexity.

Results

Overall, reviewers had a 12.1% recall rate at identifying bugs present. Here’s a confusion matrix:

| Actual      |   Flagged |   Not flagged |
|:------------|----------:|--------------:|
| Bug present |       126 |           912 |
| No bug      |       151 |          2411 |

Recall Rate and Diversity-inducing Interventions

First, I estimate how the different diversity interventions influence recall rate (i.e. identification of bugs)

Optimization terminated successfully.
         Current function value: 0.354065
         Iterations 7
                           Logit Regression Results                           
==============================================================================
Dep. Variable:               detected   No. Observations:                 1036
Model:                          Logit   Df Residuals:                     1029
Method:                           MLE   Df Model:                            6
Date:                Sat, 27 Jun 2026   Pseudo R-squ.:                 0.04344
Time:                        21:00:56   Log-Likelihood:                -366.81
converged:                       True   LL-Null:                       -383.47
Covariance Type:              cluster   LLR p-value:                 9.122e-06
===================================================================================================
                                      coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------------------------
Intercept                          -2.4738      0.313     -7.914      0.000      -3.086      -1.861
C(generator_model)[T.gemini]       -0.2731      0.300     -0.911      0.362      -0.861       0.315
C(generator_model)[T.gpt]           0.4064      0.323      1.257      0.209      -0.227       1.040
C(reviewer_model)[T.gemini]         0.7338      0.336      2.185      0.029       0.076       1.392
C(reviewer_model)[T.gpt]            0.9345      0.299      3.120      0.002       0.348       1.522
C(reviewer_persona)[T.Creative]     0.0372      0.106      0.350      0.726      -0.171       0.246
concordant                         -0.8298      0.212     -3.912      0.000      -1.246      -0.414
===================================================================================================

The coefficient on model concordance was negative–model configurations where the reviewer was the same model as the code generator had lower rates of identifying true bugs than model configurations where the reviewer was a different model from the code generator.

Note that the reviewer model effects are on the same scale as the model concordance effects. It may be the case that in the real world, the optimal strategy may not be “use a different model than the one you used to write the code”, but “use the best-performing reviewing model that is out there”.

Here’s a graph showing how this breaks down by model provider. Interestingly, GPT 5.4 mini seems to perform nearly equivalently at identifying issues regardless of the generator model.

Post-Reconciliation Error Rate and Diversity-Inducing Interventions

Now, we’ll estimate the cumulative effect on code quality via the end-to-end pipeline from different diversity configurations:

Optimization terminated successfully.
         Current function value: 0.598801
         Iterations 5
                           Logit Regression Results                           
==============================================================================
Dep. Variable:             post_error   No. Observations:                 3595
Model:                          Logit   Df Residuals:                     3588
Method:                           MLE   Df Model:                            6
Date:                Sat, 27 Jun 2026   Pseudo R-squ.:                0.002432
Time:                        21:00:58   Log-Likelihood:                -2152.7
converged:                       True   LL-Null:                       -2157.9
Covariance Type:              cluster   LLR p-value:                    0.1053
===================================================================================================
                                      coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------------------------
Intercept                          -0.7418      0.152     -4.896      0.000      -1.039      -0.445
C(generator_model)[T.gemini]       -0.2860      0.095     -3.004      0.003      -0.473      -0.099
C(generator_model)[T.gpt]          -0.1439      0.080     -1.805      0.071      -0.300       0.012
C(reviewer_model)[T.gemini]        -0.0600      0.034     -1.787      0.074      -0.126       0.006
C(reviewer_model)[T.gpt]           -0.0264      0.026     -1.018      0.309      -0.077       0.024
C(reviewer_persona)[T.Creative]     0.0020      0.010      0.206      0.837      -0.017       0.021
concordant                          0.0123      0.019      0.637      0.524      -0.026       0.050
===================================================================================================

Diversity interventions were not significant here–turns out that relatively few errors were identified, and it was very infrequent that code review changed the code state between working and non-working.

| Reviewer Model   |   Working Code Breakages |   Broken Code Fixes |
|:-----------------|-------------------------:|--------------------:|
| claude           |                        9 |                   1 |
| gemini           |                        4 |                  13 |
| gpt              |                        4 |                   4 |

Conclusions

It does seem that the practice of having distinct generator/reviewer models may improve the reviewer’s ability to identify issues upon review.

The effect of creating different “personas” on the same model was too small to measure here. I think it’s possible that this is because my “persona” prompt wasn’t detailed or long enough to shift model activations meaningfully. It could be that if I included specific examples or more details it would have led to better performance on reviews. In addition, our test did not include any code generated by the “Creative” persona which limits our ability to draw solid conclusions (what if the “cautious” persona is just way better at reviewing code and that swamps the diversity effect?).

Note that there are many different strategies that I would imagine would have larger effects than having distinct generator/reviewer models or different model personas. I think the biggest one is test-driven development (make sure that you have good test cases in place that the model can use iteratively to ensure the code works before it’s deployed) but I also think that providing project-specific code review instructions and using better models (e.g. Opus instead of Haiku) will deliver positive effects on performance.

Limitations

My theory on writing these posts is:

1. Anything worth doing is worth doing poorly
2. I’m not forcing anyone to read this stuff (so don’t have to feel bad if it’s not comprehensive)
3. I’m just in this for the love of the game (so I don’t really have to worry if people think I’m not working comprehensively enough)

So the scope of this analysis was kept pretty small. But, I’ll list out some things that I would hope to further improve on/investigate sometime in the future:

1. My coding “harness” was very limited and models did not need to use tool calls to evaluate code generated by other models. I believe that as codebases grow larger, model-specific choices for naming conventions and code organization may provide higher benefits for using a same-model reviewer to review code
2. This test involved only code generation; at companies, most AI-written code must be merged into a larger codebase
3. I used really cheap models for this test to avoid saturating the benchmarks, but I’m pretty sure very little actual coding is done by Haiku or GPT 5.4 mini in the real world.
4. I would love to look more into the “persona” perspective–I think that if I put some more time in, I might be able to write better prompts and achieve some diversity benefits there.