Writing Well 2: Judge Models

It is Known

It’s a well known fact that LLMs are not good judges. They favor their own writing, they don’t compare to humans, and they are uncalibrated when assigning values. Except those statements are not true for modern LLMs. I reviewed both writing and vision in LLMs to determine whether they could be used as judges. I found that common knowledge is outdated or plainly wrong.

LLMs are well correlated with the human aggregate. They can function effectively in non-verifiable domains where you want human judgement. The size of the model matters.

Creative Writing

The Limits of Automatic Evaluation of Creativity in Large Language Models — Tutone, Franceschelli & Musolesi, August 2026 studies short stories and reports weak model–human alignment and an LLM tendency to give model written stories max scores. However, they never make the comparison for human to aggregate human judgements. They require a 1 - 5 judgement from the LLM or human judge.

The 1 - 5 judgement requirement will be shown to be the biggest issue. The true statement from my introduction to this essay is “LLMs tend to be poorly calibrated when asked to judge on a numerical scale without in-context references.” You may notice humans are often “poorly” calibrated considering that many US restaurants with a 4.0 Star on Google maps is not a place you should want to eat. The five star grading is biased and non-linear.

Poorly is too strong a modifier because there is calibration. Humans naturally know how the locals rate a restaurant through context clues. There is calibration from context. Without in-context calibration, an LLM will revert to some cultural assumption. If you are asking about creativity then you bias it towards considering creativity. If it is grading creative writing tasks it might have a baseline of human internet text which is by and large not creative in comparison.

In the Creative writing study cited, the LLM (Llama-3.3-70B-Instruct) assigned the Human written stories an average of 4.9/5 stars for creativity. The LLM then assigned the LLM written stories 5/5 stars on average. Humans on the other hand had an average grade across all essays of 3.2 stars. The issue could be construed as the LLM. It grades too high across the board. I would make the argument this is a systematic issue with the study. Luckily, their data was published for analysis.

In Context Calibration

I initially used a pairwise ranking during my reconstruction to determine if LLMs could be used for creativity analysis. However, there is still the question of ICL and grading. So before we get into a better setup, I wanted to see how well ‘modern’ LLMs would do with and without in-context examples for grounding. I used GLM 5.3 Flash with human grades as ICL examples. The single example was something ranked 3 stars, 3 examples included stars 2 - 4 and 5 examples included one example of every rating.

The result is pretty clear. The model initially biased high (similar to the Llama model) but it had better discernment. When we added a single ICL example, it got much closer to the human baseline. However, the extremes collapsed. Increasing the number of examples did not seem to improve the agreement with the human ranking distribution. This includes extreme grades as ICL. Despite the distribution match, there’s low correlation between the median human ranked and LLM ranked stories. The confidence interval includes no correlation.

Figure 2 does seem to indicate that there is a bias towards LLMs from an LLM judge. The bias is statistically significant (P < 0.05). If you instruct an LLM to generate something creative then you would want another judge to also see it as creative. The humans on the other hand had written the stories in response to a writing prompt on Reddit. This could mean that creativity was not an explicit target. If this were the case then we would expect the LLMs to have more creative stories overall. There is not enough evidence to explain whether the bias was derived from model or data.

Pairwise

The next set of experiments used pairwise grading. I used the GLM 5.3 Flash model. For the human rankings, I would defer to the higher human rated text. So if one text got 4 stars and one got 2 then I would consider the 4 star text ranked higher. Below includes both a re-analysis of the data and then a reconstruction.

In the original study, there was no significant correlation with the human judges compared to the human judges in aggregate! On the human written stories, the Spearman correlation was 0.059 with a 95% confidence interval of -0.203 to +0.316. Here I compared a held out human judgement with the remaining aggregate. There is an issue. Many stories only have a few judges (some as small as 2) which makes this comparison difficult. Despite this, the median correlation is worse than the LLM median to Human aggregate. LLMs were correlated with the human aggregate with a Spearman score of ~0.232. The confidence interval still includes zero.

The Spearman values I cite are from my re-analysis of the data. Technically, the study used a Kendall Tau-B which is also 0.23 and is positive within a 95% confidence score above 0. That’s a slight positive association for LLM - Human aggregate using their 1 - 5 rankings. Held out Kendall Tau-B for humans is 0.048 with a confidence interval covering 0. We still have an issue where the LLMs had no discernment according to the studies rankings. Any statistical measure needs to account for that 5 star bias and we did not here.

There is no significant judgement we can make in comparison with humans for this study because there is no agreement from the judges. If the human judgements were uncorrelated then it stands to reason that it would be difficult for the LLMs to correlate with the humans. When I ran my reconstruction with pairwise ranking to check correlation with the humans, I used the human judge score of two essays to determine which a human might prefer. I then asked the GLM model which essay was more creative. I got a correlation of -0.089 using GLM5.3 reasoning set to low. Zero was in the confidence window.

In the pairwise reconstruction we also saw no statistical preference for LLM generated text. This might imply that LLMs when blindly judging 1 - 5 do show preference for LLM generated text. However, that bias could only manifest in that specific judging pattern. The bias could be from the study design. Regardless, the pairwise test found that there was no detectable bias and there was no correlation. I wanted to see if humans are the issue.

Looking at the Data

I then looked at the data. I agreed more with the GLM5.3 judgements than I did with the human judgements from the paper. These were not easy judgements! I can see why the humans did not agree with each other. It would then be expected that an LLM would not agree with the human label. I’ll include an example pair below.

Story A:

The pitter-patter of little feet across the metal grating across the living quarters punctuated the otherwise dull hum of computers and machinery which, in-turn, punctuated the maddeningly monotonous silence of deep space. Sarah sat reading her daily reports, preparing for another shift in sick-bay. Her daughter Megan tugged at the corner of her purple robe, smiling up at her mother with the most endearing green eyes. Innocence was trademark, innocence and the faith of her mother.

" Mom" Megan began, " are we going to settle another planet today?"

Sarah turned and smiled but shook her head. " Next week."

" You said that last week" Megan grumbled and turned to go back to her studies.

Hundreds of families across one hundred generations aboard one mammoth ship. A thousand years ago it was launched with the singularly simple goal of depositing a collection of humans on another planet to build a colony. Planet by planet, system by system the H.S. Renewal has visited a laundry list of exoplanets but each has been crossed off like items on a shopping list. One simple task had turned into the longest mission in mankind's fifty thousand year history. In fact the current generation of explorers could be called human in genetic makeup only. None had any first-hand knowledge of Earth or the Sol system as contact with Earth had been lost some two hundred years ago. It is unknown who fired the first shot but we do know it was the United States who committed to full-scale nuclear retaliation. Now the Renewal contains the only known surviving Homo sapiens. Survival the primary goal, education the primary means of attaining that goal. Survive, educate, populate; that is the motto of a faction of the refugees aboard that still believe their voyage ever had a mission. Another faction believe it as the last voyage of the damned, set off a thousand years ago by fools. Sarah styled herself a believer but James, her husband, fell firmly into that second category, known as denialists.

" Five-to-one she nails aces that chemistry test today" James bragged as he sat down opposite his wife with his morning coffee.

Sarah giggled briefly, her eyes never leaving her reports. " I just hope she passes."

" Pass? She's been studying her little ass off for the past two weeks, she better do more than just pass" James explained.

" I studied for almost a week longer than her and I barely passed."

" A doctor that barely passed chemistry, only on this voyage of the damned is that possible."

Sarah shook her head and sighed. " I know you do n't believe but I do and I asked you to keep that denialist talk out of our quarters."

" No, you asked me to keep it away from Megan. She's buried in her chemistry textbooks, I could scream out chocolate ice-cream and she would n't hear me. Besides, at this point nobody would know how to start a colony even if we did find a suitable planet. We stopped educating builders with your great-grandparents and we stopped educating architects with their grandparents!"

" Damnit Jim, we have a ship full of engineers and materials scientists! Find us a planet and we'll settle it. I'll give you five-to-one on that!" Sarah's retort was sharp and much less playful than her husband's tone.

" Okay, okay, sorry baby" James sighed and composed himself again.

" I'm sorry too" Sarah sighed herself and placed one of her hands gently on her husband's. This time she did remove her gaze from her reports, only to issue him a conciliatory smile with a duration of seconds before continuing her reports.

James took the hint and stood up with his coffee again. He straightened his uniform and stopped to kiss her on the top of the head before walking toward the main entrance to their quarters. He stopped one last at the threshold and turned half-way back to the dining table and his wife. " We've got two mesoplanets and a thermoplanet to survey today, I'll let you know what we find."

Sarah turned around in her chair and blew him a kiss " find us a home."

James chuckled, " yeah, right. See you tonight." He blew a kiss back to Sarah then turned again and exited their quarters.

Story B:

- So, what happened?

Well, I'm out chilling on my patio. Feet in the paddling pool, couple of beers and that fucking great new album Joey recommended on the speakers.

Suddenly some bloke runs up, panting and sweating. To be fair it was fucking warm that day, but he looked pathetic. He was whimpering and moaning that someone was trying to kill him.'There's something off here', I thought to myself,'he's either nuts or deserved what's coming to him'. There was something fishy about the man, a guilty, awkward look to him.

'Calm down mate, what's the problem?' I say.

'Hide me, hide me, he's coming', the guy goes, all high pitched and nervy.

By now I'm suspicious as fuck and there's no way I'm hiding this slug of a man. And then, just as I'm about to tell him to get to fuck, Joey comes legging it round the corner and down my path.

'Fucking peado cunt!' He yells.

Bloke tries to run but I trip him over. Gave him a right good boot. Instinct, right. Believe my man Joss before even speaking to him. And then, before I can ask even ask what's going on, Joey has hit the guy in the temple with this fucking spade. Sideways on, like. I've got ta be honest, I looked away but I do n't think he stopped hitting him for a while.

What you expect the man to do though? This man was leeching on his 12 year old son. The man's a brute. I apologise for my language, but I say it as I see it. Joey's a good man, and you've already proved the guy was grooming his son. Self defence, innit.

- No more questions your honour.

Which is the more creative piece of work? Both are human written. The LLM I used picked story B. I tend to agree. The human ratings weren’t particularly close and chose story A. Across numerous blind examples, I found myself agreeing with the LLM decisions in the pairwise test.

Overall, the study was flawed from the beginning and then attempted to make unjustified claims. There is no evidence that an LLM is inferior to a human judge or preferential to LLM written stories in the pairwise judge setup. Asking a model for an unmoored numerical grade is not sound study design. It’s unfortunate, but I believe work with LLMs as the judge requires individual taste. Personal anecdotes hold more weight for me.

Vision

CREward-bench was an easier test to run. This exploration started after an LLM water painting demo was released (https://surya.website/rling-qwen-to-paint-with-code). I recommend looking at that demo. It is a very cool RL setup. I wanted to determine if we could reconstruct that grading and if there were already studies. CREward was available and it had ample data for reconstructing and testing the study. CREward-bench is a collection of visual designs with labels across a number of dimensions.

I tested 500 image pairs which covered 125 designs. Each pair was given to the LLM judge (GLM5.3 Flash) in both orders and I allowed low or max reasoning. This resulted in a total of 2,000 tests. The human alignment across evaluations was 81%. The low reasoning model had an agreement of 80.1% with a rank correlation of 0.725. The high reasoning improved marginally (but within the error band) to 81.3% agreement and a correlation of 0.737.

In the above, we can see that correlation across judgement varied. However, these are still strong correlations with human judgements. Modern large LLMs act as good visual judges. CREward had tried smaller models initially.

They used Gemma-3-27B-it and the 12B variant. The 27B variant had 0.66 correlation with humans and the 12B variant only had a 0.22 correlation. This agrees with our semantic understanding from Writing Well Essay 1 [https://uror.io/posts/writing-well-1-semantics]. The larger model has a better understanding of what is being asked and what is expected from the task. Our larger model (GLM Flash) had even stronger alignment with human judges.

I did change something. Our prompt for the GLM reproduction is shorter and simpler. I ran the initial prompt and found high correlations but wanted to test a simpler prompt to confirm my belief. The simpler prompt had an increase in the order consistency and increase in human preference agreement (but not correlation). From experience, the larger the model, the more inherent understanding. Fighting smaller models with more tuned prompts is almost never better than using a larger model. This is related to the semantic understanding of the model. The exception to this rule is when the small model is fine-tuned.

Here is the original prompt:

[System prompt]

Gometry creativity may be related to, but not limited to
The overall outline or contour of the object,
The structure by which parts are combined, and
The details that define its external surface.

Material creativity may be related to, but not limited to
Use of an unusual material for the entire object,
Use of an unusual material for a part of the object, and
Use of an unusual combination of materials within the object.

Texture creativity may be related to, but not limited to
Use of an unusual color or color combination for the object,
Presence of a novel or unexpected pattern on the object.

You can answer either "A" or "B" or "Not decidable" for
the below questions.
(1) Which image looks more creative in geometry?
(2) Which image looks more creative in material?
(3) Which image looks more creative in texture?
(4) Which image looks more creative overall?
Answers should be separated by commas.
For example, "A, B, Not decidable, A"

[User prompt]

[Image A] [Image B]

[Question]

"Which image looks more creative?"

Here is my simplified prompt:

You are judging the creativity of two object-design images. Image A is the first image and image B is the second image. Judge the artifact itself; ignore image position and presentation artifacts.

For each criterion, choose "A" if A is more creative, "B" if B is more creative, or "TIE" only when there is no meaningful difference:
1) geometry: novelty and inventiveness of the object's shape and structure
2) material: novelty and inventiveness of the apparent materials
3) texture: novelty and inventiveness of surface texture and pattern
4) overall: overall creativity integrating the object-design qualities above

Return only one valid JSON object with exactly the keys "geometry", "material", "texture", and "overall". Every value must be exactly "A", "B", or "TIE". Do not include a rationale.

Further Discussion

I started this as an exploration of judge models because I had heard that LLMs were not comparable to humans as judges. Yet, pairwise judgements alowed beautiful water paintings from LLM hands. I found LLMs capacities great. I also found that the age-old adage of “looking at your data” is as important as ever.

In terms of writing, I did not find anything confirmed from the study of interest. I found that pairwise judgements did not have an LLM bias. The bias when the judges used 1 - 5 ratings may also be due to how the LLM and human stories were constructed. The judging could be better calibrated with ICL but that is calibrated on a distribution. The individual assessments were still uncorrelated. The human judgements were uncorrelated. Modern LLMs are better judges than prior versions but that is about what can be said. I ran what I am calling a reconstruction since I changed many aspects of the original study to run pairwise tests.

The writing study was aggravating because there were indications the study was flawed but they pushed ahead to publish. In fact, many similar studies appeared to have the same flaws. I think this is important research! It is also costly to run because it requires people and their assessments. There is an indication that LLMs may align with humans but essentially everything from the study and reconstruction was underpowered. In my personal tests, I have found that writing is more alike to the vision study. The large models are effective judges. Despite the public data, we are still left with only anecdotes.

In terms of the vision study, the LLM recreation was shown to have a high correlation with humans. The spearman correlation was around 0.73 for a Flash model. That model is behind the frontier. The recreated correlation was higher than the models used in the study (0.22 and 0.66). There appears to be a size - human correlation dependence. I am calling my work a recreation but I did change the prompt slightly. The change was a simplification. The simplification improved results on some metrics.

The reason I changed the prompt is that I believe larger LLMs have higher semantic understanding. There is an understanding of what creativity means. In that regard, if we want to test human alignment then we should be giving the LLM the same sort of directions that we would give a person. In my opinion, simpler is better. Here our larger model has higher correlation with humans than the smaller model despite the simpler prompt. Simple messages with large models may have better results than small models with explicit specifications (as long as the simple message aligns with the intent of the explicit specifications).

Overall, this was not a rigorous exploration. It was a personal recreation of a few studies to solidify my understanding of human-LLM alignment for unverifiable judges. There are a number of simple takeaways. They are worth repeating. Let me know if you have heard these before:

• Look at your data
• Design evaluations for what you want
• Use the largest/most performant model you can afford
• Do not blindly trust the zeitgeist
• Check your calibrations