∴ Syllogism Lab

Can you out-reason an LLM?

Eight short arguments. For each one, decide whether the conclusion follows from the two premises. Then see how you did against the five models from my AAAI 2026 paper.

Judge the logic, not the facts. A false conclusion can follow, and a true one can fail to.

Argument 1 of 8 keys: Y / N

Syllogisms in 30 seconds

Every sentence has one of four shapes. S, M and P are the three terms: the conclusion links S to P, and the middle term M links them in the premises.

Type	Shape	Name
A	All S are P	universal affirmative
E	No S are P	universal negative
I	Some S are P	particular affirmative
O	Some S are not P	particular negative

A mood such as AAA gives the types of the two premises and the conclusion. A figure says where M sits. In the paper, models did best when M was the grammatical subject:

Figure	Premises	M as subject	Mean accuracy, 5 models
3	M–P, M–S	2×	83.0%
4	P–M, M–S	1×	80.6%
1	M–P, S–M	1×	72.7%
2	P–M, S–M	0×	70.2%

The eight arguments here are my own examples in classical forms; I checked each answer by brute force over every way three non-empty terms can overlap. Model numbers come from the paper: 11,000 WordNet-based items, zero-shot, no chain-of-thought.