#404Self-Consistency Majority VoteEasyLLMsEvaluation MetricsAsked atOpenAI · Anthropic · Google
Self-Consistency Majority Vote
Background
Self-consistency is the dominant inference-time scaling trick for reasoning LLMs: sample independent completions, then take the answer that appears most often. Empirically it pushes pass-rate up several points on math and code benchmarks without retraining.
Problem statement
Implement majority_vote(answers, tiebreak=None) returning the mode of answers. Ties broken by the value of tiebreak (a function on candidate answer -> sort key); if None, ties broken by first-occurrence order.
Input
answers- sequence of hashable answers.tiebreak- optional function(answer) -> sortable key. None means first-occurrence.
Output
- The chosen answer (whatever type).
Examples
Input: answers=["A", "B", "A", "C", "A", "B"]
Output: "A"
Constraints
- Empty input raises
ValueError. - The output type matches the input element type.
Notes
- vs Best-of-N. Best-of-N picks by an explicit reward / verifier; self-consistency picks by frequency, no verifier needed.
Python
Loading...
▶ Run executes the 3 visible sample tests below in your browser. Submit runs the full suite — including hidden tests — on the server for an official verdict.
- •Example: a clear majority wins
- •Example: numeric answers vote
- •Reference: tiebreak function selects by key