ORPO Loss (Odds-Ratio Preference Optimisation)

Background

ORPO (Odds-Ratio Preference Optimisation) is a modern post-DPO alignment loss that combines supervised fine-tuning and preference learning into a single forward pass without a reference model. The loss adds an odds-ratio penalty to the standard SFT log-likelihood, pushing chosen completions up and rejected ones down.

Problem statement

Implement orpo_loss(chosen_logprobs, rejected_logprobs, sft_loss, beta=0.1):

$$\mathcal{L}_{\text{ORPO}} \;=\; \mathcal{L}_{\text{SFT}} \;-\; \beta\, \log \sigma\!\Bigl(\log\frac{\text{odds}_{\text{chosen}}}{\text{odds}_{\text{rejected}}}\Bigr)$$

with $\text{odds}(x) = \exp(x) / (1 - \exp(x))$ for log-probability $x < 0$. Equivalently, $\log \text{odds}(x) = x - \log(1 - \exp(x))$.

Input

• chosen_logprobs - 1-D array of per-example log-probs of chosen completions.
• rejected_logprobs - 1-D array of same length, rejected.
• sft_loss - scalar float, the SFT cross-entropy.
• beta - float >= 0, mixing weight (default $0.1$).

Output

• float, the full ORPO loss.

Examples

Input: chosen=[-1.0], rejected=[-3.0], sft_loss=2.0, beta=0.1
Output: ~ sft_loss - beta * log sigmoid(log-odds-ratio)

Constraints

• Use numerically stable $\log(1 - \exp(x))$ via log1p(-exp(x)). Inputs must satisfy $x < 0$.
• Lengths of chosen / rejected must match.

Notes

• ORPO vs DPO. DPO needs a reference model; ORPO does not. The cost is paid through the odds-ratio formulation, which behaves slightly differently in tail probabilities.