OpenEdit Leaderboard

This leaderboard presents a unified evaluation of mainstream model editing techniques across various LLMs and datasets.

To ensure consistency and comparability across methods, we adopt a unified experimental setup. For each dataset, we randomly sample 3000 instances and perform lifelong model editing. For each editing method, we explore various batch size settings and select a suitable configuration based on empirical performance.

A comprehensive evaluation of edited model is conducted after the entire editing process is completed. The evaluation setup is standardized as follows:

1. Input: using only question without additional context to align with existing literature.
2. Generation Strategy: employing autoregressive generation rather than the commonly adopted teacher forcing.
3. Output Truncation: terminating generation upon encountering predefined stop tokens (e.g., "<|endoftext|>"), rather than truncating at ground truth length, as is common in prior work.
4. Metric: adopting exact match (EM) as a strict criterion for judging whether the edited knowledge is correctly generated.

Large Language Models: LLaMA-3-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-v0.1
Editing Methods: MEMIT, WISE, AlphaEdit, RLEdit
Datasets: ZsRE, CounterFact
Metrics: Reliability, Generalization, Capability (average performance of edited models on GSM8K, MMLU, Natural Questions, WMT, and SST2)