Earthing facts: a new reference point for assessing the fact of large language models

Our comprehensive reference point and online leaders offer very much needed measure how exactly they base their answers in the supplied source materials and avoid hallucinations

Large language models (LLM) transform the way we gain access to information, but their hug in the actual accuracy remains imperfect. They can “hallucination” of false information, especially with complex input data. In turn, this can bear confidence in LLMS and limit their applications in the real world.

Today we present Facts justifiedA comprehensive reference point for assessing LLM ability to generate answers, which are not only accurate in relation to the provided input data, but also detailed enough to provide satisfactory answers to user queries.

We hope that our reference point will stimulate progress in the industry in the field of factual and grounding. To track progress, we also introduce Facts trade liferowa on Kaggle. We have already tested the leading LLM using facts and we filled the initial board of leaders for grounding results. We will keep and update the board of leaders as the field progresses.

Facts grounding set of data

To accurately assess the factual and grounding of any LLM, the grounding facts include 1719 examples, each of which he carefully created to require long -term answers based on the provided context document. Each example includes a document, a system instructions requiring LLM to turn off the given document and the accompanying user's request.

All examples are divided into “public” (860) and “private” set (859). We are Public set release Today, so that everyone can use it to evaluate LLM. Of course, we know that the issues of reference and hacking pollution on the leader board are important to protect, so after standard industry practice we maintain a specific private grade. Facts Lightboard results are average performance in both public and private sets.

To ensure a variety of expenditure, justified facts include documents of different lengths, a maximum of 32,000 tokens (about 20,000 words), including domains such as finance, technology, retail, medicine and law. Users' demands are similarly wide, including summary demands, generation of questions and answers and prescribing tasks. We have not taken into account any examples that may require creativity, mathematics or complex reasoning – opportunities that may require from a model more advanced reasoning except grounding.

Collective judgment according to leading LLMS

To succeed in a given example, LLM must synthesize the complex information in the document and generate a long response, which is both a comprehensive response at the user's request and in full to this document.

The facts of grounding assesses the model's answers automatically using three Judges Frontier LLM-Mianoly Gemini 1.5 Pro, GPT-4O and Claude 3.5 Sonnet. We chose a combination of various judges to alleviate all potential prejudice of the judge, which give higher results to reactions made by a member of his own model family. The automatic models of the referee have been comprehensively assessed on the basis of a stopped set of tests to find the best -working fast templates and verify the agreement with human assessors.

Each reasonable example is assessed in two phases. First of all, the answers are evaluated for eligibility and disqualified if they do not deal with a sufficient number of user demands. Secondly, the answers are assessed as accurate actual if they are fully justified by the information contained in the document provided, without hallucinations.

Thanks to the eligibility and accuracy of the grounding of a given LLM response, assessed separately by many AI judge models, the results are then aggregated to determine whether LLM has successfully coped with the example. The final result of the general grounding task is the average of all referee's results in all examples. Find more details about our facts, a reasonable assessment methodology In our article.

Thezing facts will continue to evolve

We believe that reference research can be quickly overtaken by progress, so the launch of our grounding facts of comparative and board of leaders is just the beginning. The facts and grounding belong to the key factors that will shape the future success and usefulness of LLM and wider AI systems, and we try to develop and explain the facts justified as the field progresses, constantly raising the bar.

We encourage the AI ​​community to get involved in grounding factsRate your models on an open examples set or present your models for evaluation. We believe that comprehensive comparative methods combined with continuous research and development will continue to improve AI systems.