WebGPT: Improving the factual accuracy of language models through web browsing

We’ve fine-tuned GPT‑3 to more accurately answer open-ended questions using a text-based web browser.

We’ve fine-tuned GPT‑3 to more accurately answer open-ended questions using a text-based web browser. Our prototype copies how humans research answers to questions online—it submits search queries, follows links, and scrolls up and down web pages. It is trained to cite its sources, which makes it easier to give feedback to improve factual accuracy. We’re excited about developing more truthful AI, but challenges remain, such as coping with unfamiliar types of questions.

Language models like GPT‑3 are useful for many different tasks, but have a tendency to "hallucinate" information when performing tasks requiring obscure real-world knowledge. To address this, we taught GPT‑3 to use a text-based web-browser. The model is provided with an open-ended question and a summary of the browser state, and must issue commands such as "Search ...", "Find in page: ..." or "Quote: …". In this way, the model collects passages from web pages, and then uses these to compose an answer.

The model is fine-tuned from GPT‑3 using the same general methods we’ve used previously. We begin by training the model to copy human demonstrations, which gives it the ability to use the text-based browser to answer questions. Then we improve the helpfulness and accuracy of the model’s answers, by training a reward model to predict human preferences, and optimizing against it using either reinforcement learning or rejection sampling.

WebGPT Answer Viewer

The model can find helpful answers to questions on the web.

How do neural networks work?

In its most basic form, a neural network has two layers: an input layer and an output layer. The output layer is the component of the neural network that makes predictions. In a feedforward network, information flows through the network in the following way: patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these in turn arrive at the output units. The network learns by a feedback process called backpropagation, which involves comparing the output a network produces with the output it was meant to produce, and using the difference between them to modify the weights of the connections between the units in the network. Over time, backpropagation causes the network to learn, reducing the difference between actual and intended output to the point where the two exactly coincide.

ELI5 results

Our system is trained to answer questions from ELI5, a dataset of open-ended questions scraped from the “Explain Like I’m Five” subreddit. We trained three different models, corresponding to three different inference-time compute budgets. Our best-performing model produces answers that are preferred 56% of the time to answers written by our human demonstrators, with a similar level of factual accuracy.

Conclusion

Human feedback and tools such as web browsers offer a promising path towards robustly truthful, general-purpose AI systems. Our current system struggles with challenging or unfamiliar circumstances, but still represents significant progress in this direction.