oracle in large language model study
Table of Contents
oracle in large language model study often refers to an ideal source of information, used to benchmark performance of the model.
1. example
1.1. oracle
- human expert
- an annotated dataset
- ground truth information (output of some system that does the same thing and is known to be very good)
1.2. using oracle
- let human expert to rate how large language model is doing
- compare the response of human expert’s with large language model’s
- compare annotated dataset’s label with that is predicted by large language model based on only the features in the dataset
- when making LLM to do math, compare output of LLM and the expression’s value with calculator(or WolframAlpha, which is known for its capability to parse and compute math).