Every Sunday, the NPR host will be Shortz, a cross -word puzzle expert on New York Times. It was written so that it could be solved without ~ Many foresight, thunderstorms usually challenge skilled participants.
That’s why some experts think it’s a promising way to test the limitations of AI’s problem -solving skills.
In recent studies, researchers from Wellesley College, Oberlin College, Oberlin College, Oustin, NortheasterN University I made a chi mark. The team says their tests reveal amazing insights, such as an inferred model such as O1, O1, and sometimes “give up” and provide answers they do not know.
“We wanted to develop a benchmark as a problem that human beings can understand with general knowledge.
The AI industry is currently in some benchmarking queries. Most tests are usually used to evaluate the AI model probe. Meanwhile, even a relatively recent benchmark is approaching a saturated point with many benchmarks.
The advantage of public radio quiz games such as Sunday puzzles is that it does not test difficult knowledge, and the problem is that the challenge is expressed to prevent the “Rote Memory” to solve the model.
GUHA said, “It is really difficult to make meaningful developments on the problem until it solves the problem. “This requires a combination of insight and removal process.”
Of course, the benchmark is not perfect. Sunday puzzles are in the center of America and only English. And because the quiz is publicly provided, GUHA says it has not seen evidence of this, but the trained model can be “deceived” in a sense.
“New questions are published every week, and the latest questions can be seen not really.” “We want to keep the benchmark fresh and trace how model performance changes over time.”
In the benchmark of a researcher, about 600 days Sunday, the reason for reasoning models such as R1 of O1 and DeepSeek is much more than the rest. The reasoning model helps to avoid the traps that usually travel to the AI model by thoroughly checking the model that completely checks themselves before the results are announced. The trade off is that it takes a little more time for the reasoning model to reach the solution. In general, it gets a few seconds to a few more minutes.
At least one model, DEEPSEEK’s R1 offers a solution that some of the puzzle questions are wrong. R1 says “I give up” with an oral, and randomly chosen a wrong answer -this is followed by actions that this man can be clearly involved.
This model makes another bizarre choice, such as withdrawing immediately, trying to make things worry, and failing again. They are also trapped in “thinking” forever, providing a meaningless explanation for answers, or reaching the right answer, but considers an alternative answer for no clear reason.
GUHA said, “There is literally ‘frustration’ about difficult problems. “It was fun to see how the model imitates human words. The frustration of reasoning still remains how it can affect the quality of the model results. ”
The best model on the benchmark is O1, which is 59%, and the recently released O3-Mini is set to high “reasoning efforts” (47%). (R1 recorded 35%.) In the next stage, researchers plan to expand the test to additional reasoning models, which hopes to help identify the areas where these models can be improved.
GUHA said, “We do not need a doctoral degree to make good reasoning, so we must be able to design a reasonable benchmark that does not require PHD level knowledge. “Through a benchmark with a wider access, more researchers can understand and analyze the results, which can lead to better solutions in the future. In addition, since the cutting -edge models are more and more deployed in the settings that affect everyone, I think everyone should be able to intuitive what these models can do and what they do. ”
