I just learned that Epoch AI, the institute that created FrontierMath, had released a statement "clarifying" its relation with OpenAI, two days before I published this post.
For some reason I had not been able to find it all last week. It's the best published account I've seen so far. (I can't vouch for its accuracy, because I have no independent source of information, but it's consistent with the other reports I've read.)
Profile Badge for the Achievement Top 1% Poster Top 1% Poster
To those mathematicians who are upset about contributing to the dataset without being informed about the exclusive content-sharing deal with OpenAI: this is the kind of thing you should expect when dealing with a Silicon Valley company! If you don't like it, then don't give any of these companies your help!"
who complain that the above post is "unreadable." I was informed about this story a week ago but my "day job" at the beginning of the semester occupies more than 12 hours/day, leaving no time to edit my own report on the scandal. It seemed to me important to share the information with mathematicians, given that the FrontierMath derives its respectability from the association of several well-known mathematicians.
Several posts on the same reddit thread provide an excellent summary of the story:
"hobo_stew
•
8h ago
Mathematicians contribute AI benchmark problems to FrontierMath. FrontierMath doesn‘t disclose (to the contributing mathematicians) that it is funded by OpenAI and that OpenAI has access to the problems.
Individual-Row-8262
•
6h ago
… and OpenAI published results that claim a major breakthrough in performance on said benchmark. It’s not too big of a scandal that they had the potential to cheat, except they started bragging about how good they did on the test they paid for and had access to all the answers, and made all the contributors sign an NDA so they can’t talk about it.
Qyeuebs
•
1h ago
Profile Badge for the Achievement Top 1% Poster Top 1% Poster
Not just access, but exclusive access. Quoting from the new announcement from the FrontierMath company:
OpenAI commissioned Epoch AI to produce 300 advanced math problems for AI evaluation that form the core of the FrontierMath benchmark. As is typical of commissioned work, OpenAI retains ownership of these questions and has access to the problems and solutions, with the exception of a holdout set discussed below.
Epoch AI is free to conduct and publish evaluations of any models using the FrontierMath problem set commissioned by OpenAI. However, we cannot share the questions and answers with other parties without written permission from OpenAI.
So it seems that the problem contributors were unwittingly creating a high-quality dataset for OpenAI exclusively to do whatever they like with, while they were all under the impression that they were contributing to a third party only interested in benchmarking various AI models.
hobo_stew
•
53m ago
The best move for everbody that contributed would be to publish their problems on the Internet and thereby spoil the benchmark."
I just learned that Epoch AI, the institute that created FrontierMath, had released a statement "clarifying" its relation with OpenAI, two days before I published this post.
https://epoch.ai/blog/openai-and-frontiermath
After reading it I am not inspired to change anything I wrote.
I also just found an article by Amanda Zhang on the scandal that was published on January 19.
https://www.ctol.digital/news/openai-hidden-involvement-frontiermath-ai-transparency/
For some reason I had not been able to find it all last week. It's the best published account I've seen so far. (I can't vouch for its accuracy, because I have no independent source of information, but it's consistent with the other reports I've read.)
Seen on reddit, to be saved for future reference:
"Qyeuebs
•
1h ago
• Edited 1h ago
Profile Badge for the Achievement Top 1% Poster Top 1% Poster
To those mathematicians who are upset about contributing to the dataset without being informed about the exclusive content-sharing deal with OpenAI: this is the kind of thing you should expect when dealing with a Silicon Valley company! If you don't like it, then don't give any of these companies your help!"
Apologies to those on this same reddit thread
https://www.reddit.com/r/math/comments/1iadcqw/the_frontiermath_scandal/
who complain that the above post is "unreadable." I was informed about this story a week ago but my "day job" at the beginning of the semester occupies more than 12 hours/day, leaving no time to edit my own report on the scandal. It seemed to me important to share the information with mathematicians, given that the FrontierMath derives its respectability from the association of several well-known mathematicians.
Several posts on the same reddit thread provide an excellent summary of the story:
"hobo_stew
•
8h ago
Mathematicians contribute AI benchmark problems to FrontierMath. FrontierMath doesn‘t disclose (to the contributing mathematicians) that it is funded by OpenAI and that OpenAI has access to the problems.
Individual-Row-8262
•
6h ago
… and OpenAI published results that claim a major breakthrough in performance on said benchmark. It’s not too big of a scandal that they had the potential to cheat, except they started bragging about how good they did on the test they paid for and had access to all the answers, and made all the contributors sign an NDA so they can’t talk about it.
Qyeuebs
•
1h ago
Profile Badge for the Achievement Top 1% Poster Top 1% Poster
Not just access, but exclusive access. Quoting from the new announcement from the FrontierMath company:
OpenAI commissioned Epoch AI to produce 300 advanced math problems for AI evaluation that form the core of the FrontierMath benchmark. As is typical of commissioned work, OpenAI retains ownership of these questions and has access to the problems and solutions, with the exception of a holdout set discussed below.
Epoch AI is free to conduct and publish evaluations of any models using the FrontierMath problem set commissioned by OpenAI. However, we cannot share the questions and answers with other parties without written permission from OpenAI.
So it seems that the problem contributors were unwittingly creating a high-quality dataset for OpenAI exclusively to do whatever they like with, while they were all under the impression that they were contributing to a third party only interested in benchmarking various AI models.
hobo_stew
•
53m ago
The best move for everbody that contributed would be to publish their problems on the Internet and thereby spoil the benchmark."