"The researchers asked an advanced model of ChatGPT to analyze the O*Net data and determine which tasks large language models could do. It found that 86 jobs were entirely exposed (meaning every task could be assisted by the tool). The human researchers said 15 jobs were. The job that both the humans and the A.I. agreed was most exposed was mathematician."
A brief look at the data and the research confirms my first impression. The authors admit as much:
"A fundamental limitation of our approach lies in the subjectivity of the labeling. In our study, we employ annotators who are familiar with LLM capabilities. However, this group is not occupationally diverse, potentially leading to biased judgments regarding LLMs’ reliability and effectiveness in performing tasks within unfamiliar occupations. We acknowledge that obtaining high-quality labels for each task in an occupation requires workers engaged in those occupations or, at a minimum, possessing in-depth knowledge of the diverse tasks within those occupations. This represents an important area for future work in validating these results."
The O*NET database lists 11 "tasks" for the occupation label "mathematician." Most of these are plausible, and a few are familiar to academic mathematicians, for example
"Maintain knowledge in the field by reading professional journals, talking with other mathematicians, and attending professional conferences."
I'll explore how the "annotators" decided that LLMs can reduce the time required for that sort of thing by 50%, among other things, in a future post.
I'd like a closer look at the data and the research. It appears "the researchers" were at OpenAI and probably have no idea what a mathematician is or does. On the other hand, this paragraph is intriguing:
"Aquent Talent, a staffing firm, is using a business version of Bard. Usually, humans read through workers’ résumés and portfolios to find a match for a job opening; the tool can do it much more efficiently. Its work still requires a human audit, though, especially in hiring, because human biases are built in, said Rohshann Pilla, president of Aquent Talent."
Can we use this in our hiring committee? Or for graduate admissions?
"The "intelligence" part of AI, for example, can only be a metaphor, a particularly tenacious one, since there is no agreed definition that embodies everything that may go by the name of "intelligence." The consequence is that discussion tends to be dominated by the definitions promoted most aggressively" – clap clap clap
Might it not be "an unreasonable effectiveness of Mathematics in National Defense" that allows for the illusion of lack of urgency to prevail among the "purest" of mathematicians; "vibrations'" amongst the "colleagues across the campus ... in the humanities and social sciences" (outside of our "departmental silo") notwithstanding?
There is a deep underground connection - maybe not even underground - between these ideas and what Roger Penrose wrote in _The Emperor's New Mind_. It's so strange how he is celebrated as a thinker, while his core point about AI in that book has been flagrantly ignored, at least by the people making decisions.
"The researchers asked an advanced model of ChatGPT to analyze the O*Net data and determine which tasks large language models could do. It found that 86 jobs were entirely exposed (meaning every task could be assisted by the tool). The human researchers said 15 jobs were. The job that both the humans and the A.I. agreed was most exposed was mathematician."
https://www.nytimes.com/2023/08/24/upshot/artificial-intelligence-jobs.html
A brief look at the data and the research confirms my first impression. The authors admit as much:
"A fundamental limitation of our approach lies in the subjectivity of the labeling. In our study, we employ annotators who are familiar with LLM capabilities. However, this group is not occupationally diverse, potentially leading to biased judgments regarding LLMs’ reliability and effectiveness in performing tasks within unfamiliar occupations. We acknowledge that obtaining high-quality labels for each task in an occupation requires workers engaged in those occupations or, at a minimum, possessing in-depth knowledge of the diverse tasks within those occupations. This represents an important area for future work in validating these results."
The O*NET database lists 11 "tasks" for the occupation label "mathematician." Most of these are plausible, and a few are familiar to academic mathematicians, for example
"Maintain knowledge in the field by reading professional journals, talking with other mathematicians, and attending professional conferences."
I'll explore how the "annotators" decided that LLMs can reduce the time required for that sort of thing by 50%, among other things, in a future post.
I'd like a closer look at the data and the research. It appears "the researchers" were at OpenAI and probably have no idea what a mathematician is or does. On the other hand, this paragraph is intriguing:
"Aquent Talent, a staffing firm, is using a business version of Bard. Usually, humans read through workers’ résumés and portfolios to find a match for a job opening; the tool can do it much more efficiently. Its work still requires a human audit, though, especially in hiring, because human biases are built in, said Rohshann Pilla, president of Aquent Talent."
Can we use this in our hiring committee? Or for graduate admissions?
"The "intelligence" part of AI, for example, can only be a metaphor, a particularly tenacious one, since there is no agreed definition that embodies everything that may go by the name of "intelligence." The consequence is that discussion tends to be dominated by the definitions promoted most aggressively" – clap clap clap
Might it not be "an unreasonable effectiveness of Mathematics in National Defense" that allows for the illusion of lack of urgency to prevail among the "purest" of mathematicians; "vibrations'" amongst the "colleagues across the campus ... in the humanities and social sciences" (outside of our "departmental silo") notwithstanding?
There is a deep underground connection - maybe not even underground - between these ideas and what Roger Penrose wrote in _The Emperor's New Mind_. It's so strange how he is celebrated as a thinker, while his core point about AI in that book has been flagrantly ignored, at least by the people making decisions.