August, 2023
In March, OpenAI released the long-awaited paper covering its new large language model (LLM), GPT-4. The paper accompanied the public release of GPT-4 itself. The model was widely hailed as a big step up from its predecessor, GPT-3, the LLM that initially powered the wildly successful ChatGPT application. Despite widespread praise for the GPT-4 model – a New York Times author said the model provoked an “existential crisis” with its uncannily intelligent responses – the artificial intelligence research community was less complimentary of the accompanying paper.
The GPT-4 “paper” had a number of issues that suggested it may have been more of a marketing ploy than a real academic paper. Instead of giving the work a title that summarized the contribution of the new model to the standing of LLM research, as they did with previous GPT papers, OpenAI referred to the writing only as the “GPT-4 technical report.” Rather than list specific individuals who worked on GPT-4, again a constant with papers for previous models, the paper listed the company as the sole author. OpenAI also published the research on their blog rather than in a peer-reviewed journal. Standard machine learning papers are at most a few dozen pages. The GPT-4 technical report was 100 pages.
The content of the technical report was even more concerning for other researchers in the field. OpenAI included almost none of the important new details about the model, the data they used for its training, or the hardware they employed–all elements usually included in papers to facilitate replication of the results. A researcher from language model company Hugging Face claimed that after reading the 100-page report, she has “more questions than answers.” OpenAI themselves even admit to obscuring the important details in a passage from the report itself: “Given the competitive landscape…this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method.”
To those unfamiliar with the machine learning research space, OpenAI’s secrecy may be unsurprising. Why would a company with a profit motive reveal the details of its primary money-making software? Yet, there is a delicate balance between the cold realities of business and the mores of the optimistic academic research community where OpenAI’s PhD-toting employees are trained, and the optimistic side usually wins out. The incentives for communication and profit are not so misaligned in other research fields like the humanities, where drumming up interest in one’s work is likely also the best way to make more money. This is not necessarily the case for computer science, where tech companies can sometimes transform the best results into cash, but only if they are among the first to put them into practice. Computer scientists have nonetheless long followed the research norm of publishing what they find for other researchers to see and build on, implicitly agreeing that the communal pursuit of knowledge is more important than individual monetary gains. There is also a long history of “techies” offering valuable intellectual property for free on the internet with roots in the free software movement of the 1980s and the modern “open source” community. There are always exceptions, but generally research findings from one company are shared in public journals or conferences. With the GPT-4 technical report, OpenAI is breaking from the traditional model of transparency.
There is a glaring irony here for those most familiar with the history of language model research and OpenAI’s models. The fundamental building block for all versions of GPT is the Transformer model (hence the name, Generalized Pretrained Transformers). The Transformer emerged from Google, the undisputed authority in AI research. Google introduced the model in the seminal 2017 paper “Attention is All You Need,” which quickly became the most cited paper in the field and, unlike the GPT-4 technical report, contained all the details necessary to understand and replicate the Transformer. While OpenAI has developed expertise in fine-tuning and data collection to maximize the performance of Google’s Transformer, the bones for GPT are in that 2017 publication. OpenAI used Google’s generosity to become its biggest new competitor, but they are not playing by Google’s rules, instead protecting their most important results.
While OpenAI undoubtedly caught Google (and the rest of the AI research community) by surprise with their secrecy and leveraged it to gain a sizable lead in the generative AI market, perhaps they showed their hand too soon. In May, Google released its new PALM-2 language model as a response to GPT-4. Rather than a concise and detailed paper like “Attention is All You Need,” Google released a long, vague “technical report” eerily reminiscent of the one OpenAI released for GPT-4. As Jesse Dodge, a researcher at University of Washington’s Allen Institute for AI, put it: “Now that LLMs are products “(not just research), we are at a turning point: for-profit companies will become less and less transparent *specifically* about the components that are most important.” Essentially, the PALM-2 report is a declaration of war on OpenAI and a commitment to the new, private style of research.
Academics like Dodge are likely to become collateral damage in the clash between massive tech corporations, since they lack the resources to compete with companies like OpenAI and Google in building state-of-the-art LLMs from scratch and will suffer most from those companies not publishing details about their work. A memo authored by a Google software engineer and leaked in May offered a glimmer of hope for academics and startups alike, arguing that open source versions of models would “pass the next milestone” in the “arms race” for AI dominance. Yet, more senior AI experts widely criticized the essay as misinformed and overestimating the value of temporary gains for open source predicated on leaked information about Meta’s LLaMa language model. Sustained progress in AI requires quantities of expensive hardware currently affordable only to a handful of the richest tech giants. As researcher Rowan Zellers explained about his decision to pursue a career in industry over one in academia: “The reality is that building systems is really hard…I think the incentive structures in academia aren't very suited for this kind of costly, risky systems-building research.” In the warped world of modern AI research, a popular paper by Salesforce, the fifty-third largest company in the world, is considered a win for the little guys.
With academia and smaller companies at least temporarily stymied by the colossal budgets of industry research labs, Google and OpenAI are on a collision course for LLM superiority. The question remains: will OpenAI’s bet on secrecy give them enough of a head start to win the LLM race against Google? OpenAI has managed to poach some of Google’s researchers and formed a strategic alliance with Microsoft. Google, however, still holds a stronger research tradition, brand name, and bevy of other products that generate revenue and collect data for its models. If Google researchers develop the next Transformer-level breakthrough, they will likely not release it to the public for competitors to wield against them. OpenAI’s bet, therefore, precariously rests on the assumption that the Transformer is Google’s last transformational contribution to language model research. To come for the AI Goliath, OpenAI changed the rules of war. It remains to be seen if they killed a goose that lays golden eggs.