Generative AI can produce text and images, blog posts, program code, poetry, and artwork. It can even win in competitions (although still controversially).
The software uses complex machine learning models to predict the next word based on previous word sequences, or the next image based on words describing previous images.
LLMs began at Google Brain in 2017, where they were initially used for translation of words while preserving context. Since then, large language and text-to-image models have proliferated at leading tech firms including Google (BERT and LaMDA), Facebook (OPT-175B, BlenderBot), and OpenAI, a nonprofit in which Microsoft is the dominant investor (GPT-3 for text, DALL-E2 for images, and Whisper for speech).
Online communities such as Midjourney (which helped win the art competition), and open-source providers like HuggingFace, have also created generative models.
These models have largely been confined to major tech companies because training them requires massive amounts of data and computing power.
GPT-3, for example, was initially trained on 45 terabytes of data and employs 175 billion parameters or coefficients to make its predictions; a single training run for GPT-3 cost $12 million.
Wu Dao 2.0, a Chinese model, has 1.75 trillion parameters. Most companies don’t have the data center capabilities or cloud computing budgets to train their own models of this type from scratch.
But once a generative model is trained, it can be “fine-tuned” for a particular content domain with much less data. This has led to specialized models of BERT — for biomedical content (BioBERT), legal content (Legal-BERT), and French text (CamemBERT) — and GPT-3 for a wide variety of specific purposes.
NVIDIA’s BioNeMo is a framework for training, building, and deploying large language models at supercomputing scale for generative chemistry, proteomics, and DNA/RNA.
OpenAI has found that as few as 100 specific examples of domain-specific data can substantially improve the accuracy and relevance of GPT-3’s outputs.
Source: Davenport, T. H., & Mittal, N. (2022, November 14). How Generative AI Is Changing Creative Work. Harvard Business Review. https://hbr.org/2022/11/how-generative-ai-is-changing-creative-work