From 117M to 175B Parameters: The Staggering Growth of AI Models

The evolution of language models from GPT-2 to GPT-3 showcases a monumental leap in scale, with model parameters increasing from 117 million to 175 billion. This growth was driven by a philosophy to bypass the finetuning stage, aiming for models that could handle diverse tasks without specific adjustments. Key technical elements like task-agnostic learning, the scale hypothesis, and in-context learning were explored to achieve this. Both models demonstrated that larger models trained on vast datasets could develop new capabilities, enhancing zero-shot and few-shot learning. GPT-2 introduced the WebText dataset, significantly larger than its predecessor’s, while GPT-3 utilized even larger datasets, with a focus on data quality. These models not only improved performance across various NLP tasks but also sparked discussions on model safety, ethics, and the potential for emergent abilities in AI.

Source: towardsdatascience.com