# Pre-trained models

## What is a pre-trained model?

It is a saved network that was previously trained on a large dataset, typically on a large-scale task.  It is possible to use the pre-trained model as it is or use transfer learning to customize it.

## Why  pre-trained model?

Training big models (with huge amounts of parameters) from scratch is expensive.

* A lot of computing power is needed
* Involves a lot of training time
* Large corpora&#x20;

Let's take a look at the parameters of some of the most powerful  language models...

### Deep learning neural network parameters until June 2021&#x20;



<figure><img src="https://heits.digital/media/article-gpt3/img6.png" alt=""><figcaption><p> <a href="https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/">Microsoft Turing NLG blogspot</a></p></figcaption></figure>



### **Neural network parameters until  december of 2022**



<figure><img src="https://4092288392-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FY8AeUFC5KYkaUv3lbFhX%2Fuploads%2F9Fm1zy7K5qhS0dcA4CXK%2F2022-Alan-D-Thompson-AI-Bubbles-Rev-6%20modelos%20ia.png?alt=media&#x26;token=bf33f6dd-105f-473e-b1a8-6235c6cfee20" alt=""><figcaption><p><a href="https://lifearchitect.ai/models/">https://lifearchitect.ai/models/</a></p></figcaption></figure>

#### Some of the biggest model language models we have nowadays&#x20;

<table data-header-hidden><thead><tr><th width="200"></th><th></th><th></th><th></th><th></th><th width="117"></th><th></th><th></th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td>Model</td><td>Lab</td><td>"Selected playgrounds"</td><td>"Parameters (B)"</td><td>"Tokens trained (B)"</td><td>"Ratio T:P (Chinchilla scaling)"</td><td>Training dataset</td><td>"Announced "</td><td>Public?</td><td>Released</td><td>"Paper/ Repo"</td><td>Notes</td></tr><tr><td>GPT-4</td><td>OpenAI</td><td></td><td>TBA</td><td></td><td></td><td>🆆 📚 ⬆ 🕸 🌋</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>-</td><td>Inflection</td><td></td><td>TBA</td><td></td><td></td><td>🕸</td><td></td><td></td><td></td><td><a href="https://www-cnbc-com.cdn.ampproject.org/c/s/www.cnbc.com/amp/2022/03/08/reid-hoffman-has-set-up-a-new-ai-company-with-deepminds-co-founder.html">🔗</a></td><td>Devs from DeepMind</td></tr><tr><td>LaMDA 2</td><td>Google AI</td><td><a href="https://youtu.be/l9FJm--ClvY">YouTube</a> (video only)</td><td><a href="https://vux.world/thoughts-on-googles-lamda-2/">178?</a></td><td></td><td></td><td>⬆ 🕸 👥</td><td>May/2022</td><td>🟡</td><td>TBA</td><td></td><td>Chatbot with tiny walled garden demo TBA</td></tr><tr><td>Fairseq</td><td>Meta AI</td><td><a href="https://playground.helloforefront.com/">Forefront</a>, <a href="https://textsynth.com/playground.html">TS</a>, <a href="https://goose.ai/playground">Goose</a></td><td>13 &#x26; 1100</td><td></td><td></td><td>🆆 📚 ⬆ 🕸 🕸 🕸</td><td>Dec/2021</td><td>🟢</td><td>Dec/2021</td><td><a href="https://arxiv.org/abs/2112.10684">🔗</a></td><td></td></tr><tr><td>GLaM</td><td>Google Inc</td><td></td><td>1200</td><td></td><td></td><td>🆆 📚⬆ 🕸 👥</td><td>Dec/2021</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2112.06905">🔗</a></td><td></td></tr><tr><td>PaLM</td><td>Google Research</td><td></td><td>540</td><td>780</td><td>2:1</td><td>🆆 📚⬆ 🕸 👥</td><td>Apr/2022</td><td>🔴</td><td>N/A</td><td><a href="https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf">🔗</a></td><td></td></tr><tr><td>MT-NLG</td><td>Microsoft/NVIDIA</td><td></td><td>530</td><td>270</td><td>1:1</td><td>🆆 📚 ⬆ 🌋 🕸 🕸</td><td>Oct/2021</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2201.11990">🔗</a></td><td></td></tr><tr><td>BERT-480</td><td>Google Research</td><td></td><td>480</td><td></td><td></td><td>🆆 📚 🕸</td><td>Nov/2021</td><td>🔴</td><td>N/A</td><td><a href="https://cloud.google.com/blog/topics/tpus/google-showcases-cloud-tpu-v4-pods-for-large-model-training">🔗</a></td><td>Submission to benchmarks. Original dataset was BookCorpus + Wikipedia: <a href="https://arxiv.org/pdf/1810.04805.pdf">https://arxiv.org/pdf/1810.04805.pdf</a></td></tr><tr><td>Gopher</td><td>DeepMind</td><td></td><td>280</td><td>300</td><td>2:1</td><td>🆆 📚 ⬆ 🕸 🌋</td><td>Dec/2021</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2112.11446">🔗</a></td><td></td></tr><tr><td>Luminous</td><td>Aleph Alpha</td><td><a href="https://app.aleph-alpha.com/playground/completion">AA playground</a></td><td><a href="https://www-heise-de.translate.goog/news/Machine-Learning-Aleph-Alpha-feilt-mit-Oracle-und-Nvidia-an-transformativer-KI-6269269.html?_x_tr_sl=de&#x26;_x_tr_tl=en&#x26;_x_tr_hl=en&#x26;_x_tr_pto=sc">200</a></td><td></td><td></td><td>🕸</td><td>Nov/2021</td><td>🟢</td><td>Apr/2022</td><td><a href="https://www.aleph-alpha.de/pricing">🔗</a></td><td>Devs from EleutherAI</td></tr><tr><td>Jurassic-1</td><td>AI21</td><td><a href="https://studio.ai21.com/">Studio</a></td><td>178</td><td>300</td><td>2:1</td><td>🆆 📚 ⬆ 🕸</td><td>Aug/2021</td><td>🟢</td><td>Aug/2021</td><td><a href="https://www.ai21.com/blog/announcing-ai21-studio-and-jurassic-1">🔗</a></td><td>Emulated GPT-3 dataset</td></tr><tr><td>BLOOMZ</td><td>BigScience</td><td><a href="https://github.com/bigscience-workshop/xmtf">https://github.com/bigscience-workshop/xmtf</a></td><td>176</td><td>366</td><td>3:1</td><td>⬆ 🕸</td><td>Nov/2022</td><td>🟢</td><td>Nov/2022</td><td><a href="https://arxiv.org/abs/2211.01786">🔗</a></td><td>fine-tuned</td></tr><tr><td>OPT-IML</td><td>Meta AI</td><td><a href="https://github.com/facebookresearch/metaseq/tree/main/projects/OPT-IML">https://github.com/facebookresearch/metaseq/tree/main/projects/OPT-IML</a></td><td>175</td><td>300</td><td>2:1</td><td>🆆 📚 ⬆ 🕸</td><td>Dec/2022</td><td>🟢</td><td>Dec/2022</td><td><a href="https://github.com/facebookresearch/metaseq/blob/main/projects/OPT-IML/optimal_paper_v1.pdf">🔗</a></td><td>Instruct</td></tr><tr><td>ChatGPT</td><td>OpenAI</td><td><a href="https://chat.openai.com/">https://chat.openai.com/</a></td><td>175</td><td>300</td><td>2:1</td><td>🆆 📚 ⬆ 🕸</td><td>Nov/2022</td><td>🟢</td><td>Nov/2022</td><td><a href="https://arxiv.org/abs/2005.14165">🔗</a></td><td>Instruct with strict policies ("extremely limited")</td></tr><tr><td>BlenderBot 3</td><td>Meta AI</td><td><a href="http://blenderbot.ai/">blenderbot.ai</a> (US only)</td><td>175</td><td></td><td></td><td>🆆 📚 ⬆ 🕸</td><td>Aug/2022</td><td>🟢</td><td>Aug/2022</td><td><a href="https://github.com/facebookresearch/ParlAI/blob/main/projects/bb3/BB3_main_tech_report.pdf">🔗</a></td><td></td></tr><tr><td>GPT-3</td><td>OpenAI</td><td><a href="https://beta.openai.com/playground">Playground</a>, <a href="https://quickchat.ai/emerson">Emerson</a></td><td>175</td><td>300</td><td>2:1</td><td>🆆 📚 ⬆ 🕸</td><td>May/2020</td><td>🟢</td><td>Nov/2021</td><td><a href="https://arxiv.org/abs/2005.14165">🔗</a></td><td>Popular: 3.1M wpm</td></tr><tr><td>FLAN</td><td>Google</td><td></td><td>137</td><td></td><td></td><td>⬆ 🕸 👥</td><td>Sep/2021</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2109.01652">🔗</a></td><td>Fine-tuned LaMDA</td></tr><tr><td>LaMDA</td><td>Google AI</td><td><a href="https://www.youtube.com/watch?v=aUSSfo5nCdM">YouTube</a> (video only)</td><td>137</td><td>168</td><td>2:1</td><td>⬆ 🕸 👥</td><td>Jun/2021</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2201.08239">🔗</a></td><td>Chatbot</td></tr><tr><td>GLM-130B</td><td>Tsinghua &#x26; Zhipu</td><td><a href="https://huggingface.co/spaces/THUDM/GLM-130B">https://huggingface.co/spaces/THUDM/GLM-130B</a></td><td>130</td><td>400</td><td>4:1</td><td>🆆 📚 ⬆ 🕸</td><td>Aug/2022</td><td>🟢</td><td>Aug/2022</td><td><a href="https://arxiv.org/abs/2210.02414">🔗</a></td><td>50% English (200B tokens), so included here</td></tr><tr><td>Galactica</td><td>Meta AI</td><td><a href="https://galactica.org/">https://galactica.org/</a></td><td>120</td><td>450</td><td>4:1</td><td>📚</td><td>Nov/2022</td><td>🟢</td><td>Nov/2022</td><td><a href="https://galactica.org/static/paper.pdf">🔗</a></td><td>scientific only</td></tr><tr><td>YaLM 100B</td><td>Yandex</td><td><a href="https://github.com/yandex/YaLM-100B">Github</a> (train/deploy)</td><td>100</td><td>300</td><td>3:1</td><td>🆆 📚 ⬆ 🕸</td><td>Jun/2022</td><td>🟢</td><td>Jun/2022</td><td><a href="https://github.com/yandex/YaLM-100B">🔗</a></td><td>Megatron-LM clone, Russian/English: <a href="https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6">https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6</a></td></tr><tr><td>Sparrow</td><td>DeepMind</td><td></td><td>70</td><td>1400</td><td>20:1</td><td>🆆 📚 ⬆ 🕸 🌋</td><td>Sep/2022</td><td>🔴</td><td>N/A</td><td><a href="https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf">🔗</a></td><td>Chatbot as a fine-tuned version of Chinchilla 70B</td></tr><tr><td>Chinchilla</td><td>DeepMind</td><td></td><td>70</td><td>1400</td><td>20:1</td><td>🆆 📚 ⬆ 🕸 🌋</td><td>Mar/2022</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2203.15556">🔗</a></td><td>First to double tokens per size increase</td></tr><tr><td>NLLB</td><td>Meta AI</td><td><a href="https://github.com/facebookresearch/fairseq/tree/nllb/">Github</a> (train/deploy)</td><td>54.5</td><td></td><td></td><td>🌋</td><td>Jul/2022</td><td>🟢</td><td>Jul/2022</td><td><a href="https://research.facebook.com/publications/no-language-left-behind/">🔗</a></td><td>54.5B MOE, 3.3B dense. 200+ languages</td></tr><tr><td>xlarge</td><td>Cohere</td><td><a href="https://os.cohere.ai/playground/large/generate">Cohere</a></td><td>52.4</td><td></td><td></td><td>📚 🕸</td><td>Sep/2021</td><td>🟢</td><td>Nov/2021</td><td><a href="https://arxiv.org/abs/2108.07790">🔗</a></td><td>Stealth 'ebooks and webpages'. 52B: https://crfm.stanford.edu/helm/v1.0/?models=1</td></tr><tr><td>RL-CAI</td><td>Anthropic</td><td></td><td>52</td><td></td><td></td><td>🆆 📚⬆ 🕸 👥</td><td>Dec/2022</td><td>🔴</td><td>N/A</td><td><a href="https://arxiv.org/abs/2212.08073">🔗</a></td><td>RLAIF=reinforcement learning with AI feedback</td></tr><tr><td>AlexaTM 20B</td><td>Amazon Alexa AI</td><td><a href="https://github.com/amazon-research/alexa-teacher-models">Github</a> (train/deploy)</td><td>20</td><td>1000</td><td>50:1</td><td>🆆 🕸</td><td>Aug/2022</td><td>🟢</td><td>TBA</td><td><a href="https://assets.amazon.science/ee/20/3abcf2304d9b8d68da2006ff7107/alexatm-20b-few-shot-learning-using-a-large-scale-multilingual-seq2seq-model.pdf">🔗</a></td><td>Wikipedia and mC4 only. seq2seq</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

| Key: | 🆆 Wikipedia      |   | 👥 Dialogue                |   |
| ---- | ----------------- | - | -------------------------- | - |
|      | 📚 Books          |   | 🆀🅰 Questions and answers |   |
|      | ⬆ Reddit outbound |   | 🌋 Special                 |   |
|      | 🕸 Common Crawl   |   | 🇫🇷 French                |   |
