We have combined a number of elements that we believe give a fairly comprehensive and overall useful overview. We compare the model’s features with some respected third-party scores and benchmarks.

• Privacy: Privacy is mostly enhanced if you own your model and infrastructure, having full control over your data. Overall, well-known LLMs such as GPT are less private than open-source ones, because with open-source models you are the one that decides where is going to be hosted and have full control over it.
• User Feedback Score: Based on the LMSYS leaderboard. LMSYS is an organization that aims to provide LLMs that are open source and available to everyone, being also the creators of Vicuna.
• Context size: The context size refers to how many tokens the LLM can handle. Tokens are the basic units of text or code that an LLM uses to process and generate language. Tokens can be characters, words, subwords, or other segments of text or code, depending on the chosen tokenization method or scheme.
• Cost: We’ve gathered information from well-known services (like OpenAI or Anthropic) and, for the open-source models, we’ve utilized different hardware based on the requirements recommended by those models, apart from the ones we have tested ourselves.
• Aggregated score: An average score between three state-of-the-art benchmarks: MMLU (Massive Multi-task Language Understanding), HellaSwag (Commonsense tasks), and ARC (Reasoning).

Selecting the right Language Learning Model is crucial because the effectiveness of your AI solution largely depends on it. Each model has its strengths and weaknesses, and some models may be more suitable for certain tasks than others. Additionally, models vary in terms of resource requirements, such as the amount of training data needed and processing power. The correct model choice can also have implications for your project's privacy and cost. Here are a few factors to consider when choosing a Language Learning Model:

• Accuracy: How well can the model perform the task you need? Can it generate accurate and coherent responses? Can it understand and generate text in the language or languages you require?
• Efficiency: How resource-intensive is the model? Does it require a lot of processing power? How long does it take to generate responses?
• Privacy: Does the model ensure data privacy? Can it handle sensitive data securely? A solid Service Level Agreement is enough or your use case requires extreme control on data access and running everything on-prem?
• Cost: What is the cost to use or refine the model? Are there associated costs with using the necessary infrastructure to run the model? What’s the cost of associated databases for long-term storage in your use case?
• Flexibility: Can the model adapt to different tasks or contexts? Is it easy to customize or adapt to your specific needs?

The market is incredibly active, with new developments happening almost daily, but today we have curated ten models that are powerful and well-established. These models have been chosen because of their popularity in both commercial (OpenAi, Google…) and open-source solutions (like falcon, currently top 1 in Hugging Face’s leaderboard).
‍
‍If you think there are others that should be included, feel free to drop us a line.

• GPT4 8k/32k: This is the model used for paying customers of the chatGPT premium plan or those using OpenAI APIs. The current versions include both 8k and 32K context size.
• GPT3.5-turbo (ChatGPT): This is the model that is used on ChatGPT’s free plan, as of June 2023.
• PaLM-2: The latest model introduced by Google, which powers PaLM APIs and Google Bard.
• Falcon-40b-instruct: From the technology innovation institute, trained with 1,000B tokens and currently (June 2023) top 1 in Hugging Face’s leaderboard.
• LLaMa-13b/Vicuna-13b: Both are based on LLaMa, a model provided by Meta (Facebook). In the case of Vicuna, it is trained with 70k conversations from ShareGPT, a platform in which users share their chatGPT conversations.
• MPT-7b-chat: Specialized in dialogue generation and provided by the MosaicML company.
• Claude v1/Instant: A model built by Anthopic. It can be used via APIs or by using their Slack integration.

The proprietary options from OpenAI, Anthropic, and Google are currently the most comprehensive and powerful.
As for open-source models, the best option available is Falcon-40B-instruct, with a reasonable cost given its benchmark scores, and new models appearing every week. Open-source models are very appealing because you can host your own LLM instance on a server under your control, offering different possibilities in terms of security, privacty and cost flexibility. Our approach at PrivateGPT is a combination of models. We're about creating hybrid systems that can combine and optimize the use of different models based on the needs of each part of the project.With the right configuration and design, you can combine different LLMs to offer a great experience while meeting other requirements in terms of security and privacy.