Are you interested in running a large language model (LLM) on your PC, without relying on the cloud? Well, you’re in luck because it’s easier than you might think. Despite the hype surrounding massive machine-learning training clusters and AI PCs, it’s actually quite feasible to run a variety of LLMs, including chat bots and source code generators, on your desktop system.
Tools like Ollama, LM Suite, and Llama.cpp are readily available and make it relatively simple to get these models up and running on your system. In this article, we’ll focus on Ollama due to its simplicity and cross-platform compatibility across Windows, Linux, and Macs.
When it comes to performance, compatibility, and AMD GPU support, it’s worth noting that large language models like Mistral or Llama 2 typically perform best with dedicated accelerators. While datacenter operators deploy GPUs in massive clusters, Ollama offers native support for Nvidia and Apple’s M-series GPUs. For example, Nvidia GPUs with at least 4GB of memory should work, and it’s recommended to have at least 16GB of memory for M-series Macs.
If you’re a Linux user with an Nvidia GPU, you’ll need the latest proprietary driver and CUDA binaries installed. Additionally, AMD provides a comprehensive guide for getting an LLM up and running on systems with Radeon 7000-series GPUs or newer.
Even if you don’t have a supported graphics card, Ollama can still run on an AVX2-compatible CPU, albeit at a slower pace. It’s also possible to use a quantized model to potentially reduce memory requirements.
Installing Ollama
Thankfully, installing Ollama is a straightforward process regardless of your operating system. As an open-source tool, it’s easily accessible for Windows, Mac OS, and Linux users.
For Windows and Mac OS users, simply visit ollama.com to download and install the application. Meanwhile, Linux users can execute a one-liner command to kickstart the installation process. Manual installation instructions are also available for those who prefer that approach.
By following these steps, you can have Ollama up and running on your PC in no time, allowing you to explore the world of large language models without the need for specialized hardware or cloud-based solutions.
