How to Run Gemma2 on Raspberry Pi 4: A Step-by-Step Guide

Running large language models (LLMs) on resource-limited devices like the Raspberry Pi 4 might seem daunting, but it’s entirely possible with the right approach. In this guide, I’ll walk you through how I set up and ran Gemma2 on my Raspberry Pi 4 using the Ollama framework.

Why the Raspberry Pi 4? Simply put, it’s what I have on hand. The Raspberry Pi 4 is an affordable, versatile, and highly portable mini-computer, making it an attractive option for experimenting with LLMs in a constrained environment. Despite its limited processing power and memory, with the correct optimizations and a lightweight setup, the Raspberry Pi 4 can handle tasks that might surprise you — including running models like Gemma2.

In this article, I’ll share the steps I took to build a lightweight environment, configure the necessary tools, and deploy Gemma2 using the Ollama framework. Whether you’re curious about pushing the limits of your Raspberry Pi 4 or looking for a cost-effective way to explore LLMs, this guide will help you get started.

Getting started

Choosing the Right OS: Raspberry Pi OS Lite vs. Ubuntu Server

When setting up your Raspberry Pi 4 to run Gemma2 with the Ollama framework, choosing the right operating system is crucial. I opted for Raspberry Pi OS Lite over Ubuntu Server for a few key reasons:

• Optimization for Raspberry Pi: Raspberry Pi OS Lite is specifically designed for the Raspberry Pi hardware. It’s lightweight, efficient, and includes drivers and configurations tailored to the Raspberry Pi 4, ensuring smoother performance.
• Resource Efficiency: With its minimal footprint, Raspberry Pi OS Lite uses fewer system resources compared to Ubuntu Server, which is particularly important given the limited RAM and processing power of the Raspberry Pi 4.
• Community Support: Raspberry Pi OS Lite has extensive community support, with numerous tutorials and resources available. This can be a significant advantage when troubleshooting or customizing your setup.

On the other hand, Ubuntu Server does offer some benefits, such as newer software packages and a familiar environment if you’re already accustomed to Ubuntu. However, its slightly heavier resource usage and the lack of Raspberry Pi-specific optimizations made Raspberry Pi OS Lite the better choice for this project.

Flashing the OS with Raspberry Pi Imager

To set up the Raspberry Pi OS Lite, I used Raspberry Pi Imager — a straightforward tool for flashing operating systems onto an SD card. Here’s how I did it:

1. Download Raspberry Pi Imager: You can download the tool from the official Raspberry Pi website. It’s available for Windows, macOS, and Linux.
2. Select the OS and Storage

• Open Raspberry Pi Imager, choose Raspberry Pi OS Lite from the list of OS options.

• Insert your microSD card into your computer and select it as the storage destination.

3. Customize Settings Before Writing

• Enable SSH: To connect to your Raspberry Pi remotely after booting, enable SSH by clicking the gear icon in the settings menu and toggling the SSH option.
• Configure Wi-Fi: Pre-configure the Wi-Fi settings by entering your SSID and password in the same settings menu. This ensures that your Raspberry Pi will connect to your network automatically on first boot.
• Set Hostname and Password: For easier identification on your network, set a custom hostname for your Raspberry Pi. Additionally, change the default password to enhance security. Both options can be configured in the settings menu before writing the OS to the SD card.

Once the settings are configured, write the OS image to the SD card. The process takes a few minutes, after which the card will be ready to use.

4. First Boot and Connection

• Insert the SD card into your Raspberry Pi 4 and power it on. Thanks to the pre-configured settings, it will connect to your Wi-Fi network automatically.
• Find the Raspberry Pi’s IP Address: To locate your Raspberry Pi’s IP address on your network, use the nmap command:

nmap -sn 192.168.1.0/24

• Replace 192.168.1.0/24 with your network’s subnet.
• Once you have the IP address, use SSH to connect to the Raspberry Pi from your laptop’s terminal:

ssh <RPi_IP_Address>

• Since you already set a hostname and password during the configuration, use these for the SSH login.

Configuring the Environment

1. Updating the System: After flashing the Raspberry Pi OS Lite or Ubuntu Server onto your SD card, the first step is to ensure that your system is up-to-date.

sudo apt update && sudo apt upgrade -y

• These commands update the list of available packages and install the latest versions of all packages.

2. Install Ollama for Linux

To run Gemma2 on your Raspberry Pi, we need to install the Ollama framework, which supports various large language models. Here’s how we can set up Ollama on Raspberry Pi OS Lite:

cd /tmp
curl -fsSL https://ollama.com/install.sh | sh

This command will download and install Ollama to the environment. It could take some time to finish, so, please be patient.

Running Ollama with Gemma2

After installing Ollama in the environment, start the engines and run the Gemma2 model. To start Ollama:

ollama serve

It’s time to select a model. There are many different models that can be used with Ollama, varying in size and capabilities. However, since the model we will choose will work in a relatively weak environment such as RPi, it would be better to choose a light model.

The 2B model in the Gemma2 model family is one of the most successful models among its peers. Therefore, it can be preferred. You can visit Ollama library to browse other models. To get gemma2-2B, simply run this on a different shell:

ollama run gemma2:2b

This will download model parameters to the environment and get them ready for queries. Let’s give it a go:

Running Gemma2 on a Raspberry Pi 4 is not only a practical exercise in optimizing large language models for resource-limited devices but also a rewarding exploration of what’s possible with affordable technology.

By following this guide, you’ve learned how to set up Raspberry Pi OS Lite, install the Ollama framework, and deploy Gemma2 effectively. This project highlights the potential for innovation and experimentation in AI, even with constrained resources. Dive into the full guide, try it out yourself, and explore the capabilities of your Raspberry Pi 4 in new and exciting ways!

That’s it! Now you have a local LLM running on Raspberry Pi. Enjoy!

#RaspberryPi4 #Gemma2 #Ollama #LLMs #MachineLearning #AI #TechTutorial #DIYTech #DemocratizedAI #AIforEveryone