Intro

In this blog we are going learn how to run already downloaded .gguf files into ollama. For this we are going to use Windows machine today. For other operating system the process is very much similar, you just have to change the path as needed.

Downloading and installing the necessary files

First you have to install ollama in your machine. You can visit and download it from there and install into your computer.

For this tutorial you need a .gguf file which you can download from various places. We are going to download it from huggingface.co. We are going to use Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf model file.

You can search this name at huggingface.co and download it from there.

Running the model into ollama

Make sure your ollama is installed properly by opening the command prompt and run ollama -v.

Then follow these steps:

• Move the downloaded Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf file to this folder C:\Users\YOUR_USER_NAME\.ollama\models of your windows.
• Create a new folder called modelfiles at this path C:\Users\YOUR_USER_NAME\.ollama
• Inside the modelfiles folder create a file called Meta-Llama ( you can use any supported name you want ) and open it in your any text editor. Don't give any file extension to it, like .txt or anything.
• At the file give the path of your downloaded .gguf file. In our case the path is **C:\Users\YOUR_USER_NAME\.ollama\models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf.

  • So in the Meta-Llama file we are going to write:

    FROM C:\Users\YOUR_USER_NAME.ollama\models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

• Now save the file and open the command prompt at modelfiles directory. (You can just press ctrl+l, type cmd and hit enter to open the command prompt )
• Now type

ollama create meta-llama -f C:\Users\YOUR_USER_NAME\.ollama\modelfiles\Meta-Llama

I am giving meta-llama as the name, you can give yours.

• It should take few moments based on your hardware and create the file.
• To confirm type: ollama list. You should see the details of the create model as NAME: meta-llama:latest.
• Now to run the model type: ollama run meta-llama. And start chatting with the model.