Cluster Summarization with Ollama Locally or in the Cloud

(forthcoming in CiteSpace)

As a convenient alterative to OpenAI's GPT API, Ollama provides a promising and flexible solution to explore a large number of large language models on your own laptop. You can easily switch between models such as llama3, the most capable openly available LLM to date, gemma, a lightweight state of the art open models from Google DeepMind.

Here is a quick summary of what Ollama can do to expand our choices when it comes to summarize clusters in CiteSpace. The initial experience on Windows 10 was quite positive. A follow-up spin with an EC2 instance is even more appealing. I plan to add this new gateway to LLMs to CiteSpace.

Download Ollama

Download Ollama from https://ollama.com/download/windows. The installer will install it on your C drive by default. If your disk space is running low, the easiest way to redirect to another drive is to launch the installer from the command line: OllamaSetup.exe /DIR=D:\somewhereelse

LLM Models

A list of models can be added or removed from Ollama. See https://ollama.com/library for the full list. For example, llama3 is tagged as the most capable openly available LLM to date, whereas gemma is featured as a family of lightweighted, state of the art open models built by Google DeepMind. There are so many interesting models. I just wish I have the time to explore more of them one by one. In this blog, I will focus on llama3 and gemma. I will use the OpenAI's gpt-4-turbo as a reference point.

To add a model to Ollama:

ollama pull llama3 or pull gemma:2b

To remove a model:

rm llama3 or rm gemma:2b

ollama serve will start ollama on your localhost:11434.

Cluster Summarization

For the purpose of comparison, the input and the prompt are kept the same as I switched from one model to another for them to summarize 5 clusters from CiteSpace.

As you can see from the table below, GPT-4 has the fastest completion time, followed by Gemma, whereas llama3 took the longest. In terms of the length of the summarization, llama3 seems to give a much longer summary than the other two. Note that longer summaries are not necessarily better. On the other hand, gemma seems to be quite terse. The overlap between their summaries is evident. They all hit the target squarely.

Here are other cluster summaries:

Cluster Labels with GPT and LLMs with Ollama

Between GPT-4 and Gemma:2B, three of the five clusters have the identical labels, but the largest two clusters differ, namely Bioterrorism Response Preparedness (GPT) vs. Threat to Public Health (Gemma), and Terrorism Medical Response (GPT) vs Trauma from Explosive Events (events). GPT-4 has a lot more parameters than Gemma:2B.

GPT-labeled Clusters

Gemma:2B-labeled Clusters

It is certainly good news to have added flexibility of choosing your own favorite LLMs to work with. Your laptop may get stretched a bit if you are running low on your disk space or your RAM. In this exercise, I used all the default settings, which may be further optimized to speed up the overall completion time. On the other hand, since you don't need to pay for its usage, it opens up other opportunities and you don't need to worry how soon your budget will run out.

Ollama on AWS EC2

Resources on my old laptop are not ideal to host models such as llama3. Using a GPU-empowered virtual machine in the cloud can be an attractive option, especially if you enjoy managing LLMs all by yourself. An AMI instance on AWS EC2 such as Amazon Linux 2 with GPU costs about $0.53 per hour. The cost is on the time consumed, as opposed to the number of tokens involved. You can easily turn on and off your rental machine as you need, while you can still have the benefits of having LLMs in your own backyard. The overall completion time is very close to the completion time from OpenAI's API (Column 4 in the table below).

I will keep looking for other viable options of working with LLMs. If you have preferences or recommendations, please let me know.