Image by Author

At this point, we’re all thinking the same thing. Is the world of LLMs really taking over? Some of you may have expected the hype to plateau, but it is still on the continuous rise. More resources are going into LLMs as it has shown a huge demand.

Not only has the performance of LLMs been successful, but also their versatility in being able to adapt to various NLP tasks such as translation and sentiment analysis. Fine-tuning pre-trained LLMs has made it much easier for specific tasks, making it less computationally expensive to build a model from scratch. LLMs have swiftly been implemented into various real-world applications, boosting the amount of research and development. 

Open-source models have also been a big plus with LLMs, as the availability of open-source models has allowed researchers and organizations to continuously improve existing models, and how they can be safely integrated into society. 

What is OpenLLM?

OpenLLM is an open platform for operating LLMs in production. Using OpenLLM, you can run inference on any open-source LLMs, fine-tune them, deploy, and build powerful AI apps with ease.

OpenLLM contains state-of-the-art LLMs, such as StableLM, Dolly, ChatGLM, StarCoder and more, which are all supported by built-in support. You also have the freedom to build your own AI application, as OpenLLM is not just a standalone product and supports LangChain, BentoML, and Hugging Face. 

All these features, and it’s open-source? Sounds a bit crazy right?

And to top it, it’s easy to install and use.

How to Use OpenLLM?

To make use of LLM, you will need to have at least Python 3.8, as well as pip installed on your system. To prevent package conflicts, it is recommended that you use a virtual environment. 

1. Once you have these ready, you can easily install OpenLLM by using the following command:

pip install open-llm

1. To ensure that it has been installed correctly, you can run the following command:

$ openllm -h

Usage: openllm [OPTIONS] COMMAND [ARGS]...

   ██████╗ ██████╗ ███████╗███╗   ██╗██╗     ██╗     ███╗   ███╗
  ██╔═══██╗██╔══██╗██╔════╝████╗  ██║██║     ██║     ████╗ ████║
  ██║   ██║██████╔╝█████╗  ██╔██╗ ██║██║     ██║     ██╔████╔██║
  ██║   ██║██╔═══╝ ██╔══╝  ██║╚██╗██║██║     ██║     ██║╚██╔╝██║
  ╚██████╔╝██║     ███████╗██║ ╚████║███████╗███████╗██║ ╚═╝ ██║
   ╚═════╝ ╚═╝     ╚══════╝╚═╝  ╚═══╝╚══════╝╚══════╝╚═╝     ╚═╝

  An open platform for operating large language models in production.
  Fine-tune, serve, deploy, and monitor any LLMs with ease.

1. In order to start an LLM server, use the following command including the model of your choice:

openllm start

For example, if you’d like to start an OPT server, do the following:

openllm start opt

Supported Models

10 models are supported in OpenLLM. You can also find the installation commands below:

1. chatglm

pip install "openllm[chatglm]"

This model requires a GPU.

1. Dolly-v2

pip install openllm

This model can be used on both CPU and GPU. 

1. falcon

pip install "openllm[falcon]"

This model requires a GPU.

1. flan-t5

pip install "openllm[flan-t5]"

This model can be used on both CPU and GPU. 

1. gpt-neox

pip install openllm

This model requires a GPU.

1. mpt

pip install "openllm[mpt]"

This model can be used on both CPU and GPU. 

1. opt

pip install "openllm[opt]"

This model can be used on both CPU and GPU. 

1. stablelm

pip install openllm

This model can be used on both CPU and GPU. 

1. starcoder

pip install "openllm[starcoder]"

This model requires a GPU.

1. baichuan

pip install "openllm[baichuan]"

This model requires a GPU.

To find out more information about runtime implementations, fine-tuning support, integrating a new model, and deploying to production - please have a look here at the one that caters to your needs. 

Wrapping it up

If you’re looking to use OpenLLM or need some assistance, you can reach out and join their Discord and Slack community. You can also contribute to OpenLLM's codebase using their Developer Guide.

Has anybody tried it yet? If you have, let us know what you think in the comments!
 
 
Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.