OpenLLM: Open Source Library for LLM

⚙️ Integrations

OpenLLM is more than a single solution; it’s a modular component that can seamlessly connect with other advanced tools.. It currently offer integration with BentoML and LangChain.

BentoML

You can use a Runner to incorporate OpenLLM models into your BentoML service. The Runner has a generate function that accepts a prompt string and produces an output string accordingly. This enables you to easily integrate any OpenLLM models with your current ML pipeline.

import bentoml
import openllm

model = "opt"

llm_config = openllm.AutoConfig.for_model(model)
llm_runner = openllm.Runner(model, llm_config=llm_config)

svc = bentoml.Service(
    name=f"llm-opt-service", runners=[llm_runner]
)

@svc.api(input=Text(), output=Text())
async def prompt(input_text: str) -> str:
    answer = await llm_runner.generate(input_text)
    return answer

Hugging Face Agents

Hugging Face Agents can easily integrate with OpenLLM.

OpenLLM client allows you to interact with the agent that is running. You can use it to pose queries to the agent:

import openllm

client = openllm.client.HTTPClient("http://localhost:3000")

client.ask_agent(
    task="Is the following `text` positive or negative?",
    text="What are you thinking about?",
)

LangChain (⏳Coming Soon!)

Soon, LangChain will let you easily use OpenLLM models with this syntax:

from langchain.llms import OpenLLM
llm = OpenLLM.for_model(model_name='flan-t5')
llm("What is the difference between a duck and a goose?")

To access an existing OpenLLM server from a different location, enter its URL in the following format:

from langchain.llms import OpenLLM
llm = OpenLLM.for_model(server_url='http://localhost:8000', server_type='http')
llm("What is the difference between a duck and a goose?")

🚀 Deploying to Production

To deploy your LLMs to production:

1. Building a Bento: OpenLLM lets you create a Bento for any model, such as dolly-v2, with the build command.:

   openllm build dolly-v2

   BentoML distributes your program as a Bento. A Bento contains your source code, models, files, artifacts, and dependencies.

2. Containerize your Bento

   bentoml containerize <name:version>

   BentoML provides a flexible and robust framework for building and deploying ML services online. For more details, see the Deploying a Bento guide.

🍇 Telemetry

To improve the product and user experience, OpenLLM tracks usage data. We only send internal API calls from OpenLLM and protect your privacy by not including any sensitive information. We do not collect any user code, model data, or stack traces. See the code for usage tracking.

To disable usage tracking, use the --do-not-track option in the CLI:

openllm [command] --do-not-track

Or by tweaking the environment variable OPENLLM_DO_NOT_TRACK=True:

export OPENLLM_DO_NOT_TRACK=True