Using Langfuse to Trace Queries in RAG

This is a simple cookbook that demonstrates how to use Langfuse to trace your queries in RAG. The RAG pipeline is implemented with LlamaIndex and Milvus Lite to store and retrieve the documents.

In this quickstart, we’ll show you how to set up a LlamaIndex application using Milvus Lite as the vector store. We’ll also show you how to use the Langfuse LlamaIndex integration to trace your application.

Langfuse is an open-source LLM engineering platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications. All platform features are natively integrated to accelerate the development workflow.

Milvus Lite is the lightweight version of Milvus, an open-source vector database that powers AI applications with vector embeddings and similarity search.

Setup

Make sure you have both llama-index and langfuse installed.

$ pip install llama-index langfuse llama-index-vector-stores-milvus --upgrade

Initialize the integration. Get your API keys from the Langfuse project settings, and replace public_key secret_key with your key values. This example uses OpenAI for embeddings and chat completions, so you also need to specify your OpenAI key in environment variable.

import os

# Get keys for your project from the project settings page
# https://cloud.langfuse.com
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region

# Your openai key
os.environ["OPENAI_API_KEY"] = ""

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager
from langfuse.llama_index import LlamaIndexCallbackHandler
 
langfuse_callback_handler = LlamaIndexCallbackHandler()
Settings.callback_manager = CallbackManager([langfuse_callback_handler])

Index using Milvus Lite

from llama_index.core import Document

doc1 = Document(text="""
Maxwell "Max" Silverstein, a lauded movie director, screenwriter, and producer, was born on October 25, 1978, in Boston, Massachusetts. A film enthusiast from a young age, his journey began with home movies shot on a Super 8 camera. His passion led him to the University of Southern California (USC), majoring in Film Production. Eventually, he started his career as an assistant director at Paramount Pictures. Silverstein's directorial debut, “Doors Unseen,” a psychological thriller, earned him recognition at the Sundance Film Festival and marked the beginning of a successful directing career.
""")
doc2 = Document(text="""
Throughout his career, Silverstein has been celebrated for his diverse range of filmography and unique narrative technique. He masterfully blends suspense, human emotion, and subtle humor in his storylines. Among his notable works are "Fleeting Echoes," "Halcyon Dusk," and the Academy Award-winning sci-fi epic, "Event Horizon's Brink." His contribution to cinema revolves around examining human nature, the complexity of relationships, and probing reality and perception. Off-camera, he is a dedicated philanthropist living in Los Angeles with his wife and two children.
""")

# Example index construction + LLM query

from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext
from llama_index.vector_stores.milvus import MilvusVectorStore


vector_store = MilvusVectorStore(
    uri="tmp/milvus_demo.db", dim=1536, overwrite=False
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    [doc1,doc2], storage_context=storage_context
)

Query

# Query
response = index.as_query_engine().query("What did he do growing up?")
print(response)

# Chat
response = index.as_chat_engine().chat("What did he do growing up?")
print(response)

Explore traces in Langfuse

# As we want to immediately see result in Langfuse, we need to flush the callback handler
langfuse_callback_handler.flush()

Done! ✨ You see traces of your index and query in your Langfuse project.

Example traces (public links):

1. Query
2. Query (chat)

Trace in Langfuse:

Langfuse Traces

Interested in more advanced features?

See the full integration docs to learn more about advanced features and how to use them:

• Interoperability with Langfuse Python SDK and other integrations
• Add custom metadata and attributes to the traces