MinecraftButlerAI is a FastAPI backend that brings a butler ("Alfred") to life inside Minecraft: it understands natural-language questions (by text or by voice), answers with real game knowledge, and executes in-world actions. Rather than a single LLM call, it's an agentic architecture where each piece solves a concrete problem, from intent routing to knowledge retrieval and voice synthesis.
The butler combines a LangGraph agent with persistent memory, a multilingual RAG system over the game documentation, and a local voice pipeline, all exposed through a JWT-authenticated HTTP API designed for production (rate-limiting, migrations, observability).
Demo
Three questions in a single session, each exercising a different capability of the agent.
How do I craft an iron sword?
Craft it with 2 iron ingots and 1 stick. The ingots go in the top two slots of the center column, and the stick in the bottom slot.
Do I have what I need to make one?
No. You're missing the iron ingots. You have sticks in your inventory but no iron ingots.
Which crops are ready to harvest?
You have 5 carrots, 2 potatoes, and 6 wheat ready to harvest.
Under the hood (LangSmith)
Agent architecture
The butler is modeled as a graph of nodes with LangGraph: a first node classifies the user's intent and the graph deterministically routes (a plain intent → node dictionary, not "LLM magic") to one of three branches: answer a question with RAG, move to a set of coordinates, or chat.
The graph state (ButlerState) is a typed TypedDict whose messages field uses LangGraph's add_messages reducer, which accumulates history automatically. The graph compiles once (a singleton guarded by an asyncio.Lock) with an AsyncRedisSaver checkpointer: each session persists its state in Redis with a TTL, giving multi-turn conversational memory per player without any client-side management.
Intent classification runs on Claude Haiku 4.5 (fast and cheap) using structured output (a validated Pydantic object, not free text to parse), while the final answer uses Claude Sonnet 4.6. Both models are obtained through a factory that abstracts by role ("classifier"/"responder") and by provider (Anthropic or OpenAI), so the code asks for capabilities, not concrete models.
Multilingual RAG
Game knowledge is indexed in English from PrismarineJS/minecraft-data and Minecraft Wiki extracts, but users ask in Spanish. To bridge that language gap, dense cross-lingual embeddings (paraphrase-multilingual-MiniLM-L12-v2) are used with cosine-similarity search in Qdrant.
The original design included a full hybrid pipeline (a sparse BM42 branch + a FlashRank reranker), which was dropped after validating with real data: both components are lexical and English-only, so with Spanish queries the sparse branch returned noise and the reranker could not reorder ES→EN results. The measured conclusion, documented in the code, is that dense multilingual retrieval alone is superior in both languages.
For wiki mechanics, Parent Document Retrieval is used: small chunks (~800 characters) are indexed for precise embeddings, but the full parent block (~2000 characters) is retrieved and passed to the LLM, resolving the tension between retrieval precision and context sufficiency.
Voice & production
The response is streamed over Server-Sent Events sentence by sentence (not token by token), because the client synthesizes it as speech (TTS) and a TTS needs complete sentences to sound natural; this gives a sense of immediacy without chopping the audio.
Voice transcription runs on-device with faster-whisper in int8 quantization (faster and using half the memory of float32, with no noticeable quality loss), with the model pre-warmed at startup to avoid cold-start and without ever sending the user's audio to third parties.
The backend is built for production: JWT authentication with roles, rate-limiting with SlowAPI, Alembic migrations, slice architecture (router/schemas/service/repository), and optional per-run observability of the agent with LangSmith.