Lumigator is a Python application designed for evaluating large language models. It provides an opinionated selection of models and evaluation metrics for specific use-cases (starting with summarization), but it also allows for extension and customization. It is built on an open stack that includes FastAPI and Ray and easily scales from private, self-hosted, local installations to bare metal servers and cloud providers. It can be used via web UI, programmatically with a python SDK, or directly via REST API. In this talk we will show how Lumigator works, explore its architecture and design choices, and demonstrate how it can be deployed on different infrastructures, extended, and customized to meet your specific needs.