This talk is about our experience building LLM based telephone agents on top of open source telephony platforms. We will detail the issues and challenges in the real time media processing, STT, TTS and large language model handling that were discovered whilst developing the llm-agent server code base and playground, look at the results we were able to achieve and where development is likely to go next.