Building distributed systems with Msgflo
Flow-based-programming over message queues
MsgFlo is a tool to build systems that span multiple processes and devices, for instance IoT sensor networks. Each device acts as a black-box component with input and output ports, mapped to MQTT message queues. One then constructs a system by binding the queues of the components together. Focus on components exchanging data gives good composability and testability, both important in IoT. We will program a system with MsgFlo using Flowhub, a visual live-programming IDE, and test using fbp-spec.
At the 2014 IoT devroom, we introduced flow-based programming (FBP) for heterogenous IoT systems, using NoFlo and MicroFlo. The programming model worked well for individual devices, but between the devices of the system our solution caused too tight coupling. When we realized this, we decided to build Msgflo, which reuses many of the same concepts and tools.
In MsgFlo each process/device is an independent participant, receiving data on input queues, and sending data on output queues. A participant do not know where the data comes from, nor where (if anywhere) the data will go. This strong encapsulation gives good composability and testability. MsgFlo uses a standard message queue protocol (MQTT or AMQP). This makes it easy to use with existing software. As each participant is its own process and communicate over networks, they can be implemented in any programming language. Convenience libraries exist for C++, Python, Arduino, Node.js and Rust. On top of the message queue protocol, a simple discovery mechanism is added. For existing devices without native Msgflo support, the discovery messages can be sent by a dedicated tool.
We have used Msgflo in a handful of real-life deployments, and will demonstrate building a simple stand-alone IoT system.