Brussels / 3 & 4 February 2024


Friend or Foe Inside? Exploring In-Process Isolation to Maintain Memory Safety for Unsafe Rust

Rust provides unsafe language features to interact with hardware or call into non-Rust libraries. However, this shifts responsibility for ensuring memory safety to the developer. Failing to do so may lead to memory-safety violations in unsafe code, which can violate the safety of the entire application. In this talk, we explore in-process isolation with Memory Protection Keys as a mechanism to shield safe program sections from safety violations that may happen in unsafe sections. Our approach is easy to use and comprehensive, as it prevents heap and stack-based violations. We further compare process-based and in-process isolation mechanisms and the necessary requirements for data serialization, communication, and context switching. We specifically explored various Rust serialization crates such as Abomonation and bincode. Our results show that in-process isolation can be effective and efficient, permits for a high degree of automation, and also enables a notion of application rewinding where the safe program section may detect and safely handle violations in unsafe code. However, even for modestly sized arguments, the context switch cost starts to get dominated by the cost of data transfer between domains. Here, the Rust data serialization method used can significantly impact performance and thus, it is crucial to optimize it for the use case at hand. We open-source our prototype and experimental evaluation data under a BSD license on GitHub:


Photo of Merve Gülmez Merve Gülmez