Securely signing Android releases, while being a critical process and operation for every AOSP-based project, has been lacking in comprehensive documentation, especially for building a production-grade and enterprise-level signing infrastructure. This talk presents our experience in designing and implementing a Hardware Security Module (HSM)-based signing solution for CalyxOS that ensures transparency and operational practicality while upholding security standards widely endorsed by security experts with limited resources.

We will walk through our process of defining criteria for secure signing operations and redesigning a signing infrastructure. In particular, we will discuss the trade-offs and our trajectory to technical decisions, including: * Security and operational pros and cons: Why use an HSM; * Our criteria for evaluating HSM solutions: Exemplified with the comparison between YubiHSM 2, Nitrokey HSM, Amazon Cloud HSM, and Entrust nShield in open-source standards, cost-effectiveness, and operational practicality; * PKCS#11 integration challenges: What it is, why it matters for HSM compatibility, and the specific code changes and scripts we made to to support it; * Key ceremony design: The use of Shamir's Secret Sharing (SSS) schema for recovery and additional backup and lessons from the provisioning process; and * Audit logging and cryptographic verification of signing operations.

In addition, this talk invites discussions from participants on experiences in operational security and building trust through transparency and communication. We will focus on how to balance complex Android development needs and overcome challenges with constrained resource and scant systematic documentation. This talk aims to start collaborations on issues such as concurrent multi-device signing, ceremony design, and community-driven criteria across FOSS development teams.