The global software ecosystem has moved to richer and richer web experience. With the addition of A/V APIs, webgl acceleration, rich media APIs, RTC and, more recently, the wide open field of web assembly-supported features, more or and more of the typical user interaction and applications happens within the browser.

However, not all processing is meant to happen browser-side. In particular, when dealing with media with potentially large resolutions, exotic formats or complex compute-heavy effects, to provide a full user experience, it might be required to move back and forth between the browser and a backend server processing.

But this comes with its own sets of challenges: what kind of processing is well suited in the browser? How to best interface a browser-based API with a backend-end based one? How is it possible to transpose a user experience that is built on javascript APIs available in the browser to a backend-based processing where these APIs typically have no bearing.

In this talk, which is based on some of the challenges faced when building Descript, a feature-rich web-based video editor, we will review some of the technologies that are available to help interfacing web and backend processing, illustrate some of the challenges that these pose and also solve and explore potential future solutions using recent or prospective technologies.