For those of us who have transitioned to working from home over the course of the last year, we must navigate a strange new manifestation of mobility.

Far-flung colleagues appear almost magically in grid format on a screen right in front of our faces, despite their remote locations. Yet at the same time, a document, presentation, piece of content, or part of a running application already at our fingertips is awkward to share with others on the same video call.

It’s a paradoxical science fiction world where far is near, and the close-at-hand dilates impossibly beyond our reach. Perhaps these surreal distortions of time and space explain in part why “video-conferencing syndrome” feels so draining.

And even as we’re stranded within the confines of our improvised home offices, we’re somehow supposed to navigate this otherworldly place—a jumbled chaos terrain of home and work, personal and professional, private and semi-public.

Moving between these realities, sometimes moment by moment, makes us nimble in a way we’ve never experienced before: our activity is mobile even as we stay put in the same location. We work in the same physical spaces, but as we navigate these transitions, we’re not in the same human places.

While 2020 has accelerated this trend, perhaps it’s inevitable—and indeed, as we point out below, in many ways this strange new mobility has been a long time coming.

Microsoft researchers are creating technologies to help people succeed in this new way of life. We are working to develop systems that help us to navigate these changes. A new world where these transitions feel less strange—and more empowering. A place appropriate to our current task, locality, and context, where “mobility” means technology that rises to the universal human need to connect and work with others seamlessly.

To that end, Microsoft researchers have published three papers—two of which appear at this year’s ACM Symposium on User Interface Software and Technology (UIST 2020)—on new technologies that redefine how we interpret this concept of place.

The first explores SurfaceFleet, a system to decouple computing from individual devices and places. The second presents Ambrosia, a system which uses resilient distributed programming techniques to unbind running programs and their state from any particular device (CPU). The third circles back on this notion of place, showing how nuanced social cues such as tilt and orientation of a display, on the adjustable Microsoft Surface Studio, can support elegant and natural transitions between different tasks and ways of using such a display.

SurfaceFleet: Mobility as transitions of user activity from one ‘place’ to another

What the Fleet system is in brief:

  • A distributed system leveraging a robust, performant declarative database foundation and building on the Ambrosia runtime
  • An exploration of novel implications for migration of user experiences across devices
  • A platform for Applets: lightweight, distributed user interface elements that unbind interactions from devices, applications, users, and time
  • A collaboration tool enabling people to work across devices and act at synchronous or asynchronous times.

The COVID-19 pandemic has brought the collision of home and work—of activity in limited physical spaces that must transition between different human places—to a critical juncture.

But two trends, manifested over the past decade already, are influencing the future of human experiences with computing technology.

The first trend concerns hardware and systems architecture. With Moore’s Law at an end, yet networking and storage exhibiting exponential gains, the future appears to favor systems that emphasize seamless mobility of data—favoring techniques that consume network and storage bandwidth rather than those using a particular CPU. Accelerated by pervasive cloud services and 5G, these computational shifts show no sign of slowing.

The second trend is one of human behavior. People now interact with more devices than ever before. Modern information work increasingly relies on multi-device workflows and distributed workspaces, with connected and interdependent devices—smartphones, desktops, tablets, and perhaps even emerging new form factors. The problem here lies in that transitioning from device to device, and more broadly place to place, can cost us precious efficiency or resources, like time or attention, leaving our activities marooned on islands of glass instead of creating a new interconnected world.

What’s needed is an ecosystem of technologies that seamlessly transitions from place to place, whether that “place” takes the form of a literal location, a different device form factor, the presence of a collaborator, or the availability of the pieces of information needed to complete a particular task at a given time. Such a “Society of Technologies” favors techniques that establish meaningful relationships between the members of this society, rather than with any particular device, to afford mobility of user activity from one place to another, in a very general sense of the word.

Through this lens, we can view the essence of mobility as the transition of user activity from one place to another. SurfaceFleet is a working system, development toolkit, and user experience that explores some implications of these challenges by decoupling computation—including its representation in the graphical user interface—from the current device.

Yet, once user interface mechanisms are decoupled from a single device, we discovered that this also has interesting carry-on implications for unbinding interaction from the current application, the current user, and the current time, as well. The Fleet system handles transitions in place—bridging the resulting gaps—across all four of these dimensions. See the embedded video above for demonstrations that show how user interfaces can “float” above the screen and transcend program state that is confined to the current device.

The Fleet system unbinds UI elements from not only the device but also the current application, user, and time. In the visible UI, Applets unbind controls from applications. Portfolios unbind tools, inputs, behaviors, and content from the current device and user. Promises unbind actions from time. Read more in the SurfaceFleet paper.

But authoring distributed programs is difficult and requires considerable expertise. How do we reimagine this notion of “device” that is so deeply baked into current development practices? This is where our journey crosses paths with a new distributed-systems technology known as Ambrosia.