The virtual reality (VR) industry is growing at a fast pace, with nearly one in five US consumers using VR in 2020. But while currently, the gaming and entertainment sectors are the biggest drivers of VR growth worldwide, this immersive technology is also making its way to other spheres of life.
Researchers at Brown University and Cornell University are working towards expanding VR usage in professional settings. It involves a telepresence robot that responds automatically and in real-time to the movements and gestures made by a remote user in virtual reality.
For this, the research team developed the robotic system called VRoxy, a controller-less, VR-controlled robotic proxy that has the potential to make hands-on collaboration between people working remotely in a small space, like an office with teammates working in a much larger space.
“The great benefit of virtual reality is we can leverage all kinds of locomotion techniques that people use in virtual reality games, like instantly moving from one position to another.”
– Mose Sakashita, Ph.D. student in Information Science at Cornell.
“This functionality enables remote users to physically occupy a very limited amount of space but collaborate with teammates in a much larger remote environment.”
So, the team created VRoxy, built off a similar robot from Cornell called ReMotion, which occupies physical space on a remote user's behalf and automatically mirrors the user's movements in real-time before conveying crucial body language that is otherwise lost in standard virtual environments. ReMotion's functionality was limited to scenarios where both users—the one at the local end and the one at the remote end—had identical hardware setups and comparably sized workspaces. However, VRoxy has overcome this limitation.
What the software program VRoxy does is, it allows a user in a remote location to put on a VR headset and be virtually transported to a space, which could be an office, laboratory, or any other setting where their colleagues are physically working. At the physical location, the remote user is then represented through a robot, which allows them to move through that environment using natural movements, like walking. On top of that, the remote user can collaborate with colleagues through gestures such as pointing at objects, head animations like nodding, and even facial expressions through the robot proxy.
Although currently in its early stages, the researchers say the software already has the potential to address some of the biggest challenges faced when using robot proxies and augmented reality (AR) software for remote collaboration.
VRoxy: Revolutionizing Remote Collaboration
The paper “VRoxy: Wide-Area Collaboration From an Office Using a VR-Driven Robotic Proxy” was presented at this year's Association for Computing Machinery (ACM) Symposium on User Interface Software and Technology (UIST) held on Oct. 29 through Nov. 1 in San Francisco. The research paper demonstrates how VRoxy can accommodate mapping multiple physical locations onto a unified virtual space.
Virtual reality provides an opportunity to create intuitive interfaces by immersing users in a 3D environment where they can view and interact with robots in shared or remote environments, allowing for easier interaction. VR interfaces have been shown to help reduce task completion time, increase operator performance, and are generally preferred over traditional interfaces. But while a significant amount of foundational work has been done in VR robot interfaces, there's still much development required in creating VR teleoperation interfaces.
“Right now, motion-controlled robots for collaboration require the physical location to have the same amount of space as the remote environment, but that's often not the case,” with some people being in home offices, classrooms, or workspace while others in kitchens or living rooms, said Brandon Woodard, a Ph.D. student at Brown University who is also a graduate researcher for the Visual Computing Group and prior to that interned for Nvidia, MIT Lincoln Laboratory, NASA's Jet Propulsion Laboratory, and Intel. He further pointed out that rooms have different dimensions, which is easily visible even when meeting with people in Zoom, the video telephony software program.
According to the paper, recent research in robotic proxies has shown that one can automatically reproduce many non-verbal cues important in co-located collaboration, but they often require a symmetrical hardware setup in each location. As a solution to this, the team introduced VRoxy, a system designed to enable access to remote spaces through a robot, using a VR headset in a much smaller space, such as a personal office. Here, researchers said the software's automatic real-time responsiveness is key for both remote and local teammates.
VRoxy maps any small movements made by the remote user in VR space to the robot's larger movements in the physical space, allowing the user to easily navigate large physical spaces. This way, through VRoxy, a VR user can quickly navigate in a low-fidelity rendering of the remote space.
When putting on a VR headset, a VRoxy user gets access to two view modes. One is live mode, which displays an immersive image of the collaborative space in real time for interactions with local collaborators. The other one is a navigational mode that shows rendered pathways of the room, allowing for quicker, smoother mobility for remote users to “teleport” to where they'd like to go while limiting motion sickness, researchers said.
Moreover, the system being automatic means teammates can simply focus on collaboration instead of manually steering the robot, which uses the feed of a 360 camera to support real-time interactions.
The system also facilitates interaction modalities by rendering the micro-mobility around shared spaces on the proxy. These multiple modalities of non-verbal cues, such as head rotation, facial expressions, pointing gestures, and eye gaze, were captured by the VR user through Quest Pro, which is then represented through a telepresence robot at the remote location.
With VRoxy, a collaborator can navigate and collaborate within a vast, remote environment from a smaller physical space as well as instantly switch between separate workspaces that are physically apart, stated the research. This demonstrates that VRoxy can adapt to both stationary and mobile robotic proxies using a consumer-grade headset.
In future work, Sakashita, the paper's lead author, aims to supercharge VRoxy with robotic arms to allow remote users to interact with physical objects in the live space via the robot proxy. Moreover, he foresees VRoxy doing its own mapping of a space, similar to the Roomba vacuum cleaner, which uses different kinds of sensors to navigate and adapt to its environment. Currently, VRoxy relies on ceiling markers to navigate a room, but support for real-time mapping could allow its deployment in a classroom, Sakashita said.
Hyunju Kim and Ruidong Zhang, both doctoral students in the field of information science, along with François Guimbretière, professor of information science in Cornell Bowers CIS and Woodard, are co-authors of the research paper, which is supported by the Nakajima Foundation and the National Science Foundation.
The Synergy Unleashed: VR and Robotics Collaboration
The concept of virtual reality has been around for a long time now, since the 1960s. However, its usage in the field of robotics has rather been limited. This is despite the fact that there is a great deal of synergy between VR and robotics.
VR can be used in robotics to provide immersive robot teleoperation and to conduct human-robot interaction and collaboration studies, while robots help provide more immersive VR experiences by providing haptic feedback to users.
The lack of VR integration in the field of robotics has been largely caused by the absence of availability and affordability of commercial VR devices, which have seen considerable advancement in the past decade. And with that, there has been an increase in combining the power of VR and robotics for human-robot interaction and collaboration.
With an increasing interest in more automation with reconfigurable systems that can be adjusted based on changing needs, we are seeing a growing usage of virtual reality in robotics to make the communication between humans and machines more successful.
These two technologies are currently used in multiple applications to boost the performance of robots while allowing a human to be in control. For instance, in medical robotics, VR training gives doctors and surgeons the chance to train in an almost real-life situation without having to deal with actual patients. VR can also be used to train soldiers to deal with military-grade drones and robots without the risk of expensive and irreplaceable equipment getting damaged. VR and robotics research are further used to optimize space exploration in complicated settings.
By leveraging VR and robotics, VRoxy can also bring about significant advancements in remote collaboration and professional settings. By allowing users present at different physical locations to collaborate seamlessly in shared spaces, this software can prove to be really valuable for remote teams working on projects that require close coordination and communication.
VR and robotics are useful technologies for training on their own, but combining them allows education in all kinds of fields to become more intuitive and immersive. They even help train smarter robots themselves.
In addition to being used for training purposes and providing remote assistance and support, VRoxy can also find its use case in situations where physical space is constrained, as well as in remote education, by providing students with immersive experiences in physical settings. It can also facilitate virtual site visits, allowing stakeholders to explore physical spaces remotely and make informed decisions.
Companies Leveraging the Power of VR and Robotics
Given the potential of VR and robotics combination, several companies are researching and working on leveraging these technologies, so let's take a look at some of the most prominent ones:
It is focused on the development and application of machine intelligence algorithms. Cambrian Intelligence is building teleoperation interfaces to intuitively control robots at a distance with hand and arm movements using gaming technologies such as virtual reality, tracking, and haptics.
As of September 21, 2023, Cambrian Intelligence has cumulatively raised a total of $3,793,919 across various funding rounds, including a significant series unknown round, a pre-seed investment, a grant from the EASME – EU Executive Agency for SMEs, and an undisclosed amount from ff Venture Capital.
One of the world's largest semiconductor chip manufacturers, Intel is the key player in the development of technologies related to robotics and VR, providing technologies for both fields. Earlier this year, Intel introduced MiDaS 3.1 for computer vision as a solution to the depth estimation challenge in creating a wide range of applications in robotics, AR, and VR.
Currently, Intel's (INTC) shares are trading at $38.73, up 46.69% this year, and the company is paying a dividend yield of 1.29%. For 3Q23, it reported a revenue of $14.2 billion, down 8% year over year (YoY) and earnings per share (EPS) of $0.07.
“We delivered a standout third quarter, underscored by across-the-board progress on our process and product roadmaps, agreements with new foundry customers, and momentum as we bring AI everywhere,” said Intel CEO Pat Gelsinger.
This company is led by a team of technologists and physicians to apply the latest innovations in robotics and virtual reality to surgical procedures. To provide patients with better outcomes and less risk or invasiveness when undergoing abdominal procedures, Vicarious Surgical makes use of a patient cart featuring two robotic arms with camera-enabled visibility and exceptional mobility along with the surgeon console to drive the robot, controllable via a 3D screen, a VR headset, and the proprietary software.
For the second quarter of 2023, Vicarious Surgical reported $82.8 million in cash, cash equivalents, and short-term investments. During this period, the company received clarity on the clinical and regulatory pathway with the expectation to complete the first ventral hernia procedure in a patient in mid-2024. It also signed a fourth major U.S. hospital system agreement, bringing its total hospital partners to over 250. Meanwhile, Vicarious Surgical's (RBOT) shares are currently trading at $0.4263, having a P/E (TTM) of -0.68 and EPS (TTM) of -0.63.
Despite being a relatively nascent technology, the adoption of virtual reality has been widespread across companies, organizations, and even governments as its potential becomes clearer with every new development. VR is not only gaining a lot of traction in gaming, sports, healthcare, education, real estate, manufacturing, marketing, automotive, and defense, but it is also changing the robotics industry in big ways.
Combining VR and robotics has the potential to save lives and push the boundaries of exploration. Though it is still in the development phase, with factors like funding, technological challenges, and regulatory considerations affecting this growth, we are constantly seeing scientists and engineers making breakthroughs in these technologies.
Given the advancement in VR and robotics, we can expect to see the application of systems like VRoxy in specific industries within the next 5-10 years, with broader adoption to take more time as the technology matures and becomes more accessible.