Amputees Control 3D Avatars By Visualizing Missing Limbs In New Israeli Study

In early 2017, almost two years after an accident led to an amputation below his shoulder, an Israeli man controlled a 3D avatar by visualizing the movement of his missing arm to propel the avatar forward on a virtual path. This achievement is one of the advances in brain-computer interface (BCI), a mind-reading technology that allows a user to control an external device with brain activity only, and the focus of a new study launched by a team of researchers from Israel’s Interdisciplinary Center Herzliya (IDC), Bar Ilan University, the Weizmann Institute of Science, and the Sheba Tel Hashomer Hospital.

The research project allowed a group of subjects, some of whom lost an arm, to navigate a virtual avatar using recorded brain patterns. The data was registered in real-time using an fMRI (functional magnetic resonance imaging) scanner and subjects used motor execution (moving fingers and toes) in order to guess what was moved while researchers like IDC PhD student Ori Cohen and Dr. Doron Friedman, head of the IDC’s Advanced Virtuality Lab (AVL), looked at their brain patterns.

The research team looked at the brain activity of seven subjects, four of whom had not had an amputation, and three who had had an arm removed at least 18 months to two years earlier. The researchers wanted to compare the brain patterns of the amputee subjects to that of the non-amputee control group to determine if there was an obvious difference in performance.

As they laid in the fMRI scanner, subjects were shown a 3D avatar with a path ahead of it on a screen. Each subject was given 40 auditory directions (left, right, forward, etc) to follow in order to move the avatar along the path through their own thought process.

The subject’s avatar is seen standing at the beginning of the 3D virtual path. Courtesy

“As a group, our amputee subjects lost their ability to control their arm,” Cohen tells NoCamels, “We didn’t know what happened to the motor regions in the brain after amputation. We showed that there is adequate representation of motor activation for high-level BCI control. i.e., the amputee subject exhibited high performance while controlling a virtual avatar (navigating a path, while collecting discs, as fast as possible and as many as they can) using their missing arm (and other limbs.)”

SEE ALSO: Israeli Scientists Discover The Brain Mechanism Behind Successful Multitasking

According to Cohen and Dr. Friedman, who also spoke to NoCamels about the experiment, the performances of the amputees were almost on par with the performances of those in the control group, which had no amputations.

Surprisingly, all subjects reported suffering from phantom pains, according to Friedman. One subject even said it was easier for him to use the missing arm than the existing arm for BCI because when he was using the intact limb, he had to keep moving his fingers for a relatively long duration, which grew tiring. The missing limb, on the other hand, would get stuck and keep activating without getting tired.

Results showed that the dominant arm was stronger than the non-dominant one for all subjects, but as expected, the amputated arm was weaker than the intact one in some cases, because the motor skills extended beyond the motoric area of the brain. The people who had amputations were mostly able to do just as well with their missing arm as with the arm that was intact.

“Our biggest contribution,” says Cohen, “is that we show that regardless of the plasticity in the brain, it is possible to use motor representation for complex tasks.” In other words, even after the brain has changed its “wiring” through something like BCI, it is possible to use a representation of the missing arm in the brain.

“The big picture shows that [the subjects] are almost at the same level, and still using the missing arm in the brain,” Cohen explained when the team presented their research at the 8th International IEEE EMBS Conference on Neural Engineering in Shanghai, China in June 2017.

To Cohen’s knowledge, this is the first time researchers have shown that there is an adequate representation of motor activation for high-level BCI control.

Cohen, Friedman, and the team of researchers were nominated for the annual BCI Award in 2017 for their study. The Award was created to recognize outstanding and innovate research in the field of brain-computer interfaces. Only 12 projects were nominated for the award worldwide. While they didn’t win, Friedman said it was a huge honor to be nominated, because the award is one of the top accolades in BCI research.

SEE ALSO: Controlling An Avatar With Your Brain? Israeli Lab Is Trying

The robot study 

The team uses the findings from their first study for another study where they are testing how an amputee would embody a humanoid robot and control it, through BCI. Friedman, Cohen, and other researchers, only some of whom had taken part in the first study, researched this option — the ability to use the BCI system for navigation tasks through a large humanoid robot.

Here, as part of a study to dissolve the boundary between the human body and a surrogate robotic representation, subjects were asked to operate an HRP-4 humanoid robot inside a room and direct the robot’s body as if it was their own. Various tests also meant to look at motor imagery and visual categories. The subject directs the robot’s arm toward one of three objects placed on a table and when the correct item has been selected, the subject navigates the robot towards the experimenter in order to stimulate delivery of the object. The subject was located in Israel and the robot was located in France, with the geographic split only made due to the availability of facilities.

The researchers said it may have been the first study of its kind.

To read the study of brain-computer interface control of an avatar using missing hand representation in amputees, click here.

Photo and Video: Courtesy