Brain Signals and Images Hold the Secret to Better Decisions

Understanding how accurately people make decisions in complex tasks has become much clearer, thanks to a new approach that combines brain signals and image information. This innovative research was recently published in the journal Scientific Reports. The research was conducted by Xuan-The Tran, a PhD student under the co-supervision of Professor Chin-Teng Lin, Professor Nikhil Pal, Professor Tzyy-Ping Jung, and Dr. Thomas Do who are affiliated with institutions including the University of Technology Sydney, the Indian Statistical Institute, and the University of California San Diego.

The researchers created a framework that uses machine learning, a type of artificial intelligence that learns patterns from data, to analyze brain activity and image details together, enabling predictions of whether a person will respond correctly in a challenging task. The method used the Segment Anything Model (SAM) to identify and isolate objects in images. It extracted features from both the target object’s characteristics and target objects’ relationships with neighboring objects to enhance prediction accuracy. Brain signals are collected using an electroencephalogram (EEG), which is a non-invasive technique. Features extracted from EEG data are then fused with image features to further improve prediction accuracy. “This advancement highlights how combining information from the brain and images can improve our understanding of how people make decisions,” explained Professor Lin. 

In the study, participants were asked to find animals in pictures. These animals were camouflaged to make the task more difficult, simulating challenges similar to real-world situations. “Unlike other studies where participants can guess correctly by chance, this setup made guessing much harder, providing a better test of how people think and decide,” explained Dr. Thomas Do. The researchers recorded the brain’s electrical activity, measured using electroencephalography, which captures brain signals through sensors placed on the scalp, and analyzed it alongside the image features to see how both influenced decision-making.

The results showed that combining brain and image data works much better than using either alone. “When tested, this combined approach achieved significantly higher accuracy in predicting correct decisions compared to models that relied on only one type of data,” said the lead author, Xuan-The Tran. This highlights the advantage of blending multiple sources of information to better understand human behavior.

“This research not only helps predict decision accuracy but also provides a framework for designing systems that can alert users to potential errors before they occur. Such systems could be vital in critical areas like healthcare or defense, where avoiding mistakes can be life-saving,” added Professor Nikhil Pal.

One key element of this success was the in-depth use of image features. The extracted features identified relationships between objects in the pictures and were transformed to integrate seamlessly with EEG neural features. “Brain signals from regions known to be involved in object detection and decision-making, such as the occipital and parietal areas, which are responsible for processing sensory information and making decisions, played a significant role in model’s performance,” added Professor Tzyy-Ping Jung. The team found that training their model on data from individual participants worked better than training it on combined data from groups, showing how decision-making can vary from person to person.

By bringing together detailed brain activity analysis and sophisticated image analysis, this research opens up exciting possibilities for developing systems that can predict how well people will perform tasks in real time. The team plans to expand their research by using more data and refining their model, making it even more practical for everyday applications.