Understanding how accurately people make decisions in complex tasks has become much clearer, thanks to a new approach that combines brain signals and image information. This innovative research was recently published in the journal Scientific Reports. The research was conducted by Xuan-The Tran, a PhD student under the co-supervision of Professor Chin-Teng Lin, Professor Nikhil Pal, Professor Tzyy-Ping Jung, and Dr. Thomas Do who are affiliated with institutions including the University of Technology Sydney, the Indian Statistical Institute, and the University of California San Diego,
The researchers created a framework that uses machine learning, a type of artificial intelligence that learns patterns from data, to analyze brain activity and image details together, enabling predictions of whether a person will respond correctly in a challenging task. The method useed the Segment Anything Model (SAM) to identify and isolate objects in images. It extracted features from both the target object’s characteristics and target objects’ relationships with neighboring objects to enhance prediction accuracy. Brain signals are collected using an electroencephalogram (EEG), which is a non-invasive technique. Features extracted from EEG data are then fused with image features to further improve prediction accuracy. “This advancement highlights how combining information from the brain and images can improve our understanding of how people make decisions,” explained Professor Lin.
In the study, participants were asked to find animals in pictures. These animals were camouflaged to make the task more difficult, simulating challenges similar to real-world situations. “Unlike other studies where participants can guess correctly by chance, this setup made guessing much harder, providing a better test of how people think and decide.” explained Dr Thomas Do. The researchers recorded the brain’s electrical activity, measured using electroencephalography, which captures brain signals through sensors placed on the scalp, and analyzed it alongside the image features to see how both influenced decision-making.
The results showed that combining brain and image data works much better than using either alone. “When tested, this combined approach achieved significantly higher accuracy in predicting correct decisions compared to models that relied on only one type of data,” said the lead author, Xuan-The Tran. This highlights the advantage of blending multiple sources of information to better understand human behavior.
“This research not only helps predict decision accuracy but also provides a framework for designing systems that can alert users to potential errors before they occur. Such systems could be vital in critical areas like healthcare or defense, where avoiding mistakes can be life-saving” added Professor Nikhil Pal.
One key element of this success was the in-depth use of image features. The extracted features identified relationships between objects in the pictures and were transformed to integrate seamlessly with EEG neural features. “Brain signals from regions known to be involved in object detection and decision-making, such as the occipital and parietal areas, which are responsible for processing sensory information and making decisions played a significant role in model’s performance” added Professor Tzyy-Ping Jung. The team found that training their model on data from individual participants worked better than training it on combined data from groups, showing how decision-making can vary from person to person.
By bringing together detailed brain activity analysis and sophisticated image analysis, this research opens up exciting possibilities for developing systems that can predict how well people will perform tasks in real time. The team plans to expand their research by using more data and refining their model, making it even more practical for everyday applications.
Journal Reference
Tran X.T., Do T., Pal N.R., Jung T.P., Lin C.T. “Multimodal Fusion for Anticipating Human Decision Performance.” Scientific Reports, 2024. DOI: https://doi.org/10.1038/s41598-024-63651-2
About the Authors
Chin-Teng Lin Distinguished Professor Chin-Teng Lin received a Bachelor’s of Science from National Chiao-Tung University (NCTU), Taiwan in 1986, and holds Master’s and PhD degrees in Electrical Engineering from Purdue University, USA, received in 1989 and 1992, respectively.
He is currently a distinguished professor at School of Computer Science and Director of the Human Centric AI (HAI) Centre and Co-Director of the Australian Artificial Intelligence Institute (AAII) within the Faculty of Engineering and Information Technology at the University of Technology Sydney, Australia. He is also an Honorary Chair Professor of Electrical and Computer Engineering at NCTU. For his contributions to biologically inspired information systems, Prof Lin was awarded Fellowship with the IEEE in 2005, and with the International Fuzzy Systems Association (IFSA) in 2012. He received the IEEE Fuzzy Systems Pioneer Award in 2017. He has held notable positions as editor-in-chief of IEEE Transactions on Fuzzy Systems from 2011 to 2016; seats on Board of Governors for the IEEE Circuits and Systems (CAS) Society (2005-2008), IEEE Systems, Man, Cybernetics (SMC) Society (2003-2005), IEEE Computational Intelligence Society (2008-2010); Chair of the IEEE Taipei Section (2009-2010); Chair of IEEE CIS Awards Committee (2022, 2023); Distinguished Lecturer with the IEEE CAS Society (2003-2005) and the CIS Society (2015-2017); Chair of the IEEE CIS Distinguished Lecturer Program Committee (2018-2019); Deputy Editor-in-Chief of IEEE Transactions on Circuits and Systems-II (2006-2008); Program Chair of the IEEE International Conference on Systems, Man, and Cybernetics (2005); and General Chair of the 2011 IEEE International Conference on Fuzzy Systems.
Prof Lin is the co-author of Neural Fuzzy Systems (Prentice-Hall) and the author of Neural Fuzzy Control Systems with Structure and Parameter Learning (World Scientific). His 948 publications include 3 books; 28 book chapters; 485 journal papers; and 432 refereed conference papers, including about 232 IEEE journal papers in the areas of neural networks, fuzzy systems, brain-computer interface, multimedia information processing, cognitive neuro-engineering, and human-machine teaming, that have been cited more than 40,065 times. Currently, his h-index is 96, and his i10-index is 464.
Nikhil R. Pal was a Professor in the Electronics and Communication Sciences Unit and was the founding Head of the Center for Artificial Intelligence and Machine Learning of Indian Statistical Institute. His current research interest includes brain science, computational intelligence, machine learning and data mining.
He was the Editor-in-Chief of the IEEE Transactions on Fuzzy Systems for the period January 2005 – December 2010. He served/been serving on the editorial /advisory board/ steering committees of several journals including the International Journal of Approximate Reasoning, Applied Soft Computing, International Journal of Neural Systems, Fuzzy Sets and Systems, IEEE Transactions on Fuzzy Systems and the IEEE Transactions on Cybernetics.
He is a recipient of the 2015 IEEE Computational Intelligence Society (CIS) Fuzzy Systems Pioneer Award and 2021 IEEE CIS Meritorious Service Award. He has given many plenary/keynote speeches in different premier international conferences in the area of computational intelligence. He has served as the General Chair, Program Chair, and co-Program chair of several conferences. He has been a Distinguished Lecturer of the IEEE CIS (2010-2012, 2016-2018, 2022-2024) and was a member of the Administrative Committee of the IEEE CIS (2010-2012). He has served as the Vice-President for Publications of the IEEE CIS (2013-2016) and the President of the IEEE CIS (2018-2019).
He is a Fellow of the West Bengal Academy of Science and Technology, Institution of Electronics and Tele Communication Engineers, National Academy of Sciences-India, Indian National Academy of Engineering, Indian National Science Academy, International Fuzzy Systems Association (IFSA), The World Academy of Sciences, and a Fellow of the IEEE, USA.
Tzyy-Ping Jung (S’91-M’92-SM’06-F’15) received the B.S. degree in electronics engineering from National Chiao Tung University, Hsinchu, Taiwan, in 1984, and the M.S. and Ph.D. degrees in electrical engineering from the Ohio State University, Columbus, OH, USA, in 1989 and 1993, respectively. He currently serves as the Co-Director of the Center for Advanced Neurological Engineering and the Associate Director of the Swartz Center for Computational Neuroscience at the University of California, San Diego. In addition, he is an Adjunct Professor in the Department of Bioengineering at UC San Diego. Dr. Jung extends his academic contributions internationally, holding adjunct professorships at Tianjin University and the University of Science and Technology Beijing in China, as well as at National Tsing Hua University and National Yang Ming Chiao Tung University in Taiwan.
Dr. Jung pioneered transformative techniques for applying blind source separation to decompose multichannel EEG, MEG, ERP, and fMRI data. In recognition of his contributions to blind source separation for biomedical applications, he was elevated to IEEE Fellow in 2015. He is also a Fellow of the Asia-Pacific Artificial Intelligence Association (AAIA). Dr. Jung’s research emphasizes the integration of cognitive science, computer science and engineering, neuroscience, bioengineering, and electrical engineering. His interdisciplinary work is highly regarded and well-cited by peers, with ~47,000 citations and an h-index of 92, according to Google Scholar.
Thomas Do is a Senior Lecturer and Co-Director of the Human-AI Interaction (HAI) Centre at the University of Technology Sydney (UTS). With a PhD in Computer Science from UTS, a Master’s in Human-Computer Interaction from the Korea Institute of Science and Technology.
His research focuses on the integration of Artificial Intelligence (AI), Brain-Computer Interfaces (BCI), Human-Computer Interaction, and Robotics, with a particular emphasis on using BCI technologies for assistive applications. Dr Do’s vision is to bridge the gap between neural engineering and practical, real-world applications by developing cutting-edge AI-powered systems that translate brain signals into actionable outputs.
