How NIRS is used in brain-computer interfaces
Imagine a person wearing a Brite and playing a demanding video game. This video game is difficult, and the mental workload is increasing drastically. Changes in blood volume, or hemodynamic changes, which are associated with the increase in workload are registered using the Brite. The signals that the Brite is measuring are sent in real time to an analysis system. This system gauges the increase in mental workload and subsequently lowers the difficulty of the video game by 5%. This is an example of a NIRS-based brain-computer interface (BCI), in which physiological signals detected from the brain are translated and used to control an output device (like a computer). This blog will expand on how a NIRS-based BCI works and what researchers have made possible using NIRS-based BCI.
ℹ This feature is outdated and support for BCI is now implemented via LSL. More information about LSL.
How it works
The most basic BCI can be built with a one-channel system, consisting of one detector and one transmitter. This means that on one place in the brain the hemodynamic changes are measured. In practice, a multi-channel system will be used more often, because this provides you with more data and brain areas to measure simultaneously. Additionally, you should think about what brain area you want to use for your BCI: all sorts of brain activation can be used. In this example, we will play the well-known ‘Snake’ game with our motor cortex. A nice piece of Matlab-code by Per-Anders Ekstrom for the game can be downloaded from this website and the interface is shown in Figure 1.
Figure 1. On the left, you can see the Matlab interface of the game ‘Brain Snake’. The snake is controlled via a NIRS-based BCI by hemodynamic signals in the motor cortex induced by finger tapping. The image on the right shows the input of hemodynamic changes displayed by our own Matlab script. If the difference is over 1 μM, the snake will turn right.
Our software OxySoft is used for recording the NIRS data. In OxySoft, the recorded data can be managed and analyzed. However, in order to do real-time analysis and data manipulation, we need to stream our data in real-time to the analysis software of choice that will be used as input for the game. For example, the data measured by the OxyMon can be streamed into the FieldTrip buffer. FieldTrip is a Matlab based toolbox and can be downloaded for free from the FieldTrip website. With FieldTrip, you can analyze NIRS data in real-time; very useful for BCI. You can also stream your data real-time with Lab Streaming Layer (LSL) to any device and software connected to your network. We linked a data stream to Matlab, and implemented a script so the snake can use the data to turn.
The top graph of Figure 2 (channel 1) shows the typical raw hemoglobin changes that are measured in rest. Finger tapping is performed between the vertical blue lines. In the raw data, you can see an oscillation at a fast frequency (the heartbeat), and other oscillations at a lower frequency (slow vasomotion). These low frequency waves have amplitudes comparable to the hemoglobin concentrations changes and thus, they can mask the changes. We can increase the success rate of this BCI significantly by adding a short separation channel to the setup (Figure 3). The graphs of channel 2 in Figure 2 shows the signals measured by the short separation channel. The filtered signal, where the short separation channel is used to filter out the heartbeat and slow vasomotion, is plotted in the bottom graph of Figure 2. In this graph, we can see a very clear response to the finger tapping. After the start of the finger tapping, with a few seconds delay, the oxyhemoglobin concentration increases (typically about 1-2 μM) and the deoxyhemoglobin concentration decreases with a smaller amount (~ 0.5 μM). You can see these changes in the bottom graph of Figure 2. When the signal amplitude increases above a certain threshold value, the snake will make a right turn. As such, the user can control the snake and direct it to the yellow blocks.
Figure 2. Typical oxyhemoglobin (red) and deoxyhemoglobin (blue) concentration changes were measured in the motor cortex (graphs made in OxySoft). The x-axis shows time in seconds and the y-axis shows concentration changes in micromolar. Finger tapping is performed between the vertical blue lines. The top graph shows the hemodynamic changes measured using 4 cm inter-optode distance (channel 1) and the graph below shows the hemoglobin concentrations at the more superficial channel with 1.5 cm inter-optode distance (channel 2). The difference between the two channels is plotted in the bottom graph. A multiplication factor (MF) is used in the subtraction (MF x channel 1- channel 2). In this case, the multiplication factor for O2Hb is 1.6. A cyclic average with standard deviation over the three finger tapping periods is plotted as well.
Figure 3. Double channel NIRS setup. A reference channel is created by adding a second transmitter with small inter-optode distance.
What has been done
BCI’s have many applications including health care, marketing, education, entertainment and security. All these applications, and more, are being explored as we speak. Currently, NIRS is being used extensively for the creation and development of BCI’s. Researchers Zephaniah and Kim (2014) wrote a review article on the recent developments and applications of NIRS-based BCI. To our delight, they mention our PortaLite and write the following sentence on it: “This is just an example of a sensor that shows convenience and portability that would make it ideal for a BCI system.” We believe this holds true for the OctaMon and the Brite as well, and we believe much more interesting research can be done with our devices. In the meantime, everyone is learning more about BCI’s every day, partly due to the great work of our customers. A few examples:
Kamavuako et al. (2018) used the OxyMon to investigate if changes in hemodynamics due to the activation of Broca’s area during overt (aloud) or covert (silent) speech can be harnessed to create a BCI that can be used by people suffering from neuromuscular disorders such as locked-in syndrome.
Hennrich and colleagues (2015) have used the OxyMon to investigate the feasibility of more complex and powerful classification approaches like Deep Neural Networks for NIRS-based BCI.
Since a robust single trial classification is key to a good BCI, Herff et al. (2014) have shown, using the OxyMon, that measuring hemodynamic responses in the PFC can be used to robustly quantify and classify mental workload.
Zhang and colleagues (2017) have explored the feasibility of recognizing motor imagery and motor execution in the same motion. The results show that fNIRS-BCI can indeed recognize both in the same motion.
Maior et al. (2018) have been working with fNIRS and Neurocinematics and may develop an interactive movie driven by hemodynamic responses in the brain using fNIRS.
“Thank you all for your work on BCI, and we are looking forward to more research in this fascinating area!”
References
1. Zephaniah, P. V., & Kim, J. G. (2014). Recent functional near infrared spectroscopy based brain computer interface systems: developments, applications and challenges. Biomedical Engineering Letters, 4(3), 223-230.
2. Kamavuako, E., Sheikh, U., Gilani, S., Jamil, M., & Niazi, I. (2018). Classification of overt and covert speech for Near-Infrared Spectroscopy-based Brain Computer Interface. Sensors, 18(9), 2989.
3. Hennrich, J., Herff, C., Heger, D., & Schultz, T. (2015, August). Investigating deep learning for fNIRS based BCI. In 2015 37th Annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 2844-2847). IEEE.
4. Herff, C., Heger, D., Fortmann, O., Hennrich, J., Putze, F., Schultz, T. (2014). Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS. Frontiers in Human Neuroscience 7, 935.
5. Zhang, Z., Jiao, X., Xu, F., Jiang, J., Yang, H., Cao, Y., & Fu, J. (2017). The investigation of brain-computer interface for motor imagery and execution using functional near-infrared spectroscopy. In International Conference on Innovative Optical Health Science (Vol. 10245, p. 102450I). International Society for Optics and Photonics.
6. Ramchurn, R.; Maior, H.; Wilson, M.; Martindale, S.; Benford, S.; Cai, M. (2018, October) fNIRS and Neurocinematics. Poster session presented at the fNIRS 2018 conference in Tokyo, Japan.
In hyperscanning, brain activity and connectivity of multiple subjects are measured simultaneously during social interaction, for instance in competitive situations. fNIRS is often used as neuroimaging technology for hyperscanning in cognitive studies due to its portability and relative insensitivity to movement artifacts. In an internal mini-study, we tested the use of Brite Frontal to perform hyperscanning while participants played a competitive game of checker.