Ambient Intelligence

A paper titled “Illuminating the dark spaces of ambient intelligence in healthcare“ has been published in Nature .
It was submitted by Stanford University, and the famous Professor Li Fei-Fei is listed as a co-author.
Above all, the title is cool!
To me, papers that use metaphorical expressions, have question marks in the title (as if they are asking a question), or give their own unique names to their techniques are cool.
Anyway, that’s all for now, let’s read the paper!

Notice

This article will introduce and summarize the research paper, so it will mainly focus on case studies from the United States.
This translation is also a two-part series.
In this article, I will introduce case studies from medical institutions, and in the next article, I will summarize case studies from living spaces, as well as technical and ethical issues.

TL;DR for busy people

Ambient Intelligence example

• Ambient Intelligence is the idea of ​​”giving intelligence to the space itself” by using various sensors .
• In recent years, ambient intelligence has become increasingly widespread due to the increasing performance and affordability of sensors and the development of deep learning technology .
• Ambient Intelligence can be either non-wearable (cameras, etc.) or wearable (wearable devices, beacons) .
• Ambient Intelligence brings light to the dark corners of healthcare.
  • Dark Side
    • Extremely overworked
    • Patient deaths due to oversights or errors in clinical decision-making or physical actions
    • Medical transcription takes a lot of time
    • If we constantly provide care to the elderly, we cannot encourage them to become independent in their daily lives.
• Using RGB and/or depth cameras, we were able to identify patient behavior and the results were impressive.
• Microphone-based speech-to-text reduces medical transcription time.
• In areas where privacy is an issue (such as showers and toilets), it seems possible to identify behavior using only audio information from a microphone .
• By using sensors to detect when someone has fallen, caregivers do not need to be nearby at all times, which will likely encourage independence.
• Appropriate data privacy and model transparency quality must be maintained.

What is Ambient Intelligence?

As mentioned at the beginning, advances in AI technology and data science have led to widespread adoption of systems that support human decision-making.
Furthermore, sensors have become more powerful and affordable, making them easier to install in living spaces.
“Ambient” means “surrounding” or “enclosed,” and “Intelligence” means “intelligence.
” “Ambient Intelligence” can be translated as “environmental intelligence” or “intelligent space .
” Rather than directly intervening in our lives and behavior, Ambient Intelligence seems to have a stronger connotation of accumulating and analyzing data from our lives and behavior to encourage better lifestyles and behavior.
Ambient Intelligence itself is not a new term; it has been widely used since the 2000s.
Ambient Intelligence (article from April 2005) Speaking of which, I remember a service called Ambient
that visualizes sensor values ​​using microcontrollers like Arduino and ESP32 . I prefer hardware that uses sensors, such as electronics, so Ambient Intelligence is a field that really aligns with my interests, and I feel like I want to do more in this area. Now, let’s get into the paper!

Ambient Intelligence Intro

Advances in data science and AI technologies have led to the widespread adoption of systems that support correct diagnostic decision-making in healthcare.
For example, systems that detect malignant tumors or abnormal trends in CT scans and X-rays.
However, little support has been provided for the physical actions of clinicians, patients, and their families, and the importance of activities performed in physical spaces, such as hospitals and homes, for promoting health remains unclear. (Here, physical actions and activities include caregiving and rehabilitation.)
To maximize the benefits of medical advances, affordable, human-centered approaches are needed to illuminate these dark and unclear areas.

The dark side of medical and nursing care

• The hectic environment makes it difficult for clinicians to keep their cognitive abilities sharp.
• The rapid increase in complexity in modern medicine means that even in busy environments, it is necessary to constantly keep up with information.
• As a result, over 400,000 patients die each year in the United States due to oversights and errors in clinical decision-making and physical actions.
• Medical writing (filling out medical records, etc.) takes up 30% of working hours
• If care is provided by constantly being present, it becomes difficult for elderly people to live independently.

The purpose of this paper

In this review paper

• Wearable devices (contact type) that collect data by being worn on the body
• Non-wearable sensors that collect data by being placed in space (non-contact type)

This project will focus on these two key factors in health and explore how we can shed light on the darker side of hospitals and everyday living spaces, two environments that contribute significantly to our health.

Introduction of specific examples

Let’s first look at the diagram.

• RGB camera
• Depth sensor (depth camera)
• thermal camera
• Wireless sensors (WiFi, radar, etc.)
• Sound sensor

These include: Ambient Intelligence can be realized by using these technologies.
For example, using an RGB camera makes it possible to detect objects and people and estimate their posture. This can also be expanded to behavioral recognition.
This year alone, thermal cameras have become extremely popular due to the spread of COVID-19.
Using thermal and face detection, it is possible to detect people with high fevers based on their body temperature (although few are accurate enough for medical use).
Wireless sensors may not be very familiar, but at this year’s HCI (Human Computer Interaction) conference, a 3D posture estimation method using WiFi signal strength was proposed.

Hospital Cases

• In 2018, approximately 7.2% of the US population experienced a hospital stay of at least one day.
• In the UK, the NHS has reported 17 million hospital admissions
• Healthcare workers are frequently overworked and understaffed with limited resources

Here, we will introduce how Ambient Intelligence can play an important role in improving the quality of medical care, increasing clinician productivity, and improving operations.

Cases in intensive care units

Intensive care units (ICUs) are a type of hospital facility that treats patients with life-threatening illnesses or severe organ failure.
In the United States, ICUs account for $108 billion in annual healthcare costs, or 13% of all hospital costs.

Patient behavior monitoring

Critically ill patients are known to develop acute limb weakness after admission to the ICU, a condition known as ICU-acquired weakness.
Reference: What is ICU-acquired weakness?
It has been reported that this condition can double one-year mortality and increase hospital costs by 30%.
Therefore, it has also been reported that encouraging early mobilization (from the ICU) can potentially reduce the relative incidence of ICU sequelae by 40%. Currently, standard mobilization assessment relies on in-person observation, but its use is limited due to issues such as cost, observer bias, and human error. Appropriate assessment requires an understanding of subtle changes in patient movement.
This is where Ambient Intelligence comes in.

Measuring Patient Mobility in the ICU
Using a Novel Noninvasive Sensor. Wearable devices attached to the arms or legs can detect movement. However, wearable devices alone cannot detect external assistance or interactions with the physical space (e.g., the difference between sitting in a chair and sitting in bed). Therefore, noninvasive sensors can provide the continuous, subtle changes necessary to accurately measure a patient’s ability to leave bed.

A computer vision system for deep learning-based detection of patient mobilization activities in the ICU.
This research team installed a Kinect in an ICU room and collected 362 hours of data from eight patients. They then used that data to train a machine learning algorithm to identify activities. As a result, the system was able to classify activities such as in bed, out of bed, and walking with 87% accuracy compared to checks by three doctors.

While these results are very promising, Li said that more insightful assessments (such as which actions affect which symptoms) rather than just evaluating a collection of individual short videos would likely yield more hierarchical results.

Intelligent ICU for Autonomous Patient Monitoring Using Pervasive Sensing and Deep LearningFor
example, the research team used cameras, microphones, and accelerometers to observe 22 patients in the ICU for seven days to study delirium.
They found that patients with delirium had significantly less head movement than patients without delirium.
Future studies could use this ambient technology to detect delirium earlier, providing researchers with a better understanding of how patient mobilization affects mortality, length of stay, and recovery.

Management of hospital-acquired infections

More than 100 million patients worldwide are affected by hospital-acquired infections each year, and up to 30% of ICU patients experience hospital-acquired infections. Compliance with hand hygiene protocols is one of the most effective ways to reduce the frequency of hospital-acquired infections.
However, measuring compliance remains challenging.
For example, wearable devices such as RFID (radio frequency identification, as used in Suica and Uniqlo tags) can detect approach to handwashing sites and roughly estimate the number of times handwashing occurs. However, they cannot identify detailed behaviors such as the ” Five moments of hand hygiene”
proposed by the WHO . Therefore, ambient sensors can more reliably monitor handwashing activities and measure true alcohol dispenser usage.

Towards vision-based smart hospitals: a system for tracking and monitoring hand hygiene compliance.
Automatic detection of hand hygiene using journal of ambient intelligence and humanized computing vision technology
. For example, a study was conducted to measure handwashing compliance using a depth sensor placed on a dispenser and a deep learning algorithm. In this study, compliance was measured over a one-hour period with 75% accuracy, compared with 63% accuracy for human observation and only 18% accuracy for RFID over the same period.

Dr. Li summarized that the next important step in improving patient outcomes is to move from “observation” with Ambient Intelligence to “clinical action.”

Cases in the operating room

Surgical skills assessment

More than 230 million surgeries are performed worldwide annually, with up to 14% of patients experiencing adverse events. Prompt surgical feedback, including frequent technical coaching, can reduce the number of errors by 50%. However, assessment is performed by peers or supervisors, which is time-consuming, infrequent, and subjective. While wearable
sensors can be used to estimate skill, they can impede manual dexterity and require complex sterilization procedures. Ambient cameras offer an alternative.

Surgeon Technical Skill Assessment Using Computer Vision-Based AnalysisIn
this study, a convolutional neural network was trained to track needle drivers during prostatectomy and estimate the classification of 12 surgeons who were classified into high-skill and low-skill groups based on peer evaluation.As a result, classification of high-skill and low-skill groups was achieved with 92% accuracy based on needle driver movements.

Dr. Li said that while video-based surgical stage recognition and other approaches may lead to improved surgical training, further clinical validation is needed, and appropriate feedback mechanisms must be validated.

Counting equipment used

Furthermore, the application of Ambient Intelligence in the operating room is not limited to endoscopic images. Another example is “surgical counting,” the process of counting used instruments. Currently, counting these objects visually and verbally requires the time and effort of dedicated staff. This can lead to mislabeling due to a lack of attention or poor communication. Therefore, introducing an automated measurement system could assist the team to compare ambient intelligence vs artificial intelligence.

Using a Data-Matrix–Coded Sponge Counting System Across a Surgical Practice: Impact After 18 Months
Effectiveness of a Radiofrequency Detection System as an Adjunct to Manual Counting Protocols for Tracking Surgical Sponges: A Prospective Trial of 2,285 Patients
This study reported that the use of barcoded sponges for abdominal surgery reduced the retained object rate (surgical instruments forgotten to be removed from the affected area) from once every 16 days (!) to once every 69 days.
Similar results were also obtained using RFID and sponges made by Raytec.

However, due to the size of barcodes and RFID tags, they cannot be applied to needles and instruments, which account for up to 55% of count discrepancies. Therefore, using an ambient camera may be able to count these small objects and staff.

A multi-view RGB-D approach for human pose estimation in operating rooms:
First-year Analysis of the Operating Room Black Box Study.
This study used ceiling-mounted cameras in the operating room to track body parts of surgical team members with an accuracy of approximately 5 cm. Ambient data collected throughout the room also allowed for the creation of a detailed log of intraoperative activity.

While these studies are promising as proof of concept (PoC), Dr. Li concluded that further research is needed to quantify the impact on patient outcomes, reimbursement, and efficiency improvements.

Other Cases

Medical Documentation

Clinicians spend up to 35% of their time on medical documentation tasks, taking valuable time away from patients. Currently, physicians complete documentation during or after each patient visit. To alleviate this burden, some clinicians hire medical scribes, resulting in 0.17 more patient visits per hour and 0.21 more relative value units per patient (i.e., 0.21 more reimbursements from payers). However, training medical scribes is expensive and staff turnover is high.
Ambient microphones have the potential to perform the same tasks as medical scribes.

In a study of speech recognition for medical conversations
, a deep learning model was trained on 90,000 patient-doctor conversations spanning 14,000 hours, and showed a word-level transcription accuracy of 80%, potentially better than the 76% accuracy achieved by medical scribes.

How Google Glass Automates Patient Documentation For Dignity HealthIn
terms of clinical usefulness, one medical institution reported that using Google Glass (glasses with a microphone) reduced the time spent on documentation from two hours to 15 minutes (!), doubling the time spent talking with patients.

management

Using Time-Driven Activity-Based Costing to Identify
Value Improvement Opportunities in Healthcare From a management perspective, Ambient Intelligence can improve the transition to activity-based costing. Traditionally, insurance companies and hospital managers have used a top-down approach called value-based accounting ($ per medical procedure) to estimate the healthcare outcomes per US dollar. Time-driven activity-based costing is a bottom-up alternative, allowing value to be calculated based on the time and cost of individual resources (e.g., 48 hours of use of a ventilator in an ICU). This can inform process redesign, as in the following example: Measuring the value
of process improvement initiatives in a preoperative assessment center using time-driven activity-based costing.
In this example, staff reductions of 17% and an increase in patient visits of 19% were achieved without worsening patient outcomes.

Currently, clinical activity and cost mapping is performed using face-to-face observations, staff interviews, and electronic medical records. Furthermore, the Ambient Intelligence system introduced in this paper can automatically recognize clinical activities, count the number of healthcare professionals, and estimate activity time.
However, the activity-based costing paradigm is unknown to hospital staff, and supporting evidence demonstrating the clinical benefits of Ambient Intelligence is still lacking. As technology advances, hospital management is expected to participate in the implementation and testing of Ambient Activity-based costing systems.

Conclusion

This has gotten quite long, so I plan to continue in a separate article.
Ambient Ai is expected to be applied not only in medicine, but also in other fields.
For example, collecting behavior logs from behavior recognition using cameras and depth sensors could be applied to managing workers in the manufacturing industry.
In the next article, I will discuss the application of Ambient Intelligence in living spaces, as well as the technical and ethical issues.