Advancing First Responder Preparedness through Digital Twin and XR-Based Indoor Drone Training

EXECUTIVE SUMMARY

First responders often work in hazardous and unpredictable environments, where traditional training can be costly, risky, and difficult to repeat. To address this challenge, we developed an immersive simulation environment that allows users to operate a drone inside a realistic virtual building during an emergency scenario. This project presents an immersive training approach using extended reality (XR) and digital twin technology to simulate indoor emergency scenarios for drone operations. We created a realistic building environment and integrated it into a physics-based XR system where participants performed tasks such as human identification and hazard identification. The system was designed to support hazard identification, human target detection, navigation practice, and decision making without exposing participants to real-world danger. Most participants were able to complete the mission and identify key hazards. While some participants experienced difficulty with drone control, the overall usability result indicates that the system was practical for training. This project can significantly enhance first responders’ preparedness while minimizing risk and cost.

BACKGROUND

In the United States, every year there are 16,500 office and store fires. These office and store fires cause over $932M in direct property damages [1]. The National Fire Protection Association (NFPA) estimates around 26,000 firefighter injuries per year, primarily due to hazardous exposure and overexertion during training [2]. Firefighter injuries remain a major concern. Drones can reduce risk by allowing responders to inspect hazardous areas remotely before entering. Drones enable responders to remotely assess hazardous environments, reducing direct risk. However, real-world drone training is limited by safety risks, high costs, and logistical constraints, so we propose XR-based systems to provide repeatable and scalable training environments [3]. Prior research shows that XR technologies improve task accuracy, safety, and situational awareness in dynamic environments [3-4].

METHODOLOGY

The emergency scenario was designed to represent an indoor fire event. The fire originated from an electrical socket near a window while open doors allowed smoke to spread through connected spaces. This setup created a realistic training condition where participants had to navigate the drone inside the building, observe environmental hazards, and identify human targets. Figure 1 shows the pipeline for development of the experimental environment. Three human targets were included in the environment: one running, one fainted, and one injured. These elements were added to test both navigation ability and situational awareness.

Figure 1. Development Pipeline

Digital Twin Development. We used a Trimble X9 Laser Scanner to capture the interior of the building. A drone was used to capture images of the exterior of the building. Data was converted into point clouds, which were then converted into a 3D model in Autodesk Revit and exported to Unity to build the simulated environment [5]. After importing the descriptive digital twin into Unity, we created interactive layouts of rooms, equipment, and pathways and added physics-based navigation. Colliders were used so the drone could interact with walls and objects. Rooms, furniture, equipment, pathways, smoke, fire, and human targets were added. The drone movement was made realistic enough for navigation practice, allowing us to maneuver the drone in a realistic simulation.

Experiment Scenario Design: Design was inspired by the NIST evaluation guidelines [6]. This is a guide used while working with drones, and we utilized Level 5 Confined Test Lane to create the placement for fire in the scenario. Participants spawned just outside the building and used the controller to pilot the drone indoors while viewing the live drone camera feed on the controller screen. Their mission was to locate fire sources and human targets inside the building then report the overall scenario. To encourage careful operation, the drone could collide a maximum of five times before being destroyed.

EXPERIMENTAL TEST PLAN

The experiment was organized into three phases. During Phase 1: Pre-experiment phase, participants provided consent, completed a demographic survey, and reviewed the building floor plan. During Phase 2: XR simulation phase, participants controlled a drone inside the virtual environment and completed mission objectives. They viewed the live drone camera feed on the controller screen. Their task was to identify the number and location of human targets and locate the origin of the fire. The mission was timed for data collection and stopped after 5 collisions, based on NIST-inspired guidelines. During Phase 3: Post-experiment phase, participants identified objectives on a map and completed survey questions about system usability and a modified version of NASA-TLX workload rating scale [7-8].

Participants. There were n = 8 participants in the age range of 21-37 years. Most had low to moderate drone experience and limited XR experience. This made the study useful for testing beginner-level training.

RESULTS AND DISCUSSION

Figure 2 shows that there was an 87.5% mission completion success rate, meaning 7 out of 8 participants completed the mission. Hazard identification accuracy was 81.25% from post-experiment map identification. This implied that most participants were able to complete the mission and identify key hazards.

The System Usability Scale score was 74.06, which is above the common benchmark score of 68. This suggests that participants found the XR training system generally usable and effective. Although some participants experienced difficulty with drone control precision, the overall usability result indicates that the system was understandable and practical for training purposes.

Figure 2. Experimental Results

In figure 3, the Modified NASA-TLX result shows that participants experienced moderate to high mental demand, which means they were actively engaged in navigation, hazard identification, and decision-making tasks. Physical demand was low, suggesting that XR-based drone operation does not require significant physical effort and is ergonomically manageable. Overall, participants reported manageable workload levels, indicating that the XR system was challenging but not overwhelming.

Figure 3. Modified NASA-TLX Workload Rating Scale

Challenges observed. Some participants had difficulty controlling the drone precisely, especially in confined indoor spaces. This suggests that beginner users may need a short practice session before completing the main mission. Through the post experiment survey, participants responded that the simulation had limitations in realism compared to real-world environments. Future improvements should focus on more realistic physics, improved environmental effects, and expanded emergency scenarios.

CONCLUSION

This project demonstrated that XR-based digital twin training provides a safe, scalable, and effective alternative to traditional drone training methods for first responders. We found that the system improves situational awareness, supports decision-making under pressure, and reduces reliance on hazardous real-world exercises. This approach can significantly enhance first responder preparedness while minimizing risk and cost.

Future work. The project should expand the system by adding more emergency scenarios, multi-user collaborative training, and validation with real emergency response agencies.

Additional improvements should include more realistic drone physics, enhanced fire and smoke behavior, improved human target animations, and built-in scoring metrics. These improvements would make the system more realistic and useful for professional first responder training.