Multimodal Input Surveillance & Tracking
Challenge + Solution
Challenged by the Department of National Defence to develop a novel aerial surveillance solution using artificial intelligence,the TerraSense development team delivered a prototype capable of real-time object detection and tracking using deep learning and multi modal data fusion on the edge. The TerraSense MIST solution analyzes data from both
colour and infrared (EO/IR) sensors already on board existing surveillance aircraft to provide intelligence, surveillance, and reconnaissance (ISR) capabilities using deep learning.
Our team partnered with leading researchers from the University of British Columbia to create a new data fusion architecture that enables MIST to operate on both EO and IR sensors simultaneously. An ensemble of neural networks interprets the video and detects, identifies, and tracks targets of interest in real-time, increasing operator situational awareness and enabling rapid post-mission search, retrieval, and playback.
MIST addresses a gap in existing ISR capability and effectively solves three key points in the IDEaS Challenge by detecting, recognizing and tracking target objects as they travel across the camera field of view, and generating and saving the resultant metadata to enable rapid search and retrieval of video clips. As Canada and our allies continue to increase aerial ISR capabilities, a great volume of video data is being collected from EO/IR sensors at a rate that exceeds human ability to process it. MIST generates scalable data bases of meta data to allow rapid search, sorting, and retrieval of this data – for instance, an analyst can rapidly query the data base for videos of a specific target within a geographical area and timeframe from any number of airborne sensors across multiple Canadian and allied operators.
Delivered to the DRDC at solution readiness level 6 and tested on real data, TerraSense has continued to refine the MIST 3D synthetic data process and has shared our findings with our academic and industry partners and competitors, contributing to a positive change in scientific knowledge. MIST 3D has the unique ability to impact the timeliness of the ISR cycle, reducing the delay between the identification o fa priority intelligence requirement and the deployment of the neural network in the collection phase.
MIST Models synthetic data enables MIST t to exceed 92% accuracy with only 300 real images.
Satisfies all ISED eligibility criteria for Investment Framework &
General/R&D/ Venture Capital ITB Investments,100% CCV
Addresses some of Canada’s important emerging KICs including remotely piloted systems, AI, and autonomous technologies
Contributes to many of the Value Proposition themes (R&D, Skills Development, SME, Exports)
Contributes to Canada’s Knowledge-Based Economy goals and
the goals set out in Strong, Secure, Engaged
Proactive member of industry/academia/government collaboration intended to ensure that Canada continues to innovate in KICs
Provides technologies that contribute to the efficiency & performance of our national defence, security, sovereignty & surveillance
Delivers competitive advantages for both Canadian and international aerial platforms and ISR solutions