Real-Time Augmented Reality for large outdoor areas: Integrating decoupled camera and sensor streams for GPS-based element placement

Abstract

ABSTRACT This research addresses a significant gap in Augmented reality (AR): the challenge of placing virtual elements in a large outdoor environment using general-purpose cameras that do not have built in sensor-unit. Although modern mobile devices including smart phones, tablets have integrated cameras with multiple sensors which can track the position and orientation of the device and the camera, many practical scenarios need to use general purpose cameras including drones, cinematic cameras and web cameras. Enabling AR experience using this kind of general-purpose camera becomes challenging task due to the lack of sensor data. This research proposes a novel approach to combine the video stream from general purpose camera and a sensor data stream from a separate set of sensors which is externally mounted to the standard camera. This co-located sensor unit provides data about GPS position, acceleration and the orientation of the camera via wireless communication channel. The implementation used web-based technologies to render the AR elements including Three.js and ar.js. WebSocket base communication protocol is used for real time communication with low latency. Web based sensor APIs are used for capturing sensor data which support multiple platforms including Android and iOS. Several Algorithms are used to convert coordinates systems to map real world coordinates with the 3D virtual world coordinates. The system architecture of the implementation consists of five main components: Sensor data acquisition module, Sensor data receiving module, Camera module, Video capturing module, AR rendering engine. These components are carefully designed to prioritize Loosely coupling, Real-time performance, Adaptability and Scalability. Furthermore, implementation used buffer-based approach to smooth the sensor data values while minimizing the errors due to noise and other factors. Experimental evaluation was conducted using the implementation in a school cricket ground to assess the performance of the system in large outdoor areas. Web camera is used as the general purpose camera and mobile phone is used as a external sensor unit. The results found from the evaluation demonstrate that proposed approach can be used to place AR elements in large outdoor areas for mid-range distances with average accuracy with the inherit limitation of mobile phone GPS sensors accuracy. Technical challenges encountered during the study include the GPS sensor accuracy