Abstract

The burgeoning demand for enhanced digital imaging solutions has catalyzed the integration of image sensors (IS) across various applications, such as smart cities, intelligent transportation systems, and robotics. The global IS market, valued at over $26 billion in 2022, is projected to reach approximately $38.6 billion by 2027. Leveraging the capabilities of visible light communication (VLC) and OCC technologies, this paper explores innovative methods for data transmission, tracking, and positioning using commercially available cameras. The dual utilization of ISs for image capturing and data communication provides significant opportunities for low-data Internet of Things (IoT) applications. This paper discusses the operation modes of ISs, particularly focusing on rolling shutter and global shutter mechanisms, and highlights challenges such as noise interference, synchronization, and non-linearity. Additionally, the study emphasizes the necessity for novel system designs, advanced signal processing techniques, and routing algorithms to enhance the efficiency and reliability of OCC systems. Ultimately, the work aims to pave the way for integrated sensing, positioning, and communication systems that can adapt to diverse environments and operational requirements.

Keywords: Optical Camera Communication (OCC), Image Sensors (IS), Visible Light Communication (VLC), Smart Cities, Internet of Things (IoT), 3D Positioning, Integrated Sensing.

Introduction

There are several applications that drive the use of digital cameras, including smart cities, smart devices and phones, intelligent transportation systems, robotics, medical, visual surveillance systems, among others. The image sensor (IS) market was valued at over $26 billion in 2022 and is projected to reach $~38.6 billion by 2027 [1]. The utilization of IS has not only revolutionized the paradigm of capturing and sharing images and videos but also has extended its application in data transmission, sensing, tracking, and positioning as well as into integrated sensing-positioning communications. In addition, estimating 3D pose of objects is desirable in many applications including human-robot interaction, manufacturing automation, intelligent transportation systems (i.e., autonomous vehicles, vehicles/drones platooning, etc. VLC employing LEDs as Txs and complementary metal-oxide-semiconductor ISs (or cameras) as Rxs is best known as OCC since 2010. In recent years, we have seen growing research and development activities in VLC-OCC technology as a promising solution for the next generation of wireless communication networks (i.e., sixth generation and beyond). More specifically, the availability of ISs in pervasive consumer electronics has created significant opportunities for the practical application of VLC-OCC by offering many interesting features as outlined in Fig. 1(a).

The practical applications of VLCOCC using cameras with a resolution on the order of megapixels are best used in low data IoT and Internet-ofEverything (IoE) in applications, see Fig. 1(b). Systems like QR codes and augmented reality allow people to access additional content in the virtual realm using their smart devices. Additionally, OCC uses advanced image processing for shape recognition and estimating depth perception [2], [3]. OCC can also be used for indoor positioning with higher accuracy (sub-centimeter). For example, 3D positioning is achieved using dual cameras on smartphones to capture images, whereby determine the distance from the light source to the Rx by comparing the disparity of corresponding projection point from each camera. RF-based technologies such as WIFI, Bluetooth and near-field communications are currently used with limited transmission capabilities [4].

The typical camera consists of a two-dimensional array of photodetector pixels that can classify multiple spatially separated light sources with a high level of accuracy. The devices are comprised of an imaging lens, an IS, colored filters, and a readout circuit for capturing images in the form of single or multiple frames and converting them into intensity (grayscale) values based on the region of interest partitioning and pixel sampling. The integrated image processor uses a demosaicing algorithm taking advantage of the built-in color filter array such as a Bayer filter to produce a colored output image, which acts as the data image for subsequent post-processing. Background lights can be easily suppressed by using band-pass optical filtering in conjunction with frame differential techniques. The IEEE 802.15.7m [5] outlines a standard that supports OCC functionalities and MAC modifications.

State of the art

In the ISs there are two modes of operation (i) global shutter – where an array of sensors is simultaneously exposed, with each pixel’s information being read out sequentially. Using this mechanism, high-quality images and moving objects can be captured and processed; and (ii) rolling shutter – where pixels in a column-bycolumn format or in a row-by-row format are sequentially exposed to the light to create an image. With the rolling shutter mode, the Rx can sample at higher rates, which results in increased data rates, enabling multiple LED states (ON/OFF) to be captured at the same time, where the captured image contains a collection of black and white stripes representing LED flickering. Note that the captured strip widths and their numbers are determined by the modulation frequencies and the distance between the camera and the light sources, respectively. The maximum symbol rate of commercial cameras is typically less than 15 sps, which is insufficient for some applications due to their low frame rates (i.e., about 30 frames per second).

In addition, in RS-based OCC, the data rate is dependent on the camera's pixel clock, frame rate, and exposure time. The first two parameters are directly dependent on the IS technology, whereas the latter, defining the bandwidth of OCC, can be controlled by the user. Therefore, the shorter the exposure time, the higher the bandwidth. Note that short exposure times, however, result in low SNR and, consequently, higher bit error rates (BERs). To reduce image processing time (i.e., latency), a combination of an automatic location-based system and an efficient segmentation algorithm can be used. As part of OCC, the region of interest is critical for identifying the communication region of bright and dark stripes, which directly impacts both the demodulation and decoding of the captured images into digital format. Complex algorithms are required for decoding optical information into digital data. Note, in OCC, data evaluation can be performed using image processing tools in the software domain (i.e., MATLAB, OpenCV and Python).

Furthermore, low frame rates result in flickering, which is not desirable since the critical flicker frequency is usually 100 Hz for human eyes. Low frame rate OCC systems can use both standard modulation schemes (e.g., OOK, FSK and multicarrier modulation) as well as signage cipher modulation for integrating data communication with cinematic contents and digital advertising, colour shift keying (CSK), optical SM, multilevel intensity modulation, distance colour-coded OOK [6][7][8] [9]. Of these, CSK has received the most attention for enhancing the transmission data rate and was originally proposed for VLC in the IEEE 802.15.7 standard. For short-range IoT applications, reliable, robust, flicker-free and low data rates (few kbps) OCC links are more important than the high-speed system. However, to increase the data rate, a MIMO-OCC system using an array of red, green and blue LEDs is one possible option, see Fig. 2. Equalization is needed to compensate for distortions experienced by the propagating optical signals over the free space channel. Several techniques have been proposed and utilized in OCC including (i) an artificial neural network equalizer; (ii) predictive equalization to deal with changing light intensity; and (iii) double-equalization to deal with spatial and time dispersions. In OCC, noise increases with the sensitivity setting in the camera, the exposure time, temperature and even varies amongst different camera models. In digital cameras there are three types of noise sources: (i) random noise (short exposure time and high ISO sensitivity) – This is characterized by fluctuations in intensity and color above and below the actual intensity of the image; (ii) fixed pattern noise (low ISO sensitivity and long exposure time) - Because the intensity of a pixel is far greater than that of the ambient random noise fluctuations; and (iii) banding noise (high ISO sensitivity) - This is a highly camera-dependent feature, which is introduced by the camera when it reads data from the IS.

Challenges and future work

OCC offers many appealing features; however, the nature of image-based communications still poses several challenges associated with the devices design, advanced signal processing, interference suppression and system/network protocol designs. These include: ▪ New system models – Most models are based on PD Rxs, which do not truly represent the IS based Rx. It should also consider (i) interference and noise due to solar irradiance, streetlights and advertising boards, which will degrade the performance seriously, and even cause saturation and blinding of the IS; and (ii) the effects of radiation interference, atmospheric conditions, presence of suspended particles in the air and the temperature variation.

▪ Capturing a small size light source in RS-based camera – The scan rate is faster than the frame rate is slower than the scanning rate by several rows vertically aligned. Therefore, portions of vertically aligned rows not capturing a light source have no contribution to captured data signal.

▪ Nonlinearity induced distortion – During the design, optimization and practical implementation processes of intensity modulation VLC-OCC links, the nonlinearity of light sources (LEDs and screen pixels) must be compensated using pre-distortion mechanisms and gamma-correction avoidance.

▪ Synchronization – This is an essential component of OCC systems, just as it is in digital transmission, particularly when dealing with high-frame-rate cameras and asynchronous protocols, which require careful design to ensure accurate data transmission. Therefore, a dedicated protocol with a timestamp for alignment between the Tx and Rx is needed. Such a protocol must be scalable and adaptable to various OCC setups. It is also necessary to adjust the data payload size by designating additional bits for synchronization based on the synchronization requirements.

▪ Energy usage – This is typically higher for cameras compared to a photodetector, thus shorter battery life. ▪ New topologies and routing algorithms – To achieve large-scale VLC-OCC networking.

▪ Data throughput - OCC systems have relatively low data throughput due to the low frame rates of common cameras. which is inversely proportional to the exposure time (i.e., the exposure duration per frame of the IS). High frame rate cameras can be used to increase the throughput but are costly.

▪ Broadcast transmission mode - Existing OCC systems transmit data in this way, only allowing reception of the transmitted data provided that the Rx is within the field of view of the Tx. In addition, it is challenging to establish bidirectional communication.

▪ Blurring and blooming effects - (i) Internal electrical noise, photon overflow and external ambient light lead to fringes blurring, where bright and dark fringes are difficult to distinguish with the increasing distance that need addressing to achieve longer range; (ii) while upper limits of exposure and camera gain values cause blooming effects with over exposed incident light on image pixels; and (iii) due to a camera not being focused leading to signal-to-noise ratio degradation and poor spatial separation of source signals.

▪ Blocking/shadowing and atmospheric conditions - Further research is needed to determine the impact of these on the performance of OCC links. Consequently, developing an adaptive OCC link could provide one option for dynamically adjusting the communication path or signal strength based on the environment.

▪ Integrated sensing, positioning and communication technology – This offers two primary advantages over dedicated sensing, positioning and communication links: (i) integration gains - from the efficient use of congested wireless/hardware resources; and (ii) coordination gains - to balance multiple functional performance or the execution of mutual assistance.

Conclusion

The advancement of Optical Camera Communication (OCC) presents a transformative approach to digital communication and sensing technologies. Despite its potential, several challenges must be addressed, including noise suppression, synchronization, and the design of new system models to effectively leverage the capabilities of image sensors. The integration of advanced signal processing techniques and innovative routing algorithms will be crucial in improving data throughput and reducing latency in OCC systems. As the market for image sensors continues to expand, this paper underscores the importance of continued research and development efforts to refine OCC technologies for practical applications in various domains, particularly in IoT and smart environments. Future work will focus on creating adaptive OCC systems that can dynamically adjust to changing conditions, enhancing both the quality and reliability of data transmission.