Fusion of colour and monochromatic images with edge emphasis

Abstract

We propose a novel method to fuse true colour images with monochromatic non-visible range images that seeks to encode important structural information from monochromatic images efficiently but also preserve the natural appearance of the available true chromacity information. We utilise the β colour opponency channel of the lαβ colour as the domain to fuse information from the monochromatic input into the colour input by the way of robust grayscale fusion. This is followed by an effective gradient structure visualisation step that enhances the visibility of monochromatic information in the final colour fused image. Images fused using this method preserve their natural appearance and chromacity better than conventional methods while at the same time clearly encode structural information from the monochormatic input. This is demonstrated on a number of well-known true colour fusion examples and confirmed by the results of subjective trials on the data from several colour fusion scenarios.

Introduction

The goal of image fusion can be broadly defined as: the representation of visual information contained in a number of input images into a single fused image without distortion or loss of information. In practice, however, a representation of all available information from multiple inputs in a single image is almost impossible and fusion is generally a data reduction task.  One of the sensors usually provides a true colour image that by definition has all of its data dimensions already populated by the spatial and chromatic information. Fusing such images with information from monochromatic inputs in a conventional manner can severely affect natural appearance of the fused image. This is a difficult problem and partly the reason why colour fusion received only a fraction of the attention than better behaved grayscale fusion even long after colour sensors became widespread.

Fusion method

Humans tend to see colours as contrasts between opponent colours and an improvement in visibility of structures from the monochrome can be achieved when they are used to encode a single HVS colour dimension consistently. The lαβ colour system effectively decorrelates the colour opponency and intensity channels and manipulating one causes no visible changes in the others. Colour fusion can be achieved by fusing one of the colour opponency channels with the monochrome image. We use the Laplacian pyramid fusion known to be one of the most robust monochrome fusion methods available. The Laplacian, also known as the DOLP (difference of low-pass) pyramid is a reversible multiresolution representation that expresses the image through a series of sub-band images of decreasing resolution, increasing scale, whose coefficients broadly express fine detail contrast at that location and scale. A simple fusion strategy creates a new fused pyramid by copying the largest absolute input coefficient at each location.

The β channel of the lαβ space represents the red-green opponency and we base our fusion on encoding this channel of the colour input with the monochrome image. This causes warmer objects (lighter in IR) to appear redder in the fused image. The fusion proceeds in several steps. Initially we transform the colour input RGB image into the lαβ image. Monochrome fusion is then performed by decomposing the β image and the normalised monochrome into their Laplacian pyramid representations. We use the select max strategy to construct the fused pyramid but we only apply this to a small number of higher resolution pyramid sub-bands. Larger scale features in lower resolution sub-band images that constitute the natural context of the scene are sourced entirely from the colour image (β). This ensures that well defined smaller objects from the IR image are transferred robustly into the fused image as well as the broad scene context from the colour input. Reconstructing the fused pyramid produces the fused β image which is combined with the original l and α channels of the colour input to produce the fused RGB colour image.

Edge Emphasis

We encode only the β channel which has only a fraction of the overall colour signal power (most is in the intensity channel) so the contrast of the monochrome image structures is still relatively modest in the fused image. We can improve their visualisation using a relatively simple effect of gradient outline enhancement. Initially, we extract gradient information from the monochrome image using 3x3 Sobel edge operators. The responses to horizontal and vertical Sobel templates, s^x and s^y, are combined to evaluate gradient magnitude at each location. To enhance the structure visualisation, prior to fusion, to the monochrome input we add its gradient magnitude image. The enhanced monochrome image  is well behaved as the used gradient filters are linear, and is used directly as the input into monochrome fusion. The gradient magnitude image effectively captures the primal sketch of the scene and encoding an opponency channel with this information improves the visualisation of the structural outline of the monochrome input in the colour fused image.

Results and Conclusion

A new “β fusion” colour image fusion method is presented that successfully both visualises important structure information from the monochrome input and preserves the natural appearance of the true colour input. Colour fusion is performed in the lαβ colour space known to decorrelate main colour opponencies seen by the human visual system. We chose the β channel representing the red-green opponency of the true colour image to encode structural information from the monochrome input by fusing them using modified Laplacian  pyramid fusion. The visualisation of important structures from the monochrome input can be improved through a simple structure encoding step using its gradient information. The method is naturally extended to video fusion. The proposed fusion methods produce colour fused images with significantly better visualisation of important information from the monochrome input while almost entirely preserving the natural appearance of the true colour input. This was demonstrated on a number of well-known colour fusion examples and measured using subjective trials on the data from multiple surveillance scenarios.