Prism-based cameras for high fidelity colour and NIR inspection
Prism-based sensors in multispectral cameras
Machine vision systems have evolved over time, from systems using monochrome cameras to systems using RGB broadband colour imaging and then to systems using several narrow bands in the visible and Near Infrared spectrums. This evolution has allowed machine vision systems to perform more and more sophisticated inspection and analysis.
The colour sensing in most 2D industrial cameras used in the machine vision industry is largely based on Bayer mosaic pattern sensors and on single line RGB, dual line RGB or trilinear RGB sensors for linescan cameras. While these reproduce colour to varying degrees of fidelity they are not enough to carry out some inspection tasks. Some applications demand unconventional RGB wavelength bands while others demand a combination of RGB and non-visible wavelengths. Others require exclusively non-visible wavelengths such as UV, NIR or SWIR, with no wavebands in the visible spectrum.
Complex metrology and imaging applications sometimes demand larger numbers of spectral channels or to create bespoke band combinations. With the traditional machine vision industry merging with intricate measurement technologies, consistent, reliable, high-fidelity colour and multispectral imaging are playing key roles in industrial quality control.
Multispectral imaging is often misrepresented because there is no commonly agreed definition in the imaging community. The literal meaning is imaging with more than one spectral band. By this definition, even an RGB camera falls under the category of multispectral imaging. A quad-band RGB-NIR camera as another example also falls under the definition.
But when does a multispectral camera have too many bands and when is it better classified as an hyperspectral camera? This definition is perhaps even less clear. There is perhaps a loose understanding within the imaging community that imaging with two to fifteen bands can be termed multispectral. Some companies extend multispectral to systems having up to 25 wavebands whereas others will call a 25 band camera hyperspectral.
Multispectral imaging can apply to a camera with discrete spectral bands designed for a specific task like red-edge NDVI in agriculture and the bands do not necessarily need to be adjacent or contiguous. Red edge NDVI cameras for example only need 2 bands, one in the red spectrum and one in the NIR where plants reflect strongly, but they often have 5 bands. The 3 extra bands are often Green and Blue (so that they capture a normal RGB image) and a band on the red-edge for other indices like NDRE.
The graph shows an example of multispectral camera response with five different wavebands.
The graph shows an example of multispectral camera response with four different wavebands.
The graph shows an example of multispectral camera response with twelve different wavebands. Source: C. Godau et. al; ISBN: 978-3-942709-08-8
A key feature that differentiates Multispectral from Hyperspectral imaging is that Hyperspectral cameras have the capability to produce continuous and contiguous wavelength bands that are adjacent and cover a full spectral range. For example a VNIR hyperspectral camera with silicon sensor might have 224 bands with the first band starting at 400nm and the last band at 1000nm, and every other wavelength in between covered by the other bands. It is also worth pointing out that Hyperspectral cameras can often be setup to output with spectral regions of interest and so can be made to behave like a Multispectral camera with non contiguous bands.
Hyperspectral imaging provides a continuous range of spectra.
Multispectral imaging consists of spectral bands which are discretely positioned from each other.
Multispectral imaging applications
Multispectral imaging enhances inspection capabilities in several applications such as agriculture, medical and industrial applications using machine vision cameras.
In agriculture, multispectral images are an effective tool for evaluating soil productivity and analysing plant health. Multispectral sensor technology allows the user to see what the naked eye cannot. Apart from estimating crop yield, multispectral imaging can help to identify damaged crops and so allow a grower to make modifications to crop-growth management. Identifying weeds, disease and pests with multispectral imaging is utilised for early detection and helps to optimise resources and avoid loss. For inspection of fruit and vegetable produce, multispectral imaging can provide a combination of visible and non-visible wavelengths to measure extrinsic features like colour and texture and intrinsic features like ripeness and moisture content at the same time.
Multispectral imaging enhances inspection capabilities in several applications such as agriculture.
Using multispectral inspection techniques can reveal unwanted dirt particles (see the NIR image), and help to ensure the right quality of spinach leaves before they are packed.
In this example, foreign objects are identified and located during the inspection of hazelnuts. The difference between foreign objects and hazelnuts is far less obvious in the RGB colour image.
In medical applications, combining colour imaging with NIR bands can help to locate and distinguish between tumours and surrounding tissues. For endoscopic surgical imaging, two or three images of different wavebands can be captured simultaneously and fused such that non-visible NIR channels are overlaid on the visible RGB image to provide an augmented view of tissues or blood vessels.
In a real surgical situation, ICG might be injected into blood vessels, tissue or lymphatic vessels. With real-time video images overlaid on the visible image, surgeons could use fluorescence to locate tumours/glands for removal, highlight key vessels, and/or monitor blood flow while operating.
Other applications include pharmaceutical tablet manufacturing, where NIR wavelengths assist in imaging through blister packs for identification and quality analysis. In PCB inspection, simultaneous imaging of RGB and NIR wavebands is used to inspect surface components and buried copper or gold conducting lines. In textile and printing inspection, multispectral cameras help to reproduce and measure accurate colour and identify garment materials such as leather, vinyl and polyester.
Multispectral Camera Technologies
There are varied technologies used to produce multispectral cameras and the following represents the main ones.
Perhaps using multiple cameras with bandpass filters is the simplest but at the same time the most difficult. The construction of a multi camera system is relatively simple but combining spectral data into a single image from multiple cameras is tricky: there is parallax because they are physically displaced due to their size; each has their own optic with its own unique distortions. Correcting for these to create a spatially correlated unified image requires a great deal of effort.
Filter-wheel multispectral cameras capture spectral images sequentially in time by rotating filters in a wheel mounted in front of the sensor or lens. This method removes the complication of parallax and distortion. Filter wheels typically support up to 12 filters and give the camera full spatial resolution per band. Drawbacks include slow imaging speeds and the inability to deal with motion, and a mechanical element that will require maintenance.
A multispectral camera using filter wheel captures multi-spectral images. This is done by rotating the filter wheel mounted in front of the lens or in-between the sensor and the lens.
Pixelated multispectral filter arrays (MSFA) extend the concept of the Bayer mosaic filter to acquire multispectral images in one shot. These snapshot mosaic sensors can support between 4 and 25 channels. Their advantage is that they are small and a single image captures all bands, so they can run at high speeds e.g. 60Hz. Ideal for drones for example. The disadvantages are that to get 25 pixels for example, they repeat a 5x5 pixel filter pattern over the sensor and so adjacent pixels with the same wavelength filter are spatially displaced by a minimum of 5 pixels (sparse sampling) making them fairly low resolution. They also have a great deal of spectral crosstalk due to two factors: the filter wavelengths overlap significantly; non-orthogonal photons hit a pixel and burrow into a surrounding pixel. Nevertheless these multispectral cameras serve a purpose for the right applications.
With snapshot mosaic sensor based cameras, it is possible to acquire a multispectral image in one shot. However, multispectral demosaicing is a challenge due to sparse sampling of spectral bands in the filter array.
Beam splitter systems introduce an element that can simultaneously capture images on multiple cameras from common optics. This alleviate image registration issues but results in a physically large and expensive system with light intensity loss.
This multispectral imaging technique uses a beam splitter. Hence, it is possible to simultaneously capture images using multiple cameras.
In this method, all the optics including the lens are common for both the sensors, unlike the previous method which uses two individual cameras with individual lenses but a common beam splitter.
Multi-sensor dichroic prism cameras mount sensors directly to prism faces, resulting in a size reduction over multi-camera beam splitter systems. Prism blocks use hard dichroic coatings to direct spectral ranges to each sensor. Each channel receives the full amount of light for its range, and full spatial resolution per waveband is achieved.
In prism-based sensors, the prism blocks consist of hard dichroic coatings which are interference filters by nature. These filters are responsible for the primary separation of incoming light.
Line scan cameras with multi-line sensors are also used for multispectral applications. A quad-line sensor can consist of R-G-B-NIR. The most popular cameras have 8 to 16 lines where each line has a unique spectral band-pass filter. The pushbroom method, traditionally used in hyperspectral cameras, can also be applied to multispectral imaging. This allows for full spatial and spectral information to be captured line by line, though speed decreases as the number of channels increases.
Line scan cameras with multi-line sensors can be used for multispectral applications, where each line of pixels has a unique spectral band-pass filter.
This method uses line scan sensors where the horizontal resolution of the sensor can be divided into multispectral domains by adding additional filters in the optical assembly. Here a trilinear sensor is divided into three spectral separations resulting in a 9-channel multispectral camera.
It is possible to do multispectral imaging with the Pushbroom hyperspectral camera technique, where full spatial and spectral information is captured line by line.
Key selection considerations
Find the Right Multispectral Solution
Every inspection task is unique. Our engineering team can help you determine whether a prism-based sensor, filter-wheel, or multispectral filter array is the most cost-effective choice for your project.