How to Look at Spectral Images?
A Tentative Use of Metameric Black for Spectral Image Visualisation
The original article on my blog website here.
Visualizing spectral images is difficult, often the spectral images are visualized as color images, but in this case, the information is limited. Because in the metameric case sometimes the colors are not distinguishable. For that, we can use a component that exists in the spectral images called the metameric black. This component contains some interesting information that we believe will help us to visualize the missing information of the spectral image. We use this information to propose a visualization that emphasizes metameric differences in colors. But the metameric black will have some negative values that we should project to something positive and more clear. So this will be a challenge to find ways to project this metameric black and to choose which approach is the best.
In industry or public, we are often faced with visual perception problems. We can not make a difference in colors between some objects, knowing that they are different materials or different colors. The problem here goes back to metamerism.
In colorimetry, metamerism is a perceived matching of colors with different (nonmatching) spectral power distributions. Colors that match this way are called metamers
Because under some illuminants we can not see the real colors if the illuminants are not sensitive to that color. Actually, there is a difference but we as humans can not see it. Even machines can not differentiate if we leave the images as they are. The only way to see the difference is to extract a component called metameric black that could help us to see the difference. So if we could see (visualize) the metameric black then the problem should be solved. This metameric black has a lot of negative pixels which makes it a challenge to visualize it, and we believe that these negative pixels are the solution to our problem.
In this project we will implement the method made by Thomas and Hardeberg then we will try to improve the results using the metameric index.
To make this job working, we need some tools. At the beginning the image is a reflectance image, then we multiply it by the illuminant and the color matching function to create the radiance image. Because the work that we are doing will be on this image.
For that in this section, we will talk a little bit about these tools.
Types of images
In this project, we will deal with different sorts of images. There is the reflectance image which is what will begin with, the radiance image which is what we see, and finally the spectral image which contains all the information that we need.
Reflectance is the proportion of the radiation striking a surface to the radiation reflected off of it. Some materials can be identified by their reflectance spectra, so it is common to correct an image to reflectance as a first step toward locating or identifying features in an image.
The reflectance image that we will work with is in figure 1.
To get the radiance image N, we need only to multiply the reflectance image R by an illuminant I. And here we can choose multiple illuminants to see which one is the more sensitive for our case. Then, we multiply the result by the color matching function C. For that, we have the real equation to have the radiance image with continuous values and another equation for the discrete values.
Here is the equation for the continuous values:
Color matching function
The CIE’s color matching functions are the numerical description of the chromatic response of the observer (described above). They can be thought of as the spectral sensitivity curves of three linear light detectors yielding the CIE tristimulus values X, Y, and Z.
Or we can call it, the observer. Because we will project each pixel of the radiance image on this function to obtain the right color of that pixel, depending to its wavelength. We use this color matching function, to obtain the tristimulus values of the image that we have so that we can convert it into RGB to visualize it.
The metameric black is the most important component in our project because it is the only component that could help us to differentiate between materials or we can say colors. In the beginning, we can’t see it as we said because our eyes are not sensitive to it. This goes back to its negative pixels. In the beginning, we said that when we see the same color in different material means that we are looking at the fundamental metamer ‘N*’. Then there is the radiance image which has the role of the spectral image ’N’. Knowing that the spectral image contains both components, fundamental metamer and metameric black ‘B’. So if we need to extract the metameric black all we need to do is to subtract the fundamental metamer ‘N*’ from the radiance image N.
Experiments and Results
Now we will talk about the algorithms that we used in this project. The data that we have is only one spectral image with 77 channels, this image contains 24 cows and each cow is divided into two. Each part has a different material or we can say different colors. But as we said, here we have the metamerism problem which makes us see the exact same color in the two parts. To visualize this image (before the processing) we need to build its RGB image, and to do that we should find the tristimulus values of this image which are x, y and z values, or we can call them the chromatic values in the colorimetry.
The RGB image
First of all, we will create the reflectance image by putting the 77 channels together. Then we should multiply each pixel of the reflectance image by a matrix constituted by the illuminant and the color matching function.
After doing that, we will obtain the XYZ image, then to find the RGB image, we need to multiply the XYZ image by the Conversion matrix to sRGB for the D65.
Finally we have got the following RGB images:
Here you can see that we have 2 results, the first one is the RGB image under the D65 illuminant and the second one is under the A illuminant. What does it mean? You remember that we said to build the tristimulus values we should multiply the reflectance image by a matrix that contains the illuminant. So here we can choose any illuminant we want and we see which one is better than the other. In this case, you can see that the image under D65 illuminant we can’t see any difference between the two parts of the cows, against the A illuminant we can see already the difference between the parts of some cows for example the cow number 2 or 3.
Visualizing the fundamental metamer
The metamerism phenomenon means that we have the same tristimulus values of the 2 materials, but actually, because we are looking at a common component which is called the fundamental metamer. Let’s take an example, we have two spectral images which have the same colors (but in reality they are different). We call them N1 and N2 referring to their radiance images, it will be like this:
We, as humans and ever machines, will see only the N* For that we see the same colors in both images. By knowing that B1 and B2 are not visible means:
which means that the radiance image of both images equal to the same fundamental metamerism.
V means We are talking about the visualization because in reality there are some values that we should add to the fundamental image to get the radiance image, these values are the metameric black.
If we want to find the fundamental metamer values, then we need to project the radiance image by multiplying it by an orthogonal matrix R. This matrix is constituted by the color matching function C.
Here is the image of the fundamental metamer:
You can see that we got the same result as the radiance image because we said that they are equal when talking about the visualization.
Visualizing the metameric black
Now comes the most important part of the project which is how to visualize the difference between the two materials. As we were saying in the last section, when we have the metamerism problem means that the radiance image which is the spectral image will appear the same in both images (the 2 parts of the cow) because it has the role of the fundamental metamer.
To see the difference between materials, we need to visualize the metameric black which is the only component that has the important information including some negative values (figure 3).
Subtracting the metameric black
Going from equation 1, having the radiance image and the fundamental metamer we could find the metameric black values.
After that we subtracted the fundamental metamer from the radiance image, we have got the pure metameric black with a lot of negative values which make it impossible to be visualized as a normal color. Here is the RGB image of the metameric black with the negative values:
Visualizing the metameric black
As you can see, the image is worse than it was, because of the negative pixels. Now comes the challenge, how can we visualize these images with normal colors, although from now we can see that there is a difference between each part of the image. But to do the analytics we need to project these negative values into something that we can see clearly. For that, we have 2 approaches that we tried and they actually worked. The first approach is to square all the pixels so that we will have only positive values. So after trying that here is the result that we have got:
We tried a second approach, which is to find the absolute values of the pixels so that we won’t touch the positive values and like this, we save the maximum information of the image. After doing that, here is the result that we have got:
Using these two methods, you can see that the metameric black is more clear then before and the important thing is that we can see the difference between each part of the cow for almost all the cows which is what we were looking for. However, we wanted to optimize the approaches that we found by adding an additional step to them.
Note: you should know that when we did that two approaches, we applied the algorithms on the pure metameric black, which means we applied it on the image with 77 channels then after finding the result we converted it into RGB using the same method that we used to find the radiance image and the fundamental metamer. So here comes the idea, we said why don’t we apply the algorithms (the approaches) directly on the tristimulus values, then we normalize the values.
Here is the result that we have got after doing that:
You can see that we have a unique result for each approach we tried. And if we talk about the colors, we can see that there are differences between each part of the cows but there is no unique color. What I’m saying is, if we look at the third cow for example, so you can see that its colors change for each approach, but it doesn’t matter because our goal here is to justify that they are two different materials, not the exact colors.
Until now, we can’t say which is the best approach between what we tried even if we can see that the last one is more clear than the others. But we should find a better way to analyze these results. That is what we will talk about in the next section.
For this part, we should select one of these approaches that we tried so that we can apply it in the industry. But we can not choose randomly. For that, we found a method that we will apply to each result to decide which one is the best.
The solution here is to calculate the metameric index for all the cows and on each result. What does it mean, we will calculate the distances between the 2 parts of the cow and if the distance will be big that means that the materials are very different else they ain’t. The approach that allows us to find the big distance for most of the cows is the best.
To do this we chose to take a small rectangle on each part of the cow then we calculate the average of this rectangle and using this mean we calculate the distance.
As you can see in figure 11, these are the rectangles that we will use to calculate the averages. After that we applied the algorithms on all the cow for the 3 approaches, here are the results that we have got:
After calculating the distances for all the results, you can see clearly which approach was the best. I’m talking about the third one. We can see that for almost all the cows, the distance is big which average that we could differentiate between the parts. Let’s take the first cow for example:
This is the same cow which means the same materials but the result differs for each approach we tried. We won’t talk about colors that changed because it doesn’t matter as we said, but talking about distances, we can see that for the third approach we have the big one and the same thing for almost all the cows. For that, we will choose the third approach that responds to our needs.
We could find a solution for the metamerism problem, and it worked for our case and we believe that it will work for another type of material. The thing that you should understand is that the metameric black is the component that has the solution. Then it depends on you and which method you want to use to visualize it. Of course, there are other ways that you can find to project the metameric black so that you won’t have the negative values. And finally, you can test the approaches using the metameric index so that you validate what you found.