Beware the Library Search My Child!

Apologies to Lewis Carroll for butchering the first line of his nonsense poem “Jabberwocky” (for a chemistry version of the poem called ”Beware the Physical Chem” click here: https://www.alphachisigma.org/Page.aspx?pid=536 ). But I do mean what I say in the title, beware of library searching. It is a frequently abused procedure in infrared spectroscopy, and if it is used improperly it can be dangerous.

Back in the day before personal computers (boy I’m giving my age away here) library searching was done by eye. Sadtler Research compiled thousands of paper copies of infrared spectra in green three ring binders, and the user had to flip through them comparing the sample to the reference visually. Thanks to modern computers the comparison job has been automated. Library search programs use algorithms instead of visual comparison to make decisions about match quality. A number, called a Hit Quality Index (HQI) is calculated for each comparison. Then, the best matches are shown in a search report.

I have seen people do a library search, look at the first result in the search report, declare “that is it” and go on their merry way without ever looking at the spectra. This is a recipe for disaster! The library search program will always give you a result, even if it is a bad result. Just because the HQI is a number that comes out of a computer does not sanctify it. Remember, computers are programmed by people, and computer programs make mistakes as easily as people do.

Another pitfall people fall into is over interpreting the HQI. When an HQI of 100 is a perfect match I’ve seen people interpret a 95 as meaning “there is a 95% probably that I identified the sample correctly, or “the samples are 95% the same” or “the spectra are 95% similar”. All of these ideas are wrong. The HQI is not a probability or a percentage, it has no units. The value of the HQI varies with a number of things including the search algorithm and spectral regions used in the search. The HQI simply orders the matches for a given search…that is all!

Another reason to beware the library search is search algorithms. The problem here is that we are trying to automate a visual comparison by substituting a calculation for it. As spectroscopists we know what the peaks mean and what noise and artifacts look like but search algorithms do not. It can happen that two spectra of radically different samples can give a high HQI if by coincidence the noise and artifacts in their spectra are similar. Additionally, the spectra of similar samples can give a low HQI if the noise and artifacts in their spectra are different by chance.

There is one simple solution to the problems of library searching, ALWAYS VISUALLY COMPARE THE SPECTRA! Look at your sample spectrum and the library matches and draw your own conclusions about what is the best match, do not rely on the HQI by itself to make the decision for you. In any competition between your eyeballs and the search algorithm, your eyeballs win. The computer is not smarter than you, it is faster than you. The purpose of a library search is to narrow down the possibilities for you. It is your job as the human being performing the library search to interpret the results to arrive at your own conclusions.

If you follow this advice, you will no longer have to beware the library search.

FTIR for Identity Testing

In many industries there is a need to identify raw materials as they come in the door, and the process is called “performing an identity” or “identity testing”. Because of its molecular fingerprinting abilities, FTIR is well suited for this type of work. Performing an identity normally entails comparing the sample spectrum to the reference spectrum of a known material.

Recently there have been moves afoot, particularly within the pharmaceutical industry, to use near infrared (NIR) spectroscopy or Raman scattering in place of FTIR for identity testing. Any identity testing technique should ideally be fast and easy, specific, and widely accepted in industry. Obtaining NIR spectra can be fast and easy, but this technique fails the specificity test. The features in NIR spectra are broad and diffuse, few in number, and difficult to interpret. Also, in general only functional groups containing O-H, N-H, and C-H bonds are observed in the NIR. These spectra simply lack the specificity we need in an identity testing method.

Recently developed handheld Raman scattering spectrometers (www.ahurascientific.com) make obtaining Raman spectra fast and easy. Raman spectra contain many sharp peaks and give much of the same information as infrared spectra, so they have the specificity needed for identity testing. However, the use of Raman spectra for identity testing is in its infancy, there are few libraries of Raman spectra available, and the technique certainly is not yet widely accepted in industry.

If two molecules have different chemical structures they have different infrared spectra…you can’t get more specific than that. From my own observation FTIRs are already used for identity testing in thousands of companies in dozens of industries, so the technique is widely accepted. In the past the main criticism of FTIR for identity testing was that the sample preparation is not fast and easy. The development of diamond ATRs has solved this problem (see previous blog posts). An FTIR equipped with a diamond ATR accessory can obtain spectra on powders, solids, liquids, and polymers in a matter of seconds. If one desires to take the instrument out of the lab to the loading dock where the sample is, some lab FTIRs such as the Bruker ALPHA that I use (http://www.brukeroptics.com/alpha.html ) are portable enough and rugged enough to be put on a cart and wheeled up to the vessel containing the sample. If one wants to take portability to its extreme, hand held FTIRs do now exist (see previous blog posts and http://www.a2technologies.com/index.html ). FTIR now satisfies all criteria for a perfect identity testing tool. Why would you bother using any other technique?

Debunking FTIR Myths II: “You Can Make a Subtraction Say Anything You Want”

Hi Folks. I’ve been away from blogging while enjoying the holidays but now I am back. I’m going to start off the New Year talking about a pet peeve of mine: FTIR myths. These are totally unfounded pieces of “wisdom” that I hear all too frequently from FTIR users, and worse yet from people who have never touched an FTIR. After looking at the title of this blog post you may be asking yourself, “Where’s Debunking FTIR Myths I?” It’s already been written, but was not given the proper title because at the time I did not realize this was going to become a series. The first entry in this series was my most recent blog post originally entitled “FTIR vs. GC-MS Smackdown.” It is now called “Debunking FTIR Myths I: “FTIR Can’t Identify Things in Mixtures.” Go read it now if you haven’t already.

This post’s myth has to do with spectral subtraction. Spectral subtraction makes use of specialized software to simplify mixture spectra and make mixture analysis easier. The attached figure illustrates the utility of spectral subtraction. The bottom spectrum in red is of the amino acid glutamine dissolved in water, and the middle spectrum in blue is of pure liquid water. The glutamine peaks in the bottom spectrum are very small and masked in part by the strong, broad absorbances of liquid water. It would be hard to tell from the bottom spectrum whether there was anything dissolved in the water at all. Using a spectral subtraction program I subtracted the spectrum of pure liquid water, known as the reference spectrum, from the sample spectrum. The subtraction result, in green, is at the top of the figure. The water peaks have now either been removed or greatly reduced in size, simplifying the mixture spectrum. The glutamine peaks are now apparent for all to see. It would be difficult or impossible to identify the glutamine in this sample without the use of subtraction.

The criticisms I hear of subtraction are that its “arbitrary”, that you are “adding something” to the spectrum when subtracting or that you can “make the result say anything you want.” This is all baloney. Spectral subtraction is like anything else in life, if it is USED PROPERLY it has great utility and legitimacy of purpose. The “arbitrary” critique probably stems from the fact that subtraction involves user interaction. Most of the time when two spectra are being subtracted from each other their absorbances are different because of differences in pathlength or concentration between the two samples. To compensate for this the absorbances of the reference spectrum are multiplied times a number called the “subtraction factor” or “scale factor” which adjusts the reference spectrum absorbances so they match those of the sample. When the subtraction is performed the reference spectrum absorbances subtract out. There is nothing arbitrary about adjusting the subtraction factor; there are rules and procedures to follow that are clearly described in books and training courses on FTIR (including my own). If done properly, there is nothing arbitrary in the setting of the subtraction factor.

The “adding something” myth is the most ridiculous. Subtraction and addition are completely opposite mathematical operations. You can’t “add” something to a spectrum by subtracting something from it; nothing is added during a spectral subtraction. The “you can make a subtraction say anything you want” criticism is the most damning, and the most wrong. If performed properly a spectral subtraction will allow you to more clearly see that which was already present in the sample. In the attached example by following proper procedures I was able to decide upon the best subtraction factor (0.96 in this case), and the result is that the glutamine peaks that were difficult to see before are now obvious. I didn’t make the result say anything I wanted, the result simply shows me more clearly that which was already present in the spectrum.

In the hands of a properly trained user spectral subtraction is a legitimate tool to simplify mixture spectra and make mixture analysis easier. Books and training courses to help you to perform subtractions properly exist (www.spectros1.com), so there is no excuse for not using subtraction in your work, and there is certainly no excuse for badmouthing a useful spectroscopic technique.