ST02 Module 1   Previous Module   Next Module   Examples   Exercises   Computing

View/Print PDF, PS

Module 1: Chemometrics and NIR spectroscopy

By Bent Jørgensen and Yuri Goegebeur

Table of Contents

1.1 The NIR frequency band

Here's looking at you, kid. [Rick Blaine, Casablanca, 1942]

We now give a short introduction to chemometrics and NIR spectroscopy, and introduce the Beer-Lambert law. More detailed introductions to NIR spectroscopy may be found in Davis (2000) or Siesler et al. (2002).

• Electromagnetic spectrum, wavelength in nm (nanometres) (from Davis, 2000):

• UV (Ultra Violet): 1-400 nm

• Visible light: 400-750 nm

• InfraRed region:
  • Near InfraRed (NIR) band: 750 to 2500 nm

  • Mid InfraRed (MIR) band: 2500-16,000 nm

  • Far InfraRed (FIR) band: 16,000-10$^{6}$ nm

• NIR wavenumbers: about 14.000-3.500 cm$^{-1}$ (see conversion table under Examples).

• NIR absorptions originate in molecular vibrations:
  • Fundamental absorptions found in MIR band (Raman spectroscopy).

  • Overtones and combinations found in NIR band (NIR spectroscopy).

• NIR spectroscopy may be divided into:
  • Transmission spectroscopy of liquids: 750 to 1100 nm band.

  • Reflection spectroscopy of powdered materials: 1100 to 2500 nm band.

• Overtones and combinations found in the NIR band provide very complex 'fingerprints' of the constituents in the sample.

• Table of characteristic wavelengths (shown under Examples) shows that one must record the spectrum at intervals of length 10 nm or less in order to pick up all the different structures.

Realizing the importance of the case, my men are rounding up twice the usual number of suspects. [Captain Louis Renault, Casablanca, 1942]

• Because of the high number of variables, such data are called multivariate, or even megavariate data (Eriksson et al., 2001).

• The different characteristics overlap and give rise to an amorphous shape of the spectrum, which is very different from a GC line spectrum or an MIR spectrum, where each feature is well defined.

• As Wold, Martens and Wold (1983) stated, we must hence utilize a combination of many spectral frequencies in order to estimate the concentrations of the constituents. This requires the use of chemometric (statistical) methods.

1.2 Other measurement techniques

Chemometrics method can be used for many other types of multivariate data, characterized by having the information spread over a range of measurements. Examples:

• Acoustic sensors.

• Mass spectrometry.

• MRI scanning.

• Multisensor data, such as the Electronic Tongue or Nose.

1.3 Absorbance and the Beer-Lambert law

• Let there be given a sample, often a compound of several constituents.

• The word sample is used here to denote a sample taken from e.g. a chemical process. This is slightly different from its use in statistics, where sample often refers to the total set of chemical samples available for analysis.

• If the sample contains only one constituent, it is called pure.

• We use analyte to indicate a particular constituent of interest.

• Absorbance of a given analyte at a particular wavelength or frequency is defined as
  $$x=-\log _{10}\left( I/I_{0}\right)$$
  ($x>0$ ). One could use $\ln$ instead of $\log _{10}$ without affecting the results. Here
  • $I_{0}$ is the intensity of the incident light, before the substance is inserted into the light path.

  • $I$ is the intensity of the transmitted light, after the substance is inserted into the light path.

• The absorbance is always positive, because $I<I_{0}$ . Beware of false light, which might in the worst case make $I>I_{0}$ and hence $x<0$ .

• For transmission spectroscopy $T=I/I_{0}$ is known as the transmittance ($0<T<1$ ).

• For reflection spectroscopy $R=I/I_{0}$ is known as the reflectance ($0<R<1$ ).

• Absorbance is a decreasing function of transmittance, respectively reflectance.

• Increasing the concentration will decrease the transmittance or reflectance, and hence increase the absorbance. So absorbance is a monotonely increasing measure of concentration.

• The following graph shows $-\ln R$ as a function of $1-R$ , where $R$ is the reflectance (or transmittance).

• Note that the graph is almost linear in $1-R$ for $1-R$ near 0, that is, for small concentrations.

• The Beer-Lambert law (simple case):
  $$x=ya,$$
  where $y$ is the concentration of the analyte, and
  $$a=\varepsilon \delta ,$$
  is the absorbance of the pure analyte; the product of
  • $\varepsilon$ : the specific absorption coefficient of the analyte;

  • $\delta$ : the path length of the light.

• Assume that $\delta$ is fixed, then $a$ is a relative measure of absorbance for the analyte.

• The Beer-Lambert law for $m$ constituents plus noise:
  $$x=\sum_{\ell =1}^{m}y_{\ell }a_{\ell }+e,$$
  where
  • $y_{\ell }$ is the concentration of the $\ell$ 'th constituent;

  • $a_{\ell }$ is the absorbance of the $\ell$ 'th pure constituent;

  • $e$ is random noise; may also reflect deviations from Beer-Lambert law.

• The Beer-Lambert law may not always hold exactly, but is a useful tool in quantitative analysis. Here is an illustration of the Beer-Lambert law.
  
• Closed system: when $y_{1}+\cdots +y_{m}=1$ , i.e. when all constituents in the compound are analyzed.

1.4 Chemometrics data

1.4.1 Spectra and concentrations

• Spectral data: absorbances $x_{1},\ldots ,x_{k}$ measured at $k$ wavelengths, numbered $j=1,\ldots ,k$ .

• Often $k$ is in the range 100-200, or even more if the spectrum is recorded at very small intervals.

• Example: The NIR spectrum of sucrose (from Davis, 2000):

• The Beer-Lambert law for $m$ constituents and $k$ wavelengths plus noise:

  $$x_{j}=\sum_{\ell =1}^{m}y_{\ell }a_{\ell j}+e_{j},$$ (1.1)

  
  
  for $j=1,\ldots ,k$ , where
  • $y_{\ell }$ is the concentration of the $\ell$ 'th constituent;

  • $a_{\ell j}$ is the absorbance of the $\ell$ 'th pure constituent at the $j$ 'th wavelength;

  • $e_{j}$ is the random noise at the $j$ 'th wavelength.

• Vector notation:
  • Vector of $m$ constituent concentrations, represented as a row vector:
    $$\boldsymbol{y}=\left[ y_{1},\ldots ,y_{m}\right]$$

  • Spectrum for compound, represented as a row vector:
    $$\boldsymbol{x}=\left[ x_{1},\ldots ,x_{k}\right]$$

  • Spectrum of $\ell$ 'th pure constituent, represented as a row vector:
    $$\boldsymbol{a}_{\ell }=\left[ a_{\ell 1},\ldots ,a_{\ell k}\right]$$
    There are $m$ such pure spectra, one for each constituent, numbered $\ell =1,\ldots ,m$ .

• In Module 2 we consider vector and matrix algebra.

1.4.2 Calibration data

• Calibration data consist of an $X$ -block and a $Y$ -block, also called a training sample.

• Assume that there are $n$ samples of varying compositions.

• $Y$ -block: concentrations measured by a reference method, one row for each sample.

• $X$ -block: spectra measured by NIR instrument, one row for each sample.

• Calibration methods study how $Y$ varies with $X$ . This results in a calibration model.

• Here we explain the method in connection with NIR spectroscopy, but calibration may, in principle, be used for linking any two given measurement methods, provided that there is linearity in the sense (1.1).

Example

• Measurement of protein and water in a sample of grain, $m=2$ .

• Assume for simplicity: $n=4$ calibration samples and $k=3$ wavelengths.

• $Y$ -block: $4\times 2$ matrix $\boldsymbol{Y}$ : protein (column 1) and water contents (column 2):
  $$\boldsymbol{Y}=\left[ \begin{array}{cc} y_{11} & y_{12} \\ ... ...22} \\ y_{31} & y_{32} \\ y_{41} & y_{42}\end{array}\right]$$
  for each of the 4 calibration samples (rows).

• $X$ -block: $4\times 3$ matrix $\boldsymbol{X}$ : absorbances at each of the three frequencies (columns):
  $$\boldsymbol{X}=\left[ \begin{array}{ccc} x_{11} & x_{12} & ... ..._{32} & x_{33} \\ x_{41} & x_{42} & x_{43}\end{array}\right]$$
  for each of the 4 calibration samples (rows).

• For example, the second sample gave the following concentrations of protein and water:
  $$\boldsymbol{y}_{2}=\left[ \begin{array}{cc} y_{21} & y_{22}\end{array}\right] ,$$
  and the following spectrum:
  $$\boldsymbol{x}_{2}=\left[ \begin{array}{ccc} x_{21} & x_{22} & x_{23}\end{array}\right]$$

• Sometimes we use column vectors, e.g.
  $$\left[ \begin{array}{c} y_{11} \\ y_{21} \\ y_{31} \\ y_{41}\end{array}\right]$$
  is the first column of $\boldsymbol{Y},$ i.e. the 4 protein contents, or
  $$\left[ \begin{array}{c} x_{12} \\ x_{22} \\ x_{32} \\ x_{42}\end{array}\right]$$
  is the second column of $\boldsymbol{X},$ i.e. the 4 light absorptions for the second frequency.

• The $X$ - and $Y$ -blocks may be joined to form a big data matrix:
  $$\left[ \boldsymbol{X},\boldsymbol{Y}\right] =\left[ \begin{... ...22} \\ y_{31} & y_{32} \\ y_{41} & y_{42}\end{array}\right]$$
  which is the way calibration data are often read into the computer.

1.4.3 Test data and prediction

• Test data consist of one or more spectra of the form
  $$\boldsymbol{z}=\left[ z_{1},\ldots ,z_{k}\right]$$
  for a sample of unknown composition.

• Based on the calibration model found in the calibration step, we may perform prediction of the unknown concentrations in the test sample, resulting in a vector
  $$\widehat{\boldsymbol{y}}=\left[ \widehat{y}_{1},\ldots ,\widehat{y}_{m}\right] ,$$
  the hats denoting predicted values.

• The calibration methods used in this course are linear, resulting in predictions of the form
  $$\widehat{\boldsymbol{y}}=\boldsymbol{z}\widehat{\boldsymbol{B}},$$
  where $\widehat{\boldsymbol{B}}$ is a matrix of regression coefficients estimated from the calibration data, see the following figure.
  
  Here the hat denotes estimated values, which means that the value $\widehat{\boldsymbol{B}}$ is calculated from the calibration sample.

• The expression $\boldsymbol{z}\widehat{\boldsymbol{B}}$ denotes a matrix product--see Module 2 concerning matrix algebra.

• There are several different calibration methods being used in practice, as explained in detail from Module 4 and on.

Example (continued)

• Consider protein and water in a grain sample again.

• Test data: absorbances at each of the three frequencies:
  $$\boldsymbol{z}=\left[ \begin{array}{ccc} z_{1} & z_{2} & z_{3}\end{array}\right] .$$

• By prediction, we obtain the vector
  $$\widehat{\boldsymbol{y}}=\left[ \begin{array}{cc} \widehat{y}_{1} & \widehat{y}_{2}\end{array}\right]$$
  which consists of the predicted concentrations of protein and water, respectively.

Bibliography

1

Abdi, H. (2004). Partial Least Squares Regression. In The SAGE Encyclopedia of Social Science Research Methods (Eds. Lewis-Beck, M.S., Bryman, A., Liao, T.F.). Sage Publications, Thousand Oaks, CA.

2

Brereton, R.G. (2003). Chemometrics. Data Analysis for the Laboratory and Chemical Plant. Wiley, Chichester.

3

Bro, R. (1996). Håndbog i Multivariabel Kalibrering, DSR forlag, (in Danish).

4

Bro, R. (1998). Multi-Way Analysis in the Food Industry. Models, Algorithms and Applications.

5

Brown, P.J. (1993). Measurement, Regression, and Calibration. Oxford: Clarendon Press.

6

Buchi FT-NIR Applications Database. Table with characteristic absorbance in NIR.

7

Carlson, R. (1992). Design and optimization in organic synthesis. Elsevier, Amsterdam.

8

Chau, F.-T., Liang, Y.-Z., Gao, J. and Shao X.-G. (2004). Chemometrics: From Basics to Wavelet Transform. Wiley-Interscience, Hoboken, New Jersey.

9

Davis, T. (2000). NIR: A Combination of Spectroscopies. Course notes.

23

Eriksson, L., Johansson, E., Kettaneh-Wold, N. and Wold, S. (2001). Multi- and Megavariate Data Analysis. Principles and Applications. Umetrics AB, Umeå.

11

Esbensen, K.H. (2001). Multivariate Data Analysis--In Practice (5th Ed.) Camo Process, Oslo. Includes trial version of The Unscrambler.

12

Hildrum, K.I., Isaksson, T. Naes, T. and Tandberg A. (1992). Near Infra-Red Spectroscopy. Bridging the Gap between Data Analysis and NIR Applications. Ellis Horwood, New York.

13

Kramer, R. (1998). Chemometric Techniques for Quantitative Analysis. Marcel Dekker, New York.

14

Massart, D.L., Vandeginste, B.G.M., Deming, S.N., Michotte, Y. and Kaufman, L. (1988). Chemometrics: A Textbook. Elsevier Science Publishers, Amsterdam.

15

Mark, H. (1991). Principles and Practice of Spectroscopic Calibration. John Wiley & Sons, New York.

16

Martens, H. and Martens, M. (2001). Multivariate Analysis of Quality. An Introduction. John Wiley & Sons, New York.

17

Martens, H. and Næs, T. (1989). Multivariate Calibration. John Wiley & Sons, Chichester.

18

McClure, G.L. (Ed.) (1987). Computerized Quantitative Infrared Analysis. ASTM STP 934, American Society for Testing and Materials, Philadelphia.

19

Pavia, D.L., Lampman, G.M. and Kriz, G.S. (2001). Introduction to Spectroscopy. A Guide for Students of Organic Chemistry (3rd Ed.) Thomson Learning, Orlando.

20

Siesler, H.W., Ozaki, Y., Kawata, S. and Heise, H.M. (2002). Near-Infrared Spectroscopy. Principles, Instruments, Applications. Wiley-VCH, Weinheim.

21

Standard Practices for Infrared Multivariate Quantitative Analysis. ASTM E, 1655-04.

22

The Web Page Chemometrics Text Book (in Portuguese).

23

Wold, S., Martens, H. and Wold, H. (1983). The Multivariate Calibration Problem in Chemistry solved by the PLS Method. Proc. Conf. Matrix Pencils, (A. Ruhe and B. Kågström, Eds.), March 1982. Lecture Notes in Mathematics 973, Springer Verlag, Heidelberg, 286-293.

24

Wold, H. (1985). Partial Least Squares. In Encyclopedia of Statistical Sciences (Eds. Kotz, S., Johnson, N.L. and Read, C.B.) Vol. 6, 581-591. John Wiley & Sons, New York.

HOME | Back Last modified January 29, 2007. Webmaster ©2001-2005 Master Of Applied Statistics