The otolith digital catalogue AFORO allows unknown otoliths to be classified automatically by using a comparison with its classified records. To do this, the otolith’s contour, which is extracted from an image, is used. In AFORO, otolith images follow a strict positional normalization. Only the left sagitta is considered, and the images must show the internal side of the whole otolith, with the sulcus acusticus visible, the dorsal side (D) placed in the dorsal position and the rostral side (R) placed on the right. The otolith in the incoming image to be classified must also follow the same positional normalization. Variations from the reference position worsen the classification results. In this article, robust contour descriptors are proposed to extend this functionality of AFORO to the images of otoliths that are poorly normalized, contain rotations, are entirely inverted or came from the right rather than the left sagitta. These descriptors are based on the discrete Fourier transform and could extend the classification functionality to incoming images that are taken and sent, for instance, from smartphones in a wide range of working conditions.

El catálogo digital de otolitos AFORO permite la clasificación automática de otolitos desconocidos mediante la comparación con sus registros clasificados. Para ello, se utiliza el contorno del otolito, que se extrae de una imagen. En AFORO, las imágenes de otolitos siguen una estricta normalización posicional. Sólo se considera la sagita izquierda, y las imágenes deben mostrar la cara interna de todo el otolito, siendo visible el surco acústico (SA), con la cara dorsal (D) colocada hacia arriba y el rostrum (R) a la derecha. De la misma manera, el otolito en la imagen entrante a clasificar debe seguir la misma normalización posicional. Las variaciones respecto a la posición de referencia empeoran los resultados de la clasificación. En este trabajo se proponen descriptores de contorno robustos para extender esta funcionalidad de AFORO a las imágenes de otolitos mal normalizados, que contienen rotaciones, totalmente invertidos o que proceden de la sagita derecho (no del izquierdo). Estos descriptores se basan en la DFT (transformada discreta de Fourier) y podrían ampliar la funcionalidad de clasificación a imágenes entrantes tomadas y enviadas, por ejemplo, desde teléfonos inteligentes y en condiciones de trabajo muy diferentes.

Otoliths are calcareous structures found in the inner ear of osteichthyan fishes, and the shape and structure are species-specific (

The AFORO website (

At present, the AFORO database contains 5793 high-resolution images that correspond to 1776 species, and its software tools have been used for otolith atlases (

Currently, AFORO’s functionality of automatic identification of fish species and populations is based on 2D otolith closed contours. It works by comparing the contour that is extracted from an external query image against those that are in the AFORO database. All images in the database have been registered following a strict acquisition protocol, which is also required for incoming query images. However, small errors in registration of the query image are common. These introduce small rotations of the contour resulting in misalignments in the contour reference, which in turn decrease the accuracy of the classifier. Furthermore, two more critical factors cause the system to fail: otoliths in the incoming image appear rotated by approximately 180 degrees with respect to normalization because the rostrum (a part of the otolith) has been misidentified or because the incoming images have been acquired under other normalization criteria.

In addition, in some practical applications only the right sagitta is available, whereas the AFORO database only stores data on the left sagittae. To improve the automatic otolith recognition functionality that is provided by AFORO, in this paper, we study how to relax and also to completely avoid the normalization that is required for images and we explore a strategy for classifying the right sagitta by taking advantage of contour symmetries it has with the left sagitta. This is done by proposing descriptors that take advantage of the well-known discrete Fourier transform (DFT) coefficients and their properties (

To highlight the contributions of this work and point out the differences from that of _{k}, b_{k}, c_{k}, d_{k}) of a contour with the same contour rotated, displaced or scaled (a_{k}’, b_{k}’, c_{k}’, d_{k}’) follows the matrix notation proposed in _{k}, b_{k}, c_{k}, d_{k}) into one (f_{k}). Those f_{k} are invariant to rotations, sampling point and direction and symmetries. However, their use for classification of otoliths provides very poor results compared with EFDs with handmade positional standardization (

In the present work, we employ the DFT coefficients, which unlike EFDs are complex, and we perform all the operations presented in

– A closed formula for the angle θ involved in the automatic positional normalization.

– A closed formula to find a starting point of sampling
$\widehat{k}$
.

– The procedure to apply both sampling and positional standardizations by performing operations directly on the phase of the DFT coefficients.

– A way to deal with rotations higher than ±90° and also with symmetries.

The most relevant result is that the new strategy maintains the classification rate of otoliths of non-standardized inputs very close to that obtained when the contours are manually orientated by an expert and the samples are prepared for this purpose. The experiments were performed with the same database and the same classifier as in

The work is organized as follows: the Materials and Methods section covers the AFORO image acquisition protocol required to correctly position the otolith, the derivation of the proposed descriptors robust to rotations of θ (|θ|<90°), the number of points that were used to represent contours, some shape symmetries and their relation to the proposed descriptors, a baseline to achieve invariance and, finally, the method of classification and validation that was employed. The Results section presents the results under two frameworks: one in which the AFORO user applies positional normalization and in which the errors that are introduced with respect to the desired reference are small; and one that evaluates the recognition performance that could be achieved without positional standardization. We conclude the paper with some discussion.

The test material comes from the AFORO database, which is regularly updated and at present (07/06/2018) contains a total of 5793 high-resolution images corresponding to 1776 species and 238 families from oceans all around the world. The specimens selected for the database are focused on representing all the possible variability within each species, such as age, length, sex and stock. The approach presented in this paper was tested with 134 otoliths from eight different species but was optionally divided into eight or nine classes. To compare the results with that of a previous study (

1. Upload only one image of the left sagitta, as shown in

2. To obtain a good representation of the sagitta contour, the image must be well contrasted. If possible, the background should be homogeneously black.

3. Sagitta position. The dorsal side (D) of the otolith must be placed facing upwards in the image. The anterior or rostral side (R) must be located on the right side of the image.

Consider a contour sampled in counter-clockwise direction in which _{k}_{k}

_{k}_{k}+ _{k}k |
(1) |

By applying DFT, we have the transformed coefficients _{l} as follows:

$${f}_{l}={\displaystyle {\sum}_{k=0}^{N-1}{x}_{k}{e}^{-j\frac{2\text{\pi}}{N}kl}l=0,1,\cdots ,N-1}$$ | (2) |

Thus, the original _{k}_{l}

$${s}_{k}=\frac{1}{N}{\displaystyle \sum _{l=0}^{N-1}{f}_{l}}{e}^{j\frac{2\pi}{N}kl}l=0,1,\cdots ,N-1$$ | (3) |

Note from (2) that _{0}=_{cm}_{cm}_{cm}_{cm}_{cm}_{cm}_{0} to be 0. Any reconstruction of _{k}_{0 }=0, will reconstruct the contour of the original shape centred on the coordinate origin.

Note that at this point we do not assume any particular position to start contour sampling. In fact, contour sampling could start at any point. We merely begin from any point belonging to the contour, with the sampling completed counter-clockwise.

With the aim of evolving the descriptors, let us first look at how we can describe the first ellipse upon which the contour is centred. It would be the most straightforward reconstruction of the original outline performed by only two complex coefficients, _{1 }and _{N}_{–1}. From (3), the ellipse _{k}

$$\begin{array}{c}{e}_{k}=\frac{1}{N}\left({f}_{1}{e}^{j\frac{2\pi}{N}k}+{f}_{N-1}{e}^{j\frac{2\pi}{N}\left(N-1\right)k}\right)=\\ =\frac{1}{N}\left({f}_{1}{e}^{j\frac{2\pi}{N}k}+{f}_{N-1}{e}^{-j\frac{2\pi}{N}k}\right)k=0,1,\cdots ,N-1\end{array}$$ | (4) |

Let us write _{1 }and _{N}_{-1} in polar form: _{1}=|>_{1}|^{jϕ1} and _{N}_{–1}=|_{N}_{–1}|^{jϕN–1}. The two terms of _{k}_{1}|+|_{N}_{–1}|) and it is reached at point $\widehat{k}$

$$\widehat{k}=\frac{N}{4\pi}({\varphi}_{N-1}-{\varphi}_{1})$$ | (5) |

which is obtained by equalizing the phases of both terms in Expression (4). Thus, by considering _{1} and _{N}_{–1 }in polar form, we have

$$\frac{2\pi}{N}\widehat{k}+{\varphi}_{1}=\frac{-2\pi}{N}\widehat{k}+{\varphi}_{N-1}$$ ,

and by isolating $\widehat{k}$, (5) becomes direct. A positive value of $\widehat{k}$ represents a clockwise displacement of $\widehat{k}$ samples from the position of the first sample of the contour sequence, while a negative value means a clockwise displacement. Therefore, the final value must be taken modulo _{$\widehat{k}$})>0, which is the one that is located in the region that is formed by the first and fourth quadrants. Thus, angle θ, which is used in the first/fourth quadrant region of the major ellipse axis with respect to the horizontal, is as follows:

$$\theta =ATAN\left[\frac{\Im m({e}_{\widehat{k}})}{Real({e}_{\widehat{k}})}\right]$$ | (6) |

In the range of −π/2<θ<π/2, we can correct the angle of rotation in the frequency domain by multiplying each coefficient by ^{–jθ}.

As in badly normalized incoming images we do not control the starting contour sampling point, we change it in order, beginning with the point $\widehat{k}$, which will pass to the first sample of the sequence. This can be done by DFT in the coefficient domain by multiplying each

$${e}^{-j\frac{2\text{\pi}}{N}\widehat{k}}$$ ,

according to the DFT property:

$$F[{s}_{k-\widehat{k}}]={f}_{l}{e}^{-j\frac{2\text{\pi}}{N}\widehat{k}l}$$ .

We achieve scale invariance by normalizing the coefficients by half of the major principal ellipse axis. Therefore, the new proposed coefficients take the following form:

$${C}_{l}=\frac{{f}_{l}}{\text{|}{f}_{1}\text{|+|}{f}_{N-1}\text{|}}{e}^{-j\frac{2\text{\pi}}{N}\widehat{k}l-j\theta},l=1,\cdots ,N-1$$ | (7) |

Note that if we apply an IDFT directly to the _{l}_{0}=0, we recover the shape of the otolith. Thus, if in the original figure the otolith was rotated between –90° and 90° concerning the required positional normalization, the recovered contour appears to be represented in the way that the principal ellipse axis remains horizontal with the rostrum (R) in the right position, as is represented in

Note that from the result of directly applying the contour descriptors defined in (7), the initial sampling point does not necessarily fall at the zero phase position (on the real axis), but varies slightly. A variant of the descriptors is proposed in (8), where the angle is corrected, but sampling is forced at the point closest to the horizontal axis (the zero phase point).

$${C}_{l}=\frac{{f}_{l}}{\text{|}{f}_{1}\text{|+|}{f}_{N-1}\text{|}}{e}^{-j\theta},\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}l=1,\cdots ,N-1$$ | (8) |

This second option allows the symmetries represented in

To be able to compare the spectral content between contours, they are resampled so that they all have the same number of

In this section, we show the relationship between reconstructions from the proposed coefficients that were obtained for otolith contours depending on their orientation in the image. _{l}_{l}

_{l}

idft^{l+1}·_{l}

idft^{l+1}·^{*}_{I}) →

idft^{*}_{l}) →

The super index ( )^{*} stands for the complex conjugate operation. When calculating the descriptors of a randomly rotated contour (360º) that has (or does not have) a possible specular symmetry, it will always fit with one of the four previous expressions. In the literature we find other ways of dealing with symmetries, such as those described in

We can see that the modules of the coefficients coming from the four expressions (corresponding to contours

$${M}_{l}=\text{|}{C}_{l}\text{|}=\frac{\text{|}{f}_{l}\text{|}}{\text{|}{f}_{1}\text{|+|}{f}_{N-1}\text{|}},\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}l=1,\cdots ,N-1$$ | (9) |

Expression (9) is similar to the one proposed in

Features must represent the contour efficiently, and through them, the amount of information to describe it must be much less than the complete information that defines the silhouette. The most energy-intensive DFT-based coefficients are those at the beginning and end of the sequence. An efficient representation requires the number 2_{c}_{c}_{c}_{c}_{l}

The

The LOOCV strategy is the option used to evaluate the performance of the classifier. LOOCV uses a single observation from the original set as the validation data and the remaining observations as training data. All the elements are tested against the rest, so we know how each particular element is classified. With LOOCV, all the observations are used for both training and validation, with each observation being used once for validation. In datasets in which there are few elements per class, as usually happens in our case, LOOCV seems to be an appropriated strategy.

Several experiments were conducted to evaluate the performance of the presented descriptors according to two different scenarios. The first scenario considers that the incoming image (the query image) has been taken by someone with sufficient knowledge to correctly distinguish the left sagitta, to determine the face of the otolith that shows the SA and to orient it correctly according to the indications of AFORO. In this scenario, the errors are only minor orientation errors and will always be between –90° and +90°. In the second scenario, we want to evaluate the capacity for identifying species just from the silhouette of the otolith, regardless of whether it is the right or the left sagitta and regardless of its orientation in the image. Therefore, no normalization positions are considered at all.

Common to all experiments, the effectiveness of the proposed contour descriptors will be evaluated using a K-NN classification test with (K=1) and using LOOCV. The first scenario was also used to assess the impact on the classifier accuracy depending on the number of coefficients and the number of points that are employed to represent the contour. To do this, we codified all the contours under a test, the coefficients _{l}_{0} was eliminated, and the 2_{l}

LOOCV involves selecting one contour and deleting its coefficients from the group. We apply a random rotation of between –90° and 90° to that contour. We codify it by applying (8) and selecting the same 2

In the most straightforward experiment completed for the second scenario, the otolith (left or right) can appear in any position and can show any face; therefore, without observing any positional normalization, the descriptors that are presented in expression (9) as well as the expressions given in

To improve the classification results in the second scenario (avoiding the positional normalization in the query image), we propose the use of the descriptors that were developed in (8) in the following form. The contour in which the otolith appears with any normalization is sampled counter-clockwise and is parameterized according to (8), as in the first scenario. Then, from these parameters, we reconstruct the contour again, obtaining any of the forms shown in

With this procedure, the classification results are close to those obtained in the first scenario. The results can be seen in

The success of automatic species recognition from the images of otoliths that are sent to AFORO requires the otolith to follow positional normalization. The determination of the contour sampling starting point depends on this. Studies that are carried out in the framework of the AFORO database show that the contour descriptors that achieve the best performances, such as those based on FT, WT, and CSS, are sensitive to the determination of that point (

Taking advantage of the use of the well-known DFT properties that relate rotations and sampling point changes in the spatial domain with changes in DFT coefficients, and specifically in their phase, our approach outperforms other Fourier-based methods that are dependent on a strict positional protocol. The proposed parameterization directly corrects the starting point of sampling and variations of between 90° and –90° with respect to the positional normalization that is required in AFORO.

To completely eliminate the positional standardization protocol (positional normalization), in the second framework we can directly employ the modules of the DFT coefficients _{i}_{i}_{1}|. For automatic recognition of shape contours, we achieve the best results by normalizing by |_{1}|+|_{N}_{–1}|, as given in (9), because |_{1}|+|_{N}_{–1}| is a magnitude that is directly related to the length of the principal ellipse axis. However, the main contribution of this paper is that accuracy can be improved using the descriptors that are proposed in (8) by merely adding four possible variants for each contour and selecting the one that obtains the best score (the minimum Euclidean distance to the closest neighbour). From

This work was partially supported by the Spanish Government projects PHENOFISH with reference CTM2015-69126-2-R. The authors are very grateful to all the people who have helped in the technical development of the AFORO website, especially Emili García-Ladona, Òscar Chic, Vicenç Parisi-Baradad, Jaume Piera, Víctor M. Tuset, Josep Forest, David Otero and Roger Olivella.