Unsupervised Segmentation of Agricultural Crops in UAV RGB Images

Segmentación no supervisada de Cultivos Agrícolas en imágenes de Vehículos Aéreos No tripulados.

Eduardo Garea LLano^1*, Dailé Osorio Roig¹ , Yasser Chacón Cabrera¹

¹División de Investigaciones CENATAV, DATYS, 7ma A N0 21406 e/ 214 y 216, Siboney, Playa, La Habana {egarea, dosorio, ychacon}@cenatav.co.cu

ABSTRACT

Crop inventory is a precision agricultural task that allows for the planning and estimation of yields per hectare cultivated. The use of Unmanned Aerial Vehicles (UAV) has gained a great boom in the development of these applications given its low cost and fast possibilities of obtaining quality images. This paper presents a method for unsupervised segmentation of agricultural UAV RGB color images. We propose the combination of a set of texture features under a segmentation framework, based on the active contour without edges model with level set representation and a connected component filtering strategy. The experiments show that it can be applied for the segmentation of agricultural crops, with an average segmentation quality of 90%. It exceeds in efficacy other methods of supervised segmentation of the state of the art. It was demonstrated the robustness of the approach for images taken with UAVs of low performance which makes cheaper its application with low costs by agricultural producers.

Key words: crop inventory, texture analysis, UAV, unsupervised segmentation.

RESUMEN

El inventario de cultivos es una tarea agrícola de precisión que permite planificar y estimar los rendimientos por hectárea cultivada. El uso de vehículos aéreos no tripulados ha ganado un gran auge en el desarrollo de estas aplicaciones debido a su bajo costo y rápidas posibilidades de obtener imágenes de calidad. Este artículo presenta un método para la segmentación no supervisada de imágenes a color tomadas por estos vehículos. Proponemos la combinación de un conjunto de características de textura en un marco de segmentación, basado en el contorno activo sin modelo de bordes con representación de nivel y una estrategia de filtrado de componentes conexas. Los experimentos muestran que la propuesta puede ser aplicada para la segmentación de cultivos agrícolas, con una calidad de segmentación media del 90%, la misma supera en eficacia a otros métodos de segmentación supervisada del estado del arte. Se demostró la robustez del enfoque para imágenes tomadas con vehículos aéreos de bajo costo que hace más barata su aplicación por parte de los productores agrícolas.

Palabras clave: análisis de textura, inventario de cultivos, segmentación no supervisada, UAV

INTRODUCTION

The use of Unmanned Aerial Vehicles (UAV) has gained in the last years a great boom because they are cheap and, at the same time, they offer high quality images. Nowadays, high-resolution aerial images are widely available due to the diffusion of advanced UAVs technologies. These developments offer new opportunities for accurate land use analysis and change detection. UAVs have several advantages over satellites and piloted aircraft: they can be deployed quickly and repeatedly; they are cheaper and safer than piloted aircraft; they are flexible in terms of flight height and mission time; and they can obtain images at the sub-decimeter resolution. However, because of the limitation of the flight height and focal length of the camera, the acquired image size is smaller and a single image cannot always cover the entire target area, more than one remote sensing image needs to be added into an UAV remote sensing image mosaic. Another problem is that although multispectral and hyperspectral sensors for UAVs are already available, these sensors still have relatively high prices in the market, so their use for studies make projects more expensive and limit their use by producers and farmers with lower economic incomes.

The applications of remote sensing in agriculture are multiple and varied. These applications are used among others, in the identification, mapping and monitoring of crops, determination of biophysical variables, estimation of biomass and yield, detection of biotic and abiotic stresses, prediction of nitrogen content, recognition and monitoring of irrigated areas, characterization of the water needs of crops, study of environmental impacts, etc. Different examples of their use in agriculture can be found in Zhang, 2012; Atzberger, 2013. In the last 3 years, there have been advances in remote sensing using UAVs (Yinjiang, 2016; Jabal, 2017). Detection of water stress, pests and diseases, weed; nutritional status, determination of biomass and yield, soil characterization, etc., are some of the agricultural applications that are being developed.

The crop inventory is another application that allows the planning and estimation of the yields per hectare cultivated. This is done from the classification of the different vegetation cover present in the study area.
In crops as in general in plants, the chlorophyll level is different. These levels are perceptible by reflectance in the near infrared band, which can be detected in the remote sensing images. In the near infrared, the spectral reflectance curve of different crops allows their differentiation so that spectral analysis is more suitable for the realization of crop inventories from remote sensing images.

The healthy crop in visible band of the spectrum has a strong absorption rate and in the near infrared band has a strong reflectance. Based on the theoretical basis of the spectrum, the monitoring of crop growth can be achieved by extracting the approach of indicators of remote sensing of crops. The normalized difference vegetation index (NDVI) are commonly used monitoring indicators. However, these analyses are possible when the images are captured by sensors that are able to work in different bands of the spectrum. However, when we have UAVs equipped only with cameras that take pictures in the visible spectrum, i.e. RGB images, it is not possible to perform this type of analysis. Then a viable alternative for segmentation of different vegetal cover in the scene could be through the description of their textures.

The term texture is used to characterize the surface of a given object or phenomenon and is undoubtedly one of the main features used in digital image processing and pattern recognition (Unser, 1986). Texture is an important feature of every image. It is a very valuable source of information for the analysis and image understanding. Texture is an innate property, virtually of all surface, so it is possible to talk about grains in the woods, the weave of a textile, the pattern of vegetation in an agricultural field, etc.

When segmenting the texture of the image, the interest is centered on obtaining a map of regions of textures, or a map of edges between textures. The region-based texture segmentation approach tries to identify regions of the image that have similar texture patterns. The local pixel regions are joined to form texture patterns based on the similarity of some texture property. Regions that have different textures are the ones that should be segmented. Region segmentation methods have the advantage that the boundaries of the regions are always closed and therefore regions with different textures are always well separated. The disadvantages are in the data needed by the methods in advance, such as: the number of different textures in the image, the similarity values between textures, or a threshold to separate them.

In this work, we propose a method for unsupervised segmentation of agricultural crops in UAVs RGB color images. We propose the combination of a set of texture features under a segmentation framework based on the active contour without edges model with level set representation and a connected component filtering strategy.

Method for unsupervised segmentation of agricultural crops in UAV images

The proposed method is based on the combination of segmentation processes using texture models, texture descriptors, feature extraction/ selection, pixel grouping and post-segmentation. The steps of the proposed methodology are as follows: (figure 1)

(1) Read the input UAV image. It can be a simple image or a video. In the video case, the selection of the best frames that correspond to the presence in the scene of the interested crops is made. (2) Apply some pre-processing to the simple image. Improvements in brightness-contrast, signal-to-noise ratio, rotating (dextrorotatory, levorotatory, horizontal, vertical), and turning it to gray, obtaining negative, edges, or other transformation. (3) Apply a combination of texture descriptors in the image. This action can generate a multi-dimensional image in the order of tens and hundreds. (4) Apply segmentation method, based on the active contour without edges model with level set representation. (5) Optional: Apply methods for feature selection and reduction (PCA, SFFS). (6) Post-processing of the segmented image. It facilitates the digital filtering by applying as a theoretical basis an analysis of the connected components (CC), and the evaluation of the quality of the segmented image using supervised metrics.

Proposed combination of texture models

The Statistical Model of First Order is one of the forms of texture representation that generates simple statistical procedures to characterize the texture quantitatively. The methods that define texture descriptors in this model work directly with the gray levels (GL) presented in the image, i.e. they are used without modification. These descriptors are computed from the intensity values of the GL of the pixels belonging to a centered neighborhood at each point of the image, or from the GLs histogram of the image. The typical parameters that can be extracted from the GL distribution of the image as texture measurements are (Ferdeghin, 1991): minimum, maximum, mean, standard deviation and variance. These features prove to be very simple and efficient discriminating simple textures. The great advantage they have is their simplicity of calculation. Their main limitation is that they cannot express the spatial characteristics of the texture. To perform the segmentation of the image based on the texture analysis, it must be ensured that the calculated and selected texture features reflect a change in the statistical values of the image. This is usually achieved when the regions have a simple texture (Theodoridis, 2009). When the texture is complex, statistical changes in spatial information can be contrasted using texture descriptors based on spectral methods.

The agricultural crops usually have orientations along an axis and have an elongated shape describing the shape of the grooves (figure 2B). This fact has a significant influence on the texture characteristics of the image captured by UAVs.

One of the spectral models that has obtained better results in the description of complex spatial oriented textures are those based on Gabor filters (Lu, 21017).
A 2D-Gabor filter consists of a sinusoidal flat wave having a frequency value and an orientation, and is modulated by a 2D-Gaussian envelope. We propose the use of a 2D Gabor filter bank at different frequencies and orientations to extract oriented texture features of the image. Each frequency and its corresponding orientation, models a channel that returns an image response. The input image I (x, y) is convolved with the 2D-Gabor g (x, y) filter to obtain the image of the Gabor feature r (x, y) as shown in (1).

Where, (x, y) E Ω; Ω: is the set of points of the image and g (x, y) is the 2D-Gabor family of functions.

Multi Phase ACWE for Unsupervised Vegetal Cover Segmentation in UAV images

The active contour approach has been widely used as image segmentation method and for texture segmentation (Savelonas, 2001). The level set theory has provided more flexibility and convenience to its implementation. The Active Contour without Edges (ACWE) (Chan, 2001) uses a level set representation, so the results do not depend on the initial position of the contour. In [16] a Multi-Phase ACWE model for vector valued images was presented. The algorithm is based on the extension of the ACWE model and level set proposed by Chan and Vese (Chan, 2001) conceived to segment an image u0: Ω → R. In this function, the active contour C ⊂Ω was represented by a level set using a Lipschitz function φ (x, y): Ω → R. With this function φ (x, y), the image can be divided into two or multiple regions.

The problem in our case is to segment a textured UAV image without any previous knowledge about the position of the vegetation textures in the image. Then we propose to use this algorithm (Vega, 2008). We will get as input a vector composed by the combination of texture features of two theoretical sources (first order statistics and a bank of 2D Gabor filters), responsible for providing differentiating properties between the types of textures present in the image and characterizing each of the vegetation cover present in the UAV scene.

Starting from the Multi-Phase ACWE model (Vega, 2008).  it is possible to use m = log2n level functions to represent n classes, of an UAV image possessing any complex topological structure, such as example of figure 2. Function (2) can segment n = 2m texture regions using m level functions and a vector made up of r texture features (first order statistics and a bank of 2D Gabor filters) that are received at

Where is the term that computes the contour length. The vector of averages is defined as , being 1≤ I ≤ r and n equal to the number of classes in which the image will be segmented. The average of the pixels of class j in feature i are calculated by the following equation (3):

Where the variable, χj (x, y) is the so-called characteristic function of the j-th class, which contains the information of the values that take the different φ functions in each pixel for that particular class. All weights λi are going to be considered as 1, since, it is not possible to assign different weights to any texture feature because we work with images without previous information.

EXPERIMENTAL RESULTAS Y DISCUSSION

In figure 3 we present the experimental design based on the flow diagram proposed in figure 1. The objective of the experiments is to verify the validity of the proposed method.

1) Pre-processing of the video sequence: In this step, the video frames corresponding to the presence in the scene of the studied crops are manually extracted. As a result, the images that will be processed in the subsequent steps are obtained.

2) Manual segmentation of the regions with the study culture: By using tools developed for this purpose, the regions containing the studied crops will be marked (by the members of the research team) in each one of the images resulting from step 1, as a result for each image we will obtain an ideal segmentation, "ground truth".

3) Application of the flow of the segmentation process using texture models. This step corresponds to the proposed methodological scheme.

4) The evaluation of the proposed method and its comparison with state of the art methods will be developed by measuring its accuracy using the “Coincidence Measure” (CM) proposed in Vega, 2008. This measure takes into account the information that brings the ideal segmentation or “ground truth”. Then 0 ≤ CM ≤ 1, if CM=1 there is a perfect coincidence.
For the comparison of the performance of our proposal with state of the art methods, we have selected two methods that are used in images taken in the visible spectrum for the texture segmentation. (Haindl, 2005 y CALDERO, 2010)

For the experiments, 21 video sequences were taken at different times and days at 100 feet (30.5 m) and 150 feet (45.7 m) on crop fields of the “Artemisa” province located about 40km west from Havana, Cuba, where sugar cane, fruits, banana, tomato, and potato crops are grown. Videos were taken by the UAV Phantom 1 quadcopter (figure 2A) equipped PHANTOM. with RGB Sensor: Sony Exmor 1 / 2.3 ", effective pixels: 12.4 m (total pixels: 12.76 m), frames size: 1920 x1080. figure 2 shows some examples of frames taken for the experiment.

CONCLUSIONS

In this paper, we proposed a method for automatic and unsupervised segmentation of agricultural crops in UAV RGB color images. We analyzed and demonstrated the pertinence of combining two models of texture descriptors: The Statistical Model of First Order, which guarantees the detection and differentiation of the simple textures of the types of vegetation and the Spectral Model, which guarantees the detection of the spatially oriented and complex textures characterizing the agricultural crops. The proposed combination under a segmentation framework, based on the ACWE model with level set representation and a connected component filtering strategy shows that the proposed approach is promising. The robustness of the approach for images taken with UAVs of low performance as the Phantom 1 in terms of flight height and standard RGB sensors was demonstrated which makes its application cheaper for farmers and producers of low economic possibilities. The future works will be oriented to determine other combinations of texture models as well as its application in algorithms of supervised image classification for the agricultural crop classification.

REFERENCIAS BIBLIOGRÁFICAS

Atzberger, C. Advances in remote sensing of agriculture: Context description, existing operational monitoring systems and major information needs. Remote Sensing, 5, 949-981. (2013)

Calderero, F. y F. Marques: Region merging parameter dependency as information diversity to create sparse hierarchies of partitions. Proceedings of 2010-IEEE 17th International Conference on Image Processing, September 26-29, Hong Kong. (2010)

Chan, T.F. L.A. Vese: Active Contours Without Edges. IEEE Transaction on Image Processing. 10(2):266-277, (2001).

Ferdeghini, E., B. Pinamonti, E. Picano, F. Lattanzi, R. Bussani. Quantitative texture analysis in echocardiography; application to the diagnosis of myocarditis. Journal of Clinical Ultrasound, 19(5): 263-270. (1991).

Haindl, M.; S. Mikeš: Colour texture segmentation using modelling approach. Lecture Notes in Computer Science, (3687):484–491, (2005)

Jabal TI, Nugroho TW, Wahono W, Sitti RM. Survey of Irrigation Area Using Micro Unmannes Aerial Vehicle (Micro- UAV) in Gumbasa irrigation Area. Agricultural Socio-Economics Journal. AGRISE,Vol 17, No 1, (2017).

Lu R.,  Dennison E., Denison H., Cooper C., Taylor M.  & Bottema M.J.  Texture analysis based on Gabor filters improves the estimate of bone fracture risk from DXA images. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization. Published on line, http://dx.doi.org/10.1080/21681163.2016.1271726, (2017).

PHANTOM. Quick Start Manual V1.7, 2013.09.25. Revision 2012-2013 DJI Innovations

Theodoridis, S. and Koutroumbas, K. Pattern Recognition. Academic Press. Fourth Edition. Book. ISBN: 978-1-59749-272-0. Printed in the United States of America. (2009).

Savelonas M.A., D.K. Iakovidis, D.E. Maroulis, "An LBP-Based Active Contour Algorithm for Unsupervised Texture Segmentation," icpr, pp. 279-282, 18th International Conference on Pattern Recognition (ICPR'06) Volume 2, (2006).

Unser, M. Local linear transforms for texture measurements. Signal Processing, vol. 2, pp. 61-79. (1986).

Vega, S.; Gil J.L. and Vera O.L. Active contour algorithm for texture segmentation using a texture feature set. ICPR2008. IEEE Computer Society. TuBCT8.32, (1-4), (2008): IEEE.

Yinjiang J. Zhongbin S., Weizheng S., Jianqing Y., Zhenan X.  UAV Technology and Its Application in Agriculture. Advanced Science and Technology Letters Vol.137 (SUComS 2016), pp.107-111. (2016).

Zhang, C., Kovacs, J.M. The application of small unmanned aerial systems for precision agriculture: a review. Precision Agriculture, 13, 693-712. (2012).

Recibido: 27/10/2017
Aceptado: 21/09/2018