Classification and localization of cotton bolls in real-time using 3-D imaging and artificial Intelligence for development of a small robotic cotton harvesting platform

Summary

The overall goal is to develop swarms of small autonomous robotic cotton harvesters that continuously harvest bolls as they open. Several key technologies must be developed in order to reach this goal. In our research program, we are determining how to detect cotton bolls and locate their 3-D position (localize) for robotic harvesting. We are using a stereoscopic camera and artificial intelligence to take images and process these images in real-time to direct the end-effector of a robotic arm to harvest the bolls.

Situation

Current cotton harvesting is an expensive production cost for farmers. The cotton harvester can cost around $800k and limits cotton production to large farms that can afford such expensive equipment. As computer processing speeds have increased, sensors have become smaller and cheaper, and robotics advanced, we have reached a point where we can develop economical mobile platforms to perform real-time agricultural operations. We are developing an artificial intelligence to classify and locate cotton bolls using an inexpensive 3-D camera and NVIDIA Jetson TX-2 processor

Response

We have developed an ensemble method of deep learning, color segmentation and image transformation to track and count cotton bolls from a camera on a mobile harvester in real-time. In each image taken by a camera, a trained deep learning method, the YOLOv2 (You Only Look Once) model is used to detect open cotton bolls and then color segmentation applied to detect bolls missed by the YOLO model. An image transformation algorithm is applied to the next image in case some bolls were detected previously but were occluded in subsequent images. Tracking was done by tracking the spatial movement of the features near the bolls using Lucas-Kanade algorithms. In case the boll that is detected previously is missed, image homography transformation and L-estimators are applied to retrieve the information of the lost boll. Each boll tracked is stored using vectors that are counted to give the total number of bolls detected. In this research, the accuracy is sacrificed for speed by using the YOLO model compared to other deep learning methods which offer higher accuracy with low processing speed. However, accuracy increased by developing an ensemble method that combines image color segmentation and image transformation to achieve accuracy and sensitivity of 93%, and precision of 99.9%. This method illustrates the ability to track and count cotton bolls in real-time for the development of a robotic cotton harvesting system.

Impact

To develop a robotic cotton harvester, identifying and locating the bolls in real-time is critical. In addition, there are several advantages to deploying a swarm of autonomous harvesters that harvest continuously as soon as bolls begin to open. Some advantages include increase in cotton quality, uniformity and less contamination. Small farmers would have access to cotton production using the appropriate number of small harvesters that would be much less expensive than the current cotton harvesters. Also, assuming several small robots, failure of one robot would have a more limited impact on harvest progress compared to a single machine covering 80 acres per day.

State Issue

Sustainability, Conservation, & the Environment

Details

• Year: 2018
• Geographic Scope: International
• County: Tift
• Program Areas:
  • Agriculture & Natural Resources