In the first part of this series I showed off my first attempt at create a fully convolutional neural network (FCN) for localizing and identifying plants as either weeds or crops. It was able to look at a generic picture of a crop field and find the plants within the image and tell if they were corn, okra, or weed. However, it was far from perfect. I used the lowest resolution FCN32 network for that version, and at best it would give a general blob overlay of the plant telling what it was (example shown in Fig. 1). At the end of that article I mentioned several things I could do to improve the performance of this FCN. In this article I will discuss my latest neural network (NN) and show off the dramatically improved results.
The original NN was built using Caffe and Digits. These tools worked great as long as I only used the pre-canned functionality that is already available. When I wanted to do anything different, like data augmentation or modifying the network, things became much more difficult. These are both great tools, but Caffe in particular has very little good entry level documentation on how to really use it and build networks. After a frustrating few weeks I started looking at other frameworks. That was when I discovered Google's TensorFlow. It blew me away! I was able to find a few different FCN implementations out there. I decided to start with this one from Shekkizh. It was short, sweet and easy to understand. It uses the ImageNet VGG model instead of AlexNet that I used before. This model produces much higher resolution output masks compared to the course FCN32 I originally used. My first attempt at just swapping to this with my existing data produced really good results. I was very surprised at the difference. I decided to implement a few of my other suggestions to see how good I could get it to work. Here are a few of the other things I changed with this network.
Once I had all those modifications in place I retrained my network using TensorFlow, and I was stunned by the beautiful results. Figure 2 below shows the images from the validation data set. The neural network was never trained on these images, and this is the actual output directly from the NN. No other post-processing has been done on them except to color them and overlay back on the original image. There are semi-transparent arrows on the images to allow you to click back and forth, or you can click on one of the dots below for a specific image. I have drawn boxes around mislabeled plants. If a plant is only slightly mislabeled I did not draw a box on it. As you are looking at the plants there are a few things you should keep in mind.
So while you keep these points in mind here are the images labeled by the FCN.
I was very happy with the results from using just an RGB camera. I attained an average 97.8% pixel level accuracy for all images in the test data set. I initially thought that I would have to have a multi-spectral camera to obtain results this good, but I was wrong. However, I still believe that using depth, calculated height above ground, surface normals, and possibly multi-spectral information could continue to improve the performance of the network. A few weeks ago I finished work on a combined depth+RGBIR camera. I plan to use this to collect the next batch of data to try and improve these results even more. One area where I believe adding depth and size information would help is with telling the difference between some of the young plant types and certain weeds.
There were two main weed types that had issues (Fig. 4). These are the nutweed and the young copperleaf plants. Nutweed has long slender leafs that look similar to the corn leafs when seen from afar. Copperleaf also had a problem in that it looks very similar to the young okra plant. So while adding the images of young corn and okra did help the network to better identify those plants, it also caused some weeds to be more likely to get confused with crops, especially where the plant image was small. If the NN also had some information about how far away, and how big the plant was I believe it would help tremendously with this problem. It will give the network more of an environmental context of how big something is in the real world. This will make it much easier to see that nutweed only ever grows to a small height, whereas the similar looking corn is much taller. At the moment, I have zoomed in pictures of okra right alongside zoomed out pictures of copperleaf, and the network has no context on how big each of these plants are in the image. It is actually amazing that the NN does so well at this task.
I believe that adding multi-spectral data will also help it with the identification task, but at this point I will need to do some tests once I have data and see if it is really helping enough to justify the added expense for a multi-spectral camera. After seeing the results from just the RGB camera my gut says that it will probably not be enough of an improvement to justify this, but we will see.
Since I have the new camera built I plan to start trying to get some new depth+RGBIR images to find new ways of improving the results even more. However, I am already so impressed with the results that I have decided to start work on the first version of the weeding robot. I plan to start by build a simple version that will find and eliminate weeds that are between rows of the crops. I purchased a Dagu Wild Thumper 6WD robot chassis with a super-twist independent suspension for each wheel, and a 75:1 gear ratio (Fig. 4). This small robot is built to handle rough terrain and should have no trouble getting around a field without getting stuck, and it is small enough that it can easily get in between the crop rows. My first step is getting it where I can drive it around using a remote control. Then I will add a small tiller on the front that can be raised and lowered. Once I have this up and going I can start adding the autonomous functionality so it can drive itself. Please sign up for my newsletter if you would like to keep informed of my progress on building this robot.
Please subscribe to my Newsletter!
NeuroRobotic Technologies is dedicated to creating the next generation of intelligent, adaptive robotic systems by building autonomous controls systems that mimic the brains of real animals.