Ferry and ship detection

DeepShip or ShipNet: Matlab Multiple Transfer Deep Learning Ship/Ferry Detection

Using Matlab and the Computer Vision System Toolbox, Image Processing ToolboxNeural Network ToolboxParallel Computing Toolbox and the Statistics and Machine Learning Toolbox, I labelled 1923 images from my web cam feed with tags, [ferry], [ship], [bird] [meteor].

How my webcam feed overlooking the Clyde Estuary is configured, is explained here (http://www.data-analyst.org/wp/category/ucam247/).

In two years over one million pictures have collected and processing these in person visually has become impossible.

To find interesting events, I designed a Deep Learning System with Matlab that makes it possible to find interesting events in all these pictures (small streak).

Meteor 1
Meteor 1
Meteor 2
Meteor 2

In the moment the system can detect Ferries with high accuracy.

Ferry Detection
Ferry Detection

The biggest challenge are waves and ships. Ships (non ferries), can be big and small at the horizon or small fisher boats. This happens in constant changing lights often frontal in changing weather conditions interrupted by waves. My weather data overlay is also poison for the largely on edge detection relying Deep Learning Matrix Algebra approach.  Additionally Dunoon, Inellan and Rothesay show lights at night. Seperating these from ships has in the moment been done by detecting only  in the water area {roi = Region of Interest].

roi = [1 631 1919 100];
[bboxes, score, label] = detect(rcnn, picture, roi, 'SelectStrongest', true, 'MiniBatchSize', 512); % 631,0 731,1920

I started with the standard CIFAR10 Transfer Learning example and the Image Labeler App

I also used the Alexnet and vgg19 pretrained networks that MATLAB offers.

I have two  NVidia Maxwell Titan X  (2 x 12 GB VRam) on a i7 5960 @ 3 GHz and 64GB RAM, so training the network is quite quick using the parallel toolbox.

In the original Cifar layers I added extra padding as I have often ferries ate the edges, I also reduced the stride in one maxpool layer.

middleLayers = [
% The first convolutional layer has a bank of 32 5x5x3 filters. A
% symmetric padding of 2 pixels is added to ensure that image borders
% are included in the processing. This is important to avoid
% information at the borders being washed away too early in the
% network.
%%convolution2dLayer(filterSize, numFilters, 'Padding', 2)
convolution2dLayer(filterSize, numFilters, 'Padding', 3)
%
% Note that the third dimension of the filter can be omitted because it
% is automatically deduced based on the connectivity of the network. In
% this case because this layer follows the image layer, the third
% dimension must be 3 to match the number of channels in the input
% image.
%
% Next add the ReLU layer:
reluLayer()
%
% Follow it with a max pooling layer that has a 3x3 spatial pooling area
% and a stride of 2 pixels. This down-samples the data dimensions from
% 32x32 to 15x15.
%maxPooling2dLayer(3, 'Stride', 2)
maxPooling2dLayer(3, 'Stride', 1)
%
% Repeat the 3 core layers to complete the middle of the network.
%%convolution2dLayer(filterSize, numFilters, 'Padding', 2)
convolution2dLayer(filterSize, numFilters, 'Padding', 3)
reluLayer()
%%maxPooling2dLayer(3, 'Stride',2)
maxPooling2dLayer(3, 'Stride',2)
%
convolution2dLayer(filterSize, 2 * numFilters, 'Padding', 2)
reluLayer()
maxPooling2dLayer(3, 'Stride',2)
%
]

Results from this were pretty good.

In the final layers I doubled the fully connected layers to 128 in one layer tp give the weights more chance to vary as the original net is very limited in it’s categories. I also played around with an extra drop out layer with varying probabilities to reduce over-fitting , but did not get good results.

finalLayers = [
%
%%dropoutLayer(0.1) %%
%
% Add a fully connected layer with 64 output neurons. The output size of
% this layer will be an array with a length of 64.
fullyConnectedLayer(128)
%%fullyConnectedLayer(64)
%
% Add an ReLU non-linearity.
reluLayer
%
% Add the last fully connected layer. At this point, the network must
% produce 10 signals that can be used to measure whether the input image
% belongs to one category or another. This measurement is made using the
% subsequent loss layers.
fullyConnectedLayer(numImageCategories)
%
% Add the softmax loss layer and classification layer. The final layers use
% the output of the fully connected layer to compute the categorical
% probability distribution over the image classes. During the training
% process, all the network weights are tuned to minimize the loss over this
% categorical distribution.
softmaxLayer
classificationLayer
]

in my options, I largely stuck with the standards

% Set the network training options
opts = trainingOptions('sgdm', ...
    'Momentum', 0.9, ...
    'InitialLearnRate', 0.001, ...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropFactor', 0.1, ...
    'LearnRateDropPeriod', 8, ...
    'L2Regularization', 0.001, ...
    'MaxEpochs', 100, ...
    'MiniBatchSize', 256, ...
    'Verbose', true);

For the R-CNN object detector I upped the learning rate and played a lot around with the bounding box Overlap Ranges for positive and negative overlap.

if doTraining
% Set training options
options = trainingOptions('sgdm', ...
'MiniBatchSize', 128, ...
'InitialLearnRate', 0.0001, ...
'LearnRateSchedule', 'piecewise', ...
'LearnRateDropFactor', 0.1, ...
'LearnRateDropPeriod', 8, ...
'L2Regularization', 0.04, ...
'MaxEpochs', 100, ...
'Shuffle','every-epoch', ...
'ExecutionEnvironment', 'auto', ...
'Verbose', true);
% Train an R-CNN object detector. This will take several minutes.
%%rcnn = trainRCNNObjectDetector(ferries, cifar10Net, options, ...
%%'NegativeOverlapRange', [0 0.3], 'PositiveOverlapRange',[0.5 1])
rcnn = trainRCNNObjectDetector(ferries, nnet, options, ...
'NegativeOverlapRange', [0 0.4], 'PositiveOverlapRange',[0.8 1], ...
'NumStrongestRegions', Inf);
else
% Load pre-trained network for the example.
%%load('rcnnStopSigns.mat','rcnn')
end

Later on I used alexnet with the same options but did not find a large divergence in the results.

The system works very well in calm water even at night. The biggest problem remains is that I have yet to feed in more birds and that small ships with a bow wave look very similar to a normal wave. As this would also be the case for humans, it will need many more training examples. Please note that while a ferry is a ship the system can detect ferries from ships.

Night Time Ferry Detection
Night Time Ferry Detection

 

Ship detection in deep Fog
Ship detection in deep Fog

 

Ferry and ship detection
Ferry and ship detection