Hi everyone!
Today I am going to share a code of object category and subcategory (instance) classification of objects using RGB, Depth and RGBD data. The dataset used is RGB- D Object Dataset.
https://rgbd-dataset.cs.washington.edu/
I am posting my code (tested on MATLAB 2016a) here, hoping to help any one out there to whom it may prove beneficial…….happy reading!
Before we go into the code, I would briefly explain the dataset to you. In this dataset, we have images of scenes with 51 different kinds of objects places around the room. Each type (category) of object can be of further different subcategories (these subcategories are called instances). For example ‘apple’ is a category name. Now, apples can be either red, green or golden. These are the three subcategories (or instances of apples).
The complete dataset can be used for scene categorization and object detection, followed by object classification. However, if you wish to do object classification alone (without detection of objects from scene images first), then you need to work on ‘evaluation database’ instead of the main database. This evaluation database has cropped images (RGB as well as depth images) of all objects, neatly arranged into folders and corresponding sub folders. Folders represent category labels and sub folders represent instances. We have a total of 51 categories and 300 instances in this data.
This evaluation database is the one that is the standard to evaluate object classification algorithms. You can download this directly from here:
http://rgbd-dataset.cs.washington.edu/dataset/rgbd-dataset_eval/
It’s a really simple MATLAB code and will help you learn how using a combination of three different feature extraction techniques (color autocorrelogram, wavelet transform and LBP features) can give you really good accuracies. When you run the code, you will get the following results using K Nearest Neighbor classifier:
| Classification | Accuracy |
| Depth category | 82.8% |
| Depth instances | 57.6% |
| RGB category | 90.4% |
| RGB instances | 83.8% |
| RGBD category | 95.41% |
| RGBD instances | 85.9% |
Approach used is this: First all data is read in the main for loop and then after that, it is randomly split into 70% for training and remaining 30% for testing. Each time you run the code, this 70 / 30 data division is done randomly, so the accuracies you get each time will not be exactly same. They would have a difference of +-3 %. So what you can do is run the code a couple of times and keep noting down values. Afterwards you can see which was the best combination of values you got, and mention them in your research results.
(P.S if you would like the same best combination of values to come each time, do let me know in the comments, I will explain that too then)
MAIN FILE – MATLAB CODE
clear
clc
path_orig_database= strcat(cd, ‘\rgbd-dataset_eval DATASET’);
category_subFolders_list = dir(path_orig_database);
category_subFolders_list= category_subFolders_list(3:end);
category_label=0;
target_set_category=[];
target_set_instance=[];
feature_set= [];
expt=0;
exceptionss=’emptyy’;
t_count= 0;
instance_count=0;
for i = 1: length(category_subFolders_list) % 51 categories
category_label= i
path_instance_folder= strcat(cd, ‘\rgbd-dataset_eval DATASET\’, category_subFolders_list(i).name);
instance_folders_in_category = dir(path_instance_folder);
instance_folders_in_category= instance_folders_in_category(3:end);
for j= 1 : length(instance_folders_in_category)
category_label
instance_count= instance_count+1
instance_sub_folder= strcat(cd, ‘\rgbd-dataset_eval DATASET\’, category_subFolders_list(i).name, ‘\’, instance_folders_in_category(j).name);
pngFiles = dir(strcat(instance_sub_folder, ‘*.png’)); % the folder in which ur images exists
for k = 1 : 15*3%length(pngFiles)
filename = strcat(instance_sub_folder, ‘\’,pngFiles(k).name);
if strcmp(filename(end-8), ‘_’) %_crop.png
% try
featurex= extract_f1(imread(filename));
t_count= t_count + 1;
feature_set(t_count, :)= featurex;
target_set_category(t_count, :)= category_label;
target_set_instance(t_count, :)= instance_count;
% catch
expt= expt+1;
exceptionss= pngFiles(k).name;
% end
end
if strcmp(filename(end-12:end), ‘depthcrop.png’) && ~strcmp(filename, exceptionss) %_crop.png
depth= im2double(imresize(imread(filename), [85, 85]));
depth_data_all(:, t_count)= depth(:);
depth_categoty_target_set(t_count, :)= category_label;
depth_instance_target_set(t_count, :)= instance_count;
end
% save RGBdonetillnow20
end
end
end
save RGBDdata
%% data division 70- 30
all_features_data= feature_set;
num_points = size(all_features_data,1);
split_point = round(num_points*0.7);
seq = randperm(num_points);
%% training and testing targets
Y_train_category = target_set_category(seq(1:split_point));
Y_train_instance = target_set_instance(seq(1:split_point));
Y_test_category = target_set_category(seq(split_point+1:end));
Y_test_instance = target_set_instance(seq(split_point+1:end));
%% training and testing RGB data
X_train_RGB = all_features_data(seq(1:split_point), :);
X_test_RGB = all_features_data(seq(split_point+1:end), :);
%% training and testing Depth data
all_features_data= depth_data_all’;
X_train = all_features_data(seq(1:split_point), :);
X_test = all_features_data(seq(split_point+1:end), :);
num_eigenfaces=51;
train_ims= X_train’;
mean_face = mean(train_ims, 2);%mean column
shifted_images_trn = train_ims – repmat(mean_face, 1, size(train_ims, 2));
[evectors, score, evalues] = princomp(train_ims’, ‘econ’);
evectors = evectors(:, 1:num_eigenfaces);
features = evectors’*shifted_images_trn;
X_train_Depth=features’;
shifted_images_tst = X_test’- repmat(mean_face, 1, size(X_test’, 2));
features = evectors’*shifted_images_tst;
X_test_Depth=features’;
%% KNN on RGB category recognition
c = knnclassify(X_test_RGB, X_train_RGB, Y_train_category);
percentErrors = sum(Y_test_category ~= c)/numel(c);
AC_rgb_cat_knn=100*(1-percentErrors)
%% KNN on RGB instance recognition
c = knnclassify(X_test_RGB, X_train_RGB, Y_train_instance);
percentErrors = sum(Y_test_instance ~= c)/numel(c);
AC_rgb_ins_knn=100*(1-percentErrors)
% KNN on Depth category recognition
c = knnclassify(X_test_Depth, X_train_Depth, Y_train_category);
percentErrors = sum(Y_test_category ~= c)/numel(c);
AC_depth_cat_knn=100*(1-percentErrors)
% KNN on Depth instance recognition
c = knnclassify(X_test_Depth, X_train_Depth, Y_train_instance);
percentErrors = sum(Y_test_instance ~= c)/numel(c);
AC_depth_ins_knn=100*(1-percentErrors)
% KNN on RGBD category recognition
c = knnclassify([X_test_RGB X_test_Depth], [X_train_RGB X_train_Depth], Y_train_category);
percentErrors = sum(Y_test_category ~= c)/numel(c);
AC_rgbd_cat_knn=100*(1-percentErrors)
% KNN on RGBD instance recognition
c = knnclassify([X_test_RGB X_test_Depth], [X_train_RGB X_train_Depth], Y_train_instance);
percentErrors = sum(Y_test_instance ~= c)/numel(c);
AC_rgbd_ins_knn=100*(1-percentErrors)
FUNCTION FILE- extract_f1.m
function featuresx= extract_f1(image)
image = imresize(image, [384 256]);
autoCorrelogram = colorAutoCorrelogram(image);
img = double(rgb2gray(image))/255;
wavelet_moments = waveletTransform(image);
M = repmat(all(~image,3),[1 1 3]); %mask black parts
image(M) = 255; %turn them white
x= histeq(rgb2gray(image), 200);
BW1 = edge(x,’prewitt’);
featuresx = [autoCorrelogram wavelet_moments, double(extractLBPFeatures(BW1))];%, [rgbhist(image, 3)]’ ];
end
FUNCTION FILE – colorAutoCorrelogram.m
function colorAutoCorrelogram = colorAutoCorrelogram(image)
% input: image in uint8 form, from wich to extract the color auto correlogram
% output: 1×64 feature vector containing the color auto correlogram
% quantize image into 64 colors = 4x4x4, in RGB space
[img_no_dither, map] = rgb2ind(image, 64, ‘nodither’);
% figure, imshow(img_no_dither, map);
rgb = ind2rgb(img_no_dither, map); % rgb = double(rgb)
% imshow(rgb);
% rgb = cat(3, r, g, b);
% clear workspace
clear(‘img_no_dither’);
% 4 predefined distances between
% neighbor pixel intensities
% according to “Image Indexing Using Color Correlograms” paper
distances = [1 3 5 7];
colorAutoCorrelogram = correlogram(rgb, map, distances);
colorAutoCorrelogram = reshape(colorAutoCorrelogram, [4 4 4]);
% consturct final correlogram using distances
colorAutoCorrelogram(:, :, 1) = colorAutoCorrelogram(:, :, 1)distances(1);
colorAutoCorrelogram(:, :, 2) = colorAutoCorrelogram(:, :, 2)distances(2);
colorAutoCorrelogram(:, :, 3) = colorAutoCorrelogram(:, :, 3)distances(3);
colorAutoCorrelogram(:, :, 4) = colorAutoCorrelogram(:, :, 4)distances(4);
% reform it to vector format
colorAutoCorrelogram = reshape(colorAutoCorrelogram, 1, 64);
end
% check if point is a valid pixel
function valid = is_valid(X, Y, point)
if point(1) < 0 || point(1) >= X
valid = 0;
end
if point(2) < 0 || point(2) >= Y
valid = 0;
end
valid = 1;
end
% find pixel neighbors
function Cn = get_neighbors(X, Y, x, y, dist)
cn1 = [x+dist, y+dist];
cn2 = [x+dist, y];
cn3 = [x+dist, y-dist];
cn4 = [x, y-dist];
cn5 = [x-dist, y-dist];
cn6 = [x-dist, y];
cn7 = [x-dist, y+dist];
cn8 = [x, y+dist];
points = {cn1, cn2, cn3, cn4, cn5, cn6, cn7, cn8};
Cn = cell(1, length(points));
for ii = 1:length(points)
valid = is_valid(X, Y, points{1, ii});
if (valid)
Cn{1, ii} = points{1, ii};
end
end
end
% get correlogram
function colors_percent = correlogram(photo, Cm, K)
[X, Y, ttt] = size(photo);
colors_percent = [];
for k = 1:K
countColor = 0;
color = zeros(1, length(Cm));
for x = 2:floor(X/10):X
for y = 2:floor(Y/10):Y
Ci = photo(x, y);
Cn = get_neighbors(X, Y, x, y, k);
for jj = 1:length(Cn)
Cj = photo( Cn{1, jj}(1), Cn{1, jj}(2) );
for m = 1:length(Cm)
if isequal(Cm(m), Ci) && isequal(Cm(m), Cj)
countColor = countColor + 1;
color(m) = color(m) + 1;
end
end
end
end
end
for ii = 1:length(color)
color(ii) = double( color(ii) / countColor );
end
colors_percent = color;
end
end
FUNCTION FILE- waveletTransform.m
function waveletMoments = waveletTransform(image)
% input: image to process and extract wavelet coefficients from
% output: 1×20 feature vector containing the first 2 moments of wavelet
% coefficients
imgGray = double(rgb2gray(image))/255;
imgGray = imresize(imgGray, [256 256]);
coeff_1 = dwt2(imgGray’, ‘coif1’);
coeff_2 = dwt2(coeff_1, ‘coif1’);
coeff_3 = dwt2(coeff_2, ‘coif1’);
coeff_4 = dwt2(coeff_3, ‘coif1’);
% construct the feaute vector
meanCoeff = mean(coeff_4);
stdCoeff = std(coeff_4);
waveletMoments = [meanCoeff stdCoeff];
end
CONTACT US
Contact us for all type of R&D projects and training on digital image processing, speech processing, Computer Vision, Robotics. AI, Machine learning, Data analysis, time series problems, regression, forecasting, numerical computation, electrical power systems, renewable energy, control systems, digital communication systems, bioinformatics and others.
Like and share our Facebook page with your friends:https://www.facebook.com/matlabexperts
https://www.facebook.com/xpertsvision/
Check a few of our projects at: http://www.eveati.com/portfolio.html
Keep visiting our blog to get valuable information:https://xpertsvision.wordpress.com/
You can share your queries/problems at:
Email: expertsvision@live.com
Contact numbers:
+966580031992 (Saudi Arabia)
+923345301505 (Pakistan)
+925187313856 (Pakistan)
+212610737228 (Morocco)
+16137106075(Canada)
WhatsApp: +966590815192
Experts Vision 