Kalman away

Well took long enough but I have a Kalman filter working. Thanks to help and code from my dissertation supervisor Hannah Dee. I have managed to tweak her code that uses Kalman2d which can be found here to fit in with my scenario. Example of what it is currently doing can be seen below.

Here shows the Kalman filter in action(the green line) however at this point it jumps between the two people in the box currently

I have added some basic checking for the HoG(Histogram of orientated gradients) and Kalman filter so that if a HoG is not detected the bounding box is drawn in the last place it was located. If it then re appears the Kalman filter then is updated with the new location. This is done by searching with a buffer with in the range of last HoG detection, this is assuming the person has not travelled far and there is no one else nearer.

The issues I am having with two people coming near to each other that the tracker swaps over is not an issue. The reason being as if two people come close to each other and one gets the tracker from the other then the other one will get their tracker. One way I can mitigate this is to start the search in the direction the person was travelling so that way the person is more likely to get their tracker back.

I am currently only tracking one person but now I have a goal I feel I can really push this through. My goal is to build a person object that will keep track of locations to help mitigate occlusions and count them. This will be using HoG detector as the initialiser, mean shift, Kalman filter and cam shift to help determine location helping keep a more accurate track of people. Background subtraction will be used as evidence to show that something is moving here chances are someone or something is here.

I am hoping to be able to determine in scene exits by determining if a number of person detectors go dead at the same location. If they are predicted to go off scene then that is a normal exit but can monitor exits to focus detectors more.

Any way this will be my last blog post for a while as I am off to sunny Lloret de Mar for my first vacation abroad with dance sport. Should be a good time sportsvest who are organising it have made it look like it is going to be an epic time. So speak in a week and a bit.

Lloret de Mar

Kalman filter pain

So over the past weeks I have been having some issues trying to get the Kalman filter to work to predict location of the people in the scene. This is going to be used to determine if they are in scene but occluded so still need to count them.

Several problems I was having involved everywhere I looked the tutorials told me to use something which wasn't in the OpenCV version I had or wasn't recognised. As such I was upgraded OpenCV to 2.4.11 rather than my current
version, 2.4.10.

Upgrading OpenCV was more of an issue than expected and had a few issues with the building. My first issue was I ran out of space on my laptop.

Turns out issue I was having was with using cv2 not cv which I just thought the way the examples were named. So after importing cv2.cv as cv I managed to get the Kalman filter working. Now I can start using it to determine where people are heading. First I need to get it so it stays on one person as HoG(Histogram of orientated gradients) sometimes picks a different person to find first.

 As well as trying to get the Kalman filter working I also did some tests to determine best moving average background subtraction to use. As well as this I calculated the time for completion on a set of 795 images, my PETs 2009 s2l1 data set. Results can be seen in the graphs below.

Compared moving average bg subtraction with photo shopped image using technique from here

I am still working on trying to get a outline of a person for my ROI(Region of interest) rather than a tight box to reduce the errors that are thrown. I am currently very close to getting this working but needs a few more tweaks and should be good.

A bit of house cleaning was in order as I had a few Python files in my directory that needed removing. It still needs a bit of a clean but it is much easier to navigate now.


 When everything is implemented properly then I can start work on different combinations of detection and see which combinations are the most accurate and the fastest. I can try this on other data sets as well to see how well they preform under different circumstances.

With the features implemented I can also catch up on my tests for them. As well as cleaning up my code with more comments and better layout.

Hopefully the next lot of stuff to do after Kalman filter will be a little less frustrating.

Points, assumptions and more graphs!

Ok so it has been a busy week so here comes a list of what I have been up to with my dissertation. Don't worry there are pictures.

First thing I got working was MoG(Mixture of Gaussian) background subtraction using this page on OpenCV. This is to allow me to detect movement within the image that can help determine the likeliness of a person existing at that point. I have also implemented moving average background subtraction from this helpful site. I will be creating a test to try different values for moving average background subtraction to see which one is the best to use.

I have also implemented Cam shift which is starts off by doing a mean shift but then updates the size of the window and calculates the best rotation of a fitting ellipse to it. Then re applies the mean shift with the new scaled search window and previous window position. This is then repeated until the desired accuracy is met. (Paraphrased from here).


I am dealing with the problem of people being obscured by using a Kalman filter to determine the direction and velocity of people. From this I can determine if they are going to end up off screen or if they are still with in the scene but obscured. The issue may arise if there is an exit on screen that the person can disappear through. This issue can be dealt with by assuming that if the person isn't detected with in a number of frames then they are likely no longer in the scene. I won't be modelling the scene as that is a whole other project.


I have also made some more wonderful graphs to show the number of correct detections that each setting for the HoG detector provides. These can be seen below

Patterns include: Multiples of 4 having 0 detections and multiples of 6 having highest number of detections.

Steady drop of accuracy after scale setting of 1.01

This peaks at 3 but may go back up for higher numbers

My code is getting a little messier than I would like it to be. For this reason I will be going over my code and modularising it and doing some more testing for the new code I have done.


I am working on a points based system that will give points to the more accurate detections that are all consistent. This will use mean shift, cam shift and hog detectors to build up a more accurate person detector. I will create an array of ROIs(regions of interest) for each person depending on the score they get to build up what to look for.


I have encountered some problems along the way. Some of these include contours from background subtraction overlapping with other HoG boundaries meaning false positive when looking for movement in a HoG detection. Another issue is a 'non convex optimisation problem' with trying to search through the most accurate window stride for HoG detector. This is trying to avoid searching through all the possible combinations by going in the direction of the best detector settings in chunks.


An interesting point that was raised in a dissertation meeting was, should you count people that are occluded. If a person was counting number of people in a scene and a person was occluded that person would not be counted. There may be times when you want to know if a person is still in a scene even when obscured. An example would be when a place is limited to a certain number of people but they will be blocked by certain elements. In this case keeping track of number of people is important.

In the case of my dissertation I will be keeping track of all people that are judged to still be with in the scene.

The ups and down of graphs!!!!

After having some issues I managed to get my graphs for the different settings for the HoG detector and this is what they look like.

This is the number of correct detections for a few HoG settings

This is the number of offsets from the Ground truth for a few HoG Settings

 

 

 

This is only based upon a few frame so that it did not take forever to complete. The graphs as well only contain a select few HoG settings, otherwise it would be to busy and un readable.

I am currently working on trying to try different values between bigger numbers. This will allow me to try more combinations without trying all the combinations between.

The mean shift, the background subtraction and lots of time

Over the past week and a bit I have had to re create my HoG(Histogram of orientated gradients) as I discovered I had the wrong values in it. This is currently running and will hopefully finish soon as it has been running for two days now. As well as this I have implemented mean shift to track people over a number of frames and looked at background subtraction to increase accuracy of detectors.


The various combinations of HoG detector settings come to a rough total of 20000 different combinations. These include settings such as scale between 1 and 1.1, window stride between 0 and 9 and scaling between 0 and 32. This will then be used to determine the best and most accurate settings to use for the HoG detector increasing the overall performance. An issue I had was that they were not in the correct order so I had to write a quick script to go through the lines and sort them out. Below you can see the output from the 20000 lines of combinations and results of the HoG detector on 8 test images.

As it shows best values and how many were correct and what the total offset is for that setting

This value takes a while to process an image so it may be worth searching different combinations in between. As currently, for padding which takes two numbers, I use the same one twice. So padding(6,6) rather than padding(4,8), the same can be said for window stride. This could be faster or result in a more accurate detector. This will be ran at a later date.

As well as using HoG I am also using mean shift to track the movement of people. This will be helpful in counting people when they are occluded and then picking them back up when they come into focus. An example of current progress can be seen below.

Mean shift over 10 frames trying to track top and bottom

As you can see the boxes are not that accurate or tight round the people so the tracking is not as good as I would like it to be. To try and fix this and deal with false positives like the window at the top of the image I have been looking at background subtraction. This will allow me to ignore the parts that are not moving such as the window and focus on the more likely areas to contain people. It may just still be a cat.


To also keep track of people and make sure the mean shift does not go to far away from the people I will be re checking the area to find people are still there using the HoG detector. This will be less cpu intensive as we can assume that the people, if still are in frame, will be near by so no need to check the entire image.


More graph to follow when I get them working properly :)