Thursday, May 1, 2014

HeadUp - A Webcam Posture Monitor


Since receiving a lot of feedback about my posture sensor solution I was thinking about how to improve it. Despite its functionality there are a few problems like the continuous clicking of the ultrasonic sensor and the way it is installed on the chair’s rest.


Ideas

One way of fixing the clicking noise would be to use an infrared-based proximity sensor. These are available in different operating ranges and prices but sadly I could not find one that was cheap enough and had the right range for the project.
Improving the chair holder is really difficult, too. It would require a lot of 3D-printed parts and still not work on chairs with low rests.
This meant the old idea had to be discarded completely and the project was suspended until I came across this great blog post about a pc-monitor-mounted sensor:

That’s how I got the idea of simply using the webcam as a sensor. Most people have cameras on their laptops and even desktop PCs so there is no additional hardware needed.


Implementation

Because I already used it in other projects I went with Processing as a programming language again:
To detect the position of the head the software has to scan the webcam image. This is a nearly impossible task without the help of a powerful library. OpenCV is the solution:
OpenCV is an Open Source library that contains computer vision functions and was originally developed by Intel. Face detection is just a small part of it.
The “OpenCV for processing” library by Greg Borenstein can easily be downloaded with the Library Manager in Processing and already contains some example sketches to start with.


The software I wrote works in a similar way as the Arduino code in the posture sensor. It scans the webcam image and compares the height and the size (~distance) of the face to limit values. If the limits are exceeded for more than two seconds the alarm will sound and then silence after a few seconds. If the user gets up from his chair or the face is not visible the alarm will not be triggered. The alarm itself is just the Windows error sound. The user interface consists of the live image in the background. Three buttons are for setting up the limit values and for pausing the alarm.
Finally Processing can export everything into a standalone application.


Download

If you have a computer with a webcam you can give “HeadUp” a try:

Instructions: 1. Download HeadUp.zip
                       2. Unzip everything into one folder
                       3. Run HeadUp.exe
Compatibility: Currently only windows is supported. And there are a few webcams it will not work with depending on the drivers. If there are problems you should try running the application in administrator mode. The newest version of java needs to be installed as well.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

I am also releasing the source code so you can customize and improve it:

Copyright (c) 2014 Coretech Robotics

import gab.opencv.*;
import processing.video.*;
import java.awt.*;
import java.awt.Toolkit;//this is needed for the windows sound

int ypos; // height
int rSig; //distance
int almTimer; 
int trigHeight = 0; //height limit
int trigDist = 0; //distance limit
boolean alm, pause;

Capture video;
OpenCV opencv;

void setup() {
  size(320, 240);
  //The next lines are for intitializing the camera and OpenCV
  video = new Capture(this, 640/4, 480/4);
  opencv = new OpenCV(this, 640/4, 480/4);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  
  video.start();
}

void draw() {
  getDistance(0);  // run the OpenCV routine
  if (trigHeight!=0 && trigDist!=0 && !pause) { //check if limits have been initialized
                                                //and if pause is off
    if (rSig > trigDist || ypos > trigHeight) { //compare values to limits
      alm = true; 
    }
    else {
      alm = false;
    }
  }

  if (alm==false) almTimer = millis()+2000;  //reset alarm timer if alarm is off
  else if (millis() > almTimer) { //check if alarm timer has expired
    if (millis()-2000 < almTimer) { //do this for additional 2 seconds
      Toolkit.getDefaultToolkit().beep(); //call the windows alarm sound
      delay(150); 
    } 
  }
  
//The following part draws the 2 buttons and checks if they were pressed
  
  textSize(14);
  fill(0, 255, 0);
  text("set distance", 18, 220);
  text("set height", 136, 220);
  text("pause", 248, 220);
  
  stroke(0, 255, 0);
  
  noFill();
    if(mousePressed && mouseOver(10, 200, 100, 30)) {
      trigDist = rSig+3;
      fill(0, 255, 0);
    }
  rect(10, 200, 100, 30);  
  
  noFill();  
    if(mousePressed && mouseOver(120, 200, 100, 30)) {
      trigHeight = ypos+3;
      fill(0, 255, 0);
    }
  rect(120, 200, 100, 30);
  
  noFill();
  if(pause) fill(0, 255, 0); // this part draws the pause switch
  rect(230, 200, 80, 30);
 
}

void getDistance(int interval) { //OpenCV functions
  //pushmatrix and popmatrix prevents the buttons from beeing scaled with the video
  pushMatrix(); 
  scale(2); // scales the video to the window size
  opencv.loadImage(video);
  
  image(video, 0, 0 ); // this draws the webcam image

  noFill();
  if (alm) stroke(255, 0, 0); //draw all lines red if alarm is active
  else stroke(0, 255, 0);
  strokeWeight(2);
  Rectangle[] faces = opencv.detect();
  int dist = 0;
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
    rSig = faces[i].height;
    ypos = faces[i].y;
    int delta = trigDist-faces[i].height;
    //the following line draws a second box with the limit distance
    if (trigDist!=0) 
    rect(faces[i].x-delta/2, faces[i].y-delta/2, faces[i].width+delta, trigDist);
  }
  //This draws a line at the limit height:
  if (trigHeight!=0) line(0, trigHeight, width, trigHeight);
  popMatrix();
}

void captureEvent(Capture c) { //important OpenCV stuff
  c.read();
}

void mouseReleased(){ //check if mouse was released so the switch gets triggered only once
  if(mouseOver(230, 200, 80, 30)) pause = !pause;
}

boolean mouseOver(int xpos, int ypos, int rwidth, int rheight){ 
  //return true if mouse is over a given rectangle
  if(mouseX > xpos && mouseX < xpos+rwidth && 
      mouseY > ypos && mouseY < ypos+rheight) return true;
  else return false;
}


Conclusion

The new software based posture sensor is reliable and has none of its predecessor’s flaws. You can easily run the program in the background while working on the computer and it will remind you if you are slouching into the chair or getting too close to the screen. 

About slouching: You should be aware that sinking into your chair and leaning back is not only more comfortable than sitting upright but does also relieve your spine. The software rather prevents you from hunching over which even unhealthier.

17 comments:

  1. Very good idea!!

    ReplyDelete
  2. a probarlo, gracias :D

    ReplyDelete
  3. I can`t run this program in Windows 7 x64

    ReplyDelete
    Replies
    1. Sorry but this depends on your webcam. If it's not compatible, there isn't much you can do besides running the app with admin rights and checking your Java version.

      Delete
  4. Hi! The idea is indeed cool.

    I've been using this for quite some time: https://github.com/keera-studios/keera-posture

    It've used it on all of my machines (win & linux) and it works very well. Plus, my back pain is gone after a few days of using it.

    I think there is a new version coming up that will (optionally) work with a kinect. If it can detect the actual limbs, it should be able to tell you more than just "your head is here or there".

    ReplyDelete
  5. Chrome flags this zip file as malicious. Any idea why?

    ReplyDelete
    Replies
    1. Crome flags every zip file with an unsigned executable inside. However it should work with most other browsers. Internet Explorer downloads it without complaining.

      Delete
  6. it's a wonderful idea. you did the fantisic job! i'm working for small customization for this. Exising application is not alerting the user when the applet program is minimized in windows task bar. But i want to work all the time. Any one has any idea about this?

    ReplyDelete
    Replies
    1. I already did that :)
      This is kind of the second version of the software:
      https://www.dropbox.com/s/ahgsp9tnk4ho6zd/headup2WoJava.zip?dl=1

      Delete
  7. Thanks for the posture alarm mention, Max! Bill, PaleoSun, Inc.

    ReplyDelete
  8. HOLA, TU IDEA PUEDE REALIZARSE EN VISUAL BASIC?

    ReplyDelete
  9. This is sooooo coooool! Very useful to me. I had a similar idea a yesterday, and I'm glad to find a solution from you!

    ReplyDelete
  10. Any updates on this? Love the idea and just want to make sure I test out the latest version of the software

    ReplyDelete
    Replies
    1. Glad you like it. I haven't actually worked on the project in the past two years.
      But there is an updated version that allows the app to be minimized to the taskbar. A little Popup window reminds you if you get too close to the camera. I just tested it and it still works on Windows 10. You only need to install the Java RTE. And there normally should be a banner image, which seems to be missing.
      https://www.dropbox.com/s/ahgsp9tnk4ho6zd/headup2WoJava.zip?dl=1
      If you are interested I can look for the Processing source files.

      Delete
  11. I get an Error: Resitry Key "Software\JavaSoft\Java Runtime Environment\CurrentVersion" hav value 1.8 but 1.7 is required,
    I tried changing it, but I am not sure how, it didn't seem to work (I believe I have the 1.7 jre also)
    Using windows 10

    ReplyDelete
    Replies
    1. This seems to be a general java error, which is not connected with the app itself.
      You can try and reinstall java or look at the answers here:
      http://stackoverflow.com/questions/29697543/registry-key-error-java-version-has-value-1-8-but-1-7-is-required

      Delete
  12. I love the idea, but the app consumes more than 60% of the CPU on a moderately modern core i5. It's too much.

    ReplyDelete