Camera Tracking with JavaScript

Tuesday, October 4th, 2016

With a variety of I/O options today's smart devices have a lot of potential to leverage user interaction. The most common peripheral which is embedded in almost all smartphones, laptops and computers is the camera. Developers can use this to build better apps that enable better ways to interact with the webpage.

In this blog I have discussed the implementation of visual object tracking using camera. I will be using the tracking.js library. The goal of this post is to show how visual object tracking can be implemented for JavaScript based applications. So I will use the WebGL app from my previous blog post to demonstrate this library.

Goal: Use visual tracking to change orientation of the JavaScript 3D object.

Step1:

Download tracking.js through this link https://trackingjs.com/ . You can either use .zip file or install it using 'bower'. In this project I have used 'bower' to install the library.

Step2:

Add video and canvas elements in index.html file under the root directory. The video element will be responsible for showing the webcam video and the canvas element will show the object co-ordinates. Also include the tracking-min.js file from your earlier bower installation.

<html>
    <body>
        <script src="assets/three.min.js"></script>
        <script src="assets/TrackballControls.js"></script>
        <script src="scripts/camera.js"></script>
        <script src="scripts/main.js"></script>


        <script src="bower_components/tracking/build/tracking-min.js"></script>
        <link rel="stylesheet" type="text/css" href="theme.css">
        <link rel="stylesheet" type="text/css" href="styles/main.css">

<div class="demo-frame">
    <div class="demo-container">
      <video id="video" width="300" height="250" preload autoplay loop muted controls></video>
      <canvas id="canvas" width="300" height="250"></canvas>
    </div>
    </div>

    </body>
</html>

Add this to main.css.

 #video, #canvas {
    margin-left: 100px;
    margin-top: 35px;
    position: absolute;
    z-index: 200000;
  }

Step3:

Create camera.js file under the scripts folder.

//global variable for storing x, y position of camera tracking.
track_position = {x: 0, y: 0};

window.onload = function() {
      var video = document.getElementById('video');
      var canvas = document.getElementById('canvas');
      var context = canvas.getContext('2d');

      //Tracking yellow color in the video.
var tracker = new tracking.ColorTracker(['yellow']);

      tracking.track('#video', tracker, { camera: true });
      tracker.on('track', function(event) {
        context.clearRect(0, 0, canvas.width, canvas.height);
        event.data.forEach(function(rect) {

          //Drawing rectangle for representation.
if (rect.color === 'custom') {            rect.color = tracker.customColor;          }          context.strokeStyle = rect.color;          context.strokeRect(rect.x, rect.y, rect.width, rect.height);          context.font = '11px Helvetica';          context.fillStyle = "#fff";          context.fillText('x: ' + rect.x + 'px', rect.x + rect.width + 5, rect.y + 11);          context.fillText('y: ' + rect.y + 'px', rect.x + rect.width + 5, rect.y + 22);

          //Update global position variables.

          track_position.x = rect.x;
          track_position.y = rect.y;
        });
      });
    };

This will help us track the position of the yellow object from the user's webcam. The track_position variable is global object that contains the x and y co-ordinates for the yellow object this is available throughout the app.

Step 4:

Update animateMotion() function in main.js file.

            function animateMotion (mesh3,mesh2,mesh4,refx2,refx3,refz3,refz4,refy4,refy2) {
                var i =0;
                setInterval(function(){
                if(i===360)i=0;
                mesh3.position.x = refx3*Math.cos(i);
                mesh3.position.z = refz3*Math.sin(i);
                mesh2.position.x = refx2*Math.cos(i);
                mesh2.position.y = refy2*Math.sin(i);
                mesh4.position.y = refy4*Math.sin(i);
                mesh4.position.z = refz4*Math.cos(i);
                
//Updating 3D camera's x and y co-ordinates using
// global track_position object. 
                camera.position.x = track_position.x*10;
                camera.position.y = track_position.y*10;
                camera.position.z = 700;


                i=i+0.04;
                render();
                },5);
            }

The only change made to the original code is that now the 3D camera's x and y positions are being updated according object tracking position in real-time.

Step 5:

Run the app.

root@ubuntu:~# node app.js

Object translation relationship:

Whenever the yellow object in the webcam moves left it updates the global x value. This enables the atom to move left. Also when the yellow object moves down starting from (0,0) the atom moves down. The representation axis is horizontally flipped for the webcam canvas so when the yellow object is physically moving left the webpage shows it moving right.

The above image shows how co-ordinates of the yellow object are represented.

Working Demo: https://comicatom-track.herokuapp.com/