6 Easy Steps to Implement a Computer Vision Application Using Tensorflow.js

In this article, we are going to see how we can implement computer vision applications using tensorflow.js models.

By Bala Venkatesh, Data Scientist

Introduction

These days, many people implement computer vision applications. Do you think is it very difficult to learn and implement? my answer is NO because nowadays a lot of libraries available to implement such a powerful computer vision application. Recently have you watched the TensorFlow summit 2020? This year TensorFlow team announced a lot of cool things for us. Here we are going to see how we can implement computer vision applications using tensorflow.js models!!

What is TensorFlow.js?

TensorFlow.js is an open-source library for machine learning application in JavaScript and Develop ML models in JavaScript, and use ML directly in the browser or in Node.js.

What is the Tensorflow.js model?

Tensorflow.js models are pre-trained models which means you need not to prepare/collect data to train the model. The models are hosted on NPM and unpkg so they can be used in any project out of the box.

In this blog, we are going to see one of the models in tensorflow.js which is the MediaPipe Facemesh model. The model is designed for front-facing cameras on mobile devices, where faces in view tend to occupy a relatively large fraction of the canvas.

Facemesh Model Demo

I have used facemesh model to move the video frame as per my face movement. See the below demo. Let me tell you what are the steps involved to implement below computer vision application.

Demo

Demonstration

Step 1:- These are the three important standalone script tag codes to implement computer vision applications.

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/facemesh"></script>

TensorFlow.js Core, flexible low-level API for neural networks and numerical computation.

TensorFlow.js Converter, tools to import a TensorFlow SavedModel to TensorFlow.js.

The facemesh package finds facial boundaries and landmarks within an image.

Step 2:- Include video HTML tag in the body content to read face through webcam.

<video width=640 height=480 autoplay muted id=”camera”></video>

Step 3:- Use canvas tag to draw graphics, on the fly, via scripting (usually JavaScript).

<canvas width=640 height=480 id=”augmented_canvas”></canvas>

Step 4:- Include a video tag to play video frames on canvas tag to move video frames as per face movement.

<video autoplay loop id=”movie” style=”visibility: hidden”>
<source src=”TensorFlowjs.mp4" type=”video/mp4"></source>
</video>

Step 5:- Load face model and estimate the face to finds facial boundaries and landmarks within an image.

//load camera stream
const frame = document.getElementById("camera");//load movie stream
const movie = document.getElementById("movie");
movie.play();//prepare canvas
const canvas = document.getElementById("augmented_canvas");
const draw = canvas.getContext("2d");const result = await model.estimateFaces(frame, false);

Step 6:- Draw video frames on the detected face using the below code.

//copy camera stream to canvas
draw.drawImage(frame, 0, 0, 640, 480);//check if face is detected
 if(result.length > 0)
 {
  for (let i = 0; i < result.length; i++) {
    const start = result[i].topLeft;
    const end = result[i].bottomRight;
    const size = [end[0] — start[0], end[1] — start[1]];   //Render a rectangle over each detected face.
   draw.drawImage(movie, start[0], start[1], size[0], size[1]);
  }
 
}

Conclusion

That’s it, we have implemented a computer vision application following the above six steps. You can get the complete source code here.

Everybody says if you want to become a data scientist you should familiarize in python or R programming language but now we can implement a machine learning application using javascript.

Source: https://miro.medium.com/max/1400/1*F-1fq9TNjDnAYPAXnZP4Ww.png

Happy Learning to all!! if you need any help or assistance please connect with me LinkedIn and Twitter.

Bio: Bala Venkatesh is a Data Scientist. He has a passion for understanding technology at a fundamental level and sharing ideas and code.

Original. Reposted with permission.

Related: