August 22, 2022 0
Face Filters

By Akash Goyal

What are Face Filters?

The new generation of technology has always been a blessing, especially for Social networks. News can be shared faster than the fire, Multiple Files get sent in Single click and thanks to modern apps, one can easily play with photos & make something amazing out of it. Speaking of photos, We all like to scroll through social media apps for the trending cool facial image and it’s really amazing to see the perfect static masking, painting, makeup or any other animated effect? Like the infamous snapchat dog tongue effect? That’s a face filter in action. 

Face filters are augmented reality effects enabled with face recognition technology. It applies/overlays/blends 3D objects and animation on the face in real-time, so you can capture engaging videos and photos. 

Popular Apps & Cool Examples

MSQRD

Launched at a time when other applications could only make changes to still pictures. MSQRD allows the users to modify their face in hilarious ways, all in real time, in an almost magical way. Take pictures or record short videos. Facebook purchased MSQRD in 2016 and it was removed from Play & App Stores in 2020. 

Muglife  

Launched in 2017, this is a one of its kind, obscenely juvenile, but really powerful. Simple Photos turns into Super-Realistic 3D Animations, putting users in jaw dropping moments.

Snapchat 

The decade old, multimedia instant messaging app. Express yourself quickly with millions of lenses. The snap filters `lenses` are very creative and have a uniqueness in the space. 

Instagram

 Photo and video sharing social networking application, now acquired by Facebook, lets you express yourself in new ways with many Insta filters and Reels features. 

Technology in use

All these apps use a MULTITUDE OF TECHNOLOGIES – Artificial Intelligence techniques combined with Augmented Reality, Computer graphics expertise with the latest advances in computer vision. To name a few – Landmark Detection, Segmentation, Active Shape Modeling, Triangulation, etc. 

Augmented Reality

Blend the digital and physical worlds. AR users experience the real-world environment with generated perceptual information overlaid on top of it. AR Filters are digitally based on responsive interactions that are applied to the user’s face or environment to enhance or modify what is being seen in the real world. Brands and social networks (SNs) are using augmented reality (AR) filters more frequently to provide users with best experiences.

Artificial Intelligence

Simulation of intelligence in machines. AI is intelligence demonstrated by machines. the ability of a digital computer or computer-controlled robot to perform tasks commonly associated with intelligent beings

Segmentation

Grouping similar regions or segments of an image under their respective class labels.Digital images are divided into several subgroups known as “image segments,” which serves to simplify subsequent processing or analysis of the image by decreasing the complexity of the original image.Segmentation in easy words is assigning labels to pixels 

Active Shape Model

Statistical models of the shapes of objects which iteratively deform to fit an object in a new image.The shape of an object is represented through landmarks which are one chain of consecutive traits points, each of which is important, point existent in most of the images being considered

Facial Points Detection

Predict keypoint positions on face images. This can be used as a building block in several applications, such as tracking faces in images and video analyzing, facial expressions, detecting dysmorphic facial signs for medical diagnosis, biometrics / face recognition

Detecting facial keypoints is a very challenging problem. Facial features vary greatly from one individual to another, and even for a single individual, there is a large amount of variation due to 3D pose, size, position, viewing angle, and illumination conditions.

The technology of MUGLIFE as described in their official website. 

Mug Life technology is broken into three stages:

1). Deconstruction – In the Deconstruction stage, Mug Life uses computer vision technology to analyze and decompose photos into 3D building blocks: camera, lighting, geometry, and surface texture.

​2). Animation – In the Animation stage, we use state-of-the-art cinema animation techniques to manipulate the extracted 3D data, while preserving photorealism.

​3). Reconstruction – Finally, in the Reconstruction stage, we leverage cutting edge video game technology to re-render your photo as an animated 3D character with stunning quality and detail.

The technology of MSQRD app is believed to play a key role in boosting Facebook’s internal portfolio of AR image and video tools, the SparkAR. Significant contribution is believed to be by LOOKSERY, which was acquired by SnapChat, in their lenses and SnapML tool. 

Creating a Simple Face Filter

Creating a lens/filter of your own is easy for these apps. Plug in a model to apply art style to the camera feed, use custom segmentation masks, attach images to custom detected objects, understand what is in the scene, and more, using SnapML(for SnapChat), SparkAR(for Meta). 

 

## import packages 

import cv2

import numpy as np

 

## load haar files 

face_cascade = cv2.CascadeClassifier(‘haarcascade_frontalface_default.xml’)

eye_cascade = cv2.CascadeClassifier(‘haarcascade_eye.xml’)

 

## method definition 

def placeSticker(imgPath, stickerPath):

    img, st = cv2.imread(imgPath), cv2.imread(stickerPath) 

    orig_st_h, orig_st_w, st_channels = st.shape 

    img_h, img_w, img_channels = img.shape 

    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) 

    st_gray = cv2.cvtColor(st, cv2.COLOR_BGR2GRAY)

 

    #use THRESH_BINARY_INV if sticker has transparent background

    ret, orig_mask = cv2.threshold(st_gray, 10, 255, cv2.THRESH_BINARY)

    orig_mask_inv = cv2.bitwise_not(orig_mask)    

    

    #find faces in image using classifier

    faces = face_cascade.detectMultiScale(img_gray, 1.3, 5)

    for (x,y,w,h) in faces:    

        face_w, face_h = w, h

        Face_x1, face_y1 = x, y

        face_x2 = face_x1+face_w

        face_y2 = face_y1+face_h

 

        #sticker size wrt face

        st_width = int(1.5 * face_w)

        st_height = int(st_width * orig_st_h/orig_st_w)

 

        # sticker position coordinates

        st_x1 = face_x2 – int(face_w/2) – int(st_width/2)

        st_x2 = st_x1 + st_width

        st_y1 = face_y1  – int(face_h*0.75)

        st_y2 = st_y1 + st_height 

 

        # out of frame check & update 

        st_x1 = 0 if st_x1<0 else st_x1

        st_y1 = 0 if st_y1<0 else st_y1

        st_x2 = img_w if st_x2>img_w else st_x2

        st_y2 = img_h if st_y2>img_h else st_y2

        st_width = st_x2 – st_x1

        st_height = st_y2 – st_y1

 

        #resize sticker

        st = cv2.resize(st, (st_width,st_height), interpolation = cv2.INTER_AREA)

        mask = cv2.resize(orig_mask, (st_width,st_height), interpolation = cv2.INTER_AREA)

        mask_inv = cv2.resize(orig_mask_inv, (st_width,st_height), interpolation = cv2.INTER_AREA)

 

       # placing sticker into bg

        roi = img[st_y1:st_y2, st_x1:st_x2]

        roi_bg = cv2.bitwise_and(roi,roi,mask = mask)

        roi_fg = cv2.bitwise_and(st,st,mask=mask_inv)

        dst = cv2.add(roi_bg, roi_fg)

        img[st_y1:st_y2, st_x1:st_x2] = dst

    cv2.imwrite(imgPath+’res.png’,img)

## test code 

imgPath = ‘image1.jpg’

stickerPath = ‘sticker2.png’

placeSticker(imgPath, stickerPath)

 

 

From applying styles on static media to processing a variety of media with effects in real-time; Face Filters have actually come a long way in a short time. It might evolve to such a stage which is not imagined yet. The evolution has expressed creativity and posed challenges as well. Same effects can be used to make or break somebody’s mood. The open technology should be used responsibly to make the world a better place to live in.

I hope this blog has added well to your understanding of the popular apps & their face filters features. Stay tuned to this space for more technical insights on the topic.

Leave a comment