
![Cover image](https://assets.zilliz.com/How_Short_video_Platform_Likee_Removes_Duplicate_Videos_with_Milvus_07bd75ec82.png "How Short-video Platform Likee Removes Duplicate Videos with Milvus")

> This article is written by Xinyang Guo and Baoyu Han, engineers at BIGO, and translated by [Rosie Zhang](https://www.linkedin.cn/incareer/in/rosie-zhang-694528149).

[BIGO Technology](https://www.bigo.sg/) (BIGO) is one of the fastest-growing Singapore technology companies. Powered by artificial intelligence technology, BIGO's video-based products and services have gained immense popularity worldwide, with over 400 million users in more than 150 countries. These include [Bigo Live](https://www.bigo.tv/bigo_intro/en.html?hk=true) (live streaming) and [Likee](https://likee.video/) (short-form video). 

Likee is a global short video creation platform where users can share their moments, express themselves, and connect with the world. To increase user experience and recommend higher-quality content to users, Likee needs to weed out duplicate videos from a tremendous amount of videos generated by users every day, which poses no straightforward task. 

This blog presents how BIGO uses [Milvus](https://milvus.io), an open-source vector database, to effectively remove duplicate videos.

**Jump to:**
- [Overview](#Overview)
- [Video deduplication workflow](#Video-deduplication-workflow)
- [System Architecture](#System-architecture)
- [Using Milvus to power similarity search](#Using-Milvus-vector-database-to-power-similarity-search)

# Overview
Milvus is an open-source vector database featuring ultra-fast vector search. Powered by Milvus, Likee is able to complete a search within 200ms while ensuring a high recall rate. Meanwhile, by [scaling Milvus horizontally](https://milvus.io/docs/v2.0.x/scaleout.md#Scale-a-Milvus-Cluster), Likee successfully increases the throughput of vector queries, further improving its efficiency.

# Video deduplication workflow
How does Likee identify duplicate videos? Every time a query video is input into Likee's system, it will be cut into 15-20 frames and each frame will be converted to a feature vector. Then Likee searches in a database of 700 million vectors to find the top K most similar vectors. Each one of the top K vectors corresponds to a video in the database. Likee further conducts refined searches to get the final results and determine the videos to be removed.

# System architecture
Let's take a closer look at how Likee's video de-duplication system works using Milvus. As is shown in the diagram below, new videos uploaded to Likee will be written to Kafka, a data storage system, in real-time and consumed by Kafka consumers. The feature vectors of these videos are extracted through deep learning models, where unstructured data (video) is converted into feature vectors. These feature vectors will be packaged by the system and sent to the similarity auditor.

![Architechure of Likee's video de-duplication system](https://assets.zilliz.com/Likee_1_6f7ebcd8fc.png "Architechure of Likee's video de-duplication system")

The feature vectors extracted will be indexed by Milvus and stored in Ceph, before being [loaded by the Milvus query node](https://milvus.io/blog/deep-dive-5-real-time-query.md) for further search. The corresponding video IDs of these feature vectors will also be stored simultaneously in TiDB or Pika according to the actual needs.

### Using Milvus vector database to power similarity search
When searching for similar vectors, billions of existing data, along with large amounts of new data generated every day, pose great challenges to the functionality of the vector search engine. After thorough analysis, Likee eventually chose Milvus, a distributed vector search engine of high performance and high recall rate, to conduct vector similarity search.

As is shown in the diagram below, the procedure of a similarity search goes as follows:

1. First, Milvus performs a batch search to recall the top 100 similar vectors for each of the multiple feature vectors extracted from a new video. Each similar vector is bound to its corresponding video ID. 

2. Second, by comparing the video IDs, Milvus removes the duplicate videos and retrieves the feature vectors of the remaining videos from TiDB or Pika. 

3. Finally, Milvus calculates and scores the similarity between each set of the feature vectors retrieved and the feature vectors of the query video. The video ID with the highest score is returned as the result. Thus the video similarity search is concluded.

![Procedure of a similarity search](https://assets.zilliz.com/02_a24d251c8f.png "Procedure of a similarity search")

As a high-performance vector search engine, Milvus has done an extraordinary job in Likee's video de-duplication system, greatly fueling the growth of BIGO's short-video business. In terms of video businesses, there are many other scenarios where Milvus can be applied to, such as illegal content blocking or personalized video recommendation. Both BIGO and Milvus are looking forward to future cooperation in more areas.

How Short-video Platform Likee Removes Duplicate Videos with Milvus


![cover image](https://assets.zilliz.com/zhentu_0ae11c98ee.png "Zhentu - the Photo Fraud Detector Based on Milvus")

> This article is written by Yan Shi and Minwei Tang, senior algorithm engineers at BestPay, and translated by [Rosie Zhang](https://www.linkedin.cn/incareer/in/rosie-zhang-694528149).

In recent years, as e-commerce and online transactions become commonplace throughout the world, e-commerce fraud also flourished. By using computer-generated photos instead of real ones to pass identity verification on online business platforms, fraudsters create massive fake accounts and cash in on businesses' special offers (e.g. membership gifts, coupons, tokens), which brings irretrievable losses to both consumers and businesses.

Traditional risk control methods are no longer effective in the face of a substantial amount of data. To solve the problem, [BestPay](https://www.bestpay.com.cn) created a photo fraud detector, namely Zhentu (meaning detecting images in Chinese), based on deep learning (DL) and digital image processing (DIP) technologies. Zhentu is applicable to various scenarios involving image recognition, with one important offshoot being the identification of fake business licenses. If the business license photo submitted by a user is very similar to another photo already existing in a platform's photo library, it is likely that the user has stolen the photo somewhere or has forged a license for fraudulent purposes.

Traditional algorithms for measuring image similarity, such as [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) and ORB, are slow and inaccurate, only applicable to offline tasks. Deep learning, on the other hand, is capable of processing large-scale image data in real-time and is the ultimate method for matching similar images. With the joint efforts of BestPay’s R&D team and [the Milvus community](https://milvus.io/), a photo fraud detection system is developed as part of Zhentu. It functions by converting massive amounts of image data into feature vectors through deep learning models and inserting them into [Milvus](https://milvus.io/), a vector search engine. With Milvus, the detection system is able to index trillions of vectors and efficiently retrieve similar photos among tens of millions of images.

**Jump to:**
- [An overview of Zhentu](#an-overview-of-zhentu)
- [System structure](#system-structure)
- [**Deployment**](#deployment)
- [**Real-world performance**](#real-world-performance)
- [**Reference**](#reference)
- [**About BestPay**](#about-bestpay)

## An overview of Zhentu

Zhentu is BestPay’s self-designed multimedia visual risk control product deeply integrated with machine learning (ML) and neural network image recognition technologies. Its built-in algorithm can accurately identify fraudsters during user authentication and respond at the millisecond level. With its industry-leading technology and innovative solution, Zhentu has won five patents and two software copyrights. It is now being used in a number of banks and financial institutions to help identify potential risks in advance.

## System structure

BestPay currently has over 10 million business license photos, and the actual volume is still growing exponentially as the business grows. In order to quickly retrieve similar photos from such a large database, Zhentu has chosen Milvus as the feature vector similarity calculation engine. The general structure of the photo fraud detection system is shown in the diagram below.

![img](https://assets.zilliz.com/Structure_of_the_photo_fraud_detection_system_cf5d20d431.png "Structure of the photo fraud detection system")

The procedure can be divided into four steps:

1. Image pre-processing. Pre-processing, including noise reduction, noise removal, and contrast enhancement, ensures both the integrity of the original information and the removal of useless information from the image signal.

2. Feature vector extraction. A specially trained deep learning model is used to extract the feature vectors of the image. Converting images into vectors for further similarity search is a routine operation.

3. Normalization. Normalizing the extracted feature vectors helps to improve the efficiency of the subsequent processing.

4. Vector search with Milvus. Inserting the normalized feature vectors into Milvus database for vector similarity search.

## **Deployment**

Here is a brief description of how Zhentu's photo fraud detection system is deployed.

![Milvus system architecture](https://assets.zilliz.com/milvus_architecture_ea45a5ab53.png)

We deployed our [Milvus cluster on Kubernetes](https://milvus.io/docs/v2.0.x/install_cluster-helm.md) to ensure high availability and real-time synchronization of cloud services. The general steps are as follows:

1. View available resources. Run the command `kubectl describe nodes` to see the resources that the Kubernetes cluster can allocate to the created cases.

2. Allocate resources. Run the command `kubect`` -- apply xxx.yaml` to allocate memory and CPU resources for Milvus cluster components using Helm.

3. Apply the new configuration. Run the command `helm upgrade my-release milvus/milvus --reuse-values -fresources.yaml`.

4. Apply the new configuration to the Milvus cluster. The cluster deployed in this way not only allows us to adjust system capacity according to different business needs, but also better meets the high-performance requirements for massive vector data retrieval.

You can [configure Milvus](https://milvus.io/docs/v2.0.x/configure-docker.md) to optimize search performance for different types of data from different business scenarios, as shown in the following two examples.

In [building the vector index](https://milvus.io/docs/v2.0.x/build_index.md), we parameterize the index according to the actual scenario of the system as follows:

```Python
index = {"index_type": "IVF_PQ", "params": {"nlist": 2048}, "metric_type": "IP"}
```

[IVF_PQ](https://milvus.io/docs/v2.0.x/index.md#IVF_PQ) performs IVF index clustering before quantizing the product of vectors. It features high-speed disk query and very low memory consumption, which meets the needs of the real-world application of Zhentu.

Besides, we set the optimal search parameters as follows:

```Python
search_params = {"metric_type": "IP", "params": {"nprobe": 32}}
```

As the vectors are already normalized before input into Milvus, the inner product (IP) is chosen to calculate the distance between two vectors. Experiments have proved that the recall rate is raised by about 15% using IP than using the Euclidean distance (L2). 

The above examples show that we can test and set Milvus' parameters according to different business scenarios and performance requirements. 

In addition, Milvus not only integrates different index libraries, but also supports different index types and similarity calculation methods. Milvus also provides official SDKs in multiple languages and rich APIs for insertion, querying, etc., allowing our front-end business groups to use the SDKs to call on the risk control center.

## **Real-world performance**

So far, the photo fraud detection system has been running steadily, helping businesses to identify potential fraudsters. In 2021, it detected over 20,000 fake licenses throughout the year. In terms of query speed, a single vector query among tens of millions of vectors takes less than 1 second, and the average time of batch query is less than 0.08 seconds. Milvus' high-performance search meets businesses' needs for both accuracy and concurrency.

## **Reference**

Aglave P, Kolkure V S. Implementation of High Performance Feature Extraction Method Using Oriented Fast and Rotated Brief Algorithm[J]. Int. J. Res. Eng. Technol, 2015, 4: 394-397. 

## **About BestPay**

China Telecom BestPay Co., Ltd is a wholly owned subsidiary of China Telecom. It operates the payment and finance businesses. BestPay is committed to using cutting-edge technologies such as big data, artificial intelligence and cloud computing to empower business innovation, providing intelligent products, risk control solutions and other services. Up to January 2016, the app called BestPay has attracted over 200 million users and become the third largest payment platform operator in China, closely following Alipay and WeChat Payment.


Zhentu - the Photo Fraud Detector Based on Milvus


This article is translated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).


Information is everywhere in our lives. Meta (formerly known as Facebook), Instagram, Twitter, and other social media platforms make information streams all the more ubiquitous. Therefore, engines dealing with such information streams have become a must-have in most system architecture. However, as a user of social media platforms and relevant apps, I bet you must have been bothered by duplicate articles, news, memes, and more. Exposure to duplicate content hampers the process of information retrieval and leads to bad user experience.


For a product dealing with information streams, it is a high priority for the developers to find a flexible data processor that can be integrated seamlessly into the system architecture to deduplicate identical news or advertisements.

[SmartNews](https://www.smartnews.com/en/), valued at [2 billion US dollars](https://techcrunch.com/2021/09/15/news-aggregator-smartnews-raises-230-million-valuing-its-business-at-2-billion/), is the most highly-valued news app company in the US. Noticeably, it used to be a user of Milvus, an open-source vector database, but later transformed into an active contributor to the Milvus project.

This article shares the story of SmartNews and tells why it decided to make contributions to the Milvus project.

## An overview of SmartNews

SmartNews, founded in 2012, is headquartered in Tokyo, Japan. The news app developed by SmartNews has always been [top-rated](https://www.businessinsider.com/guides/smartnews-free-news-app-2018-9) in the Japanese market. SmartNews is the [fastest growing](https://about.smartnews.com/en/2019/06/12/smartnews-builds-global-momentum-with-over-500-us-growth-new-executives-and-three-new-offices/) news app and also boasts [high user viscosity](https://about.smartnews.com/en/2018/07/21/smartnews-reaches-more-than-10-million-monthly-active-users-in-the-united-states-and-japan/) in the US market. According to the statistics from [APP Annie](https://www.appannie.com/en/), the monthly average session duration of SmartNews ranked first among all news apps by the end of July, 2021, greater than the accumulated session duration of AppleNews and Google News.

With the rapid growth of user base and viscosity, SmartNews has to face more challenges in terms of recommendation mechanism and AI algorithm. Such challenges include utilizing massive discrete features in large-scale machine learning (ML), accelerating unstructured data query with vector similarity search, and more.

At the beginning of 2021, the dynamic Ad algorithm team at SmartNews sent a request to AI infrastructure team that the functions of recalling and querying advertisements need to be optimized. After two months of research, AI infrastructure engineer Shu decided to use Milvus, an open-source vector database that supports multiple indexes and similarity metrics and online data updates. Milvus is trusted by more than a thousand organizations worldwide.

## Advertisement recommendation powered by vector similarity search

The open-source vector database Milvus is adopted in the SmartNews Ad system to match and recommend dynamic ads from a 10-milllion-scale dataset to its users. By doing so, SmartNews can create a mapping relationship between two previously unmatchable datasets - user data and advertisement data. In the second quarter of 2021, Shu managed to deploy Milvus 1.0 on Kubernetes. Learn more about how to [deploy Milvus](https://milvus.io/docs).

![img](https://assets.zilliz.com/image1_2a88ed162f.png "Milvus documentation.")

After the successful deployment of Milvus 1.0, the first project to use Milvus was the advertisement recall project initiated by the Ad team at SmartNews. During the initial stage, the advertisement dataset was on a million scale. Meanwhile, the P99 latency was strictly controlled within less than 10 milliseconds.

In June, 2021, Shu and his colleagues in the algorithm team applied Milvus to more business scenarios and attempted data aggregation and online data/index update in real time.

By now, Milvus, the open-source vector database has been used in various business scenarios at SmartNews, including ad recommendation.

## **From a user to an active contributor**

While integrating Milvus into the Smartnews product architecture, Shu and other developers came up with requests of functions such as hot reload, item TTL (time-to-live), item update/replace, and more. These are also functions desired by many users in the Milvus community. Therefore, Dennis Zhao, head of the AI infrastructure team at SmartNews decided to develop and contribute the hot reload function to the community. Dennis believed that "SmartNews team has been benefiting from the Milvus community, therefore, we are more than willing to contribute if we have something to share with the community."

Data reload supports code editing while running the code. With the help of data reload, developers no longer need to stop at a breakpoint or restart the application. Instead, they can edit the code directly and see the result in real time.

In late July, Yusup, engineer at SmartNews proposed an idea of using [collection alias](https://milvus.io/docs/v2.0.x/collection_alias.md#Collection-Alias) to achieve hot reload.

Creating collection alias refers to specifying alias names for a collection. A collection can have multiple aliases. However, an alias corresponds to a maximum of one collection. Simply draw an analogy between a collection and a locker. A locker, like a collection, has its own number and position, which will always remain unchanged. However, you can always put in and draw out different things from the locker. Similarly, the name of the collection is fixed but the data in the collection is dynamic. You can always insert or delete vectors in a collection, as data deletion is supported in the Milvus [pre-GA version](https://milvus.io/docs/v2.0.x/release_notes.md#v200-PreGA).

In the case of SmartNews advertisement business, nearly 100 million vectors are inserted or updated as new dynamic ad vectors are generated. There are several solutions to this:

- Solution 1: delete old data first and insert new ones.
- Solution 2: create a new collection for new data.
- Solution 3: use collection alias.

For solution 1, one of the most straightforward shortcoming is that it is extremely time-consuming, especially when the dataset to be updated is tremendous. It generally takes hours to update a dataset on a 100-million-scale.

As for solution 2, the problem is that the new collection is not immediately available for search. That is to say, a collection is not searchable during load. Plus, Milvus does not allow two collections to use the same collection name. Switching to a new collection would always require users to manually modify the client side code. That is to say, users have to revise the value of the parameter `collection_name` every time they need to switch between collections.

Solution 3 would be the silver bullet. You only need to insert the new data in a new collection and use collection alias. By doing so, you only need to swap the collection alias every time you need to switch the collection to conduct the search. You do not need extra efforts to revise the code. This solution saves you the troubles mentioned in the previous two solutions.

Yusup started from this request and helped the whole SmartNews team understand the Milvus architecture. After one and half months, the Milvus project received a PR about hot reload from Yusup. And later, this function is officially available along with the release of Milvus 2.0.0-RC7.

Currently, the AI infrastructure team is taking the lead to deploy Milvus 2.0 and migrate all data gradually from Milvus 1.0 to 2.0.

![img_collection alias](https://assets.zilliz.com/image2_96c064a627.png "Collection alis doc.")


Support for collection alias can greatly improve user experience, especially for those large Internet companies with great volumes of user requests. Chenglong Li, data engineer from the Milvus community, who helped build the bridge between Milvus and Smartnews, said, "The collection alias function arises from the real business request of SmartNews, a Milvus user. And SmartNews contributed the code to the Milvus community. This act of reciprocity is a great example of the open-source spirit: from the community and for the community. We hope to see more contributors like SmartNews and jointly build a more prosperous Milvus community."

"Currently, part of the ad business is adopting Milvus as the offline vector database. The official release of Mivus 2.0 is approaching, and we hope that we can use Milvus to build more reliable systems and provide real-time services for more business scenarios." said Dennis.

> Update: Milvus 2.0 is now general available! [Learn more](2022-1-25-annoucing-general-availability-of-milvus-2-0.md)


MilMil A Milvus-powered FAQ Chatbot that Answers Questions About Milvus

How does MilMil work?

Ask MilMil about Milvus

Don’t be a stranger

Content

Try Managed Milvus for Free

How Short-video Platform Likee Removes Duplicate Videos with Milvus

Zhentu - the Photo Fraud Detector Based on Milvus

The Story of SmartNews - from a Milvus User to an Active Contributor