Anti-face barrage, that is, a large number of barrages float by, but do not block the characters in the video screen. It looks like they are floating from behind the characters.

Machine learning has been popular for several years, but many people don’t know that these capabilities can also be run in browsers;

This article introduces the practical optimization process in video barrages, listed at the end of the article Some scenarios where this solution is applicable are described, hoping to open up some ideas.

mediapipe Demo（https://google.github.io/mediapipe/）shows

mainstream anti-face barrage implementation principle

On-demand

up Upload video

The server background calculation extracts the portrait area in the video screen and converts it into svg storage

While the client plays the video, download the svg from the server Combined with the barrage, the portrait area does not display the barrage

Live broadcast

1. When the anchor pushes the stream, the portrait area is extracted from the screen in real time (host device) and converted into svg
2. Merge svg data into the video stream (SEI) and push the stream to the server
3. At the same time as the client plays the video, parse the svg from the video stream (SEI)
4. Compile the svg with the bullet Screen synthesis, the portrait area does not display the barrage

Implementation plan of this article

While the client plays the video, the portrait area information is extracted from the screen in real time, and the portrait area information is exported into pictures and bullets Screen synthesis, the barrage will not be displayed in the portrait area.

Implementation Principle

1. Use machine learning open source libraries to extract portrait outlines from video images in real time, such as Body Segmentation (https://github.com/tensorflow/tfjs-models/blob/ master/body-segmentation/README.md)
2. Export the portrait outline into a picture, and set the mask-image of the barrage layer (https://developer.mozilla.org/zh-CN/docs/ Web/CSS/mask-image)

Compared with the traditional (live broadcast SEI real-time) solution

Advantages:

• Easy to implement; only one Video tag is required Parameters, no need for multi-end coordination
• No network bandwidth consumption

Disadvantages:

• The theoretical performance limit is inferior to traditional solutions; equivalent to exchanging performance resources for the network Resources

Problems Faced

JavaScript is known to have poor performance, making it unsuitable for CPU-intensive tasks. From official demo to engineering practice, the biggest challenge is performance.

This practice finally optimized the CPU usage to about 5% (2020 M1 Macbook), reaching a production-ready state.

Practice tuning process

Select a machine learning model

BodyPix (https://github.com/tensorflow/tfjs-models/blob/master/body-segmentation/ src/body_pix/README.md)

The accuracy is too poor, the face is narrow, and there is obvious overlap between the barrage and the character's facial edges

BlazePose (https://github.com/tensorflow/tfjs-models/blob/master/pose-detection/src/blazepose_mediapipe/README.md)

has excellent accuracy and provides limb point information. But the performance is poor

Return data structure example

[{score: 0.8,keypoints: [{x: 230, y: 220, score: 0.9, score: 0.99, name: "nose"},{x: 212, y: 190, score: 0.8, score: 0.91, name: "left_eye"},...],keypoints3D: [{x: 0.65, y: 0.11, z: 0.05, score: 0.99, name: "nose"},...],segmentation: {maskValueToLabel: (maskValue: number) => { return 'person' },mask: {toCanvasImageSource(): ...toImageData(): ...toTensor(): ...getUnderlyingType(): ...}}}]

MediaPipe SelfieSegmentation (https://github.com/tensorflow/tfjs-models/blob/master /body-segmentation/src/selfie_segmentation_mediapipe/README.md)

The accuracy is excellent (the effect is consistent with the BlazePose model), the CPU usage is about 15% lower than the BlazePose model, and the performance is superior, but the limbs are not provided in the returned data Point information

Return data structure example

{maskValueToLabel: (maskValue: number) => { return 'person' },mask: {toCanvasImageSource(): ...toImageData(): ...toTensor(): ...getUnderlyingType(): ...}}

First version implementation

Refer to MediaPipe SelfieSegmentation model official implementation (https://github.com/tensorflow/tfjs-models/blob /master/body-segmentation/README.md#bodysegmentationdrawmask), without optimization, the CPU takes up about 70%

const canvas = document.createElement('canvas')canvas.width = videoEl.videoWidthcanvas.height = videoEl.videoHeightasync function detect (): Promise<void> {const segmentation = await segmenter.segmentPeople(videoEl)const foregroundColor = { r: 0, g: 0, b: 0, a: 0 }const backgroundColor = { r: 0, g: 0, b: 0, a: 255 } const mask = await toBinaryMask(segmentation, foregroundColor, backgroundColor) await drawMask(canvas, canvas, mask, 1, 9)// 導(dǎo)出Mask圖片，需要的是輪廓，圖片質(zhì)量設(shè)為最低handler(canvas.toDataURL('image/png', 0)) window.setTimeout(detect, 33)} detect().catch(console.error)

Reduce the extraction frequency and balance the performance-experience

General video 30FPS, Try lowering the barrage mask (hereinafter referred to as Mask) refresh frequency to 15FPS, which is still acceptable in terms of experience

window.setTimeout(detect, 66) // 33 => 66

At this time, the CPU takes up about 50%

Solves the performance bottleneck

Analyzing the flame graph, it can be found that the performance bottleneck is in toBinaryMask and toDataURL

Rewrite toBinaryMask

Analyze the source code, combined with printing segmentation information, find segmentation.mask .toCanvasImageSource can obtain the original ImageBitmap object, which is the information extracted by the model. Try writing your own code to convert an ImageBitmap to a Mask instead of using the default implementation provided by the open source library.

Implementation Principle

async function detect (): Promise<void> {const segmentation = await segmenter.segmentPeople(videoEl) context.clearRect(0, 0, canvas.width, canvas.height)// 1. 將`ImageBitmap`繪制到 Canvas 上context.drawImage(// 經(jīng)驗(yàn)證 即使出現(xiàn)多人，也只有一個(gè) segmentationawait segmentation[0].mask.toCanvasImageSource(),0, 0,canvas.width, canvas.height)// 2. 設(shè)置混合模式context.globalCompositeOperation = 'source-out'// 3. 反向填充黑色context.fillRect(0, 0, canvas.width, canvas.height)// 導(dǎo)出Mask圖片，需要的是輪廓，圖片質(zhì)量設(shè)為最低handler(canvas.toDataURL('image/png', 0)) window.setTimeout(detect, 66)}

Steps 2 and 3 are equivalent to filling the content outside the portrait area with black (reverse filling ImageBitmap), in order to cooperate with css (mask-image), otherwise only when The barrages are only visible when they float to the portrait area (exactly the opposite of the target effect).

globalCompositeOperation MDN（https://developer.mozilla.org/zh-CN/docs/Web/API/CanvasRenderingContext2D/globalCompositeOperation）

此時(shí)，CPU 占用 33% 左右

多線程優(yōu)化

我原先認(rèn)為toDataURL是由瀏覽器內(nèi)部實(shí)現(xiàn)的，無(wú)法再進(jìn)行優(yōu)化，現(xiàn)在只有優(yōu)化toDataURL這個(gè)耗時(shí)操作了。

雖沒(méi)有替換實(shí)現(xiàn)，但可使用?OffscreenCanvas （https://developer.mozilla.org/zh-CN/docs/Web/API/OffscreenCanvas）+ Worker，將耗時(shí)任務(wù)轉(zhuǎn)移到 Worker 中去， 避免占用主線程，就不會(huì)影響用戶體驗(yàn)了。

并且ImageBitmap實(shí)現(xiàn)了Transferable接口，可被轉(zhuǎn)移所有權(quán)，跨 Worker 傳遞也沒(méi)有性能損耗（https://hughfenghen.github.io/fe-basic-course/js-concurrent.html#%E4%B8%A4%E4%B8%AA%E6%96%B9%E6%B3%95%E5%AF%B9%E6%AF%94）。

// 前文 detect 的反向填充 ImageBitmap 也可以轉(zhuǎn)移到 Worker 中// 用 OffscreenCanvas 實(shí)現(xiàn)， 此處略過(guò) const reader = new FileReaderSync()// OffscreenCanvas 不支持 toDataURL，使用 convertToBlob 代替offsecreenCvsEl.convertToBlob({type: 'image/png',quality: 0}).then((blob) => {const dataURL = reader.readAsDataURL(blob)self.postMessage({msgType: 'mask',val: dataURL})}).catch(console.error)

可以看到兩個(gè)耗時(shí)的操作消失了

此時(shí)，CPU 占用 15% 左右

降低分辨率

繼續(xù)分析，上圖重新計(jì)算樣式（紫色部分）耗時(shí)約 3ms

Demo 足夠簡(jiǎn)單很容易推測(cè)到是這行代碼導(dǎo)致的，發(fā)現(xiàn) imgStr 大概 100kb 左右（視頻分辨率 1280x720）。

danmakuContainer.style.webkitMaskImage = `url(${imgStr})

通過(guò)canvas縮小圖片尺寸（360P甚至更低），再進(jìn)行推理。

優(yōu)化后，導(dǎo)出的 imgStr 大概 12kb，重新計(jì)算樣式耗時(shí)約 0.5ms。

此時(shí)，CPU 占用 5% 左右

啟動(dòng)條件優(yōu)化

雖然提取 Mask 整個(gè)過(guò)程的 CPU 占用已優(yōu)化到可喜程度。

當(dāng)在畫(huà)面沒(méi)人的時(shí)候，或沒(méi)有彈幕時(shí)候，可以停止計(jì)算，實(shí)現(xiàn) 0 CPU 占用。

無(wú)彈幕判斷比較簡(jiǎn)單（比如 10s 內(nèi)收超過(guò)兩條彈幕則啟動(dòng)計(jì)算），也不在該 SDK 實(shí)現(xiàn)范圍，略過(guò)

?判定畫(huà)面是否有人

第一步中為了高性能，選擇的模型只有ImageBitmap，并沒(méi)有提供肢體點(diǎn)位信息，所以只能使用getImageData返回的像素點(diǎn)值來(lái)判斷畫(huà)面是否有人。

畫(huà)面無(wú)人時(shí)，CPU 占用接近 0%

發(fā)布構(gòu)建優(yōu)化

依賴包的提交較大，構(gòu)建出的 bundle 體積：684.75 KiB / gzip: 125.83 KiB

所以，可以進(jìn)行異步加載SDK，提升頁(yè)面加載性能。

1. 分別打包一個(gè) loader，一個(gè)主體
2. 由業(yè)務(wù)方 import loader，首次啟用時(shí)異步加載主體

這個(gè)兩步前端工程已經(jīng)非常成熟了，略過(guò)細(xì)節(jié)。

運(yùn)行效果

總結(jié)

過(guò)程

• 選擇高性能模型后，初始狀態(tài) CPU 70%
• 降低 Mask 刷新頻率（15FPS），CPU 50%
• 重寫(xiě)開(kāi)源庫(kù)實(shí)現(xiàn)（toBinaryMask），CPU 33%
• 多線程優(yōu)化，CPU 15%
• 降低分辨率，CPU 5%
• 判斷畫(huà)面是否有人，無(wú)人時(shí) CPU 接近 0%

CPU 數(shù)值指主線程占用

注意事項(xiàng)

• 兼容性：Chrome 79及以上，不支持 Firefox、Safari。因?yàn)槭褂昧薕ffscreenCanvas
• 不應(yīng)創(chuàng)建多個(gè)或多次創(chuàng)建segmenter實(shí)例（bodySegmentation.createSegmenter），如需復(fù)用請(qǐng)保存實(shí)例引用，因?yàn)椋?/li>
• 創(chuàng)建實(shí)例時(shí)低性能設(shè)備會(huì)有明顯的卡頓現(xiàn)象
• 會(huì)內(nèi)存泄露；如果無(wú)法避免，這是mediapipe 內(nèi)存泄露 解決方法（https://github.com/google/mediapipe/issues/2819#issuecomment-1160335349）

經(jīng)驗(yàn)

• 優(yōu)化完成之后，提取并應(yīng)用 Mask 關(guān)鍵計(jì)算量在 GPU (30%左右)，而不是 CPU
• 性能優(yōu)化需要業(yè)務(wù)場(chǎng)景分析，防擋彈幕場(chǎng)景可以使用低分辨率、低刷新率的 mask-image，能大幅減少計(jì)算量
• 該方案其他應(yīng)用場(chǎng)景：
• 替換/模糊人物背景
• 人像馬賽克
• 人像摳圖
• 卡通頭套，虛擬飾品，如貓耳朵、兔耳朵、帶花、戴眼鏡什么的（換一個(gè)模型，略改）
• 關(guān)注Web 神經(jīng)網(wǎng)絡(luò) API?（https://mp.weixin.qq.com/s/v7-xwYJqOfFDIAvwIVZVdg）進(jìn)展，以后實(shí)現(xiàn)相關(guān)功能也許會(huì)更簡(jiǎn)單

本期作者

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?劉俊? ?

Bilibili Senior Development Engineer

The above is the detailed content of Real-time protection against face-blocking barrages on the web (based on machine learning). For more information, please follow other related articles on the PHP Chinese website!