Drawing to a Frame (Skia)
Skia Frame Processors
It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera.
This allows you to draw onto a Frame using react-native-skia's easy to use JavaScript APIs:
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// Detect objects using GPU-accelerated ML API
const bananas = detectBananas()
// Draw banana outline using GPU-accelerated Skia drawing API
for (const banana of bananas) {
const rect = Skia.XYWHRect(banana.x,
banana.y,
banana.width,
banana.height)
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(rect, paint)
}
}, [])
..or even apply color-filters in realtime:
const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const paint = Skia.Paint(invertColorsFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
// Draw frame using Skia Shader to invert colors
frame.render(paint)
}, [])
VisionCamera Skia Integration
While the Skia Integration was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup, and fully custom Skia Preview View) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See PR #1740 for more information.
At my app development agency, Margelo, we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature reach out to us and we'll build a customized/tailored solution for your company! :)