Text Live prior to iOS 15

Thanks to the machines learning technology, Apple has announced a new feature called Live Text in iOS 15, which can detect and recognize text from camera in real-time. It is a handy function to turn any handwritten or printed notes into a digital form in mobile device. What makes it different from the commonly used OCR is it can smartly recognize a specific content type from the image instead of any text on it. For example, it can scan for email address, phone number, URL and so on from the camera image automatically. It is a super good feature to enhance the UX of any text input.

Can we do something similar in earlier iOS versions? YES!

Here are some iOS frameworks that we can use for the Live Text implementation:

UIKit 📝

AVFoundation 📷

Vision 🔍

After introducing the frameworks, let’s implement Live Text step by step! (error handling are skipped for demo purpose)

2. Implement AVCaptureVideoDataOutputSampleBufferDelegate to receive camera output as image frame.

3. Recognize text from the camera output with VNRecognizeTextRequest, which will be preformed by a VNImageRequestHandler.

4. Handle the recognized text and draw a bounding box in the camera view.

See? We can use live text before iOS 15 is released. 😎

With this Live Text technology, we can do text input more conveniently in mobile device. As it is a real-time text recognition, we can take something further. For example, the following use cases have been implemented in the demo project.

Related posts:

SSIS stands for SQL Server Integration Services and is a solution for building data integrations and workflow applications that integrate with SQL Server databases. As data integration processes…

A lot of application are using authentication, and we use many variants to handle it properly. The component system of VueJs is powerful, so let’s use it to simply handle authentication process. Note…

From the moment I could sit upright on my own, I’ve been a fidgeter. In fact, my inability to keep still led to so many incidents that my mom had me walk through stores with my hands on my stomach in…