Optical Character Recognition has existed for decades, and modern OCR engines have reached impressive levels of accuracy on clean documents. However, production systems rarely receive perfectly scanned pages. Instead, they must process screenshots, photographs taken with mobile phones, social media images, scanned books, invoices, forms, news articles, and documents containing multiple languages.

These inputs introduce challenges that conventional OCR pipelines often struggle with:

skewed or rotated text
inconsistent lighting
low image quality
complex layouts
mixed writing systems
decorative backgrounds
overlapping visual elements
varying font sizes

Simply extracting characters is no longer sufficient for AI applications. The output must preserve enough structure and context to be useful for downstream systems such as search engines, document understanding, knowledge extraction, or content generation.

This observation shaped the design of CosmicCat's OCR service.

Rather than building another standalone OCR API, we designed a semantic OCR pipeline whose purpose is to transform unstructured visual information into structured data that can immediately be consumed by other AI services.

The language model is only one component of that pipeline.

Most of the engineering effort happens before inference ever begins.

From Traditional OCR to Semantic OCR

A conventional OCR system usually follows a very straightforward workflow.

Image
   │
   ▼
OCR Engine
   │
   ▼
Extracted Text

While this architecture is simple, it assumes that the OCR engine can solve every problem on its own. In practice, the model must simultaneously identify text regions, ignore background noise, compensate for perspective distortion, understand multiple writing systems, and accurately recognize characters.

As the complexity of the input increases, recognition quality often decreases.

Instead of relying entirely on the language model, we decided to separate these responsibilities.

Computer vision techniques handle geometric problems.

Image preprocessing improves readability.

Language detection determines the most appropriate recognition path.

Finally, the vision-language model focuses on understanding text rather than locating it.

This division of responsibilities makes every component easier to optimize independently.

Designing the Architecture

One of our earliest architectural decisions was to avoid embedding the OCR model directly inside the Django application.

Although this would have been simpler initially, it tightly couples GPU workloads with the web server and makes deployment considerably more difficult.

Instead, the OCR system is implemented as an independent FastAPI microservice.

The Django application acts as the orchestration layer for the entire platform.

It is responsible for:

receiving uploaded images
validating requests
exposing configurable OCR parameters
forwarding requests to the OCR service
collecting structured results
returning consistent API responses to client applications

The FastAPI service performs the computationally intensive work.

It manages image preprocessing, text detection, language routing, model inference, GPU memory, and health monitoring independently of the web application.

This separation keeps both services focused on their own responsibilities.

                    User
                      │
                      ▼
              Django Backend
                      │
          REST Communication
                      │
                      ▼
          FastAPI OCR Service
                      │
      ┌───────────────┼────────────────┐
      │               │                │
      ▼               ▼                ▼
 Image Processing  Language Logic  OCR Models
      │               │                │
      └───────────────┼────────────────┘
                      │
                      ▼
             Structured OCR Output

This architecture provides several important advantages.

Each service can be deployed independently.

The OCR service can run on dedicated GPU hardware while Django continues running on a standard application server.

Failures inside the OCR service do not directly impact the web application, and improvements to the OCR pipeline can be deployed without modifying the frontend or API layer.

Most importantly, this architecture allows the OCR service to become reusable across multiple projects rather than remaining tightly coupled to a single application.

Treating the OCR Model as a Service

Large vision-language models consume a considerable amount of GPU memory.

Keeping them permanently loaded may be acceptable in small experiments, but production systems require finer control over hardware resources.

For this reason, the OCR service includes a dedicated model manager responsible for the complete lifecycle of the recognition model.

Instead of simply loading the model when the application starts, the service can explicitly start, stop, reload, and unload the model whenever necessary.

When the service starts, the model manager loads both the processor and the Qwen3-VL model, transfers them to the appropriate device, and switches the model into inference mode.

When the service is stopped, the process works in reverse.

The model is moved back to CPU memory when possible, released from memory, garbage collection is triggered, and CUDA caches are cleared to recover GPU resources before the service becomes inactive.

This behaviour is particularly useful in environments where multiple AI services compete for limited GPU memory.

Rather than reserving several gigabytes of VRAM indefinitely, the OCR service occupies GPU resources only while it is actively serving requests.

The model manager also performs health checks, validates inference, monitors GPU memory usage, supports automatic reloading after failures, and exposes detailed service status information through dedicated endpoints.

By separating model lifecycle management from inference itself, the OCR service remains significantly more robust than a simple wrapper around a transformer model.

This design allows the rest of the system to treat OCR as a reliable production service rather than a single Python script running inside a web application.

At this point, the architecture is in place, but no text has actually been recognized yet.

The next stage is where most of the intelligence begins.

Before a single token is generated by Qwen3-VL, every document passes through a configurable computer vision pipeline responsible for detecting text regions, grouping related elements, correcting document geometry, refining crop boundaries, and preparing each text line for the recognition model.

Those preprocessing stages ultimately determine how much useful information reaches the model—and, in many cases, they have a greater impact on OCR accuracy than changing the model itself.

Building the Computer Vision Pipeline

The OCR model is never the first component that processes an image.

Long before Qwen3-VL receives any input, the document passes through a configurable computer vision pipeline designed to transform an arbitrary image into a collection of clean, well-structured text lines.

This preprocessing stage exists for a simple reason: the quality of the input often determines the quality of the output.

Even the strongest vision-language models struggle when text occupies only a small portion of the image or is surrounded by unnecessary visual information. Rather than asking the model to solve every problem simultaneously, we remove as many of those problems as possible beforehand.

The pipeline consists of several independent stages that can be enabled, disabled, or configured depending on the document being processed.

Input Image
      │
      ▼
Text Detection
      │
      ▼
Cluster Generation
      │
      ▼
Language Detection
      │
      ▼
Deskew
      │
      ▼
Line Re-Cropping
      │
      ▼
Image Enhancement
      │
      ▼
Resize & Padding
      │
      ▼
OCR Model Selection
      │
      ▼
Recognition

Each stage improves the quality of the next one, allowing the recognition models to focus on reading text rather than compensating for poor image quality.

Detecting Text Regions

The first task is identifying where text exists inside the image.

Instead of attempting OCR immediately, the system begins by detecting candidate text regions using PaddleOCR's lightweight text detector.

At this stage, no characters are recognized.

The detector simply estimates where text is likely to appear and returns a collection of bounding boxes with confidence scores.

These detections are intentionally generous.

Their purpose is to avoid missing text, even if that means including additional surrounding pixels.

This produces a reliable starting point for the rest of the pipeline.

From Bounding Boxes to Semantic Clusters

Real-world documents rarely consist of isolated words.

A news article may contain multiple paragraphs.

A screenshot may contain menus, buttons, and captions.

A scanned page may contain several columns.

Simply processing every detected bounding box independently would destroy much of the document's logical structure.

Instead, nearby detections are merged into clusters.

The clustering algorithm considers several geometric properties simultaneously.

Horizontal distance determines whether neighbouring words belong together.

Vertical distance estimates whether boxes belong to the same paragraph.

Line-height similarity helps prevent text from different font sizes from being merged incorrectly.

Additional filtering can remove clusters that are too small, too large, contain too few detections, or have unrealistic aspect ratios.

Because these parameters are configurable, the OCR service can be adapted to very different document types without changing the underlying implementation.

The result is a collection of meaningful text regions rather than hundreds of disconnected bounding boxes.

Detecting the Reading Direction

One challenge that multilingual OCR systems face is determining which recognition strategy should be used.

Before recognition begins, the pipeline analyses each extracted line and estimates its writing system.

This lightweight language detection stage determines whether a region primarily contains Arabic or Persian characters or whether it belongs to a left-to-right language such as English.

This decision influences the remainder of the pipeline.

Rather than forcing every line through the same recognition model, the OCR service can route different writing systems to different OCR engines, allowing each model to process the data it handles best.

Correcting Document Geometry

Scanned documents are rarely perfectly aligned.

Even slight rotations can reduce OCR accuracy because character shapes no longer resemble the patterns seen during training.

To compensate for this, deskewing can be applied during both extraction and recognition.

The service supports several complementary approaches.

Hough Transform estimates dominant line orientations.

Minimum-area rectangle analysis measures the orientation of detected regions.

Projection profile analysis searches for the rotation that maximizes horizontal text alignment.

A combined mode evaluates multiple estimation methods before selecting the most reliable correction.

Because no single algorithm performs best on every document, allowing multiple deskew strategies makes the pipeline considerably more robust.

Whenever the detected skew exceeds acceptable limits, the extracted regions are rotated before additional processing takes place.

Why Re-Cropping Matters

Although the text detector usually identifies the correct region, its bounding boxes are intentionally conservative.

They often include unnecessary borders, neighbouring graphics, icons, or background textures.

While these extra pixels may appear insignificant to a human observer, they can introduce unnecessary complexity for a vision-language model.

For this reason, every detected line can optionally pass through a second refinement stage.

Instead of trusting the original bounding box, the pipeline attempts to locate the true boundaries of the text itself.

Several re-cropping strategies are available, each designed for different types of documents.

Edge-Based Re-Cropping with Canny

One refinement strategy relies on classical computer vision.

The cropped line is first converted into a representation suitable for edge analysis.

Canny edge detection is then applied using configurable upper and lower thresholds.

Rather than using the detected edges directly, the pipeline expands them through morphological dilation.

This closes small gaps between neighbouring character strokes and creates more coherent connected regions.

Contours are extracted from the resulting image.

Very small contours, which usually correspond to dust, compression artefacts, or isolated noise, are discarded using a configurable minimum contour area.

Nearby contours are then merged whenever they fall within a configurable merge distance.

Finally, a new bounding rectangle is computed around the merged contours and expanded slightly using configurable padding.

The resulting crop typically contains considerably less background while preserving every visible character.

Because every threshold can be adjusted independently, this stage can be tuned for high-resolution scans, photographs, screenshots, or heavily compressed images.

Paddle-Based Re-Cropping

Some documents benefit from a different strategy.

Instead of relying on edge detection, the pipeline can invoke PaddleOCR's detector again on the already extracted line.

This second pass frequently produces a tighter bounding box because the search area has been reduced dramatically.

When this approach succeeds, unnecessary margins are removed without requiring contour analysis.

The OCR service allows either strategy to be selected manually depending on the document type.

Automatic Re-Cropping

Selecting a single refinement strategy for every image is rarely optimal.

Some documents contain crisp printed text that responds well to edge detection.

Others contain low-contrast scans where detector refinement performs better.

For this reason, the OCR service also provides an automatic mode.

Rather than committing to a single algorithm, the pipeline evaluates the available refinement strategies and selects whichever produces the most reliable crop for the current line.

This flexibility allows the same OCR pipeline to process a wide range of document types while minimizing manual configuration.

Preparing Images for Recognition

Once each text line has been isolated, additional preprocessing can still improve recognition quality.

Different documents respond differently to different enhancement techniques.

The OCR service therefore supports multiple image variations instead of relying on a single fixed preprocessing pipeline.

Depending on the document, the extracted line may remain in its original colours or be converted into grayscale.

Additional options include normalized grayscale, adaptive black-and-white conversion, enhanced contrast, smart colour preservation, and several automatic OCR-oriented preprocessing modes.

Brightness and contrast adjustments can be applied independently.

Automatic inversion detects white text on dark backgrounds and reverses the colours when appropriate.

Optional denoising removes compression artefacts while configurable resizing and padding normalize image dimensions before recognition.

These transformations may appear simple individually, but together they significantly improve the consistency of the images presented to the recognition models.

Rather than attempting to make the OCR models more tolerant of every possible visual condition, the preprocessing pipeline instead attempts to produce images that are already close to an ideal recognition input.

This philosophy proved to be one of the most important design decisions throughout the development of the system.

Choosing the Right Recognition Model

By the time an image reaches the recognition stage, most of the difficult computer vision problems have already been solved.

The document has been segmented into meaningful text regions.

Perspective distortion has been corrected where necessary.

Unnecessary borders have been removed.

The background has been simplified.

Each text line has been normalized into a representation that closely matches the type of data modern OCR models expect.

Only then does the recognition stage begin.

One of the most important design decisions during development was recognizing that there is no universal OCR model capable of producing the best results for every language and every document type.

Instead of building the pipeline around a single model, we evaluated different recognition engines according to the writing systems they would process.

Why Qwen3-VL?

Our primary objective was accurate recognition of Persian and Arabic documents.

While many OCR engines perform well on Latin scripts, right-to-left languages introduce additional challenges.

Characters change shape depending on their position within a word.

Words are connected rather than separated into isolated characters.

Diacritics, punctuation, and handwritten variations further increase recognition complexity.

During evaluation, Qwen3-VL consistently provided the best balance between recognition quality and contextual understanding for Persian and Arabic documents.

Unlike traditional OCR engines that focus primarily on character recognition, Qwen3-VL combines visual understanding with language reasoning.

This allows the model to recover text more reliably when documents contain imperfect crops, noisy scans, low image quality, or visually complex layouts.

The ability to reason about the surrounding context often improves recognition when individual characters are partially degraded or difficult to distinguish.

For our target documents, this capability proved more valuable than maximizing raw inference speed.

Matching the Model's Training Distribution

An important observation emerged during testing.

The Persian fine-tuned version of Qwen3-VL produced its strongest results when presented with individual text lines rather than complete pages.

This is not surprising.

Machine learning models generally perform best when the inference data resembles the distribution of the data used during training or fine-tuning.

Instead of presenting an entire document containing multiple paragraphs, tables, graphics, and unrelated visual elements, we isolate every line into an independent recognition task.

Each cropped image contains a single semantic unit of text surrounded by minimal background.

This closely matches the style of data used during fine-tuning while reducing unnecessary visual complexity.

The preprocessing pipeline therefore exists not only to improve image quality, but also to transform arbitrary documents into inputs that better resemble the model's learned representation.

This architectural decision had a significantly greater impact on recognition quality than changing OCR parameters alone.

Background Normalization

Another discovery during experimentation involved the background surrounding the extracted text.

Vision-language models inevitably allocate part of their attention to the entire image rather than only the characters themselves.

Large textured backgrounds, coloured gradients, decorative elements, or surrounding graphics increase visual complexity without contributing useful information.

We found that recognition quality became noticeably more consistent when cropped text lines were presented on simple grayscale or white backgrounds.

For this reason, the preprocessing pipeline includes configurable normalization steps that remove unnecessary visual information while preserving the characters themselves.

Rather than expecting the model to ignore irrelevant pixels, the system attempts to remove those pixels before inference begins.

Different Languages Require Different Tools

Although Qwen3-VL became the preferred recognition model for Persian and Arabic, it was not the best solution for every language.

Latin scripts present a different problem.

English text generally contains isolated characters, highly standardized fonts, and abundant OCR training data.

During evaluation, PaddleOCR consistently provided the best balance of speed and recognition accuracy for Latin-based languages such as English. Because these scripts are well supported by modern OCR engines, invoking a large vision-language model would have introduced significantly higher computational cost with little measurable improvement in recognition quality.

Invoking a large vision-language model for these documents would have increased computational cost without providing a meaningful improvement in quality.

Instead, the OCR pipeline dynamically selects the most appropriate recognition engine according to the detected writing system.

Persian and Arabic text is routed toward the Qwen3-VL pipeline.

PaddleOCR serves two independent roles within the pipeline. First, its detection model identifies candidate text regions for all documents. Later, for Latin-based languages, PaddleOCR's recognition model also performs the final OCR, while Persian and Arabic text is forwarded to Qwen3-VL.

PaddleOCR Detector
│
▼
Text Regions
│
├────────► English → PaddleOCR Recognizer
│
└────────► Persian/Arabic → Qwen3-VL

Latin text is routed to PaddleOCR, while Persian and Arabic text are routed to the Qwen3-VL pipeline.

This hybrid architecture combines the strengths of both approaches while avoiding their respective weaknesses.

Rather than forcing one model to solve every OCR problem, each model is used where it performs best.

A Configurable OCR Platform

One design goal throughout development was flexibility.

Instead of exposing only a single OCR endpoint, the service allows almost every stage of the pipeline to be configured.

Depending on the application, users can adjust clustering behaviour, deskew algorithms, re-cropping methods, image preprocessing strategies, brightness and contrast, resize modes, padding, language filtering, visualization, and numerous other parameters.

This level of configurability makes the OCR service suitable for a wide variety of document types without requiring modifications to the underlying implementation.

Different datasets often require different preprocessing strategies, and exposing these controls through the API allows experimentation without changing the source code.

OCR as Part of a Larger AI Ecosystem

Within CosmicCat, OCR is not treated as an isolated utility.

It serves as one of the entry points into the broader AI platform.

Once text has been extracted and structured, the output can immediately be consumed by other services.

Structured OCR results can be indexed for semantic search, analyzed by language models, incorporated into document understanding workflows, or used as input for automated content generation.

Rather than producing plain text files, the OCR service produces structured information that integrates naturally with the rest of the platform.

This architecture transforms OCR from a simple preprocessing task into a reusable AI capability.

Conclusion

Building an effective OCR system required considerably more than selecting a powerful recognition model.

The final architecture combines classical computer vision, configurable preprocessing, intelligent language routing, modern vision-language models, and service-oriented software design into a unified pipeline.

Each stage contributes a specific responsibility.

Detection locates potential text.

Clustering reconstructs document structure.

Deskewing corrects geometry.

Re-cropping removes unnecessary background.

Image normalization prepares the input.

Language detection selects the appropriate recognition engine.

Finally, Qwen3-VL or the lightweight OCR model converts carefully prepared text lines into structured information.

By separating these responsibilities, the system achieves greater flexibility, better recognition quality, and more efficient resource utilization than relying on a single OCR engine alone.

Rather than asking one model to solve every problem, the architecture allows each component to do what it does best.

That philosophy has become a core design principle throughout the development of CosmicCat's AI services.