Introduction

Running modern AI image generation models locally sounds simple on paper: choose a model, load it into memory, and start generating images. In practice, building a reliable production service involves far more than model selection.

The goal of this project was to create a local AI image generation pipeline capable of producing high-quality cinematic images while running entirely on our own infrastructure. The system needed to be powerful enough to deliver visually impressive results, yet efficient enough to operate on a single NVIDIA RTX 3090 without requiring expensive cloud resources or multi-GPU deployments.

Finding the right balance between image quality, performance, hardware requirements, and deployment complexity quickly became the central challenge of the project. Several promising models were evaluated, implementation approaches were tested, and multiple architectural decisions had to be made before the system reached a production-ready state.

What began as a straightforward image generation experiment eventually evolved into a complete AI service with request management, task queuing, prompt enhancement, and GPU resource control. Along the way, we encountered outdated documentation, infrastructure trade-offs, and limitations that forced us to rethink parts of the system's design.

This engineering log documents the journey from initial model evaluation to a fully operational local image generation platform, along with the lessons learned from deploying and maintaining it in real-world conditions.

Searching for the Right Model

Before building the service itself, the first challenge was selecting a model that could realistically meet our requirements. We were looking for a solution capable of generating cinematic-quality images while remaining practical to deploy on a single RTX 3090.

At first, several image generation models appeared promising. Many offered impressive benchmarks, strong community adoption, or advanced capabilities on paper. However, once deployment constraints entered the equation, the list became much smaller.

One of the models we explored was Qwen. While its results were promising, the model presented challenges for our target environment. Running larger variants required significant resources, and attempts to reduce the hardware footprint through quantization often resulted in a noticeable decline in image quality. The trade-off between performance and visual fidelity was difficult to justify for a production system focused on generating high-quality images.

This highlighted an important lesson that appears frequently in AI engineering: the best-performing model is not always the best production choice. Factors such as hardware requirements, inference speed, memory consumption, deployment complexity, and long-term maintainability can be just as important as raw model capability.

As we continued evaluating alternatives, our focus shifted away from finding the most powerful model available and toward finding the model that delivered the best balance between quality, efficiency, and operational simplicity. That search eventually led us to Sana, a model that appeared capable of meeting our requirements without demanding excessive infrastructure.

At that point, the project finally seemed feasible. The next step was determining whether Sana could deliver the image quality we needed while remaining practical for real-world deployment.

Discovering Sana

After evaluating several alternatives, we eventually came across Sana. Unlike some of the larger models we had tested, Sana offered a much more attractive balance between image quality and hardware requirements.

Our primary objective was not to generate the most technically impressive images possible. We needed a model that could consistently produce cinematic visuals while remaining practical to deploy on a single RTX 3090. Sana appeared to fit that requirement surprisingly well.

Initial testing produced encouraging results. The generated images had strong composition, good lighting, and the cinematic aesthetic we were looking for. More importantly, the model could run reliably within our hardware constraints without requiring excessive optimization or infrastructure changes.

At first glance, the implementation also seemed relatively straightforward. The Hugging Face ecosystem provided the necessary resources, the model was well documented, and deploying a basic inference service looked like a task that could be completed quickly.

Those assumptions turned out to be only partially correct.

While the model itself performed well, integrating it into a production-ready system introduced several unexpected challenges. Some examples and tutorials referenced outdated implementations, and many AI assistants continued recommending pipelines that were no longer compatible with the current version of the model.

Nevertheless, the early image quality results were promising enough to justify moving forward. For the first time, it felt like we had found a model capable of serving as the foundation of a real image generation platform rather than another experiment that would be abandoned during deployment.

 

The Documentation Trap

With Sana selected and the initial tests looking promising, the next step was integrating the model into our service. On the surface, this appeared to be the easiest phase of the project. The model was available through Hugging Face, examples existed online, and most AI assistants were confident about how the implementation should be done.

Unfortunately, confidence and correctness are not always the same thing.

Our first attempts followed the pipeline that was most commonly recommended by tutorials, blog posts, and even modern AI assistants. The code looked reasonable, the explanations sounded convincing, and multiple sources pointed to the same approach. Yet the implementation simply would not work correctly.

After several rounds of debugging, it became clear that the problem was not our code. The issue was that many of the resources being recommended were based on an older Sana pipeline that was no longer the correct solution. Even more surprisingly, AI assistants consistently continued suggesting the outdated implementation despite the newer approach already being documented.

At that point, we stopped relying on secondary sources and went directly to the official Hugging Face documentation. Reading the source documentation revealed the differences between the old and new pipelines and provided the information needed to build a working implementation.

This experience reinforced an important lesson about modern development workflows. AI tools can dramatically accelerate implementation, but they should not replace official documentation. When working with rapidly evolving frameworks and models, documentation often becomes the only reliable source of truth.

In the end, the solution was not a clever workaround or a hidden configuration option. It was simply taking the time to read the latest documentation and verify assumptions instead of trusting information that had been repeated across multiple sources.

Once the correct pipeline was implemented, Sana finally began generating images reliably. With model integration complete, the next challenge emerged from a completely different area: managing access to the GPU itself.

Solving GPU Concurrency

With the correct Sana pipeline finally running, the system was capable of generating images reliably. However, another challenge quickly appeared once we started thinking beyond individual tests and toward real-world usage.

Generating a single image was not a problem. Generating multiple images at the same time was.

Unlike traditional web APIs that can process many requests concurrently, AI image generation places significant pressure on GPU resources. During testing, we discovered that allowing multiple generation jobs to run simultaneously could lead to instability, excessive memory usage, and unpredictable behavior. Even with an RTX 3090, image generation remained a resource-intensive operation.

The solution was straightforward: treat the GPU as a shared resource that could only process one generation task at a time.

Instead of allowing every incoming request to immediately start generation, we introduced a locking mechanism around the image generation process. When a request begins using the GPU, the system acquires the lock and prevents additional generation jobs from starting until the current task is completed.

This transformed the workflow from:

Request A → Generate

Request B → Generate

Request C → Generate

into:

Request A → Generate

Request B → Wait

Request C → Wait

Once Request A finishes, the next task in the queue begins processing.

While this approach may seem restrictive at first, it significantly improved stability and predictability. Rather than risking crashes or failed generations under load, users would simply wait their turn while the system processed requests safely.

This also introduced a new requirement. If requests were going to wait for GPU access, we needed a way to store and manage pending tasks. The question then became: where should those queued jobs be stored?

Building a Lightweight Task Queue

Once we decided that image generation requests would be processed one at a time, we needed a mechanism for storing pending jobs while they waited for GPU access.

The obvious solution was to introduce a dedicated database-backed queue. In many production systems, this would be implemented using PostgreSQL, Redis, or a dedicated message broker. While these solutions are powerful, they also introduce additional infrastructure, memory consumption, maintenance overhead, and operational complexity.

For our use case, that felt unnecessary.

The queue only needed to store a relatively small amount of information about pending image generation requests. We weren't dealing with thousands of jobs per minute, distributed workers, or complex scheduling requirements. We simply needed a reliable way to track which tasks were waiting and which task should be processed next.

Another factor influenced the decision. The project already contained multiple SQLite databases serving other parts of the system. Creating yet another database solely for a lightweight task queue did not seem like the most efficient use of resources. Likewise, allocating several gigabytes of memory to a dedicated PostgreSQL instance whose primary purpose would be storing a small list of pending jobs felt excessive.

Instead, we chose a much simpler approach.

The entire queue was implemented using a JSON-based storage system. Each pending task was written to a structured JSON file containing the information required for processing. When a generation completed, the next task could be read from the queue and executed.

The solution offered several advantages:

• Minimal memory usage

• No additional services to maintain

• Simple backup and recovery

• Easy debugging during development

• Fast enough for our workload

More importantly, it solved the actual problem without introducing unnecessary complexity.

This decision became another reminder that good engineering is not always about selecting the most sophisticated technology. Sometimes the best solution is the one that accomplishes the goal with the fewest moving parts.

With request management and task storage in place, the system had become stable enough for production usage. However, after users began generating images regularly, another issue became increasingly apparent: the quality of the prompts being sent to the model.

Improving Prompt Quality with DeepSeek

By this stage, the infrastructure was working reliably. The service could receive requests, queue tasks, manage GPU access, and generate images consistently through Sana.

However, a different problem began to emerge.

The quality of the generated images depended heavily on the quality of the prompts being provided by users.

While Sana was capable of producing visually impressive and cinematic results, it struggled when given short or vague instructions. Prompts such as "a futuristic city" or "a warrior standing in the rain" often lacked enough detail for the model to consistently generate the type of image users expected.

We also observed another limitation common among image generation models: rendering text and numbers accurately. While Sana excelled at creating atmosphere, lighting, and composition, it was far less reliable when prompts involved specific textual elements.

Rather than forcing users to learn prompt engineering themselves, we decided to improve the prompts automatically before they reached the image generation model.

Our first approach involved using a locally hosted Qwen language model as an intermediary. The idea was simple: users would submit a short prompt, the language model would expand it into a detailed cinematic description, and Sana would generate the final image using the enhanced prompt.

In practice, the results were mixed.

Although Qwen occasionally produced excellent prompt expansions, the output quality was not always consistent. Similar user requests could result in significantly different prompt structures, making image quality less predictable than we wanted for a production system.

To improve reliability, we replaced the middleware layer with DeepSeek.

The workflow became:

User Prompt → DeepSeek → Enhanced Prompt → Sana → Generated Image

Instead of receiving a brief request from the user, Sana would now receive a rich and detailed description containing information about composition, lighting, atmosphere, camera perspective, visual style, and other cinematic details.

This change produced a noticeable improvement in image quality and consistency. Users could continue providing simple prompts while the system handled the complex prompt engineering process automatically behind the scenes.

More importantly, it highlighted another important lesson from the project: sometimes the best way to improve an AI system is not by replacing the model itself, but by improving the quality of the information you provide to it.

With DeepSeek enhancing prompts and Sana handling image generation, the pipeline finally reached the level of quality and reliability we had been aiming for from the beginning.

Final Architecture and Lessons Learned

After multiple iterations, infrastructure changes, and deployment improvements, the project evolved from a simple image generation experiment into a complete production-ready service.

What started as a search for a model capable of generating cinematic images ultimately became an exercise in practical AI engineering. The challenges were not limited to model selection. Resource management, task scheduling, documentation accuracy, prompt quality, and deployment decisions all played a significant role in the success of the final system.

The architecture eventually settled into a simple but effective pipeline:

User → API → Task Queue → DeepSeek → Sana → Generated Image

Each component solved a specific problem.

The API provided a clean interface for receiving generation requests. The task queue ensured requests could be processed safely without overwhelming the GPU. DeepSeek transformed short user prompts into detailed cinematic descriptions, and Sana handled the final image generation process.

This architecture proved reliable enough for daily use and remained in operation for nearly a year.

Looking back, several lessons stand out from the project:

Official Documentation Matters

One of the biggest obstacles during implementation came from outdated examples and incorrect recommendations generated by AI tools. The solution was ultimately found in the official Hugging Face documentation. AI assistants are powerful productivity tools, but they should never replace primary sources.

Simplicity Is Often the Better Choice

The JSON-based task queue was never the most sophisticated solution. It was simply the solution that matched the problem. By avoiding unnecessary infrastructure, the system remained lightweight, maintainable, and easy to debug.

Infrastructure Is Part of AI Engineering

A good model alone is not enough. Queue management, GPU utilization, request handling, monitoring, and deployment decisions all contribute to the overall quality of an AI service. In many cases, these engineering decisions have a greater impact on user experience than the model itself.

Better Inputs Produce Better Outputs

Adding DeepSeek as a prompt-enhancement layer dramatically improved the consistency of generated images. The project reinforced the idea that improving inputs can often be more effective than continuously replacing models.

Every Model Has a Lifecycle

Although Sana served the project well and produced many impressive results, the AI landscape evolves quickly. New models continue to appear, offering improvements in quality, efficiency, and capabilities. After nearly a year of usage, we eventually identified a stronger alternative that better matched our evolving requirements.

That migration introduced an entirely new set of engineering challenges, architectural decisions, and lessons learned.

But that's a story for the next Engineering Log. 🚀