Building AI Under Extreme Constraints: Engineering an LLM Pipeline on Legacy Hardware

Introduction

Most AI engineering articles begin with powerful GPUs, cloud infrastructure, and virtually unlimited computing resources. Ours didn't.

This project started in an environment where simply getting the required software running was a challenge. The hardware was old, memory was limited, and the operating system imposed restrictions that made many standard AI development workflows impractical. Even technologies that most developers consider routine—such as Docker containers or Linux-based tooling—required careful workarounds before any model could be tested.

At the same time, the problem we wanted to solve wasn't getting any smaller. The planning assistant needed to understand natural language, work with large engineering datasets, and produce reliable answers without exposing the underlying database or requiring users to know internal project codes. Achieving that required more than choosing a capable language model—it demanded an architecture designed around the project's constraints.

Rather than treating those limitations as obstacles to overcome with more hardware, we approached them as engineering requirements. Every design decision, from the development environment to the AI pipeline itself, was driven by the question:

How can we build an intelligent system that remains reliable, efficient, and practical on hardware that wasn't designed for modern AI workloads?

That question shaped every stage of the project and ultimately led to an architecture that relied on thoughtful system design instead of brute-force computation. Before discussing the solution, it's important to understand the constraints that made a conventional approach impossible.

The Real Constraints

At first glance, the project looked straightforward: connect a large language model to an engineering database and allow users to ask questions in natural language. In practice, every layer of the system presented a different constraint, and together they made a conventional implementation impractical.

Legacy Hardware

The development environment was far from ideal. Instead of modern AI workstations with abundant memory and dedicated acceleration, the project began on aging hardware with limited computational resources. Every decision—from selecting development tools to choosing runtime environments—had to account for performance, memory usage, and compatibility.

Tasks that normally take minutes on modern systems often required significantly more planning. Running multiple services simultaneously, processing datasets, and experimenting with language models all competed for the same limited resources.

Large Engineering Datasets

The planning assistant wasn't designed around a small demonstration database. It needed to work with real engineering information spread across multiple datasets containing projects, departments, production stages, priorities, aliases, and status codes.

While the datasets were well structured for software systems, they were not structured for human conversation. Simply providing the entire database to a language model was neither practical nor efficient. Beyond the computational cost, doing so would have introduced unnecessary context, slower responses, and a greater risk of inaccurate reasoning.

Context Window Limitations

Large language models operate within a finite context window. Every table, description, alias, and user question consumes valuable tokens. As the amount of information grows, so does the likelihood that important details are omitted or diluted.

This meant that a naïve approach—placing the complete database schema and relevant records into every prompt—would quickly become unsustainable. Response times would increase, operating costs would rise, and model performance could actually decline because the essential information would be buried inside irrelevant context.

The challenge was no longer about finding a capable model; it was about determining what information should reach the model in the first place.

Engineering Under Time Constraints

Like most real-world software projects, development wasn't performed in a research laboratory with unlimited time for experimentation. Each architectural decision had to balance implementation effort, maintainability, and long-term scalability.

Rather than pursuing the most technically impressive solution, the objective was to build a system that was dependable, understandable, and practical. Every additional layer of complexity had to provide measurable value.

---

Individually, none of these constraints were impossible to solve. Together, however, they made it clear that simply connecting an LLM to a database would not produce a reliable planning assistant. The solution would require rethinking the architecture itself, starting with the development environment and the infrastructure needed to support it.

Building a Development Environment That Shouldn't Have Worked

Before designing the AI pipeline, there was a much more fundamental problem to solve: creating an environment capable of running it.

Many modern AI development workflows assume a Linux-based ecosystem with containerized services, package managers, and orchestration tools readily available. In our case, those assumptions didn't hold. The development machine imposed significant platform and hardware limitations, meaning even setting up the required infrastructure became an engineering challenge.

Docker, which would normally be the foundation of the development environment, wasn't immediately usable. Standard installation methods failed because of operating system limitations, and several of the supporting tools expected by modern development stacks simply weren't available.

Rather than abandoning the project or replacing the hardware, we adapted the environment step by step.

The first milestone was enabling the Windows Subsystem for Linux (WSL), allowing a lightweight Linux environment to coexist with the existing operating system. From there, a minimal Ubuntu installation provided access to the Linux tooling required by the project's backend services.

Even then, the process wasn't entirely straightforward. Certain components behaved differently from a native Linux installation, requiring additional configuration and careful testing before they became reliable enough for daily development. Eventually, the combination of WSL, Ubuntu, and Docker provided a stable platform capable of running the project's databases, APIs, and supporting services.

This may seem like infrastructure work, but it had a direct impact on the AI system itself. Reliable development environments enable reliable software. Without a reproducible platform for running containers, databases, and backend services, it would have been impossible to iterate on prompts, test retrieval strategies, or evaluate architectural decisions consistently.

More importantly, this experience reinforced an important engineering principle:

Good AI systems are built on good software engineering. Before models can solve problems, the infrastructure supporting them must be dependable.

With the development environment finally in place, attention could shift to the core question of the project: How should an AI planning assistant actually retrieve information from a large engineering database without sacrificing accuracy, performance, or safety?

Rethinking the Architecture

With the development environment stable, the next challenge became much more interesting: designing an AI system that could answer engineering questions accurately without overwhelming the language model.

The most obvious solution was also the least practical—send the entire database schema, every relevant table, and the user's question to an LLM, then let it figure everything out.

While this approach may work for small demonstrations, it quickly breaks down in production.

First, the amount of information exceeds what should reasonably be included in a single prompt. Every unnecessary table, column, or record consumes valuable context that could instead be used for reasoning. Larger prompts also increase response times and operating costs.

Second, exposing the entire database structure introduces unnecessary complexity. The model must spend part of its reasoning capacity deciding which information is relevant before it can even begin answering the user's question. That is an inefficient use of both tokens and computation.

Finally, unrestricted access to database structures raises important safety concerns. A planning assistant should retrieve only the information required to answer a question—not expose internal schemas or generate unrestricted database queries.

These observations led to a different philosophy.

Instead of asking a single model to solve every problem at once, we divided the task into a sequence of smaller, specialized stages. Each stage would perform one responsibility well before passing a smaller, cleaner piece of information to the next.

Rather than processing everything, the system would progressively reduce complexity.

The architecture evolved into a pipeline where:

• The user's natural language question is interpreted first.

• Only the relevant concepts, entities, and intent are identified.

• A restricted, read-only database query is generated.

• Only the necessary records are retrieved.

• A second language model receives the filtered data and produces the final response.

By separating understanding from data retrieval and response generation, the language model no longer had to reason over an entire database. Instead, it received exactly the context it needed—nothing more and nothing less.

This architectural decision proved to be one of the most important in the project. The objective was no longer to build the largest AI pipeline possible, but to build the smallest pipeline capable of producing reliable answers.

That principle would shape every engineering decision that followed, from semantic matching and alias detection to safe SQL generation and context-aware response synthesis. The next challenge was teaching the system something humans do naturally—but computers often struggle with: understanding that different words can refer to the same thing.

Bridging the Language Gap

The next obstacle wasn't technical—it was linguistic.

Humans rarely refer to data the same way databases do.

A production planner might ask:

"What's happening with the paint shop?"

Meanwhile, the database might store that same department as Surface Coating, Paint Line, or even an internal code that means nothing to the end user.

Similarly, project names, production stages, equipment, and departments often had multiple ways of being referenced depending on who was asking the question. Experienced employees naturally understood these differences because they had years of domain knowledge. A language model, however, only sees the words it receives.

If the user's wording didn't closely match the terminology stored in the database, even a perfectly generated SQL query could retrieve the wrong information—or nothing at all.

The straightforward solution would have been to maintain a large dictionary of aliases manually.

While possible, this approach quickly becomes difficult to maintain. New projects are introduced, terminology evolves, and different teams naturally develop their own vocabulary over time. A manually curated synonym list would constantly require updates and would still struggle with unexpected phrasing.

Instead, we decided to let the language model solve the part it excels at: understanding language.

Before generating any database query, the system sends the model a lightweight description of the available entities and their meanings rather than the full dataset. This semantic reference acts as a vocabulary guide, explaining how different terms relate to the underlying data.

For example, the prompt may describe concepts such as:

• Available departments and their official names

• Project naming conventions

• Common aliases used by employees

• Production stages

• Frequently confused terminology

The model's first responsibility is not to answer the user's question—it is simply to determine what the user actually means.

Once the intent has been identified, the model converts informal language into standardized internal terminology. A phrase like "paint shop" can therefore be mapped to the correct database entity, regardless of the wording originally used by the user.

Only after this normalization step does the system proceed to generate a database query.

This separation proved to be a significant improvement. Instead of forcing the database to understand human language, we allowed the language model to translate human language into the database's language first. The result was a retrieval process that became both more accurate and more resilient to variations in wording, while remaining independent of hard-coded synonym lists.

In other words, the model wasn't searching the database—it was acting as an interpreter between two worlds: the way people naturally communicate and the way structured data is organized.

Safe Query Generation

Understanding the user's intent solved only half of the problem.

The next challenge was transforming that understanding into a database query without compromising security or performance.

Allowing a language model to generate unrestricted SQL is rarely a good idea. Even when prompted carefully, an unrestricted model could produce inefficient queries, request unnecessary information, or attempt operations that should never be executed in a production environment.

For this reason, the database was never exposed directly to the model.

Instead, the language model received a carefully defined representation of the database schema together with a strict set of rules describing what it was—and more importantly, what it was not—allowed to generate.

The generated SQL had to satisfy several requirements:

• Read-only operations only. Any statement capable of modifying data, such as INSERT, UPDATE, DELETE, or schema-altering commands, was strictly prohibited.

• Retrieve only what is necessary. Queries were expected to return the smallest useful dataset instead of entire tables. Selecting every column or every record was intentionally discouraged.

• Encourage specificity. If a user's request was too broad, the system would not attempt to guess their intent. Instead, it would ask a follow-up question or request additional details before generating a query.

For example, a request such as:

"Show me information about Project Alpha."

would not retrieve every available record related to that project.

Instead, the system would first provide a concise summary—perhaps its current status, completion percentage, responsible department, and priority level. If the user wanted production logs, scheduling information, or capacity planning details, they could ask a more specific follow-up question.

This approach offered several advantages.

From a performance perspective, it reduced unnecessary database traffic and kept response times predictable.

From an AI perspective, it prevented the second language model from receiving excessive context, allowing it to focus on generating a concise, accurate answer rather than filtering through irrelevant information.

Most importantly, it introduced an additional layer of safety. The language model was never treated as an unrestricted database client. Instead, it acted as an intelligent query planner operating within clearly defined engineering boundaries.

By limiting both what could be queried and how much could be retrieved, the system remained efficient, secure, and scalable—qualities that become increasingly important as datasets continue to grow.