Engineering AI Assistant Evaluation: A Buyer’s Framework

A Buyer’s Framework for Evaluating an AI Assistant for Siemens NX and Teamcenter Engineering Teams

§ 02 — Who this is for and how to use it

This paper is written for the person who has to sign off on the purchase: a head of engineering, a CAD/PLM methods lead, or an IT architect asked whether an engineering assistant is worth a budget line. It assumes a Siemens NX or Teamcenter environment, but the framework holds for any deep, versioned, vendor-specific toolchain.

Use it in this order. Read section 4 and decide which grounding layers your questions need — that alone eliminates most of the market. Read section 5 so you can tell a real retrieval architecture from a wrapper. Score candidates with section 7. Run the protocol in section 8 against the two survivors, using the question-set method in section 9. Check section 10 before you talk price, and section 11 while you talk price. Sections 12 and 13 cover the mistakes that happen before and after signature.

The industry context is worth one paragraph and no more. Adoption is not the problem: in the Microsoft and LinkedIn 2024 Work Trend Index, a survey of 31,000 people across 31 countries, 75 percent of knowledge workers already used generative AI at work and 78 percent of them brought their own tools. Conversion is the problem. MIT NANDA’s The GenAI Divide: State of AI in Business 2025 found that 95 percent of enterprise pilots produced no measurable return, and the OutSystems Enterprise AI Agent Report 2026 found 94 percent of organisations treating AI sprawl as a growing source of complexity, technical debt and security risk while only 12 percent ran a central platform. The gap between “people use AI” and “the company gets value” is an evaluation and grounding gap.

§ 03 — Why general assistants fail engineering questions

A general assistant fails an engineering question in a specific and predictable way: it answers fluently from public text, and engineering questions are mostly not answerable from public text. Three failure modes recur, and each has a different fix.

Version drift. Siemens ships roughly twice a year — NX 2606 and Teamcenter 2606 both landed in June 2026, the latter on 12 June. An assistant trained on a corpus spanning a decade of releases will answer confidently with a method that was removed two versions ago, and nothing in the answer signals which release it belongs to. The engineer only finds out at compile time, or worse, at runtime on a production part. Version drift is not a model quality problem; it is a retrieval problem, and it is fixed by pinning retrieval to a release, not by asking the model to be more careful.

Invented API surface. This is measurable rather than anecdotal. In our own published measurement, frontier models invent 4 to 6 percent of package and API names, and the rate climbs on niche APIs such as NXOpen, where the public training material is thin. A hallucinated method name is worse than no answer: it survives review because it looks exactly like a real one — correct casing, plausible namespace, sensible arguments — and it costs an engineer an hour to disprove. Multiply that by a team and the assistant is a net negative.

No access to what is yours. Every NX installation of any age carries a layer nobody outside the company knows: custom macros written by someone who left, VB.NET automation from three NX versions ago, in-house APIs with no documentation, attribute and naming conventions that exist only in people’s heads, and the checker rules that decide whether a part is releasable. None of it is public, so none of it is in any base model. An assistant cannot be grounded in it unless someone deliberately puts it there.

None of these is fixed by a better model. All three are fixed by grounding — and grounding is an architecture, not a feature checkbox.

§ 04 — The four grounding layers

Every assistant on the market covers some subset of four layers. Naming them turns a vague comparison into a short checklist, and it makes obvious why two good products can look completely different in a demo.

The layers are independent, and that independence is the whole point. A PLM assistant can be excellent at layer 4 and have nothing at layers 2 and 3 — Siemens’ own Teamcenter Copilot is exactly that shape, by design and not by oversight, because it is grounded in Teamcenter records and Teamcenter documentation. A code-focused assistant can be strong at 1 and 2 and blind to 4. The mistake is not choosing the wrong product; it is not noticing which layers your questions need.

A diagnostic that takes twenty minutes. Take the last twenty engineering questions your people asked each other in chat or at a desk, and tag each with the layer that answers it. The distribution is your requirement, written by your own organisation rather than by a vendor. In the NX teams we work with, layers 2 and 3 dominate, and layer 3 is the one almost nobody plans for. If your distribution is layer-4-heavy, buy a PLM assistant and stop reading comparisons of CAD-side products.

§ 05 — What actually sits behind a grounded assistant

This section exists because “we use RAG” has stopped being information. Every vendor says it. What differs is the architecture underneath, and five specific decisions in it determine whether the assistant is trustworthy on engineering content. Ask about each one by name.

And the answer format. Source citation is not decoration. An engineer who can see which document and which release an answer came from verifies it in seconds; an engineer who cannot has to verify it from scratch, which means the assistant saved nothing. Treat citation as a functional requirement.

§ 06 — The deterministic boundary

This is a doctrine rather than a criterion, and we hold to it on every project: an assistant may help a person decide, and must never itself decide anything that has to be defensible.

We wrote about why the update model matters more than the language in NXOpen versus Knowledge Fusion; the same logic applies to AI. What the assistant should do is everything around that boundary: explain why a check failed, draft the code that implements a rule, find the guideline that applies, summarise thirty pages of specification, teach a new engineer what a named reference is. That is a large and valuable surface, and none of it requires trusting the assistant with a verdict.

Ask every vendor where they place this boundary. A vendor who wants their model inside a release decision either has not thought about auditability or is hoping you have not.

§ 07 — Twelve scored criteria

Score each candidate 0, 1 or 2 per criterion. Below 16 of 24 is not a serious candidate for an engineering environment; a 0 on any criterion marked critical is disqualifying regardless of total.

§ 08 — The test protocol: ten questions, one afternoon

Run these against every candidate, with identical wording, and write the answers down. Do not let the vendor drive the session. Questions 2 and 8 are the two that most products fail and the two most often absent from a demo.

Record pass, partial or fail in words, with a sentence of evidence each. Do not score with colours alone — three weeks later the notes are what you argue from, and a verdict needs to be readable as a word.

§ 09 — Building your own question set

The protocol above is the vendor-independent half. The other half is twenty questions only your organisation can write, and how you write them determines whether the evaluation means anything.

Source them from reality, not from imagination. Pull the last twenty engineering questions from chat history, the helpdesk queue or a methods engineer’s inbox. Questions people actually asked have the right difficulty distribution; questions invented for an evaluation are systematically too easy or too exotic.

Tag each with its grounding layer. This is the diagnostic from section 4, and it doubles as your requirement document.

Write the gold answer before you run anything. For each question, write what a correct answer must contain — the key facts, not the phrasing. Without a gold answer written in advance, scoring becomes a debate about tone, and the loudest person in the room wins.

Decide who scores, and how many. Two independent raters on the same answers, disagreements discussed rather than averaged. If you want to check whether your scoring is even reliable, the research convention is weighted kappa above roughly 0.6 between two raters, or Krippendorff’s alpha at or above 0.8 with more than two.

On using an LLM to score. It works better than most people expect — published evaluations put LLM judges at 80 to 90 percent agreement with human judgement at a small fraction of the cost — and it is the right tool for a large regression suite later. It is the wrong tool for the purchase decision, because a judge inherits the blind spots of the thing it is judging. The consensus in the 2025 to 2026 literature is that judges augment human review rather than replace it, with humans looking at flagged cases. For twenty questions and two candidates, score them yourself.

Keep the set. It becomes your regression suite. Every release of the assistant, and every NX upgrade, is a reason to re-run it — and that is how you notice a product quietly getting worse.

§ 10 — Deployment, security and regulation

Settle these before price, because each one can end the conversation. Two items on this list changed in 2026 and are worth reading even if you evaluated a product last year.

If it has to run inside your walls. On-premise is no longer exotic, and the sizing arithmetic is public: budget roughly 0.5 GB of VRAM per billion parameters at 4-bit quantisation or 2 GB at 16-bit, then reserve a further 20 to 40 percent for the key-value cache and activations, and validate under your real concurrency rather than a single-user benchmark. The serving stacks — vLLM, NVIDIA NIM, SGLang, TensorRT-LLM — are production-hardened rather than experimental. A genuinely air-gapped setup means model weights, embedding model, vector database and orchestration all live on local hardware with no outbound path, and it is achievable; the honest constraint is that most companies do not have the GPU capacity in-house, which is why a hybrid pattern — your infrastructure, your compute servers reached over a controlled API — is the common middle ground.

§ 11 — What this costs over three years

We are not going to print a price, because any number we invent would be wrong for your situation. What we can give you is the structure, so that your business case has the right lines in it rather than only the visible one.

Two observations from the projects we have run. First, the licence is rarely the decisive cost, and optimising it while ignoring the ingestion and operations lines is how a cheap product becomes an expensive programme. Second, if you are considering on-premise, the crossover is real but load-dependent — published guidance puts entry-level private serving hardware in the region of ten thousand US dollars, reaching parity with cloud API spend within months at high query volumes and never reaching it at low ones. Model your own volume before you take a position.

The line we would defend hardest in a budget review is evaluation. A question set that gets re-run every release costs very little and is the only mechanism that tells you whether the thing you bought is still the thing you have.

§ 12 — Five procurement mistakes

Evaluating on the vendor’s questions. A demo is a curated best case. Bring your own twenty questions from section 9, or you are scoring a sales script rather than a product.

Buying capability without contract terms. Data residency, retention, indemnity, EU AI Act documentation and exit belong in the same decision as answer quality — not in a later legal review that discovers a blocker after the budget is committed and the champion has staked their credibility on it.

No baseline. If you cannot say what a question costs today — minutes to an answer, escalations per week, how often a junior interrupts a senior — you will not be able to show what changed. The renewal conversation then becomes a matter of opinion, and opinions lose to spreadsheets.

Skipping the refusal test. An assistant that never says “I do not know” is not confident, it is ungrounded. See protocol question 2, and treat a fluent answer to an invented method as disqualifying rather than as a curiosity.

Not budgeting layer 3. Getting your own macros, standards and conventions into the assistant is usually the largest line item and the one omitted from the business case. It is also where most of the value is, which makes omitting it doubly expensive: the project looks cheap, underdelivers, and gets blamed on the technology.

§ 13 — After you buy: the part that decides whether it works

Procurement ends at signature; value starts later. Three things separate a deployment that sticks from one that quietly stops being used.

Put it where the work is. An assistant in a separate browser tab competes with the engineer’s deadline and loses. Inside the CAD or PLM session it becomes a reflex. This is why session integration belongs in the workflow criteria rather than in a list of conveniences.

Make onboarding a path, not a project. If adding a new use case or a new document set requires the platform team every time, the platform team becomes the bottleneck — and a bottleneck is indistinguishable from a policy that says no. Ask, before signature, what a department has to do to add its own material.

Name an owner and a review rhythm. Someone owns the knowledge base, someone approves new sources, someone re-runs the question set each release. Without those three names the assistant decays: the corpus goes stale, the answers drift, and nobody notices until an engineer stops trusting it. This is the same lesson we took from building a company-wide GenAI platform for a large manufacturer — the architecture was never the hard part; the operating model was.

§ 14 — Disclosure: where Vision4NX sits

Related reading: our review of why ChatGPT gets Siemens NX wrong covers the measurement behind layer 2.