Limit retrieval to documents that apply to the identified asset and that the technician is authorized to see. Present the edition, passage and original source together. When records are incomplete or editions conflict, identify the missing evidence and route the question to its owner. Pilot on a fleet with a controlled revision history before expanding.
What are we evaluating?
Review basis. Documentation review. Conclusions drawn from official documentation and published material, without a test of our own.
Methodology. Read Microsoft’s account of TK Elevator and Anthropic’s workflow guidance. Designed a hypothetical documentation retrieval review for a separate equipment fleet. No hands-on or production runs were performed.
Why can a correct passage still be the wrong answer?
A passage can quote a manual accurately and apply to the wrong equipment. For a field service team, a search assistant earns its place by helping an authorized technician locate the applicable document and verify it. A confident summary of a superseded manual can cost a site visit even when every sentence is faithfully grounded in the retrieved text.
Microsoft describes TK Elevator's work to bring operational information, technical documents and service notes into reach of technicians. Its account also describes the challenge of servicing equipment from different manufacturers. The published example prompted the architecture question here. The proposed pilot concerns a hypothetical commercial equipment fleet and makes no claim about the design or performance of TK Elevator's system.
Limit the first use case to finding documentation. A request for the manual that applies to an installed controller has a narrower scope than a request to diagnose a fault or authorize a repair. Agree with the service owner which tasks the assistant can support. Keep maintenance decisions with qualified personnel working under the applicable procedures.

What identifies the equipment at the point of retrieval?
Start with the asset record that the service organization trusts. A model name printed on the enclosure might identify a product family while several controller revisions exist inside it. Establish which attributes distinguish applicable manuals. These may include serial ranges, region, installed component and modification history. Document what to do when the nameplate and service record disagree.
Build a fixture for two visually similar units whose manuals diverge on one important point. Give the assistant the same broad question about both units and see which document it proposes. The result should include the identifiers that made the document eligible. A ranking score alone gives the technician no means of checking the choice.
Choose how the assistant behaves when an identifier is missing. Asking for a serial number can be useful if the technician can obtain it through an approved process. If the relevant modification history exists only in a controlled service system, route the query to someone with access. Repeatedly asking the user to invent a more precise description creates apparent certainty without improving the data.
Which documents belong in the search collection?
Build an inventory with a responsible owner, edition, applicability and effective date for each controlled document. Distinguish approved manuals from service bulletins, training notes and technician observations. All can be valuable search results, but they carry different authority. A field note describing a previous fix should not silently override an active bulletin.
Decide what happens to superseded content. Removing every older manual may make it impossible to service installed equipment that still depends on it. Keeping every edition without an applicability filter can return conflicting passages. Represent both the publication history and the equipment ranges covered. Give the document owner a way to mark conflicts that require human review.
Design ingestion around changes that matter in the field. If a bulletin replaces one section of a manual, the association between the two matters as much as the text of either document. Capture that relationship and verify it survives extraction and indexing. Include a test where the older passage is more similar to the technician's wording than the newer bulletin.
How can the interface help a technician verify a result?
Return a short passage with the document title, edition, section and a path to the authorized original. Show why it matched the asset and which input remains unverified. A technician should be able to open the original in context and see nearby conditions or exceptions. The interface should also allow a concise report that the suggested manual is wrong for the equipment.
Respect the permissions of the person making the request. A search result can disclose sensitive information through a title, excerpt or thumbnail even when the original file refuses access. Exercise the retrieval path with two accounts that have different entitlements. Look for evidence that a restricted document appears in suggestions, citations, cached summaries or follow-up responses.
Consider connectivity at the site. If the original cannot be opened, indicate that limitation and offer an approved route to the controlled document. A cached excerpt may be useful for navigation yet too incomplete for a work decision. Let the service organization determine where offline use is allowed and how a technician can recognize an outdated copy.
How do you evaluate retrieval separately from writing quality?
Score the selected sources before assessing the wording of the answer. For each synthetic request, let a documentation owner specify the eligible editions, forbidden editions and whether the correct outcome is to request clarification. A fluent response should receive no credit for retrieving a source that cannot apply to the identified asset.
Use deliberately difficult pairs. Change only one serial range, introduce a local bulletin or remove the service history required to choose between variants. Include a newly withdrawn document with particularly attractive wording. Measure how often the assistant recognizes that the collection does not support a unique answer. A system that always offers one citation can appear efficient while failing this test repeatedly.
Then ask technicians to use permitted results for the narrow lookup task. Record time to a verified original, reasons for abandoning the result and corrections they would make to the asset metadata. Documentation staff can use these observations to distinguish retrieval defects from a catalogue that nobody could disambiguate reliably, even manually.
When should the system decline to choose?
Decline when two sources of equal authority conflict, a required attribute is unavailable, or the original cannot be verified for the task at hand. Name the missing information and the owner most likely to resolve it. Giving the user the two relevant document references can still save time if they are authorized to see both and understand the unresolved difference.
Take a controlled group of assets with a complete manual history into the first pilot. Prefer a small, well-maintained family to a larger collection whose revisions are poorly recorded. Include a document withdrawal during the pilot and establish how quickly search stops presenting the obsolete material. Tie release criteria to the service team's ability to correct the collection as well as to the quality of retrieval.
Widen the pilot when the team can trace a result to an eligible original, reproduce a disputed recommendation and withdraw bad material across every place it appears. The strongest improvement might turn out to be a corrected inventory or a clearer bulletin. The benefit to the technician is finding the right source at the time of work, regardless of which component improved.
What does one disputed answer reveal about the system?
Stage a disagreement between a technician and a documentation owner using synthetic asset data. The assistant returns a manual for the right product family and the technician recognizes a connector that does not match the installed unit. Preserve the asset identifiers the assistant received, the candidate editions it considered and the section it ultimately displayed. That record lets the team examine the decision without relying on memory of the conversation.
Check the catalogue before tuning the model. If the correct manual was absent, a better ranking rule cannot retrieve it. If both editions were present but the newer one incorrectly claimed every serial range, the catalogue owner needs to repair applicability. If eligible candidates were correct but the assistant cited an ineligible excerpt from an earlier turn, investigate how the application carries context between questions.
Write a correction for the smallest responsible layer and retest a neighboring asset that should still use the old edition. Suppressing the wrong manual globally might fix the reported case and create a new problem for an older installation. A test pair that differs only by the relevant hardware revision exposes that mistake more clearly than a large collection of unrelated everyday questions.
When the correction is ready, verify each delivery surface. A mobile cache, a search suggestion and a saved job packet may all retain references to the old result. Agree which of those surfaces can be updated automatically and which must display a warning when reopened. The person withdrawing a manual should know where the withdrawal has taken effect.
Finally, review the language of the response when applicability is uncertain. A useful answer can explain which identifier separates the two editions and where an authorized user can obtain it. That still advances the job while preserving the technician's responsibility for selecting the controlled procedure.
Document who can resolve a dispute about authority when two departments maintain overlapping instructions. A faster search index only surfaces that conflict sooner. Set an owner for resolving the document relationship, record any temporary restriction on use and verify that the corrected relationship appears in the technician’s next search before treating the case as closed.
What are the limitations and trade-offs?
- The example evaluates document lookup only.
- Equipment applicability and authorized work procedures must be confirmed by the responsible service organization.
Sources and methodology
- Data and AI agents give a lift to TK Elevator service technicians (opens in a new tab). Microsoft · Accessed 4 Nov 2026
- Building effective agents (opens in a new tab). Anthropic · Accessed 4 Nov 2026
About the author. Oleksandr Veklych, Co-founder & CTO, ACW Soft Solutions. Oleksandr Veklych is the CTO and co-founder of ACW Soft Solutions, where he leads technical strategy and delivery for custom software: SaaS platforms, marketplaces, internal business systems, mobile apps and AI-based solutions. His focus is turning growing complexity into clear architecture, reliable delivery and scalable technical decisions, so teams can move faster without rebuilding everything. He holds a bachelor’s degree in cybersecurity from Taras Shevchenko National University of Kyiv. View profile

