Haruki Tan is Chief Technology Officer at a Vietnamese insurance company that operates across four Southeast Asian markets. In 2024, his team evaluated thirty-seven distinct AI tools, platforms, and models over twelve months. They adopted nine. They piloted but ultimately rejected eleven. They deprioritized seventeen without formal evaluation, because earlier in the year they had developed a framework that let them screen out tools failing basic criteria within a one-day assessment. "The mistake most organizations make," Haruki says, "is that they spend as much time evaluating the next promising AI newsletter tool as they spend evaluating a model that will affect how they price fifty thousand policies. You have to decide quickly what deserves serious attention. Otherwise the evaluation process becomes a distraction from the work."

The Problem With AI Hype Cycles

Emerging AI technologies arrive in a pattern that repeats reliably. A new capability is announced or demonstrated. Early adopters write enthusiastic blog posts and case studies. Analyst firms publish projections. Vendors build products around the capability. Organizations feel pressure to adopt so as not to be left behind. Many adopt quickly. Two years later, a substantial portion have abandoned the technology or limited its use significantly, while a smaller group have integrated it into genuine business capability.

The Gartner Hype Cycle captures this pattern visually: a Peak of Inflated Expectations followed by a Trough of Disillusionment before reaching the Plateau of Productivity. The organizations that avoid the trough are not the ones that adopted later. They are the ones that evaluated rigorously before adopting, understood what the technology could actually do in their specific context, and built the adoption infrastructure that made it stick. Waiting is not a strategy; it is simply a delay that leaves you making the same unexamined decision later than everyone else.

The practical implication is a redefinition of what evaluation is for. Your job when assessing an emerging AI technology is not to determine whether it is impressive. Impressive is easy to establish and tells you almost nothing. Your job is to determine whether the technology will generate specific, measurable value in your specific organizational context, and whether you can manage the risks of adopting it at your current maturity level. Those are two separate questions, and a technology can pass the first while failing the second badly.

The Four-Stage Evaluation Framework

Haruki's team developed a four-stage evaluation framework that scales the depth of assessment to the potential impact of the technology. This prevents the equal-time-on-everything problem that consumed disproportionate evaluation resources before the framework existed. The governing principle is that most candidates should exit at the cheapest stage, and the framework is working when the funnel is steep.

Stage 1: Rapid screen

The rapid screen answers one question: does this technology warrant serious investigation? It covers four criteria. Relevance asks whether the technology addresses a problem or opportunity that is on the current strategic agenda. Feasibility asks whether the organization has, or could reasonably acquire, the data, infrastructure, and talent the technology requires. Vendor stability asks whether the organization behind the technology is a credible entity with a track record. Risk profile asks whether there are obvious regulatory, security, or ethical concerns that would make adoption inadvisable regardless of the value on offer.

If any criterion produces a clear negative, the technology is deprioritized without further investment. The screen should take one analyst roughly one day using publicly available information, and the output is a single recommendation: investigate further, monitor and revisit in six months, or deprioritize. The middle option matters more than it looks. A technology that fails on feasibility today may pass in six months once a data platform lands, and recording that explicitly is what stops the organization from re-litigating the same candidate every time someone reads about it.

Stage 2: Technical assessment

Technologies that pass the rapid screen receive a technical assessment: hands-on testing by a small internal team, ideally two to four people including at least one technical expert and one business domain expert, over two to four weeks. The team establishes whether the technology actually does what it claims, how it performs on data resembling the organization's real data, what integration requirements adoption would create, and what the failure modes are, including how the system behaves when it fails rather than only how well it performs when it works.

The assessment should use the organization's own data wherever possible, not the vendor's demonstration data. A model that performs impressively on curated examples may perform very differently on the messy, incomplete, idiosyncratic data that characterizes real organizational environments. This is where many technologies that looked compelling in a presentation turn out to be significantly weaker than expected, and finding that out in a two-week assessment is dramatically cheaper than finding it out after integration. The presence of a business domain expert on the team is not a formality; they are the person who can tell whether an output is wrong in a way a technical evaluator would score as correct.

Stage 3: Proof of concept

Technologies that pass the technical assessment proceed to a proof of concept: a limited deployment in a real business context, with a specific use case, real users, and defined success metrics, running six to twelve weeks. The proof of concept answers whether the technology produces the expected value in the actual operating environment, what adoption challenges emerge once real people are asked to change how they work, and what integration and operational requirements were not anticipated in the earlier stages.

It needs a defined scope, a defined timeline, and pre-agreed success criteria that determine whether the technology proceeds to broader adoption or is rejected. Without criteria set in advance, proof-of-concept evaluations tend to expand indefinitely, or to conclude with a recommendation driven by the sunk cost of the time already invested rather than by evidence of value. Agreeing the bar before you see the results is the only reliable protection against a team talking itself into a technology it has grown attached to.

Stage 4: Adoption decision

Based on the proof of concept, the organization makes an explicit adoption decision: broad deployment, limited deployment in specific contexts, continued monitoring without deployment, or rejection. The decision should be documented with its reasoning, and the documentation should include an explicit assessment of the risks being accepted by adopting and the costs of the risks being accepted by not adopting. Both halves belong in the record. Declining a technology is also a decision with consequences, and an organization that only documents its adoptions will never learn anything from its refusals.

Evaluating Adoption Risk

Every emerging technology adoption carries risk. The evaluation process must surface these risks explicitly rather than managing them as a side consideration, and they are genuinely separate dimensions. Collapsing them into a single intuition about whether something "seems ready" is how organizations end up surprised by a category of problem they never assessed.

Technology immaturity risk. A technology that is genuinely new, less than two years from public release with limited production deployments, carries the risk that its failure modes are not yet well understood. The vendor's documentation may not reflect real-world edge cases, and the integration challenges may not be fully known even to the people who built it. Adopting immature technology means accepting that you will discover some of its limitations in production, which is considerably more expensive than discovering them in evaluation.

Documentation risk. Related but distinct: insufficient documentation is its own adoption hazard, and it is one of the easiest to detect early and one of the most commonly waved through. Thin, incomplete, or rapidly changing documentation means your team will reverse-engineer expected behavior from experimentation, that onboarding new engineers takes longer than planned, and that you have no authoritative reference when a system behaves unexpectedly in production. Assess documentation quality during the technical assessment as a first-class criterion, covering not only whether the documentation exists but whether it describes limitations and failure modes rather than only capabilities.

Vendor risk. AI tool vendors include many early-stage companies with limited capitalization and uncertain futures. If a vendor providing a capability you have built into your operations fails or is acquired, the disruption can be significant. Evaluate financial stability, the depth of the customer base, and the existence of contractual protections including data portability if you need to move to another provider. The question to answer is not only whether the vendor will survive, but what specifically happens to your operations if it does not.

Integration risk. AI tools rarely operate in isolation. They integrate with data sources, workflows, and downstream systems, and integration complexity is consistently underestimated in evaluations that focus on the tool itself. Assess integration requirements during the technical assessment and budget for integration costs that are typically two to three times the cost of the tool itself. An evaluation that compares licence prices across candidates without comparing integration burden is comparing the smaller number.

The first-mover advantage in AI is real but often smaller than it appears. The fast-follower advantage, learning from the adopter who discovered the integration problems, the vendor instability, and the compliance gaps, is also real. Know which you are choosing, and choose it deliberately.

The point of the quotation is not that following is safer. It is that the choice is usually made by default rather than by decision. An organization that adopts early because a competitor announced something, or delays because no one wanted to sign off, has selected a position without evaluating whether it suits their maturity level, their risk appetite, or the specific technology in question. Making the call explicitly, and recording which advantage you are pursuing and why, converts a reflex into a strategy.

Maintaining Technology Awareness

Evaluation only works if your team is aware of relevant emerging technologies in the first place. This requires deliberate investment in technology scanning: someone in the organization whose job includes monitoring developments in AI research, vendor announcements, competitor adoptions, and regulatory guidance. Without a source of candidates, the four-stage framework is a well-designed funnel with nothing entering the top of it.

Scanning should not be left to individuals who happen to find it personally interesting. It should be a defined function with defined outputs: a monthly technology briefing, a quarterly horizon assessment, and an annual review of the technology roadmap against what actually developed. Defined outputs matter because they create a cadence that survives the departure of the enthusiast who has been doing the reading, and because a briefing with a due date gets written whereas general awareness does not.

Partnerships are a cost-effective supplement to internal scanning, and the three types return different things. Vendor briefings keep you aware of what is coming before it is publicly released. Academic connections surface research that will become commercially available in twelve to eighteen months, which is roughly the horizon at which strategic planning is still useful. Industry peer groups and events reveal what organizations in similar sectors are evaluating and what they have found, which often provides the most practically useful signal of the three, because peers share the failures that vendors do not and operate under constraints that resemble your own. Participation in industry events is worth budgeting deliberately rather than treating as discretionary travel; the conversations between sessions are usually where the useful information is.

Communicating the Technology Roadmap

Evaluation produces decisions, and decisions that stay inside the evaluating team create a predictable set of problems. Business units make plans that assume a capability the organization has already rejected. Teams independently evaluate a technology that was assessed six months ago. Leadership hears about a competitor's adoption and asks why nothing is happening, when in fact a proof of concept is underway. The remedy is a published technology roadmap that states what the organization is adopting and when.

A useful roadmap has three registers. It names what is being adopted now, with the timeline and the business areas affected, so that dependent planning can proceed with confidence. It names what is under evaluation, with the stage each candidate has reached and the approximate date a decision is expected, which stops parallel evaluation and sets realistic expectations. And it names what has been assessed and deliberately deprioritized, with the reason, which is the register most organizations omit and the one that saves the most time. A recorded decision not to adopt, with its reasoning, prevents the same question being reopened every quarter by someone who has just encountered the technology for the first time.

Roadmap communication is also how the evaluation function earns the standing it needs. A team that is seen to make timely, reasoned, documented decisions gets consulted before business units commit to tools independently. A team whose work is invisible gets routed around. The roadmap should be updated on the same cadence as the horizon assessment so that the published picture and the actual pipeline do not drift apart, and it should be written for business readers rather than for the evaluators who produced it.

Anti-Patterns

• Equal time on every candidate. Spending as much effort on a minor productivity tool as on a model affecting core pricing is not rigor. It is the failure mode a staged framework exists to prevent.
• Evaluating on the vendor's demonstration data. Curated examples are selected to look good. Performance on your own messy, incomplete data is the only result that predicts production behavior.
• Starting a proof of concept without pre-agreed success criteria. Evaluations then expand indefinitely or conclude on sunk cost, driven by the time already invested rather than the evidence produced.
• Treating readiness as a single intuition. Technology immaturity, documentation quality, vendor stability, and integration complexity fail independently. An overall sense that something "seems ready" assesses none of them.
• Costing the tool and not the integration. Comparing licence prices while ignoring integration burden compares the smaller number, when integration typically runs two to three times the cost of the tool itself.
• Leaving scanning to whoever finds it interesting. Awareness that depends on one enthusiast's reading habits disappears when that person changes roles, and produces no dated outputs anyone can rely on.

Practice Prompts

• Write a one-page rapid screen for a technology currently attracting attention internally, covering relevance, feasibility, vendor stability, and risk profile, and produce one of the three permitted recommendations.
• For a tool currently in evaluation, define the proof-of-concept success criteria before any results exist, then circulate them and get agreement in writing.
• Assemble a representative sample of your real operational data, including the incomplete and irregular records, and hold it ready as the standard test set for the next technical assessment.
• Draft the three registers of a technology roadmap for your organization: adopting now, under evaluation, and deliberately deprioritized with reasons.
• Decide, for one technology on your horizon, whether you are pursuing first-mover or fast-follower advantage, and write down why that suits your current maturity level.

Reflection

Think about the last emerging AI technology your organization decided not to pursue. Can you find the decision written down anywhere, with the reasoning attached? In most organizations the answer is no, which means the assessment work was done and the institutional memory of it was not retained; the next person to encounter that technology will start from zero and very likely reach the same conclusion at the same cost. Then total your own evaluation effort over the past year and ask how much of it went to technologies that could materially change how the organization operates, and how much went to candidates a single day of scrutiny would have removed. Haruki's team deprioritized seventeen of thirty-seven candidates without formal evaluation, and that discipline is exactly what created the capacity to evaluate the rest properly.

Glossary

• Rapid screen: A one-day assessment using public information that tests relevance, feasibility, vendor stability, and risk profile, returning one of three recommendations: investigate, revisit in six months, or deprioritize.
• Technical assessment: Hands-on evaluation by a small mixed team over two to four weeks, testing claimed capability against the organization's own data and establishing integration requirements and failure modes.
• Technology immaturity risk: The exposure created by adopting something less than two years from public release with limited production deployments, whose failure modes are not yet well understood.
• Documentation risk: The exposure created by thin, incomplete, or rapidly changing documentation, which forces teams to reverse-engineer expected behavior and leaves no authoritative reference when systems behave unexpectedly.
• Technology scanning: A defined function with dated outputs, including a monthly briefing, a quarterly horizon assessment, and an annual roadmap review, monitoring research, vendor announcements, competitor adoptions, and regulatory guidance.

Related Lessons

• Innovation Portfolio Management covers how evaluated technologies are balanced and sequenced across a portfolio of initiatives.
• Internal Innovation Structures addresses the organizational arrangements that house the scanning and evaluation functions described here.
• Building Innovation Partnerships goes further into the vendor, academic, and peer relationships that feed the evaluation pipeline.
• Vendor & Platform Landscape Assessment extends the vendor stability criterion into a fuller assessment method.
• Ongoing Vendor Management & Governance picks up after the adoption decision, covering the relationship you now have to manage.

Closing

Rigorous evaluation is often mistaken for caution, and it is not. Haruki's team adopted nine technologies in a single year, which is not the behavior of a conservative organization. What the framework bought them was the ability to move quickly on the things that mattered, because they were not spending their evaluation capacity on candidates a single day of scrutiny would have removed. The habits that produce this are unglamorous and cumulative: screen fast and consistently, test on your own data, set the bar before you see the results, assess immaturity, documentation, vendor stability and integration as four separate questions, document what you decided and why including what you declined, and publish the roadmap so the rest of the organization can plan against what you know. None of it requires additional headcount. It requires the discipline to do the cheap steps properly so the expensive ones are reserved for technologies that deserve them.

Key Takeaways

• Evaluate depth proportionate to potential impact. A four-stage framework scales evaluation to significance, and it is working when most candidates exit at the cheapest stage.
• Rapid screens prevent wasted effort. A one-day assessment of relevance, feasibility, vendor stability, and risk profile filters out tools that do not merit deeper evaluation, creating organizational time for what matters.
• Test on your data, not the vendor's. Demonstration data is curated to look good; performance on your actual messy data determines real-world value. A business domain expert on the team is what catches outputs that are wrong in domain-specific ways.
• Pre-agree proof-of-concept success criteria. Without defined criteria, evaluations expand or conclude on sunk cost rather than evidence. Set the bar before you start, not after you see the results.
• Immaturity, documentation, vendor risk, and integration complexity are four separate dimensions. Evaluate each explicitly rather than forming a combined intuition about readiness. Integration alone typically costs two to three times the tool.
• Choose first-mover or fast-follower deliberately. Both positions are legitimate. The failure is arriving at one by default rather than by decision.
• Technology scanning is a function, not a hobby. It needs defined responsibility and dated outputs. Vendor, academic, and peer relationships each return a different kind of signal, and industry events are worth budgeting rather than treating as discretionary.
• Document decisions, including rejections, and publish the roadmap. Stating what is being adopted, what is under evaluation, and what has been deliberately deprioritized stops repeated assessments and keeps the evaluation function in the path of decisions rather than beside it.

Frequently Asked Questions

How do we decide how much evaluation a technology deserves? Scale the depth to the potential impact. Everything enters through a one-day rapid screen, and only candidates that could materially affect how the organization operates progress to a technical assessment or a proof of concept. The framework is working when most candidates exit at the cheapest stage; a shallow funnel means the screen is not doing its job.

What if a technology fails the rapid screen only on feasibility? That is what the middle recommendation is for. Record it as monitor and revisit in six months, with the specific reason. A candidate that fails because a data platform is not yet in place may pass once that platform lands, and having the reason written down prevents the organization re-litigating the same question each time someone new encounters it.

How should we assess vendor risk on an early-stage company? Look at financial stability, the depth of the customer base, and the contractual protections available to you, particularly data portability if you need to move providers. The question worth answering is not only whether the vendor survives, but exactly what happens to your operations if it does not, and whether you could execute that path.

Who should own technology scanning? A defined role or function, not an interested individual. The distinguishing feature is dated outputs: a monthly briefing, a quarterly horizon assessment, and an annual review of the roadmap against what actually developed. Supplement internal scanning with vendor briefings for pre-release awareness, academic connections for research that becomes commercially available in twelve to eighteen months, and peer groups and industry events for what comparable organizations have actually found.