[AI Agent Diaries] Thinking Fast and Slow About AI Agent Development

Using Daniel Kahneman's "Fast and Slow" framework to recognise the traps our mind sets ourselves when working on AI projects.

Thinking Fast and Slow

This week my colleague -

Maaike Coppens

- started writing about AI Product Development. Her excellent post on the challenges AI Product Managers face inspired me to run a favorite thought exercise of mine - reviewing the challenges of a specific space through the lens of Daniel Kahneman's "Thinking, Fast and Slow".

In the book "Thinking, Fast and Slow", psychologist Daniel Kahneman provides a conceptual framework that can help us better understand how we humans approach a wide range of issues. His basic tenet of two competing thought systems is particularly relevant when considering how we develop and deploy AI agents, especially in an environment of intense hype and competition.

System 1, as Kahneman calls it, is fast, instinctive, and emotional. System 2 is more deliberative, rational, and effortful.

When it comes to AI, we often allow System 1 to lead us down the wrong path by anthropomorphizing AI systems or relating them to patterns we've seen in science fiction and popular media, rather than understanding their true capabilities and limitations. We do not give ourselves the time to engage with System 2 and think through the harder problems.

AI projects, it could be argued, are almost perfectly designed to set these cognitive traps for our minds and lead us down the wrong path. Here are some examples.

Framing

What on the surface can appear as a simple request such as "we need an AI Agent," once unpacked, opens up a significant number of questions that go well beyond what people intuitively consider the domain of AI development. This framing significantly influences our subsequent choices. However, more often that not I come across teams that start by framing the problem of creating an AI agent in terms of "which large language model should we use" as opposed to "what user needs are we trying to address". The former will lead us down a very different (and not very useful) path.

Framing influences our future choices - Photo by Jaredd Craig on Unsplash

For example, a System 1 response might be to immediately jump to implementing GPT-4o or, dare I say, DeepSeek because it's the most advanced or most talked about model available. However, a System 2 analysis might reveal that a smaller, fine-tuned model would be more appropriate for specific use cases like customer service queries about product specifications.

Substitution Problem

The rapidly evolving nature of AI technology means there is an unusually high level of noise within the available information. It's easy to be misled about the actual complexity of implementing AI solutions and become overly concerned with specific technical aspects. Kahneman's substitution problem is particularly relevant here - we are naturally inclined to substitute complex questions (e.g., "How can we ensure consistent and reliable responses?") with apparently simpler, but less useful, questions (e.g., "What's the best prompting technique to use?").

Breaking through the noise to get to what is really useful challenges how we naturally think about AI.

The Planning Fallacy and Optimism Bias

Implementing AI solutions requires a variety of different disciplines to complete. AI strategy, prompt engineering, user experience design, data science, and software engineering all come into play. Each brings its own terminology and set of norms, and stakeholders are often asked to make quick decisions to keep the project on track and within budget. Unfortunately, as Kahneman explains, humans are afflicted by the planning fallacy. We tend to consistently underestimate the time required to develop and deploy AI systems, while optimism bias means we overestimate their benefits and capabilities.

The Availability Heuristic and Risk Assessment

Kahneman's research on the availability heuristic, our tendency to assess probability based on how easily examples come to mind, is also very relevant. When evaluating AI risks, teams often focus disproportionately on widely-publicized failures while overlooking more common but less sensational issues.

Teams might invest heavily in preventing chatbots from generating clearly inappropriate content (a highly publicized risk) while underinvesting in:

• Monitoring for subtle gender or racial bias in model responses (I know its not cool to talk about these things anymore but there is still there…)

• Testing for degradation in model performance over time

• Evaluating fairness across different user demographics (see earlier comment about uncool things)

• Measuring and improving accuracy for less common but critical edge cases

To counter this bias, teams should implement structured risk assessment frameworks that include:

• Systematic logging of all types of failures, not just dramatic ones

• Regular analysis of "quiet" failures that don't make headlines

• Quantitative metrics for measuring bias and fairness

• Automated testing for common failure modes

Think Slow to Allow AI to Act Fast

Understanding both how we think about AI and how users will interact with our AI systems is central to delivering successful AI projects. We improve our chances of success by recognizing that designing, planning, and deploying AI systems requires deliberative, rational, and effortful thought. A particularly challenging task these days where fast, impulsive and emotional action seems to be the norm.

We have to avoid numerous pitfalls and stereotypes and challenge our assumptions about AI to create something that achieves our goals. Framing these challenges as simply human traits that are perfectly normal and can be overcome by recognising them and talking about them helps teams reason about them more carefully.

Successful AI implementation requires a team that recognises our limitations as humans so that we can avoid these cognitive pitfalls. This might include comprehensive discovery workshops that directly address the framing issue by getting all stakeholders aligned on the actual problem we're trying to solve with AI. Regular technical debates and continuous learning ensure teams don't shy away from hard questions and avoid the substitution problem. Finally, cross-functional teams working closely with stakeholders through an appropriate methodology help keep the project state in check and maintain realistic planning focused on deliverable outcomes.

In short, we need to engage our deliberative and rational side when developing AI systems to produce solutions that are efficient, intuitive, and naturally interactive.