your ai thinks you're a genius (it's wrong)

garry tan, the ceo of y combinator, published his claude code setup. workflows for planning, reviewing, and shipping software, with prompts that give the agent different roles. people treated it like a revelation.

i find the enthusiasm around these setups strange. useful prompts are worth sharing. but the way people talk about them, you'd think installing someone else's workflow could give you their judgment too. put the model in the role of an engineering manager, ask it to review the plan, and apparently you've had an engineering review.

what bothers me is how easily the approval travels. ai helps you build something and tells you it's good. you post about it. people who haven't used it or read the code tell you it's impressive. by the end of this, you can feel very confident about work that hasn't had much independent scrutiny at all.

and the confidence feels earned. that's the problem.

why it keeps agreeing with you

ai assistants are trained partly around what people prefer to hear. reinforcement learning from human feedback, or rlhf, uses human judgments to shape a model's responses. those judgments are meant to reward useful, accurate answers. but people also like answers that agree with them, and that preference can work its way into the training.

anthropic's research on sycophancy found that humans and preference models sometimes favored convincingly written agreement over a correct answer. optimizing against those preferences could sacrifice truthfulness. the tendency has a name, sycophancy, but you've probably encountered it in less formal language.

"great approach." "that's a solid architecture." "this is clean and well-structured."

you've seen these. you've probably felt good reading them. i have too. they're particularly reassuring when you've spent a while on something and aren't sure whether it's any good. the model sounds like someone who has looked at your work and knows what they're talking about.

but you don't know how much of that response comes from the quality of the work and how much comes from the way you presented it. tell it confidently that an abstraction is necessary, and it may help you justify the abstraction. challenge its answer, and it may back down without much reason. you can end up steering the evaluation while believing you're receiving one.

this has been bad enough to require an actual rollback. in april 2025, openai withdrew a gpt-4o update after it became overly flattering and agreeable. their explanation said they'd put too much weight on short-term feedback. people liking the responses hadn't been enough to establish that the responses were good for them.

that's what makes me distrust the praise. it's so easy to mistake being pleased with an answer for having learned something from it.

experience doesn't protect you

there's a comforting version of this story where inexperienced developers are the ones who get fooled. they don't know enough to evaluate the code, so they trust the model. experienced developers can spot the mistakes and will be fine.

i'm not convinced. experience gives you better ways to check an answer, but it also gives you opinions you're used to trusting. you've spent years deciding how software should be structured. now there's a tool that can produce a convincing explanation for almost any decision you put in front of it.

say you want a service layer. you can get a service layer, an explanation of its benefits, and a review that finds it well organized. none of that necessarily answers whether a function would have done the job. if you never ask that question, the whole conversation can proceed on an assumption that needed challenging at the start.

an opinionated colleague might catch it. someone who has to maintain the code with you has a reason to ask why you've made their job harder. they remember the last abstraction that sounded clever and caused trouble. they can be annoying about it, and sometimes they're right.

an ai can push back too. it can find mistakes and offer useful criticism. but i don't trust it to keep holding a position when i start arguing for the answer i wanted. and when i'm the one giving it the context, i can leave out the very thing another person would have noticed.

that's a poor substitute for someone who can look at your pull request and ask why it exists. especially when you're pleased enough with the result that you don't really want to hear the question.

feeling productive is easy to get wrong

metr's early-2025 developer study is worth reading here. sixteen experienced open-source developers worked on real tasks in repositories they knew well. before the study, they expected ai to make them 24 percent faster. with ai available, the tasks took 19 percent longer. afterwards, they still estimated that ai had made them 20 percent faster.

that's one setting with the tools available at the time. it doesn't establish that ai slows everyone down, and it doesn't establish that flattery caused the mismatch. what it does show is that people can be badly wrong about whether the tool is helping them, even after using it on work they understand.

i find that more unsettling than the slowdown itself. at least if you notice a tool wasting your time, you can change how you use it. if the experience feels productive enough, you can keep making the same mistake and recommend it to everyone else.

code appearing on the screen gives you something to react to. the agent fixes an error, the tests pass, another task gets ticked off. all of that feels like movement. it's harder to notice the time spent reviewing suggestions that don't fit, or the simpler approach you didn't take because you were busy making the generated one work.

and once output becomes the thing we're proud of, we start measuring more of it. lines of code, commits, pull requests per week. we used to be suspicious of those numbers as productivity metrics. i'm still suspicious when the code comes from a model. a large diff can mean there's more work to review, more behavior to understand, and more to maintain. i want to know what got better.

the part i don't want to outsource

this is also where vibe coding starts to bother me. the app runs, but you can't explain why. when it breaks, you ask the same tool to fix it. you can keep going like that for a while, especially if the model is good enough to get you past the next error. but each successful fix can hide how little you understand about the thing you're now responsible for.

using a tool doesn't excuse you from learning what you've built. if anything, generating code faster makes that responsibility harder to keep up with. i don't find an app particularly reassuring when its creator needs another chat session to work out what it does.

i use claude at work. it makes some parts of my job easier. privately, i prefer writing the code myself because i like sitting with a problem long enough to understand why it breaks. debugging can be frustrating, but following a failure through several layers and finding the wrong assumption teaches you something you can use again.

when i ask an agent to get me past the failure, i have to make a separate effort to get that understanding. otherwise, the problem disappears and so does my reason to investigate it. it's tempting to move straight on to the next thing. i don't think that's harmless if it becomes the way you work every day.

this is why i'm skeptical of the idea that developers should hand over more and more of the job until we're mostly approving what the agents produce. approving it requires understanding it. i don't see how that requirement gets smaller just because the implementation was automated.

check it anyway

so yes, use the tool where it helps. but i'd be much less trusting of how it makes you feel about yourself. ask it to find a bug, then check whether the bug exists. ask it to propose a solution, then work out whether the solution fits. a cheerful review isn't a reason to skip those steps.

when it tells you an approach is good, you should still be able to explain why it's good without repeating its explanation. when it writes code, you should be able to follow what happens when that code fails. those are fairly basic expectations, and i don't want us to lower them because the new workflow is exciting.

what worries me most is how comfortable it is to stop checking. the code runs. the model approves. the replies are enthusiastic. you could spend the evening finding out whether the work holds up, or you could start another thing and get all that approval again.

there is no shortage of encouragement now. i'd rather keep the people and habits that make me reconsider.