Jared Kaplan, a co-founder of Anthropic, measures Claude's growing intelligence in terms of how many human thought-minutes Claude could replace. Others in Silicon Valley often fantasize about the day when doctors, lawyers, writers, and filmmakers are either completely replaced, or have their job made significantly easier by artificial intelligence. This is the dominant approach to building AI products: look at tough human tasks and try to do them as well, or even better, with an AI model. I think this approach doesn't dream big enough.
Beyond Being There, a popular HCI paper from 1992, has a related insight about designing interpersonal experiences on the internet. Given that there are many very special things about face-to-face interaction like eye contact, physicality, and zero-latency communication, experiences on the internet ought not to try to be as good as face-to-face communication, but be different from it by taking advantage of the unique characteristics of online communication. In other words, online experiences should have some creative aspect that makes them impossible to occur in person. In the best case, this experience may so novel that users right next to each other would voluntarily use their devices.
People should approach building AI products in the same way: beyond being human. Since there is something very special about the way the human mind works, a great AI product should not simply take something a human could do and try to replicate it with an AI model, but should try to create new experiences that no interaction with a human could create. ChatGPT already has some of characteristics of this style of product: instantaneous response times, superhuman knowledge, willingness to do mundane tasks, etc. But there are also some aspects that are not being put to work by any products on the market: massive parallelization of responses, selective amnesia, potential to generate tokens 24/7, and probably some others too!
A truly perfect AI product will create an output that could never be replicated by any amount of humans in any amount of time. Until then, I think we are not taking full advantage of the technology.
This threshold might be too high. AlphaFold, one of the most impressive research productions ever, technically could have its output produced by thousands of the smartest biochemists over the next thousands of years. I'm not trying to argue that products that speed up human tasks are useless. But, as a design philosophy, I think it is a mistake to constrain oneself by adopting the human-thought-minute measure of intelligence.