On this page
One of the peculiar things about research is how much progress you can make on a question without ever knowing whether it was a particularly good question to ask.
Once you begin working on something, the feedback is often quite tangible. You run an experiment, find something unexpected, and spend the next few days trying to understand it. Eventually you have a better explanation, which opens up another set of experiments. There is a certain satisfaction in this process because you can usually point to something you understand today that you didn’t understand a week ago.
The original decision to work on the problem is much harder to evaluate. Suppose you spend six months pursuing an idea and eventually get a convincing result. You have learned something about the idea, and perhaps produced work you’re proud of. But deciding whether those six months were well spent requires thinking about all the other questions you might have pursued, including some you weren’t knowledgeable enough to recognize at the time. The evidence you would need to make that comparison barely exists.
I find this an interesting problem because it creates an unusual relationship between competence and judgment. As you become better at executing on ideas, you can make progress on an increasingly large number of them. Some are important, some are interesting mostly because they’re technically difficult, and others have a way of becoming compelling simply because you’ve already invested so much time in them. The ability to make progress gives you more freedom in choosing what to work on, while also making it easier to spend a long time pursuing something that deserved less attention.
I’ve started noticing a smaller version of this in my own life.
Throughout university, I developed a habit of pursuing almost anything that seemed interesting. I did research, worked at different companies, taught courses, and regularly found myself involved in projects that I knew very little about when I first agreed to them. I enjoyed being around people who were better than me at things I wanted to learn, and I was usually happy to figure out the rest as I went along.
At eighteen, this was a pretty effective way to spend my time. I had strong opinions about what I wanted to do with my life, considering how little evidence I had to support them. Almost every new experience corrected some part of my understanding. A few months doing research could teach me more about whether I enjoyed research than an entire year spent imagining what being a researcher would be like.
Over time, though, I noticed that I had become very good at finding interesting things to do. There was always another project I could convince myself was worth exploring, particularly in machine learning, where following one interesting question tends to expose several more. I liked this about myself, and I still do. Curiosity has taken me into some of the most rewarding experiences I’ve had.
Since joining Cohere, I’ve had more time to work on the same technical problems, and I’ve started paying closer attention to how people decide what to investigate. Two people can understand the same underlying concepts and still have very different opinions about which experiment is worth running or which assumption deserves a closer look. What’s interesting to me is how much experience sits behind those decisions. You can watch someone become faster at implementing ideas, but it’s much harder to see how they developed a sense of which ideas deserve the effort in the first place.
It’s made me curious about how that judgment develops, particularly because the process seems much harder to observe than the acquisition of technical skills. You can watch someone become faster at implementing ideas or more proficient at running experiments. The gradual change in what they choose to investigate is less visible, even though it may eventually matter much more.
The changing economics of learning
One thing I find interesting about learning is how much its value depends on what you already know. The same experience can be almost useless to one person and completely change how another person thinks about a problem. And as you become more knowledgeable, your ability to extract value from new information changes in ways that you wouldn’t expect beforehand.
Consider reading a research paper in an area you know very little about. You might understand the main contribution, recognize why the result is impressive, and come away with a few ideas worth trying. however, there’s a limit to how much you can get out of it because you have very little context / experience against which to evaluate the claims. you don’t know which decisions are routine, which assumptions are questionable or how the proposed approach reacts when you extrapolate it.
Spend six months working on similar problems and read the paper again. A sentence in the methodology which you barely noticed before might become very important. Even the decision to pursue this problem in the first place becomes a lot more interested / clear once you understand the trade offs.
I like this example because the information available to you has stayed exactly the same. What changed was your ability to make use of it.
This suggests that the returns to learning have an interesting dependency structure. Understanding one thing can increase the value of understanding something else. A few months spent developing intuition for a system might make the next paper you read considerably more useful, because you now have somewhere to put the information.
I’ve noticed this in technical work. You spend hours trying to explain a strange result or tracing a failure through a codebase, and by the end the explanation can feel too specific to be useful again. Then a few weeks later, you encounter a completely different problem and recognize something familiar in its behavior. A detail you spent an afternoon investigating suddenly becomes useful. Even the explanations you ruled out have value because you know which possibilities aren’t worth revisiting.
This is what I’ve come to appreciate about advice I heard years ago: chase knowledge & understanding. You often have very little idea which parts of what you’re learning will eventually become useful. Some of the most valuable things you know today probably felt like unnecessary details when you first encountered them.
I think this is an underappreciated aspect of developing expertise. The knowledge accumulates in an uneven fashion, and a lot of its value becomes evident only when something happens to draw on it. While you’re acquiring this intuition, it’s hard to see the bigger picture i.e you have very little evidence that your understanding of this subject is improving in a meaningful way. You could spend several days going after an idea or explanation that turns out to be wrong. However, it might turn out to be useful later in the future because you ruled out an entire class of explanations that are adjacent which you might’ve tried again.
This also makes the cost of switching between projects more complicated than I once appreciated.
When you spend enough time with a problem, you carry around a surprisingly detailed understanding of it. You remember the experiments that failed for reasons you couldn’t fully explain, the parts of the system you distrust, and the questions you’ve been meaning to revisit. Some of that can be written down, but a lot of it exists as an intuition that you haven’t quite learned how to articulate. Leaving the problem interrupts the process through which those observations might eventually develop into something more useful.
During university, I rarely thought about that cost. Moving between research, courses, internships, and other projects was such a normal part of my life that starting over felt almost free. I enjoyed the early stages of learning something unfamiliar, and I became reasonably good at getting up to speed quickly. What I had less experience with was staying somewhere long enough to see what happened after getting up to speed stopped being the main challenge.
Working at Cohere has made me appreciate the amount of context that sits behind apparently simple technical judgments. Someone can look at an approach and have a fairly strong opinion about whether it’s worth investigating. From the outside, that can look like intuition in the almost mysterious sense of the word. Usually, there’s an enormous amount of experience compressed into the judgment. They’ve encountered enough related problems to recognize which details deserve attention, and they’ve developed a sense of how likely certain approaches are to succeed.
Expertise can also make exploration surprisingly productive. Imagine spending months trying to understand where an inference system is losing time. You’ve developed a fairly detailed picture of its bottlenecks, including which parts of the system behave differently under load. Then you come across a scheduling technique from operating systems that you’d previously had little reason to care about. You might immediately recognize where it could be useful because you’ve spent so much time understanding the particular inefficiency it addresses. Someone encountering the technique without that context could understand it perfectly well and never make the connection.
So the returns to exploring unfamiliar ideas can rise as your expertise grows. The important variable is how much useful understanding you can bring to an unfamiliar problem, and that depends heavily on what you’ve spent the preceding years learning.
That creates a peculiar difficulty. You have to decide whether a problem deserves more of your time using your current understanding of it, while knowing that the most important thing you might gain from staying is a better understanding of what makes the problem worth pursuing.
And this is where I’ve become increasingly interested in research taste. If your ability to recognize promising problems develops through experience, how do you tell whether you’re spending your time acquiring that judgment or simply becoming more comfortable with the problems you’ve already chosen?
How a problem earns your attention
When someone describes a new research direction, it’s usually possible to construct a convincing argument for pursuing it. There’s some limitation in the current approaches, an idea for addressing it, and an expectation of what might improve if the idea works. At that level of abstraction, an enormous number of projects sound promising. You can spend an afternoon reading papers and come away with a list of questions that could plausibly occupy the next six months.
The difficult part is figuring out which of those questions becomes more interesting when you examine it closely.
Consider a model that performs poorly on a particular reasoning benchmark. One approach would be to improve its performance by experimenting with a new training method. You might spend a few weeks implementing the approach and eventually get a measurable improvement. That’s useful work, and depending on the circumstances, it might be exactly what the system needs.
But suppose you spent the first few days looking closely at where the model was failing. You might discover that the errors are concentrated in examples requiring information from distant parts of the context. Or perhaps the model produces a reasonable answer, but the evaluation penalizes it for disagreeing with an unnecessarily narrow reference. You could also find that the failures are distributed across tasks that initially seemed unrelated, suggesting some common limitation you hadn’t considered.
Each observation changes your understanding of the problem. What initially looked like a general deficiency in reasoning might become a much more specific question about how the model uses context, or about whether the evaluation measures the behavior you care about.
The value of those first few days comes from the decisions they allow you to make afterward. Understanding the failure might save you weeks of optimizing the wrong thing. It might also reveal a considerably more interesting problem than the one you began with.
Spending a few days understanding a failure before jumping into a solution sounds straightforward, although I’ve found it takes some discipline. Once I have an idea for a technical approach, my instinct is usually to start implementing it. There’s an appealing clarity to having something concrete to do, particularly when the alternative involves sitting with a poorly understood problem and figuring out which questions to ask.
It’s also much easier to measure progress once you’ve decided what you’re optimizing. You can track the experiments you’ve completed and see whether the metric is improving. Investigating the problem itself produces less orderly progress. You might spend a day examining failures and end up realizing that your original framing was wrong, which can feel frustrating despite having learned something important.
The distinction matters because experiments can serve two purposes. Some help you improve a system you already understand reasonably well. Others help you figure out what the system is doing and where further work might matter. Early in a research project, I’ve become increasingly interested in the second kind.
One question I find useful is whether an experiment could change my mind about what to work on next.
Suppose I’m considering two explanations for a failure. One suggests that I should investigate the training data, while the other points toward a limitation in the model’s ability to use information from a long context. An experiment that helps distinguish between those explanations could be valuable even if it produces no improvement in performance. Its result would determine where I spend the following weeks.
Now imagine an experiment where I already know that, regardless of the result, I’ll probably continue trying variations of the same approach. There may be good reasons to run it, especially if I’m refining a solution that already looks promising. But if I’m still trying to decide whether the direction deserves substantial investment, it tells me comparatively little.
This suggests a practical way to think about the early stages of research: choose investigations that improve your ability to choose what to investigate next.
The harder decision comes when a problem keeps producing reasons to investigate it further.
I’ve noticed that certain technical problems become increasingly interesting as I understand them. You begin with a small anomaly, investigate the obvious explanations, and discover that none accounts for what you’re seeing. Then you find another observation that seems related. You start to suspect that both are consequences of something more general.
At that point, the original problem has become a way of investigating a larger question. The work has given you a reason to care about something you could barely have formulated when you started.
I think this is worth paying attention to because it’s difficult to predict from a project’s initial description. Some problems look impressive from a distance and become fairly routine once you understand them. Others appear narrow until you discover how many things depend on the same underlying mechanism.
There is also a danger of confusing this growing interest with attachment. Spending enough time on a problem makes almost any problem feel significant. You accumulate unresolved details and become personally invested in explaining them. I think the useful distinction is whether the work is changing your understanding of something beyond the immediate task. Are you discovering constraints that affect other decisions? Does the explanation help account for behavior elsewhere in the system? Would resolving the uncertainty change what you or someone else should do?
Those seem like better reasons to continue than the fact that you’ve already spent a month working on it.
Of course, some important questions resist small experiments, and their significance may take years to establish. There are limits to how much uncertainty you can resolve before committing. But even in those cases, I think it’s valuable to understand what you expect the next period of work to teach you. You can be uncertain about the eventual outcome while still having a clear reason for the next experiment.
I’ve been noticing how much of this applies to decisions outside work, especially now that I’ve graduated.
There are surprisingly few things in life you can evaluate properly before experiencing them. You choose a job after a handful of conversations, move somewhere knowing very little about what your days there will feel like, or commit to something because it seems worth pursuing at the time. A year later, you understand the decision differently. You’ve lived through enough ordinary days to notice things that would have been impossible to anticipate from the outside. Some experiences become more rewarding as you grow familiar with them. Others gradually lose the appeal that made you pursue them in the first place.
I think that’s part of what has made leaving university interesting. For years, there was always another semester approaching, bringing with it some reason to reconsider how I was spending my time. Now I have considerably more freedom to let things continue. I could settle into a routine I enjoy and, without making any particularly consequential decisions, find myself living roughly the same way five years from now. I’ve been trying to think about that possibility carefully, because there are parts of my life I’d be quite happy to see remain familiar and others I hope will change considerably.
I’ve also become more aware of how impatient I can be with things that take time to become meaningful. During university, I developed a habit of looking forward to whatever was coming next. I enjoyed having something new to work on, somewhere to go, or another experience to look forward to. That made those four years incredibly full, although I sometimes wonder how much I missed by being so occupied with what was around the corner. There are things whose value you begin to appreciate only after they’ve become part of your ordinary life. Friendships deepen through years of conversations that seem unremarkable when you’re having them. A place starts to feel like home because of all the small memories you’ve accumulated there. Even your interests change as you spend enough time with them to discover what initially attracted you was only a small part of what they had to offer.
I like where I am right now. I’m enjoying my work and there are several things I’m excited to keep exploring. I expect my understanding of what matters to me will change quite a bit over the next few years. Looking back at university, some of the experiences that ended up shaping my life began as decisions I barely thought twice about.
There’s something comforting about remembering that. I have a better sense of myself than I did four years ago, and I still have so much to learn about what makes a life worth spending time on. I’m curious to see which of the things I’m doing today I’ll look back on with the same affection.