What Kind of Feedback Do You Need?
Feedback shows up in many different forms for different types of learning projects.
Getting good at stand-up comedy and learning to write computer programs involve very
different kinds of feedback. Learning higher math and learning languages are going to use
feedback in different ways. The opportunities for seeking better feedback will vary
depending on what you’re trying to learn. Rather than try to spell out exactly what
feedback you need for your learning project, I think it’s important to consider different
types of feedback, along with how each one can be used and cultivated. By knowing what
kind of feedback you’re getting, you can make sure to use it best, while also recognizing
its limitations. In particular, I want to consider three types of feedback: outcome feedback,informational feedback, and corrective feedback. Outcome feedback is the most common
and in many situations the only type of feedback available. Informational feedback is also
fairly common, and it’s important to recognize when you can split apart outcomes to get
feedback on parts of what you’re learning and when feedback only on holistic outcomes is
possible. Corrective feedback is the toughest to find but when employed well can
accelerate learning the most.
Outcome Feedback: Are You Doing It Wrong?
The first type of feedback, and the least granular, is outcome feedback. This tells you
something about how well you’re doing overall but offers no ideas as to what you’re doing
better or worse. This kind of feedback can come in the form of a grade—pass/fail, A, B, or
C—or it can come in the form of an aggregate feedback to many decisions you’re making
simultaneously. The applause Tristan de Montebello received (or the crickets he heard)
after a speech is an example of outcome feedback. It could tell him if he was getting
better or worse, but it couldn’t really say why or how to fix it. Every entrepreneur
experiences this kind of feedback when a new product hits the market. It may sell wildly
well or abysmally, but that feedback comes in bulk, not directly decomposable into the
various aspects of the product. Did the product cost too much? Was the marketing
message not clear enough? Was the packaging unappealing? Customer reviews and
comments can provide clues, but ultimately the success or failure of any new product is a
complex bundle of factors.
This type of feedback is often the easiest to get, and research shows that even getting
this feedback, which lacks a specific message about what you need to improve, can be
helpful. In one study, feedback for a task involving visual acuity facilitated learning, even
when it was delivered in blocks that were too large to get any meaningful information
about which responses were correct and which were incorrect. 4 Many projects that wholly
lack feedback can easily be changed to get this broad-scale feedback. Eric Barone, for
instance, provided a development blog to publish work on his game and solicit feedback
from early drafts. It couldn’t provide him with detailed information about what exactly to
improve and change, but his simply being immersed in an environment that provided
feedback at all was helpful.
Outcome feedback can improve how you learn through a few different mechanisms. One
is by providing you with a motivational benchmark against your goal. If your goal is to
reach a certain quality of feedback, this feedback can give you updates on your progress.
Another is that it can show you the relative merits of different methods you’re trying.
When you are progressing rapidly, you can stick to those learning methods and
approaches. When progress stalls, you can see what you might be able to change in your
current approach. Although outcome feedback isn’t complete, it is often the only kind
available and can still have a potent impact on your learning rate.
Informational Feedback: What Are You Doing Wrong?
The next type of feedback is informational feedback. This feedback tells you what you’re
doing wrong, but it doesn’t necessarily tell you how to fix it. Speaking a foreign language
with a native speaker who doesn’t share a language with you is an exercise in
informational feedback. That person’s confused stare when you misuse a word won’t tellyou what the correct word is, but it will tell you that you’re getting it wrong. Tristan de
Montebello, in addition to the overall assessment of his performance by audience
members at the end of a speech, can also get live informational feedback about how it’s
going moment to moment. Did that joke work? Is my story boring them? This is something
you can spot in the distracted glances or background chatter throughout your speech.
Rock’s stand-up experiment is also a type of informational feedback. He can tell when a
certain joke lands or doesn’t, based on the reaction of the audience. However, they can’t
tell him what to do to make it funnier—he’s the comedian, not them.
This kind of feedback is easy to obtain when you can get real-time access to a feedback
source. A computer programmer who gets error messages when her programs don’t
compile properly may not have enough knowledge to understand what she’s doing wrong.
But as errors increase or diminish, depending on what she does, she can use that signal tofix her problems. Self-provided feedback is also ubiquitous, and in some pursuits it can be
almost as good as feedback from others. When painting a picture, you can simply look at
it and get a sense of whether your brushstrokes are adding to or detracting from the
image you want to convey. Because this kind of feedback often comes from direct
interaction with the environment, it often pairs well with the third principle, directness.
Corrective Feedback: How Can You Fix What You’re Doing Wrong?
The best kind of feedback to get is corrective feedback. This is the feedback that shows
you not only what you’re doing wrong but how to fix it. This kind of feedback is often
available only through a coach, mentor, or teacher. However, sometimes it can be
provided automatically if you are using the right study materials. During the MIT
Challenge, I did most of my practice by going back and forth between assignments and
their solutions, so that when I finished a problem, I was shown not only whether I had
gotten it right or wrong but exactly how my answer differed from the correct one.
Similarly, flash cards and other forms of active recall provide corrective feedback by
showing you the answer to a question after you make your guess.
The educators Maria Araceli Ruiz-Primo and Susan M. Brookhart argue, “The best
feedback is informative and usable by the student(s) who receive it. Optimal feedback
indicates the difference between the current state and the desired learning state AND
helps students to take a step to improve their learning.” 5
The main challenge of this kind of feedback is that it typically requires access to a
teacher, expert, or mentor who can pinpoint your mistakes and correct them for you.
However, sometimes the added edge of having corrective over merely informational
feedback can be worth the effort needed to find such people. Tristan de Montebello
worked with Michael Gendler to help him with his public speaking performance, and that
helped him spot subtle weaknesses in his presentations that would have gone unnoticed
by himself or by a less experienced audience member giving broader feedback.
This type of feedback trumps outcome feedback, which can’t indicate what needs
improving, and informational feedback, which can indicate what to improve but not how.
However, it can also be unreliable. Tristan de Montebello would often get conflicting
advice after delivering a speech; some audience members would tell him to slow down,
while others said to speed up. This can also be a situation in which paying for a tutor can
be useful, because that person can spot the exact nature of your mistake and correct it
with less struggle on your part. The self-directed nature of ultralearning shouldn’t
convince you that learning is best done as an entirely solitary pursuit.Further Notes on Types of Feedback
A few things are worth noting here. First, you need to be careful when trying to “upgrade”
feedback from a weaker form to a stronger form if it’s not actually possible. To switch
from outcome feedback to informational feedback, you need to be able to elicit feedback
on a per element basis of what you’re doing. If instead the feedback is being provided as
a holistic assessment of everything you’re doing, trying to turn it into informational
feedback can backfire. Game designers know to watch out for this, because asking play
testers what they don’t like about a game can often return spurious results: for example,
they don’t like the color of the character or the background music. The truth is, the
players are evaluating the game holistically, so they often can’t offer this kind of
feedback. If their responses come from using it as a whole, not from each aspect
individually, asking for greater specificity may lead to guesses from those giving
feedback.
Similarly, corrective feedback requires a “correct” answer or the response of a recognized
expert. If there is no expert or a single correct approach, trying to turn informational
feedback into corrective feedback can work against you when the wrong change is
suggested as an improvement. De Montebello noted to me that the advice most people
gave him wasn’t terribly useful, but the consistency of it was. If his speech elicited wildly
different reactions each time, he knew there was still a lot of work to do. When the speechstarted to get much more consistent comments, he knew he was onto something. This
illustrates that ultralearning isn’t simply about maximizing feedback but also knowing
when to selectively ignore elements of it to extract the useful information. Understanding
the merits of these different types of feedback, as well as the preconditions that make
them possible, is a big part of choosing the right strategy for an ultralearning project.
How Quick Should Feedback Be?
An interesting question in the research on feedback is how quick it should be. Should you
get immediate information about your mistakes or wait some period of time? In general,
research has pointed to immediate feedback being superior in settings outside of the
laboratory. James A. Kulik and Chen-Lin C. Kulik review the literature on feedback timing
and suggest that “Applied studies using actual classroom quizzes and real learning
materials have usually found immediate feedback to be more effective than delay.” 6
Expertise researcher K. Anders Ericsson agrees, arguing in favor of immediate feedback
when it assists in identifying and correcting mistakes and when it allows one to execute a
corrected version of their performance revised in response to the feedback. 7
Interestingly, laboratory studies tend to show that delaying the presentation of the correctresponse along with the original task (delayed feedback) is more effective. The simplest
explanation of this result is that presenting the question and answer again offers a
second, spaced exposure to the information. If this explanation were correct, all it would
mean is that that immediate feedback is best paired with delayed review (or further
testing) to strengthen your memory compared with a single exposure. I’ll cover more on
spacing and how it impacts your memory in the next chapter on retention.
Despite the superficially mixed results on the timing of feedback from the scientific
literature, I generally recommend faster feedback. This enables a quicker recognition of
mistakes. However, there’s a possible risk that this recommendation might backslide into
getting feedback before you’ve tried your best to answer the question or solve the
problem at hand. Early studies on feedback timing tended to show a neutral or negativeimpact of immediate feedback on learning. In those studies, however, experimenters
often gave subjects the ability to see the correct answer before subjects had finished
filling out the prompt. 8 That meant subjects could often copy the correct answer rather
than try to retrieve it. Feedback too soon may turn your retrieval practice effectively into
passive review, which we already know is less effective for learning. For hard problems, I
suggest setting yourself a timer to encourage you to think hard on difficult problems
before giving up to look at the correct answer.
How to Improve Your Feedback
By now you see the importance of feedback to your learning efforts. I’ve explained why
feedback, especially when delivered to others, can sometimes backfire. I’ve also showed
how the three types—outcome, informational, and corrective—have different strengths
and the preconditions that need to be in place in order to make them effective. Now I
want to focus on some concrete tactics you can apply to get better feedback.
Tactic 1: Noise Cancellation
Anytime you receive feedback, there are going to be both a signal—the useful information
you want to process—and noise. Noise is caused by random factors, which you shouldn’t
overreact to when trying to improve. Say you’re writing articles that you post online,
trying to improve your writing ability. Most of them won’t attract much attention, and
when they do, it’s often because of factors outside of your control; for example, just the
right person happens to share it, causing it to spill across social networks. The quality of
your writing does drive these factors, but there’s enough randomness that you need to be
careful not to change your entire approach based on one data point. Noise is a real
problem when trying to improve your craft because you need to do far more work to get
the same information about how to write well. By modifying and selecting the streams of
feedback you pay attention to, you can reduce the noise and get more of the signal.
A noise-cancelling technique used in audio processing is filtering. Sound engineers know
that human speech tends to fall within a particular range of frequencies, whereas white
noise is all over the spectrum. They can boost the signal, therefore, by amplifying the
frequencies that occur in human speech and quieting everything else. One way to do this
is to look for proxy signals. These don’t exactly equal success, but they tend to eliminate
some of the noisy data. For blog writing, one way to do so would be to use tracking code
to figure out what percentage of people read your articles all the way to the end. This
doesn’t prove your writing is good, but it’s a lot less noisy than raw traffic data.
Tactic 2: Hitting the Difficulty Sweet Spot
Feedback is information. More information equals more opportunities to learn. A scientific
measure of information is based on how easily you can predict what message it will
contain. If you know that success is guaranteed, the feedback itself provides no
information; you knew it would go well all along. Good feedback does the opposite. It is
very hard to predict and thus gives more information each time you receive it.
The main way this impacts your learning is through the difficulty you’re facing. Many
people intuitively avoid constant failure, because the feedback it offers isn’t alwayshelpful. However, the opposite problem, of being too successful, is more pervasive.
Ultralearners carefully adjust their environment so that they’re not able to predict
whether they’ll succeed or fail. If they fail too often, they simplify the problem so they can
start noticing when they’re doing things right. If they fail too little, they’ll make the task
harder or their standards stricter so that they can distinguish the success of different
approaches. Basically, you should try to avoid situations that always make you feel good
(or bad) about your performance.
Tactic 3: Metafeedback
Typical feedback is performance assessment: your grade on a quiz tells you something
about how well you know the material. However, there’s another type of feedback that’s
perhaps even more useful: metafeedback. This kind of feedback isn’t about your
performance but about evaluating the overall success of the strategy you’re using to
learn.
One important type of metafeedback is your learning rate. This gives you information
about how fast you’re learning, or at least how fast you’re improving in one aspect of your
skill. Chess players might track their Elo ratings growth. LSAT studiers might track their
improvements on mock exams. Language learners might track vocabulary learned or
errors made when writing or speaking. There are two ways you can use this tool. One is to
decide when you should focus on the strategy you’re already using and when you should
experiment with other methods. If your learning rate is slowing to a trickle, that might
mean you’re hitting diminishing returns with your current approach and could benefit
from different kinds of drills, difficulties, or environments. A second way you can apply
metafeedback is by comparing two different study methods to see which works better.
During the MIT Challenge, I’d often split up questions from different subtopics before
testing myself on an exam and try different approaches side by side. Does it work better
to dive straight into trying to answer questions or to spend a little time to try to see that
you understand the main concepts first? The only way you can know is to test your own
learning rates.
Tactic 4: High-Intensity, Rapid Feedback
Sometimes the easiest way to improve feedback is simply to get a lot more of it a lot
more often. This is particularly true when the default mode of learning involves little or
infrequent feedback. De Montebello’s strategy of improving public speaking relied largely
on getting far more frequent exposure to the stage than most speakers do. Lewis’s
language immersion exposes him to information about his pronunciation at a point when
most students still haven’t uttered a word. High-intensity, rapid feedback offers
informational advantages, but more often the advantage is emotional, too. Fear of
receiving feedback can often hold you back more than anything. By throwing yourself into
a high-intensity, rapid feedback situation, you may initially feel uncomfortable, but you’ll
get over that initial aversion much faster than if you wait months or years before getting
feedback.
Being in such a situation also provokes you to engage in learning more aggressively than
you might otherwise. Knowing that your work will be evaluated is an incredible motivator
to do your best. This motivational angle for committing to high-intensity feedback may
end up outweighing the informational advantage it provides.Beyond Feedback
Receiving feedback isn’t always easy. If you process it as a message about your ego
rather than your skills, it’s easy to let a punch become a knockout. Though carefully
controlling the feedback environment so it is maximally encouraging may be a tantalizing
option, real life rarely affords such an opportunity. Instead, it’s better to get in and take
the punches early so that they don’t put you down for the count. Though short-term
feedback can be stressful, once you get into the habit of receiving it, it becomes easier to
process without overreacting emotionally. Ultralearners use this to their advantage,
exposing themselves to massive amounts of feedback so that the noise can be stripped
away from the signal.
Feedback and the information it provides, however, is useful only if you remember the
lessons it teaches. Forgetting is human nature, so it is not enough to learn; you also need
to make the information stick. This brings us to the next principle of ultralearning,
retention, in which we’ll discuss strategies that will ensure the lessons you learn aren’t
forgotten.
Carruse
