MENU

Carruse

lunes, 27 de abril de 2020

How to be more perceptive.

Perception refers to the way we understand and interpret the information we sense and take in. Often it also refers to the things we sense but cannot explain. Learn to be more perceptive by reading people's body language, by trusting your gut, by being a perceptive listener, and by practicing meditation.

Reading Body Language

  1. Learn about body language. Ninety percent of human communication is non-verbal. A person's body language can be voluntary or involuntary, and it is both genetic and learned. It is a strong indicator of how a person feels, but it can vary from one culture to the next. Those discussed in this article are indicators of body language in Western cultures.
2
Understand the six facial expressions. Psychologists have classified six involuntary facial expressions that they consider to be nearly universal across cultures. They are happiness, sadness, surprise, fear, disgust and anger. Each has its own signals or clues, and can reveal a person's feelings. But remember they are often fleeting, and some people mask them well. [2]
  • Happiness is exhibited by a raising or lowering of the corners of the mouth.
  • Sadness is shown by a lowering of the corners of the mouth and by the raising of the inner portions of one's eyebrows.
  • Surprise can be seen when eyebrows arch, eyes open wide to expose more white area, and when the jaw drops slightly.
  • Fear is revealed through the raising of eyebrows, when eyes open after being shut or narrowed, and when the mouth opens slightly.
  • Disgust is displayed when the upper lip raises, the bridge of the nose wrinkles, and cheeks raise.
  • Anger exhibits itself when eyebrows lower, lips press together firmly, and eyes bulge.
  • 3
    Know what eye movements mean. Many believe that eyes are the windows to the soul. This belief drove many psychological and cognitive researchers to investigate if our involuntary eye movements held meaning. Research has shown our eyes make predictable movements when a person is processing a thought or question. The notion that you can tell if someone is lying based upon the direction of his or her eye movement, however, is a myth. Here's what we do know.[3]
    • Eye movements in any direction increase when a person is recalling information.
    • Our eye movements stop when something interests us. We also avert our gaze when thinking through something, such as the answer to a question. And our eyes stop moving when we're trying to shut out distractions and focus.
    • Eyes move from left to right (or vice versa) when we are either problem-solving or processing and recalling information. And the harder the problem the more your eyes move.[4][5]
    • Eyes blink at a normal rate of 6-8 times per minute. When a person is under stress, that number rises dramatically.
    • Raised eyebrows not only show fear but also genuine interest in a topic. Furrowed eyebrows signal confusion.

    Perception refers to the way we understand and interpret the information we sense and take in. Often it also refers to the things we sense but cannot explain. Learn to be more perceptive by reading people's body language, by trusting your gut, by being a perceptive listener, and by practicing meditation.

    Method 1 of 4:
    Reading Body Language

    1. 1
      Learn about body language. Ninety percent of human communication is non-verbal. A person's body language can be voluntary or involuntary, and it is both genetic and learned. It is a strong indicator of how a person feels, but it can vary from one culture to the next. Those discussed in this article are indicators of body language in Western cultures.[1]
    2. Image titled Be More Perceptive Step 2
      2
      Understand the six facial expressions. Psychologists have classified six involuntary facial expressions that they consider to be nearly universal across cultures. They are happiness, sadness, surprise, fear, disgust and anger. Each has its own signals or clues, and can reveal a person's feelings. But remember they are often fleeting, and some people mask them well. [2]
      • Happiness is exhibited by a raising or lowering of the corners of the mouth.
      • Sadness is shown by a lowering of the corners of the mouth and by the raising of the inner portions of one's eyebrows.
      • Surprise can be seen when eyebrows arch, eyes open wide to expose more white area, and when the jaw drops slightly.
      • Fear is revealed through the raising of eyebrows, when eyes open after being shut or narrowed, and when the mouth opens slightly.
      • Disgust is displayed when the upper lip raises, the bridge of the nose wrinkles, and cheeks raise.
      • Anger exhibits itself when eyebrows lower, lips press together firmly, and eyes bulge.
    3. Image titled Be More Perceptive Step 3
      3
      Know what eye movements mean. Many believe that eyes are the windows to the soul. This belief drove many psychological and cognitive researchers to investigate if our involuntary eye movements held meaning. Research has shown our eyes make predictable movements when a person is processing a thought or question. The notion that you can tell if someone is lying based upon the direction of his or her eye movement, however, is a myth. Here's what we do know.[3]
      • Eye movements in any direction increase when a person is recalling information.
      • Our eye movements stop when something interests us. We also avert our gaze when thinking through something, such as the answer to a question. And our eyes stop moving when we're trying to shut out distractions and focus.
      • Eyes move from left to right (or vice versa) when we are either problem-solving or processing and recalling information. And the harder the problem the more your eyes move.[4][5]
      • Eyes blink at a normal rate of 6-8 times per minute. When a person is under stress, that number rises dramatically.
      • Raised eyebrows not only show fear but also genuine interest in a topic. Furrowed eyebrows signal confusion.[6]
    4. Image titled Be More Perceptive Step 4
      4
      Look at the way a person's mouth moves. Researchers say the mouth's movements reveal a great deal about how a person feels. Pursing your lips, for example, is a sign of anger.[7] Happiness, as discussed, is displayed when the corners of the mouth curve up. However, researchers have found that different smiles mean different things.
      • Natural, spontaneous smiles appear gradually, last briefly and are shown repetitively.
      • Genuine delight is expressed by a series of short smile “bursts” and creases that form at the corners of the eyes.
      • Fake smiles are about 10 times larger than natural, spontaneous ones. They also appear suddenly, last longer than a natural smile, and disappear abruptly.
      5
      Observe head movements. A person tilts his or her head when actively listening to a topic that interests him or her. Nodding your head indicates you are interested in the topic and want the other person to continue talking. And rubbing one's forehead or earlobes indicates a person feels uneasy, nervous or vulnerable.
    Watch hand and arm movements. People move their hands and arms more than normal when they are speaking or answering a question.[11] They also touch both things and other people more when they respond to intimate questions or when they are physically close to others.
    • Hiding your hands, like in your pockets or behind your back, can indicate deceit.
    • Crossing your arms doesn't always indicate anger, but it can be a defensive posture. It can also mean you feel uneasy with the other person.

    Perception refers to the way we understand and interpret the information we sense and take in. Often it also refers to the things we sense but cannot explain. Learn to be more perceptive by reading people's body language, by trusting your gut, by being a perceptive listener, and by practicing meditation.

    Method 1 of 4:
    Reading Body Language

    1. Image titled Be More Perceptive Step 1
      1
      Learn about body language. Ninety percent of human communication is non-verbal. A person's body language can be voluntary or involuntary, and it is both genetic and learned. It is a strong indicator of how a person feels, but it can vary from one culture to the next. Those discussed in this article are indicators of body language in Western cultures.[1]
    2. Image titled Be More Perceptive Step 2
      2
      Understand the six facial expressions. Psychologists have classified six involuntary facial expressions that they consider to be nearly universal across cultures. They are happiness, sadness, surprise, fear, disgust and anger. Each has its own signals or clues, and can reveal a person's feelings. But remember they are often fleeting, and some people mask them well. [2]
      • Happiness is exhibited by a raising or lowering of the corners of the mouth.
      • Sadness is shown by a lowering of the corners of the mouth and by the raising of the inner portions of one's eyebrows.
      • Surprise can be seen when eyebrows arch, eyes open wide to expose more white area, and when the jaw drops slightly.
      • Fear is revealed through the raising of eyebrows, when eyes open after being shut or narrowed, and when the mouth opens slightly.
      • Disgust is displayed when the upper lip raises, the bridge of the nose wrinkles, and cheeks raise.
      • Anger exhibits itself when eyebrows lower, lips press together firmly, and eyes bulge.
    3. Image titled Be More Perceptive Step 3
      3
      Know what eye movements mean. Many believe that eyes are the windows to the soul. This belief drove many psychological and cognitive researchers to investigate if our involuntary eye movements held meaning. Research has shown our eyes make predictable movements when a person is processing a thought or question. The notion that you can tell if someone is lying based upon the direction of his or her eye movement, however, is a myth. Here's what we do know.[3]
      • Eye movements in any direction increase when a person is recalling information.
      • Our eye movements stop when something interests us. We also avert our gaze when thinking through something, such as the answer to a question. And our eyes stop moving when we're trying to shut out distractions and focus.
      • Eyes move from left to right (or vice versa) when we are either problem-solving or processing and recalling information. And the harder the problem the more your eyes move.[4][5]
      • Eyes blink at a normal rate of 6-8 times per minute. When a person is under stress, that number rises dramatically.
      • Raised eyebrows not only show fear but also genuine interest in a topic. Furrowed eyebrows signal confusion.[6]
    4. Image titled Be More Perceptive Step 4
      4
      Look at the way a person's mouth moves. Researchers say the mouth's movements reveal a great deal about how a person feels. Pursing your lips, for example, is a sign of anger.[7] Happiness, as discussed, is displayed when the corners of the mouth curve up. However, researchers have found that different smiles mean different things.
      • Natural, spontaneous smiles appear gradually, last briefly and are shown repetitively.
      • Genuine delight is expressed by a series of short smile “bursts” and creases that form at the corners of the eyes.
      • Fake smiles are about 10 times larger than natural, spontaneous ones. They also appear suddenly, last longer than a natural smile, and disappear abruptly.[8]
    5. Image titled Be More Perceptive Step 5
      5
      Observe head movements. A person tilts his or her head when actively listening to a topic that interests him or her. Nodding your head indicates you are interested in the topic and want the other person to continue talking. And rubbing one's forehead or earlobes indicates a person feels uneasy, nervous or vulnerable.[9][10]
    6. Image titled Be More Perceptive Step 6
      6
      Watch hand and arm movements. People move their hands and arms more than normal when they are speaking or answering a question.[11] They also touch both things and other people more when they respond to intimate questions or when they are physically close to others.
      • Hiding your hands, like in your pockets or behind your back, can indicate deceit.
      • Crossing your arms doesn't always indicate anger, but it can be a defensive posture. It can also mean you feel uneasy with the other person. [12]
    7. Image titled Be More Perceptive Step 7
      7
      Notice body posture and movements. Learning toward another person is a relaxed and interested indicator. Friendly feelings exist. Leaning in too close, on the other hand, can be experienced as a hostile or dominating gesture. Angling yourself toward another person while you're both standing signals respect. It's also often a sign of deference.[13]
      • Adopting postures similar to others tends to increase group or interpersonal rapport. It tells others you are open to their thoughts.
      • Standing with your legs wide apart is either a traditional stance for those in positions of power or a dominating position.[14]
      • A slumped posture indicates boredom, alienation or feeling ashamed.
      • An upright posture asserts confidence, but it also can exude hostility or a sense of uprightness.

  • 1
    Relax and become aware of what you're hearing. Studies show that talking elevates a person's blood pressure. Listening brings it back down. Listening relaxes us, which allows us to pay attention to our surroundings (and those in it).[15] Perceptive listening goes beyond actively listening, which focuses on listening to another person, reflecting on what he or she says, and sharing your thoughts.
    • It also requires you to think about what the other person is thinking and how he or she acts while talking.[16]
    • It demands focus and being mindful of and present in the conversation by paying close attention to cues from the other person and providing feedback relevant to the discussion.
    Remember that listening requires interpretation. The need to interpret information limits people in their ability to understand what a message means. These interpretations are often dictated by a person's life experiences. Therefore, they are also limited by these experiences.
    • This leaves a lot of room for error in understanding what another person means.[17]
    Remember that listening requires interpretation. The need to interpret information limits people in their ability to understand what a message means. These interpretations are often dictated by a person's life experiences. Therefore, they are also limited by these experiences.
    • This leaves a lot of room for error in understanding what another person means.[17]
    Master perceptive listening. Listening is not an involuntary, automatic response to hearing what someone is saying. It involves a conscious effort on your part and requires practice. Foremost is that you respect the speaker as a human being who deserves to be heard. An effective listener will validate and empower others. This improves the relationship and often leads to future discussions that are straight forward and detailed. Here are some tips to being a more effective listener.
    • Focus your attention, shut out distractions and listen closely to what's said. You cannot assess a statement's logic or the speaker's true intentions if you're unfocused.[18]
    • Respond to what's said so the speaker feels heard and believes you understand what he or she is saying. This feedback also allows you to clear up any misinterpretations in your processing.
    • Don't interrupt when providing feedback. Wait for natural breaks in the conversation and for cues from the speaker, such as “Does that make sense?”
    • Ask questions at appropriate times to draw out what the speaker might not have said otherwise.[19]
    • Pay attention to the speaker's mannerisms and tone, and what they might mean. Consider the context within which the message is delivered and observe what is not said. Meaning is not always openly expressed.[20]
    • Don't fill silences simply to avoid them. Give the person time to think through what he or she is thinking and wants to say.
    • Be open-minded to messages that you do not agree with (e.g., prejudicial comments and opposing viewpoints). Allow the speaker to fully explain him or herself.
    • Seek to understand and interpret a message's meaning by using the cues you've been paying attention to and by drawing on your experiences.
    • Make a conscious and active effort to remember what's said. Retaining information is necessary for assessing its relationship to other aspects of the conversation – in the moment. It's also needed to later process the information, which alone can alter your perception and handling of related situations.[21]

  • 4
    Avoid roadblocks that prevent perceptive listening. Try not to ask “why” questions because this can make people feel defensive. Avoid advising the person about what you think should be done unless you're asked. Don't make quick reassurances, such as, “Don't worry about that.” The latter can indicate you're not fully listening or taking the discussion seriously.[22]
Practice perceptive listening in other areas of your life. Listen to the sounds around you and observe how they make you feel. Notice when you aren't noticing sounds and stop, close your eyes, relax and focus. The more you do this, the more you will become aware of the world around you. This will also help you detect odd, unusual and pleasing sounds and become more perceptive about their meanings, in addition to the situations that might accompany them.[23

Method 3 of 4:
Trusting Your Intuition

  1. Image titled Be More Perceptive Step 13
    1
    Understand intuition and its role in your life. Most people have, at some point, experienced a “gut feeling.” It seems to arise from nowhere but is quite distinct. Gut feelings make people feel in any number of ways. They can also cause a person to sense and know things without any logical explanation. And, at times, they spur a person to do things he or she might not otherwise. [24]
    • The famous psychotherapist, Carl Jung, said that all people use intuition as one of the four ways we function in life. The other three are feeling, thinking and sensing. This makes intuition distinct and not determined by the others.[25]
    • Even though many people dismiss intuition as being either nonsense or luck, scientists are now saying it's a very real ability that's been identified in lab settings and on bran scans.[26]

Discover traits of the intuitive person. Researchers say that everyone is born with intuition, but not everyone is open to believing in it or willing to listen to it. And some people are more intuitive than others. That may be because they were naturally born with a heightened awareness. It could be because they've seen it work in their lives. And it might be because, along the way, they learned to notice and pick up on subtle cues from others and the environment.
  • Often those who are very intuitive are also particularly people-focused. They are more easily able to sense what others are feeling.
  • They're generally more emotionally oriented than analytical.
  • They frequently make decisions quickly and efficiently. They're able to do this because they use past experiences and emotions to guide them.
  • Women are often more intuitive than men.[citation needed] This could be the result of an evolutionary process that's made women particularly conscious of human responses and social stimuli.
  • And there is evidence that some people can go a step beyond what's even normal in this realm. There's documentation of people knowing events have occurred far away, even though they have no prior knowledge of the event or factual basis to explain how they knew.[2
3
Recognize some of the signs. Scientific studies have shown that highly intuitive people experience changes in their heart rate and get sweaty palms when they are exposed to deceit. They believe it is a response to the stress of subconsciously knowing or suspecting they were being tricked. This seems to indicate that our instincts kick in and cause physical sensations first. Our minds catch up quickly, but second
4
Learn to become more intuitive. Though intuition varies, there are things you can do to become more intuitive if you practice and have an open mind. The most basic way is to still (calm) your mind so you can a) listen to your inner voice and b) learn to be more observant of your environment and the people in it.
  • Pay attention to sensations that seem to come out of the blue and have no logical explanation. Our brain's amygdala, which generates the fight-or-flight instinct, is able to activate, process and respond to cues and information before we are consciously aware they exist. It can also process images (and initiate our response to them) that pass before our eyes so fast that we cannot even see them.
  • Researchers believe this stems from our very ancient ancestors' need to quickly gather and assess information if they wanted to survive.
  • Get plenty of REM sleep. During REM, our brains problem solve, connect pieces of information and are most in tune with emotions.
  • Before you go to sleep, write down a problem or worry you have. Think about it for a bit, and then let your brain come up with an intuitive solution during REM.
  • Distract your conscious mind so your intuitive mind has a chance to do its work. Research shows that our intuitive mind processes information even when we're not consciously paying attention to that information.
  • In fact, the decisions a person makes when distracted have been shown to often be the correct ones. If you have a problem or concern, think through your options. Then stop and concentrate on something else. Go with the first solution that comes to you
Check your gut decisions against the facts. Growing scientific evidence supports the wisdom of many intuitive decisions. Issues such as extreme distress can distort that intuitive processing, though, and lead to bad decision-making. Gut reactions and hunches are not always accurate. A smart approach is to listen to your gut and, at the same time, evaluate what it's telling you against the evidence.
  • Also factor in your emotions. Were they extreme when you got the gut feeling?
  • Method 4 of 4:
    Practicing Meditation

    1. Image titled Be More Perceptive Step 18
      1
      Meditate to improve perception. Buddhists have been practicing meditation for more than 2,500 years. Now roughly 10% of Americans meditate as well. Several studies have been conducted showing that meditation can significantly improve perception. Participants in one study were able to detect small visual variations. They had abnormally long attention spans as well.[31] Another demonstrated that the regions in the brain associated with a) sensitivity to body signals and b) sensory processing have increased grey matter when a person regularly meditates.[32]
      • Grey matter is a type of tissue in the Central Nervous System that processes information and triggers a sensory response to it.[33]
      • It is believed that meditating creates more neural connections in the prefrontal cortex of the brain. This region processes sensory information, handles rational decision-making and regulates the amygdala.
      • Teaching yourself to relax, to tune things out, and to be receptive – rather than reactive – to the moment enhances your ability to receive cues around you.[34]
      • 2
        Learn about the types of meditation. Meditation is an umbrella term for the ways you can achieve a relaxed state of being. Different types of meditation have different meditative processes. Here are some of the most widely practiced types of meditation.
        • Guided meditation is led by a teacher, therapist or guide who talks you through visualizing images of people, places, things and experiences that you find relaxing.
        • Mantra meditation involves repeating a calming word, thought or phrase to prevent distracting thoughts from entering your mind.
        • Mindfulness meditation asks that you focus on the present moment and your breathing. Observe your thoughts and emotions without judging them harshly.
        • Qi gong combines meditation, physical movement, breathing exercises and relaxation to restore balance in your thinking.
        • Tai chi is a form of the Chinese martial arts, but movements and postures are slow. You need to also focus on deep breathing.
        • Transcendental meditation involves silently repeating a personal mantra – a word, sound or phrase – to get your body into a state of deep relaxation. Here your mind can strive for inner peace.
        • Yoga is the practice of performing a series of postures and breathing exercises to create a more flexible body and a calm mind. Going from one pose to another requires concentration and balance. Therefore, the emphasis is on thinking only of the present moment.[35]
        Discover ways to practice it daily. You can practice meditation on your own at any point in the day. You don't need a formal class. The length of time you meditate is not as important as doing it regularly, and to the point of relaxation.
        • Breathe deeply and slowly through your nose. Concentrate on feeling and listening as you inhale and exhale. When your mind wanders, focus back on your breathing.
        • Scan your body and become aware of any sensations you feel. Focus your attention on different parts of your body. Combine this with breathing exercises to relax each part of your body.
        • Create your own mantra and repeat it throughout the day.
        • Walk slowly, anywhere, and focus on the movement of your legs and feet. Repeat action words in your mind, such as “lifting” or “moving," as you place one foot in front of another.
        • Pray in spoken or written form using your own words or those written by others.
        • Read poetry or books that are sacred to you, and then reflect on the meaning of what you read. You can also listen to music or spoken words that are inspiring or relaxing. Afterward, write down your reflections or discuss them with another, if you choose.
        • Focus on a sacred object or being and think loving, compassionate and grateful thoughts. You can also close your eyes and visualize the object or being.[36]

viernes, 24 de abril de 2020

  1. 21 Irrefutable Laws of Leadership by John C. Maxwell - We are all Leaders in our lives and taking in these 21 Irrefutable Laws is a great way to start leading your life in the best way possible.
  2. The Power of Now by Eckhart Tolle - Learn to stop thinking and start living. Enjoy the moment, work in the moment towards your future, let go of worries, anxieties and take some of the best advice ever written.
  3. The New Psychocybernetics by Dr. Maxwell Maltz and Dan Kennedy - Changing how you view yourself is a key part in self-improvement. Instead of growing to the point where you cannot help but view yourself in a positive light why don’t you do the easier thing and change how you view yourself first and then let your actions flow accordingly.
  4. Learned Optimism by Martin P. Seligman - Depression and Anxiety disorders are incredibly prominent and widely spread, especially in today's society. Learn to go against this by becoming more optimistic, leading a happier life and becoming more resilient to all the negativity of the world!
  5. The Way of the Superior Man by David Deida - What are the real interactions in relationships? How do you hold an amazing relationship and how do you best interact with your partner? Become a superior man than you were before!
  6. Cultivate an Unshakable Character by Jim Rohn - There are only a few things you need to cultivate an unshakable character, but these things are crucially important. Learn to develop:
    1. Integrity
    2. Honesty
    3. Perseverance
    4. Wisdom
    5. Personal Responsibility
    6. Humor
    7. Flexibility
    8. Patience
    9. Confidence
    10. Good Health
    11. Achievement
    12. Courage
  7. The Last Lecture by Randy Pausch - This book describes the speech Randy Pausch gave in 2007. It was to be his last speech. He touched on topics such as: What matters to you? What matters in life? And what should you value?
    I would highly recommend reading this book, and better yet, to watch this lecture, which is free online.
  8. The 4-hour Workweek by Timothy Ferris - How much could you improve yourself if you only had 4 hours to work a week? To get there you have to do a lot of improvement from the get-go, which is what Timothy Ferris beautifully explains in this book.
    If you are looking to get out your comfort zone with hands on challenges, then I would highly recommend this book if only for the Q&A’s (Questions & Actions).
  9. The War of Art by Steven Pressfield - Fight the resistances, distractions and boring tasks by learning to “dance in the rain”. Life is one big war, how are you going to persist through it?

EDIT: There are plenty more great books but a lot of them are included in the answer wiki already!

miércoles, 22 de abril de 2020

Knowledge Elicitation Tool Classification

Knowledge Elicitation Tool Classification

Janet E. Burge
Artificial Intelligence Research Group
Worcester Polytechnic Institute



Knowledge Elicitation Methods *
KE Methods by Interaction Type *
Interviewing *
Case Study *
Protocols *
Critiquing *
Role Playing *
Simulation *
Prototyping *
Teachback *
Observation *
Goal Related *
List Related *
Construct Elicitation *
Sorting *
Laddering *
20 Questions *
Document Analysis *
KE Methods by Knowledge Type Obtained *
Procedures *
Problem Solving Strategy *
Goals/Subgoals *
Classification *
Dependencies/Relationships *
Evaluation *
References *

Table 1. KE Techniques Grouped by Interaction Type *
Table 2. Interview Methods *
Table 3. Case Study Methods *
Table 4. Protocol Methods *
Table 5. Critiquing Methods *
Table 6. Role Playing Methods *
Table 7. Simulation Methods *
Table 8. Prototyping Methods *
Table 9. Teachback Methods *
Table 10. Observation Methods *
Table 11. Goal Related Methods *
Table 12. List Related Methods *
Table 13. Construct Elicitation Methods *
Table 14. Sorting Methods *
Table 15. Laddering Methods *
Table 16. 20 Questions Method *
Table 17. Document Analysis Methods *
Table 18. Methods that Elicit Procedures *
Table 19. Methods that Elicit Problem Solving Strategy *
Table 20. Methods that Elicit Goals/Subgoals *
Table 21. Methods that Elicit Classification of Domain Entities *
Table 22. Methods that Elicit Relationships *
Table 23. Methods that Elicit Evaluations *
 Knowledge Elicitation Methods
 Many Knowledge Elicitation (KE) methods have been used to obtain the information required to solve problems. These methods can be classified in many ways. One common way is by how directly they obtain information from the domain expert. Direct methods involve directly questioning a domain expert on how they do their job. In order for these methods to be successful, the domain expert has to be reasonably articulate and willing to share information. The information has to be easily expressed by the expert, which is often difficult when tasks frequently performed often become 'automatic.' Indirect methods are used in order to obtain information that can not be easily expressed directly.
 Two other ways of classifying methods are discussed in this document. One classifies the methods by how they interact with the domain expert. Another classifies them by what type of information is obtained.
 Other factors that influence the choice of KE method are the amount of domain knowledge required by the knowledge engineer and the effort required to analyze the data.
KE Methods by Interaction Type There are many ways of grouping KE methods. One is to group them by the type of interaction with the domain expert. Table 1 shows the categories and the type of information produced.

CategoryExamplesTypeResults
InterviewStructured
Unstructured
Semi-Structured
DirectVaries depending on questions asked
Case StudyCritical Incident Method
Forward Scenario Simulation
Critical Decision Method
DirectProcedures followed, rationale
ProtocolsProtocol AnalysisDirectProcedures followed, rationale
CritiquingCritiquingDirectEvaluation of problem solving strategy compared to alternatives
Role PlayingRole PlayingIndirectProcedures, difficulties encountered due to role
SimulationSimulation
Wizard of Oz
DirectProcedures followed
PrototypingRapid Prototyping
Storyboarding
DirectEvaluation of proposed approach
TeachbackTeachbackDirectCorrection of Misconceptions
ObservationObservationProcedure followed
Goal RelatedGoal Decomposition
Dividing the Domain
DirectGoals and subgoals, groupings of goals
List RelatedDecision AnalysisDirectEstimate of worth of all decisions for a task
Construct ElicitationRepertory Grid
Multi-dimensional Scaling
IndirectEntities, attributes, sometimes relationships
SortingCard SortingIndirectClassification of entities (dimension chosen by subject)
LadderingLaddered GridIndirectHierarchical map of the task domain
20 Questions20 QuestionsIndirectInformation used to solve problems, organization of problem space
Document AnalysisDocument AnalysisIndirect (usually)Varies depending on available documents, interaction with experts

Interviewing Interviewing consists of asking the domain expert questions about the domain of interest and how they perform their tasks. Interviews can be unstructured, semi-structured, or structured. The success of an interview session is dependent on the questions asked (it is difficult to know which questions should be asked, particularly if the interviewer is not familiar with the domain) and the ability of the expert to articulate their knowledge. The expert may not remember exactly how they perform a task, especially if it is one that they perform automatically". Some interview methods are used to build a particular type of model of the task. The model is built by the knowledge engineer based on information obtained during the interview and then reviewed with the domain expert. In some cases, the models can be built interactively with the expert, especially if there are software tools available for model creation. Table 2 shows a list of interview methods.

MethodTypeOutputReference
Interviewing (structured, unstructured, semi-structured)DirectProcedures followed, knowledge used (easily verbalized knowledge)[Hudlicka, 1997], [Geiwitz, et al., 1990]
Concept MappingDirectProcedures followed[Hudlicka, 1997], [Thordsen, 1991], [Gowin & Novak, 1984]
Interruption AnalysisDirectProcedures, problem-solving strategy, rationale[Hudlicka, 1997]
ARK (ACT-based representation of knowledge) (combination of methods)DirectGoal-subgoal network
Includes production rules describing goal/subgoal relationship
[Geiwitz, et al., 1990]
Cognitive Structure Analysis (CSA)DirectRepresentational format of experts knowledge; content of the knowledge structure[Geiwitz, et al., 1990]
Problem discussionDirectSolution strategies[Geiwitz, et al., 1990]
Tutorial interviewDirectWhatever expert teaches![Geiwitz, et al., 1990]
Uncertain information elicitationUncertainty about problems[Geiwitz, et al., 1990]
Data flow modelingDirectData flow diagram (data items and data flow between them – no sequence information)[OTT, 1998], [Gane & Sarson, 1977]
Entity-relationship modelingDirectEntity relationship diagram (entities, attributes, and relationships)[OTT, 1998], [Swaffield & Knight, 1990]
Entity life modelingDirectEntity life cycle diagram (entities and state changes)[OTT, 1998], [Swaffield & Knight, 1990]
Object oriented modelingDirectNetwork of objects (types, attributes, relations)[OTT, 1998], [Riekert, 1991]
Semantic netsDirectSemantic Net (inc. relationships between objects)[OTT, 1998], [Atkinson, 1990]
IDEF modelingDirectIDEF Model (functional decomposition)[OTT, 1998], [McNeese & Zaff, 1991]
Petri netsDirectFunctional task net[OTT, 1998], [Coovert et al., 1990], [Hura, 1987], [Weingaertner & Lewis, 1988]
QuestionnaireDirectSequence of task actions, cause and effect relationships[OTT, 1998], [Bainbridge, 1979]
Task action mappingDirectDecision flow diagram (goals, subgoals, actions)[OTT, 1998], [Coury et al., 1991]
User Needs Analysis (decision process diagrams)DirectDecision process diagrams[OTT, 1998], [Coury et al., 1991]

Case Study In Case Study methods different examples of problems/tasks within a domain are discussed. The problems consist of specific cases that can be typical, difficult, or memorable. These cases are used as a context within which directed questions are asked. Table 3 shows a list of methods that use cases to obtain information.

MethodTypeOutputReference
Retrospective case descriptionDirectProcedures followed[Geiwitz, et al., 1990], [Cordingley, 1989]
Critical incident strategyDirectComplete plan, plus factors that influenced the plan.[Geiwitz, et al., 1990], [Cordingley, 1989]
Forward scenario simulationDirectProcedures followed, reasons behind them[Geiwitz, et al., 1990], [Cordingley, 1989]
Critical Decision MethodDirectGoals considered, options generated, situation assessment[Hudlicka, 1997], [Thordsen, 1991], [Klein et al., 1986]
Retrospective case descriptionDirectProcedures used to solve past problems[Geiwitz, et al., 1990], [Cordingley, 1989]
Interesting casesDirectProcedures used to solve unusual problems[Geiwitz, et al., 1990], [Cordingley, 1989]

Protocols Protocol analysis [Ericsson and Simon, 1984] involves asking the expert to perform a task while "thinking aloud." The intent is to capture both the actions performed and the mental process used to determine these actions. As with all the direct methods, the success of the protocol analysis depends on the ability of the expert to describe why they are making their decision. In some cases, the expert may not remember why they do things a certain way. In many cases, the verbalized thoughts will only be a subset of the actual knowledge used to perform the task. One method used to augment this information is Interruption analysis. For this method, the knowledge engineer interrupts the expert at critical points in the task to ask questions about why they performed a particular action.
 For design, protocol analysis would involve asking the expert to perform the design task. This may or not be possible depending on what is being designed or the length of time normally required to perform a design task. Interruption analysis would be useful in determining why subtasks are performed in a particular order. One disadvantage, however, is that the questions could distract the expert enough that they may make mistakes or start "second guessing" their own decisions.
 If time and resources were available, it would be interesting to perform protocol analysis of the same task using multiple experts noting any differences in ordering. This could obtain both alternative orderings and, after questioning the expert, the rationale for their decisions.
 Table 4 lists protocol analysis.

MethodTypeOutputReference
protocol analysis (think aloud, talk aloud, eidetic reduction, retrospective reporting, behavioral descriptions, playback)DirectProcedures, problem-solving strategy[Hudlicka, 1997], [Ericsson & Simon, 1984], [Geiwitz, et al., 1990]

Critiquing In Critiquing, an approach to the problem/task is evaluated by the expert. This is used to determine the validity of results of previous KE sessions. Table 5 lists critiquing methods.

MethodTypeOutputReference
CritiquingDirectEvaluation of a problem solving strategy compared to alternatives[Geiwitz, et al., 1990], [Cordingley, 1989]

Role Playing In Role Playing, the expert adapts a role and acts out a scenario where their knowledge is used [Geiwitz, et al., 1990]. The intent is that by viewing a situation from a different perspective, information will be revealed that was not discussed when the expert was asked directly. Table 6 shows role playing.

MethodTypeOutputReference
role playingIndirectProcedures, difficulties encountered due to role[Geiwitz, et al., 1990], [Cordingley, 1989]

Simulation In Simulation methods, the task is simulated using a computer system or other means. This is used when it is not possible to actually perform the task. Table 7 shows simulation methods.

MethodTypeOutputReference
wizard of ozDirectProcedures followed[Geiwitz, et al., 1990], [Cordingley, 1989]
SimulationsDirectProblem solving strategies, procedures[Geiwitz, et al., 1990], [Cordingley, 1989]
Problem analysisDirectProcedures, rationale (like simulated interruption analysis)[Geiwitz, et al., 1990]

PrototypingIn Prototyping, the expert is asked to evaluate a prototype of the proposed system being developed. This is usually done iteratively as the system is refined. Table 8 shows prototyping methods.

MethodTypeOutputReference
System refinementDirect
New test cases for a prototype system[Geiwitz, et al., 1990]
System examinationDirectExperts opinion on prototype’s rules and control structures[Geiwitz, et al., 1990]
System validationDirectOutside experts evaluation of cases solved by expert and protocol system[Geiwitz, et al., 1990]
Rapid prototypingDirectEvaluation of system/procedure[Geiwitz, et al., 1990], [Diaper, 1989]
StoryboardingDirectPrototype display design[OTT, 1998], [McNeese & Zaff, 1991]

Teachback In Teachback, the knowledge engineer attempts to teach the information back to the expert, who then provides corrections and fills in gaps. Table 9 shows teachback methods.

MethodTypeOutputReference
teachbackDirectCorrection of misconceptions[Geiwitz, et al., 1990], [Cordingley, 1989]

Observation In Observation methods, the knowledge engineer observes the expert performing a task. This prevents the knowledge engineer from inadvertently interfering in the process, but does not provide any insight into why decisions are made. Table 10 shows observation methods.

MethodTypeOutputReference
Discourse analysis (observation)DirectTaxonomy of tasks/subtasks or functions[OTT, 1998], [Belkin & Brooks, 1988]
On-site observationDirectProcedure, problem solving strategies[Geiwitz, et al., 1990], [Cordingley, 1989]
Active participationDirectKnowledge and skills needed for task[Geiwitz, et al., 1990], [Cordingley, 1989]

Goal RelatedIn Goal Related methods, focused discussion techniques are used to elicit information about goals and subgoals. Table 11 shows goal related methods.

MethodTypeOutputReference
Goal DecompositionDirectGoals and subgoals[Geiwitz, et al., 1990]
Dividing the domainDirectHow data is grouped to reach a goal[Geiwitz, et al., 1990], [Cordingley, 1989]
ReclassificationDirectEvidence needed to prove that a decision was correct[Geiwitz, et al., 1990], [Cordingley, 1989]
Distinguishing goalsDirectMinimal sets of discriminating features[Geiwitz, et al., 1990], [Cordingley, 1989]
Goal Directed Analysis (goal-means network)DirectGoal-means network[OTT, 1998], [Woods & Hollnagel, 1987]

List Related In List Related methods, the expert is asked to provide lists of information, usually decisions. Table 12 shows list related methods.

MethodTypeOutputReference
Decision analysisDirectEstimate of worth for all possible decisions for a task[Geiwitz, et al., 1990], [Cordingley, 1989]

Construct Elicitation Construct Elicitation methods are used to obtain information about how the expert discriminates between entities in the problem domain. The most commonly used construct elimination method is Repertory Grid Analysis [Kelly, 1955]. For this method, the domain expert is presented with a list of entities and is asked to describe the similarities and differences between them. These similarities and differences are used to determine the important attributes of the entities. After completing the initial list of attributes, the knowledge engineer works with the domain expert to assign ratings to each entity/attribute pair. Table 13 shows construct elicitation methods.


MethodTypeOutputReference
repertory gridIndirectAttributes (and entities if provided by subject)[Hudlicka, 1997], [Kelly, 1955]
multi-dimensional scalingIndirectAttributes and relationships
proximity scalingIndirectAttributes and relationships[Hudlicka, 1997]

SortingIn sorting methods, domain entities are sorted to determine how the expert classifies their knowledge. Table 14 shows sorting methods.

MethodTypeOutputReference
card sortingIndirectHierarchical cluster diagram (classification)[1], [Geiwitz, et al., 1990], [Cordingley, 1989]

Laddering In Laddering, a hierarchical structure of the domain is formed by asking questions designed to move up, down, and across the hierarchy. Table 15 shows laddering methods.

MethodTypeOutputReference
Laddered gridIndirectA hierarchical map of the task domain[Geiwitz, et al., 1990], [Cordingley, 1989]

20 Questions This is a method used to determine how the expert gathers information by having the expert as the knowledge engineer questions. Table 16 shows the 20 questions method.

MethodTypeOutputReference
20 questionsIndirectAmount and type of information used to solve problems; how problem space is organized, or how expert has represented
Task-relevant knowledge.
[Cordingley, 1989], [Geiwitz, et al., 1990]

Document Analysis Document analysis involves gathering information from existing documentation. May or may not involve interaction with a human expert to confirm or add to this information.
Table 17 shows documentation analysis methods.

MethodTypeOutputReference
Collect artifacts of task performanceIndirectHow expert organizes or processes task information, how it is compiled to present to others[Geiwitz, et al., 1990], [Cordingley, 1989]
Document analysisIndirect (Usually)Conceptual graph[OTT, 1998], [Gordon et al., 1993]
Goal Directed Analysis (goal-means network)DirectGoal-means network[OTT, 1998], [Woods & Hollnagel, 1987]

KE Methods by Knowledge Type Obtained Besides being grouped into direct and indirect categories, KE methods can also be grouped (to some extent) by the type of knowledge obtained. For example, many of the indirect KE methods are best at obtaining classification knowledge while direct methods are more suited for obtaining procedural knowledge. This does not, however, mean that the techniques can not be used for other knowledge types. Since some designers may not be able to directly express how they perform a design task, it might be useful to use an indirect method in conjunction with a direct method to obtain this information.
 Information types used here are:
  • Procedures
  • Problem solving strategy/Rationale
  • Goals, sub-goals
  • Classification
  • Relationships
  • Evaluation
Many methods fit into more than one category and are listed more than once. Also, this classification shows the information most commonly extracted using a method and does not imply that only that type of information can be elicited.
Procedures These are methods that can be used to determine the steps followed to complete a task. Table 18 lists methods used to elicit procedures.

MethodCategoryOutputTypeReference
Interviewing (structured, unstructured, semi-structured)InterviewingProcedures followed, knowledge usedDirect[Hudlicka, 1997], [Geiwitz, et al., 1990]
Concept MappingInterviewProcedures followedDirect[Hudlicka, 1997], [Thordsen, 1991], [Gowin & Novak, 1984]
Interruption AnalysisInterviewingProcedures, problem-solving strategy, rationaleDirect[Hudlicka, 1997]
Problem discussionInterviewSolution strategiesDirect[Geiwitz, et al., 1990]
Tutorial interviewInterviewWhatever expert teaches!Direct[Geiwitz, et al., 1990]
Entity life modelingInterviewEntity life cycle diagram (entities and state changes)Direct[OTT, 1998], [Swaffield & Knight, 1990]
IDEF modelingInterviewIDEF Model (functional decomposition)Direct[OTT, 1998], [McNeese & Zaff, 1991]
Petri netsInterviewFunctional task netDirect[OTT, 1998], [Coovert et al., 1990], [Hura, 1987], [Weingaertner & Lewis, 1988]
QuestionnaireInterviewSequence of task actions, cause and effect relationshipsDirect[OTT, 1998], [Bainbridge, 1979]
Task action mappingInterviewDecision flow diagram (goals, subgoals, actions)Direct[OTT, 1998], [Coury et al., 1991]
Retrospective case descriptionCase StudyProcedures followedDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Critical incident strategyCase StudyComplete plan, plus factors that influenced the plan.Direct[Geiwitz, et al., 1990], [Cordingley, 1989]
Forward scenario simulationCase StudyProcedures followed, reasons behind themDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Retrospective case descriptionCase StudyProcedures used to solve past problemsDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Interesting casesCase StudyProcedures used to solve unusual problemsDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
protocol analysis (think aloud, talk aloud, eidetic reduction, retrospective reporting, behavioral descriptions, playback)ProtocolsProcedures, problem-solving strategyDirect[Hudlicka, 1997], [Ericsson & Simon, 1984], [Geiwitz, et al., 1990]
TeachbackTeachbackCorrection of misconceptionsDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
CritiquingCritiquingEvaluation of a problem solving strategy compared to alternativesDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
role playingRole PlayingProcedures, difficulties encountered due to roleDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
wizard of ozSimulationProcedures followedDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
SimulationsSimulationProblem solving strategies, proceduresDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Problem analysisSimulationProcedures, rationale (like simulated interruption analysis)Direct[Geiwitz, et al., 1990]
On-site observationObservationProcedure, problem solving strategiesDirect[Geiwitz, et al., 1990], [Cordingley, 1989]

Problem Solving Strategy These methods attempt to determine how the expert makes their decisions. Table 19 lists methods that elicit a problem solving strategy.


MethodCategoryOutputTypeReference
Interviewing (structured, unstructured, semi-structured)InterviewingProcedures followed, knowledge usedDirect[Hudlicka, 1997], [Geiwitz, et al., 1990]
Interruption AnalysisInterviewingProcedures, problem-solving strategy, rationaleDirect[Hudlicka, 1997]
Problem discussionInterviewSolution strategiesDirect[Geiwitz, et al., 1990]
Tutorial interviewInterviewWhatever expert teaches!Direct[Geiwitz, et al., 1990]
Uncertain information elicitationInterviewUncertainty about problemsDirect[Geiwitz, et al., 1990]
Critical incident strategyCase StudyComplete plan, plus factors that influenced the plan.Direct[Geiwitz, et al., 1990], [Cordingley, 1989]
Forward scenario simulationCase StudyProcedures followed, reasons behind themDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
protocol analysis (think aloud, talk aloud, eidetic reduction, retrospective reporting, behavioral descriptions, playback)ProtocolsProcedures, problem-solving strategyDirect[Hudlicka, 1997], [Ericsson & Simon, 1984], [Geiwitz, et al., 1990]
critiquingCritiquingEvaluation of a problem solving strategy compared to alternativesDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
wizard of ozSimulationProcedures followedDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
SimulationsSimulationProblem solving strategies, proceduresDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Problem analysisSimulationProcedures, rationale (like simulated interruption analysis)Direct[Geiwitz, et al., 1990]
ReclassificationGoal RelatedEvidence needed to prove that a decision was correctDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
On-site observationObservationProcedure, problem solving strategiesDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Goal Directed Analysis (goal-means network)Interview/Document AnalysisGoal-means networkDirect[OTT, 1998], [Woods & Hollnagel, 1987]
20 questions20 QuestionsAmount and type of information used to solve problems; how problem space is organized, or how expert has represented
Task-relevant knowledge.
Indirect[Cordingley, 1989], [Geiwitz, et al., 1990]
Cloze experimentsIndirectModel of decision-making rules and structuresIndirect[Geiwitz, et al., 1990]

Goals/Subgoals These are methods that are concerned with extracting the goals and subgoals for performing the task. These methods are listed separately from procedures since ordering is not necessarily provided. Table 20 lists methods that elicit this information.

MethodCategoryOutputTypeReference
ARK (ACT-based representation of knowledge) (combination of methods)InterviewGoal-subgoal network
Includes production rules describing goal/subgoal relationship
Direct[Geiwitz, et al., 1990]
Task action mappingInterviewDecision flow diagram (goals, subgoals, actions)Direct[OTT, 1998], [Coury et al., 1991]
Critical Decision MethodCase StudyGoals considered, options generated, situation assessmentDirect[Hudlicka, 1997], [Thordsen, 1991], [Klein et al., 1986]
goal decompositionGoal RelatedGoals and subgoalsDirect[Geiwitz, et al., 1990]
Dividing the domainGoal RelatedHow data is grouped to reach a goalDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
ReclassificationGoal RelatedEvidence needed to prove that a decision was correctDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Distinguishing goalsGoal RelatedMinimal sets of discriminating featuresDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Goal Directed Analysis (goal-means network)Interview/Document AnalysisGoal-means networkDirect[OTT, 1998], [Woods & Hollnagel, 1987]

Classification These methods are used to classify entities within a domain. Figure 21 lists methods concerned with classification.

MethodCategoryOutputTypeReference
Cognitive Structure Analysis (CSA)InterviewRepresentational format of experts knowledge; content of the knowledge structureDirect[Geiwitz, et al., 1990]
Data flow modelingInterviewData flow diagram (data items and data flow between them – no sequence information)Direct[OTT, 1998], [Gane & Sarson, 1977]
Entity-relationship modelingInterviewEntity relationship diagram (entities, attributes, and relationships)Direct[OTT, 1998], [Swaffield & Knight, 1990]
Entity life modelingInterviewEntity life cycle diagram (entities and state changes)Direct[OTT, 1998], [Swaffield & Knight, 1990]
Object oriented modelingInterviewNetwork of objects (types, attributes, relations)Direct[OTT, 1998], [Riekert, 1991]
Semantic netsInterviewSemantic Net (inc. relationships between objects)Direct[OTT, 1998], [Atkinson, 1990]
Distinguishing goalsGoal RelatedMinimal sets of discriminating featuresDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Decision analysisList RelatedEstimate of worth for all possible decisions for a taskDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Discourse analysis (observation)ObservationTaxonomy of tasks/subtasks or functionsDirect[OTT, 1998], [Belkin & Brooks, 1988]
Collect artifacts of task performanceDocument AnalysisHow expert organizes or processes task information, how it is compiled to present to othersIndirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Document analysisDocument AnalysisConceptual graphIndirect[OTT, 1998], [Gordon et al., 1993]
repertory gridConstruct ElicitationAttributes (and entities if provided by subject)Indirect[Hudlicka, 1997], [Kelly, 1955]
multi-dimensional scalingConstruct ElicitationAttributes and relationshipsIndirect
proximity scalingConstruct ElicitationAttributes and relationshipsIndirect[Hudlicka, 1997]
card sortingSortingHierarchical cluster diagram (classification)Indirect[1], [Geiwitz, et al., 1990], [Cordingley, 1989]
laddered gridLadderingA hierarchical map of the task domainIndirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Ranking augmented conceptual rankingOtherConceptual Ranking (ordering by value)Direct[OTT, 1998], [Chignell & Peterson, 1988], [Kagel, 1986], [Whaley, 1979]

Dependencies/Relationships Table 22 lists methods that obtain relationships between domain entities.

MethodCategoryOutputTypeReference
Data flow modelingInterviewData flow diagram (data items and data flow between them – no sequence information)Direct[OTT, 1998], [Gane & Sarson, 1977]
Entity-relationship modelingInterviewEntity relationship diagram (entities, attributes, and relationships)Direct[OTT, 1998], [Swaffield & Knight, 1990]
Object oriented modelingInterviewNetwork of objects (types, attributes, relations)Direct[OTT, 1998], [Riekert, 1991]
Semantic netsInterviewSemantic Net (inc. relationships between objects)Direct[OTT, 1998], [Atkinson, 1990]
QuestionnaireInterviewSequence of task actions, cause and effect relationshipsDirect[OTT, 1998], [Bainbridge, 1979]
Discourse analysis (observation)ObservationTaxonomy of tasks/subtasks or functionsDirect[OTT, 1998], [Belkin & Brooks, 1988]
multi-dimensional scalingConstruct ElicitationAttributes and relationshipsIndirect
Proximity scalingConstruct ElicitationAttributes and relationshipsIndirect[Hudlicka, 1997]
card sortingSortingHierarchical cluster diagram (classification)Indirect[1], [Geiwitz, et al., 1990], [Cordingley, 1989]
Laddered gridLadderingA hierarchical map of the task domainIndirect[Geiwitz, et al., 1990], [Cordingley, 1989]

Evaluation Table 23 lists methods that are used for evaluation of prototypes or other types of KE session results.

MethodCategoryOutputTypeReference
teachbackTeachbackCorrection of misconceptionsDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
critiquingCritiquingEvaluation of a problem solving strategy compared to alternativesDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
System refinementPrototypingNew test cases for a prototype systemDirect
[Geiwitz, et al., 1990]
System examinationPrototypingExperts opinion on prototype’s rules and control structuresDirect[Geiwitz, et al., 1990]
System validationPrototypingOutside experts evaluation of cases solved by expert and protocol systemDirect[Geiwitz, et al., 1990]
Rapid prototypingPrototypingEvaluation of system/procedureDirect[Geiwitz, et al., 1990], [Diaper, 1989]
StoryboardingPrototypingPrototype display designDirect[OTT, 1998], [McNeese & Zaff, 1991]
Decision analysisList RelatedEstimate of worth for all possible decisions for a taskDirect[Geiwitz, et al., 1990], [Cordingley, 1989]
Ranking augmented conceptual rankingOtherConceptual Ranking (ordering by value)Direct[OTT, 1998], [Chignell & Peterson, 1988], [Kagel, 1986], [Whaley, 1979]

References
Atkinson, G. (1990). Practical experience using an automated knowledge acquisition tool. Proceedings of the Second Annual Conference of the International Association of Knowledge Engineers, 87-97.
Bainbridge, L. (1979). Verbal reports as evidence of the process operator's knowledge. International Journal of-Man-Machine Studies11, 411-436.
Belkin, N. J., Brooks, H. M. (1988). Knowledge elicitation using discourse analysis. In B. Gaines and J. Boose (Eds.) Knowledge based systems, Vol. 1, pp 107-124. Academic Press Limited.
Chignell, M. H., Peterson, J. G. (1988). Strategic issues in knowledge engineering. Human Factors, 30(4), 381-394.
Coovert, M. D., Cannon-Bowers, J. A., & Salas, E. (1990). Applying mathematical modeling technology to the study of team training and performance. Paper presented at the 12th Annual Interservice/Industry Training Systems Conference, Orlando, FL, November. 
Cordingley, E. S. (1989). Knowledge elicitation techniques for knowledge-based systems. In D. Diaper (Ed.), Knowledge elicitation: Principles, techniques and applications. Chichester, England: Ellis Horwood Ltd.
Coury, B. G., Motte, S., & Seiford, L. M. (1991). Capturing and representing decision processes in the design of an information system. Proceedings of the Human Factors Society 35th Annual Meeting, 1223-1227. Santa Monica, CA: Human Factors Society. 
Diaper, D. (Ed.). (1989). Knowledge elicitation: Principles, techniques and applications. Chicester, England: Ellis Horwood Ltd.
Ericsson, K.A., Simon, H.A. (1984). Protocol Analysis: Verbal Reports as Data. Cambridge, MA: The MIT Press.
Gane, C., Sarson, T. (1977). Structured Systems Analysis:--Tools and Techniques. Unpublished document! McDonnell Douglas Corporation. 
Geiwitz, J., Kornell, J., McCloskey, B. (1990). An Expert System for the Selection of Knowledge Acquisition Techniques. Technical Report 785-2, Contract No. DAAB07-89-C-A044. California, Anacapa Sciences.
Gordon, S. E., Schmierer, K. A., & Gill, R. T. (1993). Conceptual graph analysis: Knowledge acquisition for instructional system design. Human Factors, 35, p. 459-481. 
Gowin, R., Novak, J.D. (1984). Learning how to learn. NY: Cambridge University Press.
Hudlicka, E. (1997). Summary of Knowledge Elicitation Techniques for Requirements Analysis, Course Material for Human Computer Interaction, Worcester Polytechnic Institute.
Hura, G. S. (1987). Petri net applications. IEEE Potentials, October, 25-28.
Kagel, A. S. (1986). The unshuffle algorithm. Computer Language, 1(11), 61-66.
Kelly, G. (1955). The Psychology of Personal Constructs. New York: Norton.
Klein, G. A., Calderwood, R., Clinton-Cirocco, A. (1986). Rapid decision making on the fireground, Proceedings o fthe 30th Annual Human Factors Society, 1, 576-580. Dayton, OH: Human Factors Society.
McNeese, M. D., Zaff, B. S. (1991). Knowledge as design: A methodology for overcoming knowledge acquisition bottlenecks in intelligent interface design. Proceedings of the Human Factors Society 35th Annual Meeting, 1181-1185. Santa Monica, CA: Human Factors Society.
OTT (1998), http://www.ott.navy.mil/2_2/2_2_6/ , Task Analysis, Chief of Naval Operations' Office of Training Technology.
Riekert, W. (1991). Knowledge acquisition as an object-oriented modeling process. In M. J. Tauber and D. Ackermann (Eds.) Mental models and human computer interactions, 373-381. Amsterdam: Elsevier Sciences Publishers B. V.
Swaffield, G., Knight, B. (1990). Applying system analysis techniques to knowledge engineering. Expert Systems1, 82-93.
Thordsen, M. (1991). A Comparison of Two Tools for Cognitive Task Analysis: Concept Mapping and the Critical Decision Method. Proceedings of the Human Factors Society 35th Annual Meeting.
Weingaertner, S. T., Lewis, A. H. (1988). Evaluation of decision aiding in submarine emergency decision making. In J. Ranta (Ed.) Analysis, Design, and Evaluation of Man-Machine Systems: Selected Papers from the 3rd IFAC/IEA/IFORS Conference, 1 95-201. Oxford, UK: Pergamon.
Whaley, C. P. (1979). Collecting paired-comparison data with a sorting algorithm. Behavior Research Methods and Instrumentation11, 147-150.
Woods, D. D., Hollnagel, E. (1987). Mapping cognitive demands in complex problem-solving worlds. International Journal of Man-Machine Studies, 26, 257-275.