Outcomes Assessment With A Focus on
Critical Thinking in Psychology: Sources And Ideas

by Dr. James Bell, Howard Community College, Columbia, MD
[[Note:: Our college is into outcomes assessment by courses as well as for graduates. Here are the sources I have found in psychology with an emphasis on critical thinking. I am interested in sources you know. Please email me at jbell@howardcc.edu Updated as of 1/97. ]]

TABLE OF CONTENTS
Focusing on Psychology
Recent Attempts to Define Critical Thinking
Articles on Teaching and Assessing Critical Thinking
General Sources on Outcomes Assessment

Focusing on Psychology

Allegretti, C., & Frederick, J. (1995, February). A model for thinking critically about ethical issues. Teaching of Psychology, 22, 1, 46-48.
"Teaching students to think critically is a widely endorsed goal of higher education. Although writers disagree about the definition of critical thinking, many agree that its focus is the analysis and evaluation of claims. . . . Critical thinking has many functions, such as (a) evaluating the arguments of others, (b) evaluating and gaining confidence in one's own arguments, (c) resolving conflicts, and (d) understanding and coming to a resolution in complex problems. . . . A primary objective in teaching students to think critically is for students to learn to use these skills beyond the classroom. One method to promote the transfer of these skills is to give students practical situations in which they apply the strategies and practice thinking critically." (p. 46)
Angelo, T. (1995, February). Classroom assessment for critical thinking. Teaching of Psychology, 22, 1, 6-7.
"What is critical thinking? Despite years of debate, no single definition is widely accepted." (p. 6) The evidence suggests that critical thinking does not develop just as a result of maturing. Students do not find learning to think critically easy. ". . . most college faculty agree that critical-thinking skills are notoriously difficult to teach and develop." p. 6) "There is wide agreement that college students learn more and better when they (a) are actively engaged and personally invested, (b) receive comrephensible and timely feedback, and (c) work cooperatively with peers and teachers." (p. 6) Critical thinking can be improved by having students discuss the connections between prior and current learnings, explicitly teach the skills, include guided and regular practice, provide feedback, model critical thinking, teach metacognitive thinking, use brief classroom assessment techniques, and help students learn to self assess. (pp. 6-7)
Bensley, D., & Haynes, C. (1995, February). The acquisition of general purpose strategic knowledge for argumentation. Teaching of Psychology, 22, 1, 41-45.
"Critical thinking was defined as the evaluation of evidence relevant to a claim so that a reasonable conclusion about the truth of the claim can be made. . . (a) identify the claim used in an argument; (b) evaluate the evidence relevant to the claim, comparing and weighing evidence both for and against the claim; and (c) draw a reasonable conclusion about the truth of the claim." (p. 42) Students were asked to write an outline for a persuasive essay, used the 3 steps above on an article on a topic, and wrote persuasive essays. The first two activities were used for research comparisons.
Brewer, C. (1980). On testing and evaluating. Paper presented as a Faculty Seminar on Evaluation and Testing, Furman University, April 2, 1980.
"If a teacher can do anything important for the self-education of students (and self-education is the only kind of any consequence), it is to encourage them to go beyond mere facts." (p. 7) ". . . you must first decide upon objectives or goals for your course. . . . not all of your goals can be measured by a test. . . . concentrate on some of these goals. . . . if you want to emphasize certain higher-level processes. . . . then questions will be more difficult to write and to evaluate. In any case, it is important always to relate your test items to your objectives for the course." (p. 7) "Students can handle small chunks of material better than large chunks. . . . memory is aided by repetition and rehearsal. . . . knowledge of results is essential for learning to proceed. . . . give students as much feedback as possible." (pp. 9-10)
Carlson, E. (1995, February). Evaluating the credibility of sources: A missing link in the teaching of critical thinking. Teaching of Psychology, 22, 1, 39-41.
An area of critical thinking that is often overlooked is judging credibility of the source. Students need to be taught how to determine the credibility of a source. Seven criteria for evaluating the credibility of sources are listed.
Cooper, J. (1995, February). Cooperative learning and critical thinking. Teaching of Psychology, 22, 1, 7-9.
"First, there is no generally agreed-on definition of critical thinking." (p. 7)
Derry, S., Levin, J., & Schauble, L. (1995, February). Stimulating statistical thinking through situated simulations. Teaching of Psychology, 22, 1, 51-57.
"We expect students enrolled in our course will improve their ability to comprehend, develop, and critique evidential arguments in general and statistical arguments in particular. They will improve their ability to read and analyze critically news reports of research. They will develop an ability to make effective persuasive presentations that are enhanced by statistical evidence and; graphical presentations of data. They will learn to distinguish between unexamined beliefs and those based on evidence. They will acquire new criteria for judging the quality of evidence and argument observed in presentations and in interactions of social groups." (p. 56)
Hirose, S. (1992, September). Critical thinking in community colleges. ERIC Digest, ED 348 128.
Critical thinking has become increasingly important in the community colleges. "Critical thinking has been defined as utilizing a number of cognitive processes and attitudes that undergird intelligent action in diverse situations and fields. Critical thinkers are able to discern the thought patterns and beliefs in the works of others, and to reflect upon their own beliefs, decisions, and actions." (From the Abstract)
Hubbard, R., & Ritchie, K. (1995, February). The human subjects review procedure: An exercise in critical thinking for undergraduate experimental psychology students. Teaching of Psychology, 22, 1, 64-65.
"Much critical thinking in psychology focuses on evaluating and analyzing research results used to support arguments made by theorists and researchers (Bell, 1991, Mayer and Goodchild, 1990)." (p. 64)
King, A. (1995, February). Inquiring minds really do want to know: Using questioning to teach critical thinking. Teaching of Psychology, 22, 13-17.
"Simply put, good thinkers are good questioners....Good thinkers are always asking What does this mean?, Why is this happening?, What is the evidence for this?, and How can I be sure? Asking questions such as these and using them to understand the world is what characterizes critical thinking." (p. 13) Without training students will ask questions that are mainly factual. With training students can ask critical thinking questions. Students are given a generic list of questions which are demonstrated by the teacher. Students practice alone, in pairs, and in small groups to generate questions that the students do not currently have answers for. After a lecture they create questions; after homework readings they create questions.
Koteskey, R. (1990, October). Assessing effectiveness of undergraduate instruction in psychology. Teaching of Psychology, 17, 3, 195-196.
Comprehensive test of knowledge was given before PY101, at the end, at the end of PY102, and for graduating seniors. "Existing comprehensive tests were prohibitively expersive and inappropriatel normed." (p. 195) Use items from test bank from the publisher of the textbook used for the course, random generated. Faculty do not see test before it is given each semester. 20 chapters are covered and 5 per chapter are randomly selected. Analysis can then be done by chapters. No discussion of the motivation of students taking the test at the start of the course. Students were given points for taking the tests after the pre-test. Looks like 24 students took the exit PY101 exam. 11 took the exit PY102 exam.
Lawrence, G. (1996, January). Assessment of the psychology program in a small state college. Paper presented at the meeting of the 18th Annual National Institute on the Teaching of Psychology, St. Petersburg Beach, FL. 2 pages. No list of references.
Psychology majors were assessed just before graduation. There was a comprehensive content multiple choice test and a survey about student attitudes toward the psychology program. Students graded each psychology courses on an A-F scale on 7 criteria: "(a) course subject was important for my major, (b) the course increased my understanding of psychology, (c) the course should be a required course, (d) the course was important to my future plans, (e) the material I learned helped me in my non-psychology courses, (f) the grade I give to the subject matter, and (g) the grade I give to the course instructor." Program Goals:
  1. "Comprehension of the facts, theories, and issues of the discipline of psychology to understand behavior.
  2. Comprehension of research methods, statistical skills to demonstrate critical thinking and reasoning, and analytical skills.
  3. Comprehension of the rules of writing in the style described by APA."
McBurney, D. (1995, February). The problem method of teaching research methods. Teaching of Psychology, 22, 1, 36-38.
"The concept of critical thinking is quite broad, and the literature is extremely heterogeneous." (p. 37) Students are given a research problem to help them to critically think about research design. A week before class time will be used to discuss the problem all students are given the problem to think about. About 20% do a written evaluation of the problem to turn in. Students can work alone or in pairs. One class period is used to discuss the problem.
McKeachie, W., & Solomon, D. (195? ). Retention of general psychology. The Journal of Educational Psychology, ?, 110-112.
Students who had taken PY101 were tested at the start of the next course a year later. A 40 item test was used. Items common to the previous PY101 final and the test in the second course showed retention in the 80% range.
Pellegrino, J. (1995, February). Technology in support of critical thinking. Teaching of Psychology, 22, 1, 11-12.
"To start, we must recognize that there is no simple definition of critical thinking or its components. . . . Frequently, critical thinking and problem solving are used synonymously, but when I use the term problem solving, I am referring to a specific sequential model that involves moving successfully from an initial state to a goal state.
Price, W., Wilmes, R., & Turmel, M. (1994). General education assessment in introductory psychology. Journal of Excellence in College Teaching, 5(2), 121-133.
General education competencies were specified and North Country Community College in NY decided to assess individual courses. Using a course-embedded assessment methodology, PY101 assessed psychological knowledge, critical thinking, and writing ability. Students were both pre- and post-tested with an objective test and an essay question. The tests were created by their faculty. 90% of the students were full time. The college GPA was 2.46; the psychology students GPA was 2.55; and the mean psychology grade was 2.15. The multiple choice test was 36 items and the essay was a design a study to test a hypothesis on the effects of TV. Critical thinking was measured by the essay question. "We included the essay question because there has been greater support recently for having students write open-ended responses instead of taking multiple-choice exams." (p. 125) ". . . standardized tests designed to measure critical thinking may not directly and effectively test for many significant aspects of critical thinking (Ennis, 1993). Also, the effectiveness of general instruments in measuring change in thinking skills within a particular course or over a relatively short period of time (one semester) requires further investigation (Chovan, 1991)." (pp. 128-129)
Reboy, L., & Semb, G. (1991, December). PSI and critical thinking: Compatibility or irreconcilable differences? Teaching of Psychology, 18, 4, 212-215.
"PSI has been used successfully in many courses in which students learn complex material and use higher order cognitive skills." (p. 213) "Transfer studies suggest that PSI students learn general skills that transfer to new situations." (p. 213)
Smith, P. (1995, February). Assessing writing and statistical competence in probability and statistics. Teaching of Psychology, 22, 1, 49-50.
Students are asked to explain in writing their thinking on statistical problem. Is the source a good source? Why or why not? What is the evidence? Is opinion separated from data? Are clear definitions used? Is evidence presented? How much? What is its quality? Students are also taught how to self assess their written work.
Underwood, M., & Wald, R. (1995, February). Conference-style learning: A method for fostering critical thinking with heart. Teaching of Psychology, 22, 1, 17-21.
"If there is a single goal on which most educators agree, it is that we seek to teach students skills in critical thinking. . . . psychologists and educators agree much less about exactly how critical thinking should be defined." (pp. 17-18)
Wade, C. (1995, February). Using writing to develop and assess critical thinking. Teaching of Psychology, 22, 1, 24-28.
". . . writing is an essential ingredient in critical-thinking instruction. . . the ability and willingness to assess claims and make objective judgments on the basis of well-supported reasons. .. . we decided to focus on eight general activities that critical thinkers. . . should be able to perform. These activities ask students to
  1. Ask questions and be willing to wonder,
  2. Define problems clearly,
  3. Examine the evidence,
  4. Analyze assumptions and biases,
  5. Avoid emotional reasoning,
  6. Avoid oversimplication,
  7. Consider alternative interpretations, and
  8. Tolerate uncertainty. There are other valid and useful ways to divide up the critical thinking pie." (pp. 24-25)
Six written assignments are required. They are read, receive feedback, but are not graded. Incomplete assignments can be redone so six are completed. After each chapter students are give two or three questions to think and write on. Each students pick the six topics they wish to write on. Each unit has a deadline for that written assignment. Good answers are read to the class.
Back to the Top

Recent Attempts To Define Critical Thinking

Ennis, R. (1993, Summer). Critical thinking assessment. Theory Into Practice, 32 (3), 179-186.

Facione, Peter. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (executive summary). In The Delphi Report, Millbrae, CA: California Academic Press.

Facione, P., & Facione, N. (1992). Test manual: The California critical thinking dispositions inventory.Millbrae, CA: The California Academic Press.

Halonen, J. (1995). Demystifying critical thinking. Teaching of Psychology, 22, 1, 75-81.

Jones, Elizabeth et al. (1994, December). Essential skills in writing, speech and listening, and critical thinking for college graduates: Perspectives of faculty, employers, and policy makers. Project Summary. University Park, PA: National center on Postsecondary Teaching, Learning, and Assessment.

Marzano, R. et al. (1988). Dimensions of thinking: A framework for curriculum and instruction. Alexandria, VA: Association for Supervision and Curriculum Development.

Paul, R., & Nosich, G. (nd). A model for the national assessment of higher order thinking. Santa Rosa, CA: Foundation for Critical Thinking.

Terezini, Patrick et al. (1994, November). The multiple influences of college on students' critical thinking skills. Paper presented at the meeting of the Association for the Study of Higher Education, Tuscon, AZ.

Back to the Top

Articles on Teaching and Assessing Critical Thinking

        Browne, M. Neil. (1986). Preconditions for encouraging critical thinking on the campus. International Journal of Social Education, 3l, 18-27.
"A college catalog that failed to praise critical thinking or to pledge that graduates will think more critically when they leave than when they entered would be an anomaly." (p. 18)

Ask question often. Ask higher level questions. Teach students a variety of ways of looking at things. Learning goes beyond college. Fewer errors in the future can be due to more errors now if we learn from them. There is little evidence that teachers try to teach critical thinking. Use essay format for testing.

        Browne, M., Haas, P., & Keeley, S. (1978, January). Measuring critical thinking skills in college. The Educational Forum, 43, 219-226.
Critical thinking skills:
  1. "Identifying a controversy and conclusions,
  2. Identifying major arguments pertaining to the controversy,
  3. Identifying and analyzing implicit premises according to their level of abstraction,
  4. Recognizing language difficulties (e.g., ambiguity, vagueness),
  5. Evaluating the validity of individual arguments and truth of individual premises,
  6. Formulating a conclusion from premises based on number 5,
  7. Recognizing alternative inferences that could be drawn from premises supporting the conclusion, and
  8. Suggesting a rudimentary technique for verifying these alternative inferences." Used Watson-Glaser Test at first, but did not correlate with student performance in the courses. In the test students are to recognize something rather than apply something. Students are given 10 pages of information on a topic that includes both pro and con information. Students are given two days to think about the material. Students then have up to four hours to write their exam using their critical thinking. Two people read each essay. 90% of scores were within one grade level of each other. Tests are scored using an explicit scoring rubic.
        Brown, M., & Keeley, S. (1988, Spring). Do college students know how to think critically when they graduate? Research Serving Teaching, 1(9), Center for Teaching and Learning of Southeast Missouri State University.
"While the study indicated that most studetns are able to identify some flaws in statistical reasoning, they generally failed to recognize ambiguities, questionable assumptions, and value preferences, important compoents of critical evlauation." (p. 1)
        Ennis, R. (1993, Summer). Critical thinking assessment. Theory Into Practice, 32(3), 179-186.
"Although critical thinking has often been urged as a goal of education throughout most of this century. . . , not a great deal has been done about it. Since the early 1980s, however, attention to critical thinking instruction has increased significantly--with some spillover to critical thinking assessment, an area that has been neglected even more than critical thinking instruction." p. 179 An Annotated List of Critical Thinking Tests - p. 183 "Regrettably, I can find no subject-specific critical thinking tests (that is, critical thinking tests whose primary purpose is to assess critical thinking in a subject matter area..." p. 182 "In making your own test, it is probably better that it be at least somewhat open ended anyway, since making good multiple-choice tests is difficult and time consuming, and requires a series of revisions, tryouts, and more revisions." p. 184
Keeley, S., & Browne, M. (1986). How college seniors operationalize critical thinking behavior. College Student Journal, 20, 389-395.

"We believe that a multiple choice test is not a valid indicator of a person's capacity to actively critically evaluate." (p. 389) Students were given a 500 word essay and asked to critically evalute it in two hours. Looks like they were paid per valid point made.

        Keeley, S., Browne, M., & Kreutzner, J. (1982). A comparison of freshmen and seniors on general and specific essay tests of critical thinking. Research in Higher Education, 17, 139-154.

        Kiah, C. (1993, Nov.). A model for assessing critical thinking skills. Paper presented at the Annual Student Assessment Conference of the Virginia Assessment Group and the State Council of Higher Education for Virginia. (ERIC Document Reproduction Service No. ED 367 400).

Decided to measure critical thinking by focusing on problem solving. Community college graduates were verbally interviewed. Report is short with no specifics. No references.
        Kuhn, L. (1988, Spring). What reasoning skills are important in graduate school? Research Serving Teaching, 1(10), 1-2. Center for Teaching and Learning of Southeast Missouri State University.
"Browne and Keeley (1986) provide evidence that critical thinking skills must be explicitly taught, further, they report that many graduating seniors lack important critical reasoning skills. Given the pattern of significant interdisciplinary differences in faculty perceptions reported by Powers and Enright (1987), it would seem especially important that individual faculty members incorporate into their classes those critical thinking activities most central in their discipline." (p. 2)
        Lehman, D., Lempert, R., & Nisbett, R. (1988, June). The effects of graduate training on reasoning: Formal discipline and thinking about everyday-life events. American Psychologist, 43(6), 431-442.
"Both psychology and medical training produced large effects on statistical and methodological reasoning, and psychology, medical, and law training produced effects on ability to reason about problems in the logic of the conditional. Chemistry training had no effect on any type of reasoning studied." (p. 431) "The results are thus quite consistent with the view that reasoning can be taught and that different graduate disciplines teach different kinds of reasoning to different degrees." (p. 438) "The truth is that we know very little about reasoning and how to teach it." (p. 441)
        Lehman, D., & Nisbett, R. (1990). A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology, 26(6), 952-960.
"Social science training produced large effects on statistical and methodological reasoning" but little effect on conditional logic. (p. 952)
        Matulich, L. (April, 1993). Critical thinking or cony cozenage. (Paper presented at a Symposium of the American Society for Engineering Education, Klamath Falls, OR, April 29, 1993). (ERIC Document Reproduction Service No. ED 373 824)
FIPSE Grant allowed faculty across four disciplines to share ideas on teaching critical thinking. Faculty shared course goals, course content, and ideas on teaching critical thinking. They also compiled resources and research on critical thinking. They focused on the problems of defining terms (Older children are bigger than younger children.), the problems of identifying the writer's purpose, and the problems of unstated assumptions. No references listed.
        Powers, D., & Enright, M. (1987). Analytical reasoning skills in graduate study. Journal of Higher Education, 58(6), 658-682.

Faculty in six different graduate programs listed and then rated the specific thinking skills of most importance. There were differences among the programs.

        Presseisen , B. (1986). Critical thinking and thinking skills: State of the art definitions and practice in public schools. Paper presented as the annual meeting of the American Educational Research Association, San Francisco, CA on April 20, 1986.

She reviews the history of critical thiking: l938 to 1960, 1961 to 1980, and current interests. She reviews the ideas of Glaser, Paul, and Ennis.
        Wright, B. 1991, September). Discipline-based asessment: The case of sociology. AAHE Bulletin, 14-16.
"Mostly, when we talk about assessment, we mean assessment of the major; so far, we've done almost no assessment of the sociology or social-science component in general education." p. 16.
Back to the Top

General Sources on Outcomes Assessment

        Banta, T. (Ed.). (1993). Making a difference. San Francisco: Jossey-Bass. LA227.4.M35

        Banta, T. et al. (1996). Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey-Bass.

        Burger, D. (Winter 1995). Designing a sustainable standards-based assessment system. What's noteworthy on learners, learning, schooling. McREL. ?

        Chovan, W. (1991). Exploring criteria for evaluatring athe effects of a thinking program. Teaching Thinking and Problem Solving, 13(1), 10-15.

        Clift, J., & Irmrie, B. (1981). Assessing students, appraising teaching. NY: Wiley

        Cochran-Smith, M., & Lytle, S. (Eds.) (1993). Inside/outside: Teacher research and knowledge. NY: Teachers College Press.

        Ennis, R. (1993). Critical thinking assessment. Theory into Practice, 32, 179-186.

        Erwin, T. (1991). Assessing student learning and development. San Francisco: Jossey-Bass. LB2822.75.E78

        Gaither, G. (Ed.). (1995, Fall). Assessing performance in an age of accountability: Case studies. San Francisco: Jossey-Bass.

        Green, C. (1986). Guidelines and resources for assessing your sociology program. Washington, DC: American Sociological Association. LB3051.G73 1986

        Halpern, D. (1988). Assessing student outcomes for psychology majors. Teaching of Psychology, 15, 181-186.

        Mayhew, L., Ford, P,., & Hubbard, D. (1990). The quest for quality: The challenge for undergraduate education in the 1990s. San Francisco: Jossey-Bass. LA227.4.M39

        McTighe, J. (1995). Developing performance assessment tasks: Templates for designers. Frederick, MD: Maryland Assessment Consortium.

        McTighe, J., & Ferrara, S. (1996). Assessing learning in the classroom. Washington, DC: National Education Association.

        Miller, R. (Ed.). (1988). Evaluating major components of two-year colleges. Washington, DC: College and University Personnel Association. LB2328.M544

        Paul, R., & Nosich, G. (1991). A model for the national assessment of higher order thinking. Santa Rosa, CA: The Foundation for Critical Thinking.

        Price, W., Wilmes, D., & Turmel, M. (1994). General eduication assessment in introductory psychology. Journal on Excellence in College Teaching, 5(2), 121-133.

        Sheehan, E. (1994). A multimethod assessment of the psychology major. Teaching of Psychology, 21, 74-78.

        Walvoord, B., & McCarthy, L. (1990). Thinking and writing in college. Urbana, IL: National Council of Teachers of English. LB2395.35.T47

        Whitaker, U. (1989). Assessing learning: Standards, principles, & procedures. Philadelphia: Council for Adult and Experiential Learning. LB2331.W5472

        White, E. (1993). Assessing higher-order thinking and communication in college graduates through writing. Journal of General Education, 42, 105-122.

        Wingspread Group on Higher Education. (1993). An American imperative: Higher expectations for higher education. USA: The Johnson Foundation.

        Wright, B. (1991). Discipline-based assessment: The case of sociology. AAHE Bulletin, 44(3), 14-16.

See on the Internet: http://www.unc.edu/depts/ct1/101h.htm

Back to the Top

Back to Howard Community College List of Annotated Documents

Back to MCCCTR Homepage