Learning About Therapy Theories

This article was downloaded by: [University of Tasmania] On: 14 November 2014, At: 22:39 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Journal of Teaching in Social Work Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wtsw20 Learning About Therapy Theories Daniel Coleman MSW, PhD, LICSW a a Graduate School of Social Work , Portland State University , P.O. Box 1283, Portland, OR, 97207, USA Published online: 07 Sep 2008. To cite this article: Daniel Coleman MSW, PhD, LICSW (2003) Learning About Therapy Theories, Journal of Teaching in Social Work, 23:3-4, 73-89, DOI: 10.1300/ J067v23n03_07 To link to this article: http://dx.doi.org/10.1300/J067v23n03_07 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, http://www.tandfonline.com/loi/wtsw20 http://www.tandfonline.com/action/showCitFormats?doi=10.1300/J067v23n03_07 http://www.tandfonline.com/action/showCitFormats?doi=10.1300/J067v23n03_07 http://dx.doi.org/10.1300/J067v23n03_07 sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 http://www.tandfonline.com/page/terms-and-conditions Learning About Therapy Theories: An Empirical Test of an Experiential Technique Daniel Coleman ABSTRACT. A scale of therapist theoretical orientation, the Theoreti- cal Evaluation Self Test (TEST), was tested as a tool for increasing content knowledge about psychotherapy theories, and as a means to stimulate reflective learning. MSW students displayed statistically sig- nificant gains in content knowledge after completing the scale and re- lated self-scoring materials, as measured by a pre- and posttest design. Both MSW students and practitioners rated TEST as an effective edu- cational experience. The most frequent qualitative response was that TEST increased insight and understanding of theoretical preferences. Applications of TEST as an educational tool for graduate students and practicing clinicians are discussed. [Article copies available for a fee from The Haworth Document Delivery Service: 1-800-HAWORTH. E-mail address: Website: © 2003 by The Haworth Press, Inc. All rights reserved.] KEYWORDS. Psychotherapy theory, clinical training, education Daniel Coleman, MSW, PhD, LICSW, is Assistant Professor, Graduate School of Social Work, Portland State University, P.O. Box 1283, Portland, OR 97207. The author would like to thank Sharon Imber, BA, for her assistance with the qualita- tive analysis and with website development. The Theoretical Evaluation Self Test (TEST) is available to download in MS-Word for- mat at http://www.people.bu.edu/coleman1/test.html or is available at the same site in an interactive web version. Journal of Teaching in Social Work, Vol. 23(3/4) 2003 http://www.haworthpress.com/store/product.asp?sku=J067  2003 by The Haworth Press, Inc. All rights reserved. 10.1300/J067v23n03_07 73 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 Psychotherapy theory has an obvious and central role in educating and training mental health professionals, a role heightened by recent trends in the mental health field. One trend evident from surveys of cli- nicians over the past forty years is the increased use of multiple theories (Garfield, 1994). Another more recent trend is a growing movement of clinicians and researchers who argue that specific treatments are clearly more efficacious for specific disorders, and that clinicians should be trained to apply these treatments (Beutler, 2000; Perez, 1999). These trends make it imperative for clinicians to be able to recognize the com- ponents of the major psychotherapy theories, to reflect about how theo- ries are used in their practice, and possibly to make the adjustments needed to apply a specific treatment to a specific client. The Theoretical Evaluation Self Test (TEST), a scale of therapist theoretical orientation, offers an educational opportunity to address these areas in an experiential manner. TEST consists of 30 items of statements about psychotherapy, the human psyche, and the therapeu- tic process. Respondents rate their agreement with the statements in a seven-point Likert scale format. TEST has shown good preliminary va- lidity and reliability (see Methods section). For classroom use, TEST is configured to be self-scoring so students get immediate feedback on their level of agreement with the major clinical approaches and can compare their scores with scores from previous samples. It is designed to improve students’ ability to recognize the distinguishing elements and boundaries of the dominant psychotherapy theories. Equally impor- tant, this process of feedback is intended to stimulate students’ reflec- tions about theoretical inclinations and how they approach their work with clients. TEST as an educational experience bridges both experien- tial learning methods and traditional content oriented learning, a long- standing pedagogical tension. LITERATURE REVIEW A Tension in Educational Theory and Practice Educational approaches tend to cleave in two divergent directions. On one hand there is the traditional approach to education that values content learning and seeks outcomes measurable by standardized test- ing. The central educational paradigm in this model is the transmission of knowledge or skills from one who knows (even if it is a computer program) to one who does not know. Lecture and assigned reading with 74 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 periodic testing is the most familiar transmission modality. Pro- grammed instruction, grounded in behavioral theory, raises this ap- proach to the central educational method even to the exclusion of human teachers (Vargas, 1991). Programmed instruction differs from other transmission based models in the elements of reinforcement, and assessment of mastery to move to the next sequence of learning. On the other hand, throughout the history of Western educational philosophy, there has been a steady critique of these overly content and goal oriented approaches to education. In the eighteenth century, Rous- seau (1762/1974), in Emile, advocated a liberal view of children as in- herently good and needing freedom to explore and express themselves. Dewey (1966) also emphasized the educational role of experience. Freire (1993) characterizes the transmission model as the “banking” theory of education where the teacher deposits knowledge in the empty accounts of the students. As an alternative, he advocated for engaging learners as critical participants in constructing the learning experience. Adult learning theorists form another tradition that has critiqued tra- ditional educational models. Knowles (1990) rediscovered the term “andragogy” (the teaching of adults) as an alternate to “pedagogy” (the teaching of children). Knowles’ educational approach emphasizes re- specting the innate knowledge drawn from life experience in the adult learner, as well as engaging the learner in defining goals and means of the educational experience. This approach resonates with clinical edu- cation as the students are often older, and there is an expectation that classroom learning and field practice will inform one another. Gardiner (1998) reviewed the empirical literature on student learning in higher education. The overwhelming conclusion is that students learn more when experiential elements are integrated into the classroom. One study found that learning outcomes were associated with the percent of time students talked in the class, and negatively related to the amount of time the teacher spoke. Another study reported that 70-90% of profes- sors studied use the traditional lecture as their primary instructional strategy. A study asking students to track their attentional and thought processes during lecture classes showed that direct attention to the lec- ture was sustained for only 10-20 minutes (Gardiner, 1998). Clearly tra- ditional pedagogical methods in themselves are inadequate. Kramer and Wrenn (1994) studied the use of andragogical and peda- gogical techniques in two social work practice classes. Surveys of the students indicated that they perceived andragogical elements as useful for their learning. Similar assessment for pedagogical elements were not reported, but qualitative responses were summarized as reflecting Daniel Coleman 75 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 that the students appreciated the andragogical elements, but also wanted retention of pedagogical elements. The success of programmed instruc- tion (see Harrington (1999) for a social work example) indicates the power of transmission based learning. Of course we all learn from lec- tures, readings, and carefully constructed educational modules. These methods by themselves, however, often lead to learner fatigue and an absence of placing the educational content into a personal, meaningful context. It is natural to hypothesize, then, that the most successful edu- cational approaches will balance the transmission and experiential di- mensions of the educational process. The Educational Dynamics of TEST The use of TEST as an educational tool advances knowledge about the educational use of scales in general, as well as specifically in clini- cal education. The usefulness of a scale as an educational tool rests on several ideas. One, a scale accesses attitudes and beliefs of which the subject may not be aware. The instrument elicits and makes conscious cognitive structures. Two, subjects are encouraged to critique the con- structs of the scale itself. This allows reflective thinking about the ideas that the scale is based upon, as well as the subject’s own understanding and use of these ideas. The use of psychometric scales and tests as a classroom learning ex- ercise is an underdeveloped area. In a search of the PsycINFO and ERIC databases, there were only two published articles discussing this area (Beere, King, & King, 1991; Hynan, 1997). Beere et al. (1991) de- scribed the use of instruments designed to elicit beliefs about sex roles and gender orientation in classroom exercises. The items of the instru- ments were examined to make clear the constructs the scales were at- tempting to measure, and the critique of these constructs and their assumptions were used to fuel class discussion. Paradoxically, the edu- cational value of scales flows both from their validity in tapping into be- liefs and ideas that students may not be aware of, and in deconstructing the scale to analyze the constructs that the scale measures. One framework that addresses this quality of bringing beliefs into awareness is a construct from social and cognitive psychology–naive or implicit theories (Anderson, 1998; Najavits, 1997). Implicit theories are described as causal knowledge structures used to explain phenomena and to guide decision-making and behavior. Explicit theories learned early in a career may become implicit over years of practice. Implicit theories also suggest something personal, idiosyncratic, perhaps even 76 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 unconscious, that contribute to a belief system. Kottler (1986) states: “All therapists are theoreticians. We harbor our own unique ideas about how the world works and how therapy ought to be conducted. No matter what school of thought we align ourselves with, we have our own indi- vidual notions about how . . . to work” (p. 137). Depending on a subject’s level of awareness of explicit psychother- apy theories, TEST accesses implicit theories of practice as well as con- scious theoretical preferences. The educational potential of TEST is best brought to realization when a seminar discussion is possible. While the instructions in the self-scoring section attempt to invite critical re- flection, in-depth exploration is facilitated by a group discussion. This study tests the effectiveness of TEST in transmitting content in- formation about the major psychotherapy theories, and helping subjects to delineate the limits and boundaries of the theories. It also gathers qualitative information about subjects’ responses to provide more ex- ploratory information about the educational dynamics of the experi- ence. In addition the differential responsiveness of subjects will be examined by gender, race, and status as an MSW student or a practitio- ner. The results of this study will provide educators with a tested educa- tional tool that speaks to current trends and controversies in the field and clinical education. METHODS This study utilized purposive sampling. Subjects were sought to rep- resent a spectrum of social workers, and secondarily, other mental health professionals. The sampling groups are: a mailing to subjects randomly selected from a national list of Clinical Social Workers (the NASW Clinical Register); practicing Master’s and Doctoral level clini- cians from several agencies and universities around the country; and an MSW student sample from a West Coast university. The agency and university subjects were a convenience sample, drawn through the re- searcher’s contacts. Response rate for the NASW mail sample was 28%. Since the other samples were purposive, response rate is not necessarily meaningful, and for some settings response rate was not available. Agencies where the research was conducted had response rates of 50%, 60%, and two of 100%. All but one of the agencies were private, non-profit outpatient mental health centers. One was a hospital providing primary medical care for children. The agencies were evenly divided between those lo- Daniel Coleman 77 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 cated in Northern California and New England. Three of the classroom administrations had response rates above 90%, with one exception where response was less than 50%. All subjects met criteria to be defined as “community clinicians”: working, or having had significant work experience, in community treatment settings, or being engaged in graduate education and training to work in these areas. “Community treatment settings” was defined as public or private not-for-profit agencies that serve a broad cross-section of the population. The total sample was 130, 102 (78.5%) women, and 28 (21.5%) men. The average age of subjects was 35 (SD = 13.7). Fifty-six (43%) of the subjects were MSW students, and 74 (57%) were practicing clinicians. The average practice experience of practitioners was 13.8 years (SD = 11.2). Ninety-three (72%) were social work associated (students and practitioners), and 37 (28%) were from the other mental health profes- sions. Due to sample size, there were not enough subjects to calculate group differences for specific ethnic groups. The less than ideal com- promise was to break the sample into white (n = 91, 70%) and people of color (n = 39, 30%). The major heterogeneity built into the sample was between the student and practicing clinician subsamples. However, there was a range of variability in terms of age and practice experience within each of these subsamples that mitigated possible sample effects, and makes it reasonable to view the sample as fulfilling the goal of a group of clinicians diverse in age and experience. The Theoretical Evaluation Self Test (TEST), a scale of theoretical orientation, was administered to subjects along with a questionnaire seeking demographic, training, and practice information. All adminis- trations were in paper and pencil format. TEST was developed through a generation of a large item pool guided by previous scales of theoreti- cal orientation and literature on psychotherapy theory and orientation. The item pool was reviewed in an iterative manner by a three-member expert panel of two psychologists and one social worker, and was pre- tested with a range of subjects. This process led to a 36-item measure. Preliminary validation tests included construct, convergent and known-groups. Construct validity was tested through exploratory factor analysis. The validity and reliability data are reported more at length in Coleman (2000). The most recent version of TEST consists of 30 items. Exploratory factor analysis was selected over a confirmatory analy- sis as this study included previously unstudied subscales, and sampled a different population, so a structural model could not be specified a priori (Bollen, 1989). A principal factors extraction with an orthogonal varimax 78 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 rotation was selected as the optimal solution as it accounted for the greatest percentage of variance, and yielded interpretable factors with acceptable item loadings and saturation (Guadagnoli, 1988). The seven factor derived scales are: psychodynamic, biological, family therapy, eco-systems, cogni- tive-behavioral, pragmatic case management, and humanistic. See Table 1 for the 29 items retained in the factor scales, the item loadings and internal reliabilities of the seven scales. The sample size (n = 130) is a concern. Methodological investiga- tions, however, have found that stable factor solutions are possible at much smaller sample sizes than previous rules of thumb had indicated. Guadagnoli and Velicer (1988) found that item loading and factor satu- ration were more crucial for stable solutions than either sample size or variable to sample size ratio. By these author’s guidelines, this factor solution meets criteria for a stable solution at this sample size in having several items loading at .60 or above on most factors, and having rea- sonable factor saturation (the number of variables loading above .32 on each factor). Velicer and Fava (1998) replicated Guadagnoli and Velicer’s (1988) earlier findings. These methodological studies used random sampling, so generalizing the guidelines to a selectively sam- pled study requires an unknown level of inference. The primary dependent variable is the change in scores on the knowl- edge pre- and posttests, calculated by subtracting each subject’s pretest score from the posttest score. The quantitative pre and post sections are tests of the subject’s ability to match statements with theoretical orien- tations, each consisting of twelve theoretical statements. Subjects took the 12-item pretest, completed the 36 items of TEST and scored their own inventory, then completed a posttest that consisted of the same 12 items as the pretest arranged in a different order. For validity, subjects were instructed to not look ahead or back when completing the pre- and post-tests. The 12-item pre- and posttests were constructed by the au- thor and reviewed by the four member expert panel for content validity. The subjects’ rating of the effectiveness of the scale will also serve as a dependent variable. Effectiveness was rated on a seven-point scale ranging from 1 (very ineffective) to 7 (very effective) with four an- chored as “neither effective nor ineffective.” In addition to the quantitative information gathered by survey items and the theoretical orientation instrument, subjects were prompted by two open-ended questions. One question asked subjects about “ . . . use of theoretical orientation(s),” and the other to “ . . . assess . . . TEST as an educational . . . tool.” The responses were analyzed following the tradi- tional qualitative methodology of iterative reading and coding (Buston, Daniel Coleman 79 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 80 JOURNAL OF TEACHING IN SOCIAL WORK TABLE 1. Factor Analysis Principal Factors Analysis, 7 Factors Retained Accounting for 86% of Shared Variance Loadings Factor One: Psychodynamic, Theory Driven. Cronbach’s alpha (�) = .77 proportion of variance (pv) = .24 0.71 32. Change occurs in therapy because of the client’s insight into characteristic ways of relating with others set in early childhood. 0.68 24. It is important to attend to what the client is projecting onto the therapist. 0.67 36. Denial, repression, intellectualization and other defense mechanisms are important to understanding psychology. 0.65 11. I believe dreams discussed in therapy can uncover significant unconscious wishes, conflicts and feelings. 0.56 1. One central therapeutic factor is the symbolic recreation of a nurturing care- taker relationship with the therapist. Loadings Factor Two: Biological Orientation. � = .65 pv = .17 0.61 35. There is strong evidence that most mental health problems have biological causes. 0.59 21. Most mental health problems are best treated with medication. 0.55 27. Most clients can benefit from psychiatric medication. 0.43 9. Psychoeducation about the benefits and side effects of medications is an important part of treatment. Loadings Factor Three: Family Therapy. � = .71 pv = .14 0.67 30. Change occurs in therapy through restoring healthy family structures. 0.62 26. When one person in a family is experiencing problems, it is usually the ex- pression of family commun. and relationship problems. 0.61 16. Primary emphasis should be placed on the client’s interactions with his or her family. 0.56 5. It is important for therapists to see clients together with their families. Loadings Factor Four: Eco-Systems/Cultural. � = .66* pv = .10 *dropping item 20 0.66 28. It is important to assess not only the person seeking services, but his or her environment as well. 0.65 31. It is essential for therapists to be aware of the values and worldview of their own culture and how they might affect clients. 0.55 23. Client’s problems are often contributed to by social problems and gaps in the social service system. 0.42 20. Psychological problems vary with the culture of the client. D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 1997). A qualitative analysis software package was used to organize coding and to export matrices to a statistical package. Line by line cod- ing by two coders served as a procedural check insuring that the re- searchers’ thinking stayed close to the actual words spoken by the subjects. The two coders engaged in an iterative process of reading, de- veloping coding, and discussing discrepancies. The categories of ideas, and connections between ideas, led to the development of more abstract ideas, or theory. This method of coding and grouping in hierarchical categories is the core qualitative analytic technique in common with many specific methods (Denzin, 1994). Daniel Coleman 81 TABLE 1 (continued) Loadings Factor Five: Cognitive-Behavioral. � = .63 pv = .08 0.55 34. It is helpful to ask questions to lead the client to realize their mistakes or misperceptions. 0.53 25. The therapist should teach clients techniques to address problem areas. 0.45 2. The therapist should educate the client about the relationship of patterns of cognition and many mental health problems. 0.39 19. Client’s problems are often caused by negative patterns of thinking. 0.39 7. Human behavior is shaped by patterns of reinforcements and punishments in the environment. Loadings Factor Six: Pragmatic Case Management. � = .72 pv = .07 0.67 22. The therapist should be active, directive and goal-oriented. 0.56 18. The role of the therapist is to advise and guide the client. 0.55 6. The therapeutic alliance is important primarily to provide a foundation for collaborative case management. 0.46 14. Advocacy with other providers on behalf of clients is a central role of the therapist. 0.41 12. Most psychotherapy theories are distractions from the central task of solv- ing the client’s problems. Loadings Factor Seven: Humanistic Orientation. � = .44 pv = .06 0.6 15. It is important for the therapist to respond to clients with spontaneous, gen- uine affect. 0.48 8. Change occurs in therapy because of the therapist’s empathic, non-judg- mental, positive attitude towards the client. D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 This study employed relatively simple statistical methods: t-tests, correlation, and chi-square. Where appropriate, effect size statistics were also calculated. For the comparisons of mean differences, Glass’ � (Glass, 1981) was selected. Glass’ � is calculated and interpreted in an identical manner with Cohen’s d, with the exception that the standard de- viation of the baseline or pretest group is used instead of the pooled standard deviation. For comparison of categorical variables, the odds- ratio was used as a measure of effect size (Rosenthal, 1996). RESULTS Quantitative Responses The validity of the 12-item pre- and posttests was supported by statis- tical analysis of the results. Those who would be expected to know more about psychotherapy theories, practicing clinicians, on average scored 2.3 points higher than MSW students on the pretest (students– M = 6.9, SD = 2.1; practitioners–M = 9.1, SD = 1.97), a statistically signif- icant difference (t (108) = 5.59, p < .0001). This difference corresponds to a large effect (Glass’ � = 1.05), which means the average practitioner scored higher than 86% of the MSW students. Change from the pre- to the posttest was assessed using a dependent t-test. One case was removed from the analysis as an outlier because the change from pretest to posttest was greater than 2.5 standard deviations from the mean, such a large difference suggesting an error had been made. For the whole sample, the mean change from pre- to posttest was not statistically significant. As a subgroup, practicing clinicians overall also did not show a statistically significant mean change from pretest to posttest. The MSW student subsample, however, showed a significant mean gain on the posttest of .69 (SD = 1.12) (t (35) = 3.67, p < .001). This difference corresponds to a small-medium effect size (Cohen, 1988) (Glass’ � = .33), which means the average subject who took TEST would know more about psychotherapy theories than 61% of hypothetical controls. (The mean change was tested against a null hypothesis that change = 0, in effect assuming a control group with a mean change of 0.) TEST is beneficial for MSW students as a content learning tool, but not for practicing clinicians. There were no statistically significant dif- ferences in change from pretest to posttest by sex, or by race. In addition to the relatively concrete assessment of change from a pre- to posttest, it was hypothesized that the use of TEST would stimu- 82 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 late reflective thinking about practice and practice theories. As this is more abstract, it is also more difficult to measure. To quantitatively as- sess reflective learning, subjects were asked to rate the effectiveness of TEST as a reflective learning tool. Effectiveness was rated on a seven- point scale ranging from 1 (very ineffective) to 7 (very effective) with four anchored as “neither effective nor ineffective.” The effectiveness ratings were strongly positive with little variation across different subgroups of subjects. The mean effectiveness ratings for all subjects who completed this section (n = 100) was 5.4 (SD = 1.1). There were no statistically significant differences in mean effectiveness ratings by gender, MSW student or practitioner, or race (white/person of color). Consistent with the hypothesis, TEST was rated as consis- tently effective across all subjects. Of the subjects who completed the effectiveness rating (n = 100), 13 rated it as neutral or ineffective. In an interesting result, those who rated TEST as less effective tended to show greater content learning through getting more items correct on the posttest (r (100) = �.20, p < .05). Qualitative Responses The educational effects were also evaluated through qualitative analysis of subjects’ written responses to two open-ended questions. As discussed in the methods section, thematic coding was developed by two coders. Seventy-one percent (n = 92) of subjects made some evaluative qualitative response. There were no statistically significant differences between re- sponders and non-responders in gender, age, or race. Counts of major themes were generated and exported to the quantitative dataset to test if the frequency of themes differed by any of the major quantitative variables. The non-responders were excluded from these analyses. Three broad overlapping categories of evaluative comments were even- tually settled upon: positive, negative, and mixed. Of those who made qualitative responses (n = 92), 69% (n = 62) made positive comments, 36% (n = 33) made negative comments, 14% (n = 13) made discrete positive and negative comments so were coded to both, and an additional 14% (n = 13) made mixed statements. Note that due to the categories not being mutually exclusive, the total percentage is more than 100. Of the positive statements (n = 62), 42% (n = 26) stated that using TEST resulted in better understanding, or insight, about their personal theoretical orientation. Other positive statements included that subjects felt TEST was helpful for students or new practitioners, and that it con- firmed their own sense of their theoretical orientation. The negative Daniel Coleman 83 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 comments (n = 33) included subjects who questioned the wording of items, the validity of Likert scaling, and felt the process of using TEST was cumbersome or difficult. White subjects were six times more likely than subjects of color to state that TEST led to new insights about theoretical orientation (odds-ratio = 6, χ2 (1, N = 92) = 6.36, p < .05). There were no statisti- cally significant differences in the frequency of themes by gender, age, or status as a practitioner or a student. Representative Quotations In this section verbatim extracts of subject’s comments are provided. Quotations were selected that were representative of the most frequent themes in the subject’s responses. Of those subjects who identified that TEST was helpful in increasing understanding or insight, some emphasized content learning. One subject commented: “ . . . by looking at various items, I could see in practical ways how these orientations are carried out.” Another stated: “The TEST influenced me in the sense that what I had understood to be certain cornerstones of certain orientations were actually quite different. Also, I had considered myself more of ‘one thing’ but was actually, at times, more of ‘another’.” A third subject saw the benefit of TEST in defining the theories: “It was useful in providing one definition in which to have a discussion, of- ten times people throw around orientation labels and they seem mean- ingless, because they often overlap or mean something different to another clinician.” Another set of subjects discussed how TEST stimulated more per- sonal insights. One stated: “Fascinating exercise. What I liked the most was simply how it got me thinking about both my current practice and my pre-conceived notions about my practice.” A second subject identified the value based reflections initiated by using TEST: “Very thought-provoking–made me think about what I value in these theories and think about what elements I believe are im- portant for change with mental health.” In the following quotation, a subject indicates that TEST helped un- cover implicit theoretical models: “It is very useful to look at what tech- niques you are using every day in your practice–it helped me to see that there is a method to what often feels like madness.” 84 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 In the next quotation, a subject discussed how TEST induced a desire to learn more about specific areas of theoretical orientation: “Pro- voke(d) desire to learn more, (to) assess professional development. I found it quite thought-provoking and useful in terms of practitioner self-reflection. It influenced me in that I realize how loose my interpre- tations of theory are! I need to go back and learn more!!!” DISCUSSION This study measured the effectiveness of a discrete educational expe- rience in a short time span: TEST takes approximately 15 minutes to complete and score. Given the discrete nature of the educational stimu- lus, the small-medium effect size in content learning for MSW students is notable. Overall subjects rated TEST highly as a learning experience. Approximately 10% rated it as neutral or ineffective. There were roughly twice as many positive qualitative comments about TEST as negative comments. The most frequent type of positive comment iden- tified TEST as increasing insight and understanding about the content and boundaries of theories and the subjects’ use of them in practice. TEST, then, is an empirically supported educational tool. Instructors in clinical social work, counseling and clinical psychology, psychiatric nursing, and psychiatry should consider using TEST in any course that is providing a survey of psychotherapy theory. While practitioners are less likely to show gains in content learning, they did give TEST high ratings for effectiveness. The differential response of students and prac- titioners to the content learning component suggested that TEST func- tions differently at different levels of knowledge and experience. Students are still developing intellectual and experiential knowing about theories in practice, so they benefit in part from reviewing the content of theories, and having the scope and boundaries of each theory pointed out. It is also interesting that subjective ratings of effectiveness were negatively correlated with magnitude of change from pretest to posttest: Those who learned more tended to rate TEST lower. Reflection is a process of being able to be aware of formal relation- ships but being able to fluidly work with these formal boundaries (Schon, 1995; Schon, 1983). Practitioners are likely to have a more de- veloped understanding and an intuitive sense of the formal properties of different types of therapy both in theory and practice. TEST stimulates reflection about the way each individual uses these ideas by providing the feedback of scale scores. For clinicians dynamically engaged in Daniel Coleman 85 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 their practice, and in their evolving understanding and development as a therapist, TEST is one input in this process of growth. Another feature of differential response to TEST was the low likeli- hood of subjects of color identifying insight as a positive effect. Insight in this context means identifying new knowledge or improved self- understanding of theoretical orientation as a result of TEST. This might indicate that this type of educational experience is not culturally syn- tonic for subjects of color, but there were no other statistically signifi- cant differences between whites and people of color on other response variables. Other research with TEST found subjects of color were more likely to agree with a pragmatic approach to practice, and less likely to agree with a psychodynamic, insight oriented style of practice (Coleman, 2000). It is certainly arguable that an insight orientation is a feature of middle and upper class Anglo-European culture. Several strands of research have had related findings. Some clinical process research has found subjects of color prefer active-directive ap- proaches as clients, at least in the initial stage of treatment (Sue, 1987). An exploratory study found that alexithymia (low awareness of emo- tion) was associated with early stages of ethnic identity development in African-American women (Dinsmore & Mallinckrodt, 1996). There are several likely contributors to this range of findings: cultural differences in how inner life and the “self” is conceptualized and experienced; the confound of socioeconomic status (SES) and differential educational experiences at various level of SES; and the effect on subjects of color of adapting to a dominant culture while attempting to develop and pre- serve a positive ethnic identity. It is likely that insight itself is experi- enced and understood in unique ways in different cultures. More collectivist cultural groups may value an insight that is more interper- sonal and here-and-now, as opposed to an insight that focuses on intrapsychic structures formed in childhood (Anonymous reviewer, personal communication, 12/2001). Clearly, educators have to be aware of how the individual differences of students’ backgrounds will have a complex relationship with their theoretical preferences. The construct of ethnic identity development formulates early stages as a denial of ethnocultural differences, and (as noted above) this denial was not surprisingly associated with low awareness of emotion (Dinsmore & Mallinckrodt, 1996). It seems likely that this association would also be found in white students. This suggests that supporting the development of ethnic identity develop- ment in all students is important not only for sociocultural reasons, but also directly relates to clinical skills. Graduate clinical training can pro- 86 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 vide a milieu where the encounter of sociocultural differences expands students’ awareness of human experience (empathy), and their own sociocultural identity (insight). Many schools of social work include content on diversity and ethnic identity, but this may be omitted by other clinical disciplines. These processes are also likely to be different in politically conservative environments, such as some of the reli- giously affiliated schools of social work. Another implication of this educational inquiry with TEST is that it is the first empirical demonstration of the use of a psychometric for con- tent and reflective learning. Beere et al. (1991) report the impression that using scales in the classroom was effective, but did not provide any quantitative data to support this impression. Scales and measures are useful in helping students grasp the defining concepts of an area of study. The results of this study suggest that psychometrics are effective content teaching experiences when the subjects know relatively little about the area of the scale. Possible applications include using scales such as the Beck’s Depression Inventory (BDI) (Beck, Ward, Mendelson, Mack, & Erbaugh, 1961), or the Automatic Thoughts Questionnaire (ATQ) (Hollon & Kendall, 1980), as part of teaching students about the cognitive perspective on depression. Limitations of the study include sample size and a non-random sam- ple. Replication of this study with a larger, randomly selected sample with a strong response rate would increase confidence in generalizing the findings. A large, diverse sample would also provide for better evi- dence of culturally based differences. For validation purposes, a ran- dom sample is not as critical as it is sufficient to show that the scale can discriminate different subjects. Despite emerging guidelines about the stability of factor solutions at smaller sample sizes, the validity of the scale should be considered preliminary because of sample size and a non-random sample. A strength of the sample is that it included a broad range of clinicians from real-world practice and training settings. The inclusion of a control group that completed the pre- and posttests with- out using TEST would have controlled for any learning that might take place simply from taking the pre- and posttests. TEST provides a means of helping students be able to distinguish be- tween the major theoretical orientations, and to reflect about their per- sonal inclinations and preconceptions about these theories. Later in the course of study students can be exposed to more sophisticated ac- counts of these theories, and even be trained in the empirically tested manualized therapies such as Time Limited Dynamic Therapy (Strupp & Binder, 1984), or Beck’s Cognitive Therapy (Beck, Rush, Shaw, & Em- Daniel Coleman 87 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 ery, 1979). The use of TEST as a training exercise in clinical settings could help advance thinking and discussion among clinicians that could lead to increased use of empirically supported practices. Within an agency, theoretical preferences could be identified by TEST, and work- ing groups of clinicians formed to train in an empirically supported therapy that matches with the group’s theoretical preference. REFERENCES Anderson, C. A., & Lindsey, J. J. (1998). The development, perseverance and change of naive theories. Social Cognition, 16(1), 8-30. Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depres- sion. New York: Guilford Press. Beck, A. T., Ward, C. H., Mendelson, M., Mack, J., & Erbaugh, J. (1961). An inven- tory for measuring depression. Archives of General Psychiatry, 4, 561-571. Beere, C. A., King, L. A., & King, D. W. (1991). Gender related instruments as instruc- tional devices in the communication classroom. Communication Education, 40(1), 73-93. Beutler, L. E. (2000). David and Goliath: When empirical and clinical standards of practice meet. American Psychologist, 55(9), 997-1007. Bollen, K. (1989). Structural equations with latent variables. New York: Wiley. Buston, K. (1997). NUD*IST in action: Its use and its usefulness in a study of chronic illness in young people. Sociological Research Online, 2(3), . Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum. Coleman, D. (2000). The Theoretical Evaluation Self Test (TEST): A Scale of Thera- pist Theoretical Orientation for Clinical Research and Education. Unpublished Doctoral Dissertation. University of California, Berkeley. Berkeley, CA. Denzin, N. K., & Lincoln, Y. S. (1994). Entering the field of qualitative research. In Y. S. Lincoln & N. K. Denzin (Eds.), Handbook of qualitative research. Thousand Oaks, CA: Sage Publications. Dewey, J. (1966). Selected educational writings. London: Heineman. Dinsmore, B. D., & Mallinckrodt, B. (1996). Emotional self-awareness, eating disor- ders, and racial identity attitudes in African American women. Journal of Multicul- tural Counseling and Development, 24(4), 267-277. Freire, P. (1993). Pedagogy of the oppressed (revised ed.). New York, NY: Continuum Publishing. Gardiner, L. F. (1998). Why we must change: The research evidence. Thought & Ac- tion, 14(1), 71-88. Garfield, S. (1994). Eclecticism and integration in psychotherapy: Developments and issues. Clinical Psychology: Science and Practice, 1(2), 123-137. Glass, G. (1981). Meta-analysis in social research. Beverly Hills: Sage. Guadagnoli, E., & Velicer, W. F. (1988). Relation to sample size to the stability of component patterns. Psychological Bulletin, 103(2), 265-275. 88 JOURNAL OF TEACHING IN SOCIAL WORK D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14 Harrington, D. (1999). Teaching statistics: A comparison of traditional classroom and programmed instruction/distance learning approaches. Journal of Social Work Ed- ucation, 35(3), 343-352. Hollon, S. D., & Kendall, P. C. (1980). Cognitive self-statements in depression: Devel- opment of an automatic thoughts questionnaire. Cognitive Therapy and Research, 4(4), 383-395. Hynan, L. S., & Foster, B. M. (1997). A project for developing tests in a psychological testing and measurement course. Teaching of Psychology, 24(1), 52-54. Knowles, M. (1990). The adult learner: A neglected species. (4th ed.). Houston, TX: Gulf Publishing. Kottler, J. (1986). On being a therapist. San Francisco: Jossey Bass. Kramer, B. J., & Wrenn, R. (1994). The blending of andragogical and pedagogical teaching methods in advanced social work practice courses. Journal of Teaching in Social Work, 10(1/2), 43-64. Najavits, L. M. (1997). Psychotherapists’ implicit theories of therapy. Journal of Psy- chotherapy Integration, 7(1), 1-16. Perez, J. E. (1999). Clients deserve empirically supported treatments, not romanticism. American Psychologist, 54(3), 205-206. Rosenthal, J. A. (1996). Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21(4), 37-59. Rousseau, J. J. (1762/1974). Emile (B. Foxley, Trans.). New York: Dutton. Schon, D. (1995). Reflective inquiry in social work practice. In P. M. Hess, & E. Mullen, (Eds.), Practitioner-researcher partnerships: Building knowledge from, in, and for practice. Washington, DC: NASW Press. Schon, D. A. (1983). The reflective practitioner. New York: Basic Books. Strupp, H. H., & Binder, J. L. (1984). Psychotherapy in a new key. New York: Basic Books. Sue, S., & N. Zane. (1987). The role of culture and cultural techniques in psychother- apy: A critique and reformulation. American Psychologist, 42 (1), 37-45. Vargas, E. A., & Vargas, J. S. (1991). Programmed instruction: What it is and how to do it. Journal of Behavioral Education, 1(2), 235-251. Velicer, W. F., & Fava, J. L. (1998). Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231-251. Daniel Coleman 89 D ow nl oa de d by [ U ni ve rs ity o f T as m an ia ] at 2 2: 39 1 4 N ov em be r 20 14