LATEX: 2004/2005 PROJECT
Teacher Evaluation of a Textbook using Think Aloud Protocols
The LATEX research group focuses on expertise in language teaching. In the academic year 2004/5, the members of the group decided to look at one important aspect of language teaching: materials evaluation. Specifically, we were interested in how teachers evaluate textbooks. In the last two terms of the academic year we undertook a mini-study in this area, using think-aloud protocols to gain an insight into how teachers approach this task.
This section very briefly describes what was done in this project. This is followed by a brief overview of some of the methodological considerations , and then a presentation some of the preliminary findings of the research. Finally, further references are given in three bibliographies: Textbook Evaluation Bibliography, Expertise Bibliography, and Think-Aloud Protocol Bibliography.
1. Outline of the project
1.1 Procedure
The first issue we discussed was the suitable profile of the teacher-participants in the study. We decided that it would be desirable to find teachers with similar backgrounds (e.g. first language, teaching in similar contexts, etc.). This was not easy, but fortunately we were helped by Isabel Donnelly, from the University of Central Lancashire , to find three teachers who were involved in teaching similar groups of students at that university. We were looking for teachers with similar backgrounds but at different stages in their careers: one 'experienced' (which we defined as more than 10 years teaching experience) one 'novice' (less than 2 years experience), and one 'intermediate', between the two (around 6 years of experience). We were also looking for a textbook that was unknown to them, but at the same time suitable for use by them in their classes. We also have to thank Isabel, who helped us to identify such a textbook: 'Just Right' by Jeremy Harmer (2004, London : Marshall Cavendish, ELT) The idea was to ask them to evaluate the textbook and to think aloud as they did so. Different issues were raised in the process of preparing the think-aloud sessions. We decided that we should videotape the sessions as well as audio-tape them, so as to capture non-verbal reactions and facilitate the matching of comments to particular pages. The necessity of providing the participants with some activities in order to familiarise them with think-aloud techniques, and give them some practice before the evaluation, was discussed at some length. We eventually decided to offer three different types of activity and let the participants choose whichever one they found most appealing. The training activities on offer were: Tower of Hanoi puzzle, a colourful jigsaw puzzle and an anagram. In fact, all three participants chose the jigsaw puzzle to use as a training activity. The type of instructions provided to the participants before the session, and how to act with different possible problems during the session, were also discussed. After the data collection sessions, a questionnaire was also distributed to the participants so we could have a clear picture of their backgrounds which would inform our analysis. These are reproduced below. The recordings took place on two consecutive days, and the participants provided us with lengthy and rich data, each taking around an hour to evaluate the book. The three recordings were transcribed by an external professional, and then cross-checked against the videos by the researchers before the analysis of the data was begun. Each researcher considered the data individually and then through group discussions, the aspects of the data that we considered were worth closer analysis were identified. In our first analysis we have considered aspects such as the likes and dislikes of the evaluators, the flexibility they showed towards the book, how they sequenced their evaluation, the terminology they use and the theoretical considerations they seem to apply when evaluating the textbook. We expect that further and more detailed analysis may be available as a published paper in the future.
1.2 Instructions
These instructions were read to the participants, and were also provided on an A4 sheet of paper so that they could refer to them as they wished throughout the think aloud process:
Please evaluate the suitability of the textbook provided for use with the group in the following scenario:
A group of 15-20 intermediate-level learners, predominately of Chinese nationality, who are learning basic English skills without any ESP (English for Special Purposes) or EAP (English for Academic Purposes) focus within a university setting.
During this study, our aim is to get a clear picture of your thought processes about textbook evaluation. And since there are no correct or incorrect answers, during the evaluation process, please remember to:
|
1.3 Questionnaire
LATEX Materials Evaluation Research
Thank you very much for helping us with this research. Please could we ask you to spend a very short time answering the questions below, to help us analyse the data you have given us?
YOUR NAME:
YOUR TEACHING EXPERIENCE:
[A] How long have you been working as an ESOL teacher (e.g., years)?
[B] Has this experience been full-time or part-time?
[C] In which countries have you worked as a teacher?
[D] In which of the following contexts do you have teaching experience?
- Primary education:
- Secondary education:
- Higher education:
- Further education:
[E] Do you have any specific ELT qualification(s)? YES - NO
If yes, which one(s)?
[F] Beside ELT qualifications, do you have any degree qualification(s) related to linguistics (e.g., BA)? YES - NO
If yes, which one(s)?
YOUR TEXTBOOK EVALUATION EXPERIENCE:
[G] Have you had any specific training in textbook evaluation (e.g., courses)? YES - NO
If yes, what kind of training?
[H] Have you had any previous experience in textbook evaluation (e.g., at your work)? YES - NO
If yes, what kind of experience?
Is there anything else you would like to mention regarding your teaching experience, previous textbook evaluation experience, and/or the evaluation you did in March for the LATEX Materials Evaluation Research (e.g., I wanted to have commented more on something about..., I felt it too unnatural, etc.)?
Thanks for your help. Please return to k.johnson@lancaster.ac.uk
2. Research methodology issues
2.1 Some General Criticisms of the Think Aloud Protocol Method
Generally speaking, no data collection process is entirely without drawbacks; the best a researcher can do is to select a method that best fits the situation and yields the most information of the type required. In this section criticisms levelled at Verbal Reporting / Think Aloud protocols will be discussed, followed by an overview of particular issues we dealt with in our study.
One of the most commonly heard criticisms of the Think Aloud protocol is that it does not take place in a natural situation. This, however, is true for a lot of data collection techniques (unless an ethnographic study is feasible). Time constraints dictate that these 'artificial' situations must be set up, since the opportunities to observe teachers in their 'natural' environments evaluating textbooks may not arise at predictable or convenient times. In addition, a study like this affords the researcher the luxury of controlling for some variables among the participants, such as first language, range of experience (measured in years) and the particular textbook to be evaluated, all of which add to the reliability of the study. Our study was conducted with three EFL teachers, the nature of whose work, it could be argued, may have made them more used to 1:1 situations with strangers than teachers in other domains.
Some critics have - quite legitimately - suggested that a Think Aloud protocol cannot possibly give complete records of the thought processes. This is not a unique drawback of verbally reported data, though: it is virtually impossible to obtain complete records of complex human thought processes, some of which will be hidden from the thinkers themselves. Using concurrent verbalisation data collection methods does give some insights that other methods would not be able to reveal. Even pauses in the verbalisation can tell experienced researchers something about what's going on: when intense thinking is taking place (or none at all!).
There is an argument that not all participants are good at thinking aloud, or like doing it. The same can also be said for other more common methods of data collection such as interviews or questionnaire filling. Amongst teachers, classroom observations must surely rank as one of the least popular activities, and yet they are often the main method used for uncovering teaching and learning processes. Think aloud protocols at least have the benefit of privacy, which may make them more attractive to the participants. A short period of training or practice can help familiarise participants with the procedure, and enable them to successfully undertake the tasks that the researcher requires of them.
Perhaps the most serious drawback associated with verbal reporting is the possibility that thinking aloud may distort the very thinking process that is under investigation. This is a criticism mainly levelled at retrospective reporting, where there is a much greater burden placed on the memory than would normally be the case, and especially where the participants are asked to analyse their thought processes, rather than just to report them. Many studies which have employed this technique have been investigating second language learning, but this study was conducted in the participants first language, thus reducing to a minimum the additional cognitive load. This issue has been the subject of several studies (Ericsson & Simon, 1993; Leow & Morgan-Short, 2004) all of which failed to find evidence of reactivity in concurrent think aloud protocol data. An evaluation task conducted in the participants' first language (Biggs, Rosman & Serfenian, 1993) found that the only difference in task performance, was that it took a little longer than was the case in the control group. In our study, time was not a factor, and the participants were able to take as long as they liked.
2.2 Issues Specific to our Study
Apart from these general issues that had to be taken into account, there were a number of specific questions pertaining to this study that had to be clarified before the data could be collected. A large proportion of the preparatory discussion concerned the precise form that the instructions and the training should take.
It was agreed that the instructions should not to be too complicated, but that they should give clear information about what we expected the participants to do. The instructions were read aloud to them, and they were also provided with a written version to refer to if they wanted to. The instructions did not specifically ask them to explain their reasons when they did or did not like something, which may have seemed to be a useful request, but there is some evidence that this may have altered the thought process, and so we adopted a very hands-off approach to the evaluation process. One difficulty for the 'interviewer' was knowing how much and when to interfere with the thinking aloud, by prompting for more verbalisation. The key was to balance non-intervention with the aim of obtaining data that were as revealing as possible. In the event, it was only felt necessary to prompt once, to guide the participant towards evaluating the textbook. This suggests that the training procedure was adequate in most cases.
In order to help the participants prepare for the evaluation, it was decided that a choice of three different problem solving activities should be offered, so that the teachers could choose the one they felt most comfortable with. The final selection consisted of a partially-completed jigsaw puzzle, an anagram, or the classic ' Tower of Hanoi ' puzzle; all three chose to complete the jigsaw puzzle. This training phase has been shown to help even most reticent participant to contribute and is thus a crucial part of the study. With hindsight, it may have been prudent to ask the participants to attempt two of the activities, to provide more practice in thinking aloud as they worked. Other training activities, such as listening to a tape of someone doing a think aloud activity could also have been usefully employed.
Another methodological issue that had to be resolved was that of how to analyse the data once they had been gathered; the method that was eventually adopted evolved from the observation of the data and could thus be said to be extremely grounded. During the initial viewing of the videotapes, the team identified a range of promising aspects to explore in more detail: the sequence of the evaluation, what aspects of the book the teachers liked and disliked, the terminology used by each teacher, the references to methodology, the apparent flexibility of each participant with relation to the book. Each member took responsibility for one area, coded the data and analysed the findings. These findings were then discussed by the group, and, where necessary, re-categorised. The first two aspects were discovered to be the most fruitful areas for comparison across the different experience levels of the teachers, and the data were then cross-checked by other members of the team to verify the findings in these two respects. More work could be done on the other aspects in the future, but given the time constraints it was felt that more focus was necessary.
In any research project, one factor that should never be underestimated is that of the personalities involved, both the researchers and the respondents. Within the team there was a range of experience, both as regards research in general, and Think Aloud protocols in particular. Since there was little time for training, it was imperative that all the members took their share of the work load and worked to their individual strengths; this strategy worked well, and the team quickly cohered. It was harder to predict what effect the interaction of the three different participants would have, and this will always be an unknown variable in social science research. Ideally the same interlocutor would have conducted all three sessions, but due to the need to accommodate the participants' timetables, this proved to be impossible, and a change of interlocutor for the final session was necessary. The effect this may - or may not - have had on the performance of the third teacher cannot be known. Nor can it be estimated what, if any, effect the 'test' situation had on the teachers.
One final issue that has yet to be resolved is the relationship between experience and expertise. The participants' relative experience was measured initially only in terms of years of teaching, and was then further investigated using a questionnaire to ascertain whether they had had any training in textbook evaluation. It was found that there was a correlation between length of service and training in this aspect, which went some way towards validating the use of the terms 'experienced teacher' and 'novice'. However, without a much larger sample, it is difficult to separate the evaluations and reactions to the book that were due entirely to personal preference, from those that stemmed directly from experience.
3. Preliminary findings in the recent project
3.1 Sequencing
There was a clear difference among the three teachers in their sequencing while they were evaluating the textbook. Before proceeding with each teacher's different sequencing, it might be helpful to describe what materials were given to the teachers. They are the Scenario , Students' book , and Teacher's Book . The Scenario explains the context of teaching (Who the learners are, what their learning purpose is, etc.). The Students' book has 14 units, each of which is divided into a number of sections. In most (but not all) of the units are: Vocabulary, Grammar, Functional language, Speaking, Reading , Listening, and Writing. In the Students' book is also contained a Mini Book in which audio scripts and grammar are provided. The Teacher's book comprises: a statement of principles, methodological guidelines, a description of the Students' book and the Workbook, an introduction to the lesson-handling notes, lesson-handling notes of Unit 1 - Unit 14, and Workbook answers.
Novice teacher
- The Scenario
- Contents pages
- Unit 1 (activity 1 + activity 2)
? Unit 5 (activity 1 + activity 5)
? Teacher's book (description of the Workbook + Workbook answers + methodological guidelines)
? Unit 1 (Grammar)
? Mini Book
? Unit 1 (Functional language + Writing + Review))
? Unit 3 (Just check if Vocabulary section is followed by Pronunciation)
? Unit 10 (Just check if Vocabulary section is followed by Pronunciation)
- Unit 9 ? Unit 1 ? Unit 7 ? Unit 12 ? Teacher's book (Unit 7) ? Unit 14
(Check only Speaking section, looking at all of these units)
- Unit 7 ? Unit 10 ? Unit 13 ? Unit 8 ? Unit 6 ? Unit 11
(Check only Functional Language)
- Unit 10 ? Unit 11 ? Unit 12 ? Unit 5
(Check only Dictionary section)
- Overall comments
- Contents pages
- Teacher's book (Just to check if it contains Answer Key)
- Back cover
- Teacher's book (A statement of principles + Methodological guidelines + A description of the Student's book and the Workbook + An introduction to the lesson-handling notes + Unit 1 + Unit 2)
Intermediate teacher
- Identifying Student's book (with the Mini Book: audio scripts and mini grammar) & Teacher's book
- Contents pages
- Unit 1 ? Unit 2 ? Unit 3 ? Unit 4 ? Unit 5 ? Unit 6 ? Unit 7 ? Unit 8 ? Unit 9 ? Unit 10 ? Unit 11 ? Unit 12 ? Unit 13
(Looking at different parts or sections in different units)
- Overall evaluation (by the Researcher's request)
Experienced teacher
- Scenario
- Front and back cover
- Just flicking through the pages
- Contents pages
- Unit 4 (Looking at all of the 33 activities in great detail)
- Teacher's book
- Unit 8 (Check briefly most of the sections)
- Overall evaluation
Compared with the novice teacher's 'back and forth' approach, the experienced teacher's sequencing was very systematic. The intermediate teacher also showed a kind of systematicity. However, his evaluation might be called superficial because, after looking at the contents pages, he looked through almost all the units, i.e. thirteen units out of the fourteen, with very brief comments on different parts of different units, e.g., discussing 'topic' in Unit 4, 'vocabulary' in Unit 5, 'grammar' in Unit 6. He did not seem to achieve the experienced teacher's systematic approach at a deeper level. He appeared to know not only what to do in evaluating textbooks but also what to do with the textbook for its evaluation. We think this difference could be important because if textbooks play an important role in language education, which has been said in much of the literature (e.g. Cunningsworth, 1984; Hutchinson & Torres, 1994; Macian, 1986; Rivers, 1981), teachers need to know how to evaluate textbooks which they will use with their own students.
3.2 The Teachers' Likes and Dislikes:
During the process of analysing the data, we also found that all three participants mentioned several aspects that they liked, and things that they disliked about this textbook, in their evaluations. So we considered this area as one of the categories in our analysis. Here are just a few examples of what we found in this area. We tried to find out their respective likes and dislikes, along with the reasons they provided. From this framework we got the following results:
• First, we found out that the novice teacher mentioned 17 points of likes and dislikes (4 points are dislikes); Intermediate teacher mentioned 18 points (2 points are dislikes); experienced teacher mentioned 27 points (4 points are dislikes). It is obvious that they are more vocal about what they like than what they dislike although they have their own individual criteria of likes and dislikes. We also noticed that there are some overlaps in only one category. That is, all the three subjects liked the way of learning a new word provided in this textbook. They think it is better and more helpful to provide students with vocabularies along with the listening and pronunciation practice so that students have the opportunity to correct their pronunciation, practice stress, familiarize themselves with the different accent and work out what they really mean.
• We also found that among the 17 points given by the novice teacher, 12 of them are provided with the reasons (17:12); among the 18 points given by the intermediate teacher, 10 of them are presented with the reasons (18:10); among the 27 points given by the experienced teacher, 18 of them are provided with the reasons (27:18). If we take their own ratio into account, we can see that there isn't much difference between the novice teacher and the experienced one, but calculating from the overall number (40 reasons in all), the reasons given by the experienced teacher cover nearly 45% among the 40 reasons given by all the subjects.
Finally, it was also found that some motivations for the reasons were personal preference. Some are based on the practical implications on classroom practice, and some are theoretical considerations.
3.3 The Teachers' Use of Terminology
Another area of inquiry that we have pursued is what kind of terminology the three evaluators use. We have identified two broad categories of discourse used by the evaluators:
- TESOL discourse
- Subject-matter discourse - discourse about the English language and its structure.
Our findings in this area are preliminary, but some general trends seem to emerge. Let us consider the use of TESOL discourse first:
- TESOL discourse
novice practice of TESOL
intermediate
experienced theory of TESOL
There seems to be a cline with the novice and the experienced teacher at the two ends. The novice teacher exclusively uses terms that pertain to the practical aspects of TESOL (types of activities: warmers, warm up activities, extra activities, gap fill exercise, extension activities, speaking activities; types of materials: dictionaries ). She does not seem concerned with more theoretical issues. This is in keeping with novice teachers' preoccupation with 'survival' in the classroom, and how this survival is dependent on being able to provide enough 'activities' and 'materials' to fill the allotted lesson time.
The experienced teacher, on the other hand, has the richest and the most varied TESOL-related discourse of all three evaluators and the one that contains the most references to theoretical issues ( a skills-driven syllabus, activity-led teaching). Also, he is the only evaluator who consistently uses terms that belong to the 'jargon' of TESOL materials evaluation ( They even start off with the present tenses which is pretty normal, the book seems to have settled down a bit into a sort of rhythm , there's a lot of white space ).
The intermediate teacher is in the middle. He uses a wider range of TESOL-related terminology than the novice teacher, although it is still related to the practical/procedural side rather than the theoretical one. On occasion he mentions theoretical issues for which he lacks the specific terms and resorts to paraphrase and circumlocution ( we're almost backtracking to a certain extent because we've done something before and then we come back to use it again, which is good )
If we consider subject-matter discourse now, we can see a similar cline at work:
- Subject - matter discourse
novice mined from TB 'taken on trust'
intermediate
experienced evaluator-appropriated nominated
The novice teacher mines from the textbook all the linguistic terms she mentions. She never elaborates on them but takes them 'on trust' (e.g. for her grammar is grammar - end of the story). It is important to point out that on one occasion the evaluation process helps the novice teacher 'notice' one term ( functions/functional language ) and develop her own working definition of what it stands for. When she encounters the term for the first couple of times, she doesn't 'notice' it; only on her third encounter does she start paying attention to it; she looks for clues as to its meaning and the evaluation process turns into a consciousness-raising exercise for her:
grammar and functional language ...followed by pronunciation
ok so when the words and the pronunciation are linked to the previous language, whether it be vocab or functional language
right. let's see. what are these functions all about?...so the function is different things in each chapter....ok is the functional language more aimed at common usage things that they're going to be using in the situations they find themselves in?..... the functional stuff looks as though it's addressing situations that they might find themselves in on a more regular basis, rather than the topics which are just used to facilitate the exercises I think......
OK so functional stuff is based predominantly by the looks of things, on situations that they will find themselves in away from home, which is very good......does it say anything in the actual exercises with the point of the functional ...?
So that they get to practise things that they are going to use perhaps on a more regular basis, at least weekly, if not daily.
The experienced teacher nominates several terms related to the subject matter himself ( I'm just wondering if they're corpus-based or not; they're encouraged to think about things in these semantic sets ) and even when he lifts terms from the textbook, he appropriates them, he gives them his personal interpretation ( coherence and cohesion..... so where is the cohesion? Presumably to put it back together I need to have a grasp of the cohesion ).
The intermediate teacher still relies on the textbook a great deal, but is slightly more adventurous than the novice teacher. Indeed, he comes up with his own labels for some linguistic phenomena ( so we're looking really at combined/combination noun and combination word ) and mentions concepts which he labels with the wrong term ( also practise different stress giving a completely new meaning to what you're trying to say. So you could be angry, or sad, or happy and so ).
The findings we have reported are, as we have said, still tentative. However, they seem to confirm that of the three evaluators the experienced teacher is the one who has the best grasp of the TESOL profession and all its complexities.
3.4 Flexibility
One of the ways we tried to analyse the transcripts was based on the idea of flexibility: how teachers use the textbook when they do not like the ideas presented in the textbook and ways they change activities. This was analysed by using a grounded theory approach. After several discussion meetings, we finally decided on eight categories, these are: Supplementing or doing other activities, Re-ordering activities, Trying out activities to see if they work, Explaining new vocabulary, Clarifying unclear symbols, Thinking how the mini-grammar book is used, Thinking how the key is used, Setting up a purpose for the activity, Teaching without the Teacher's book.
Instead of showing results of each category here, only findings in the first category-Supplementing or doing other activities-are presented in detail. The table below shows that both the intermediate and experienced teacher in our study provided the same number of examples in their awareness of supplementing the material. However, there is a slight difference between their evaluations. As shown below in the table, the intermediate teacher described the activities he wanted to use in a very specific way as though he was designing activities for a lesson, whereas the experienced teacher just mentioned in a very general way by saying that he wanted to "do extra things" or needed "some supplementing". This difference further lead us to believe that the experienced teacher's evaluation fitted in with what Patel & Groen (1991) and Hunt (1989) found about specialists and novices in their medical diagnosis where the former ones knew how to "short-cut" the procedure to avoid the hard work and arrive at the solution by the quickest way, while the latter worked very hard and seemed to consider many possibilities to reach the conclusion. As described, the intermediate teacher in our study worked very hard and spent much time on describing activities, in contrast, the experienced teacher was aware of the deficit in the textbook but he did not go into details because he knew that at that moment he did not have to design detailed activities and this gave him more time to consider other issues related to the textbook evaluation rather than painstakingly struggle for detailed activity design.
As to the comparison of novice and experienced teacher, although the novice teacher evaluated the textbook the same as the experienced teacher did, in terms of the total number of examples provided, the novice teacher was found to have not yet reached the same level of flexibility in using the textbook as the experienced teacher had.
To conclude this part, the above analysis reveals that the experienced teacher in our study, in comparison with the intermediate and novice teacher, seems to be more flexible in evaluating and using the textbook in terms of the idea of supplementing the material . Besides this finding, the experienced teacher was also discovered not only to express his interests in a more insightful way, but also to be able to consider ideas and terms in language teaching in his evaluation. This finding is further explained in the 'terminology' section above. In order to confirm the hypothesis that the novice teacher is less flexible in evaluating the textbook than the experienced teacher, we need to do some further detailed analysis in the other categories mentioned. The further results will be updated as soon as the analysis has been done.
Flexibility
Examples:
Categories |
NOVICE Teacher |
INTERMEDIATE Teacher |
EXPERIENCED Teacher |
Supplementing or doing other activities |
1. Some students may work a lot faster so it might be necessary to incorporate extra activities from a different text book series to these. 2. OK, so I'd need to add some warmers into this I think at the beginning. It's not something that I'd go straight into. It's not particularly appropriate I don't think for the start of a new class. |
1. So that could be a nice little role play in the classroom. Where we could have a copper and a thief, or whatever, and an observer who could then tell us what was happening. 2...looking at Reading : Shakespeare, asking about the plot, the type of play, film, actors, characters and so on. Why not get them-they probably are-asked to write something themselves about what they've just seen. 3. So the student can talk about traveling, or indeed that could segue into talking about your own country and encouraging me to visit. So we've got there a tourist, a travelogue or something which we could develop there. 4. They don't actually have an opportunity to make their own flow chart of something. Why not? That would be quite good. Get them to work it out themselves. 5. I wonder if they'd like to bandage somebody, demonstrate how it's done. At the same time they could tell us how they're doing it so we've got them thinking aloud. 6. It doesn't appear to play any music although we're talking about music, and yet we've got tapes. So why not have music? It's all very well talking about it. It would be even better if you could show them or listen. 7. So we're looking really at combined/combination noun and combination word, and yet we don't seem to exploit it. Further on we could do more with it. |
1. Maybe I would want to do a bit more input on that and have them work up to a debate. 2. Here we are, talking about timing. 6-8 hours of classroom time. So I said 16 hours, but I said I was going to do extra things. 3. There may be room to jump off and do things for a while. 4. But I'd want closer to 15 or 16 hours if I was teaching it because I could jump off and do other things. 5. It's going to need some supplementing if it is a longer course; it's going to need some supplementing and some ideas for jumping off points. 6. Space for notes at the back - kind of nice. I like that, that's good. That's a plus. That means I can do a lot more with it than I thought I could originally. 7. So if I did that as a lesson, I think I'd have to supplement it. I'd have to have, or at least brain storm on the board, some phrases and some useful expressions for getting the things they wanted to say, and so on. |
References:
Biggs, S. F. Rosman, A.J. & Serfenian, G.K. (1993). 'Methodological Issues in Judgment and Decision-making research: Concurrent verbal protocol validity and simultaneous traces of process data'. Journal of Behavioural Decision Making 9, p.234 - 248
Cunningsworth, A. (1984). Evaluating and selecting EFL teaching materials. London : Heinemann Educational Books.
Ericsson, K. & Simon, H. (1993). Protocol Analysis: verbal reports as data (Rev. Ed.) Cambridge MA ; MIT Press
Harmer, J. (2004). Just Right London : Marshall Cavendish, ELT
Hunt, E. (1989) 'Cognitive science: definition, status and questions' Annual Review of Psychology 40:603-629.
Hutchinson, T. & Torres, E. (1994). 'The textbook as agent of change'. ELT Journal 48(4): 315-328.
Leow, R. & Morgan-Short, K. (2004). 'To Think Aloud or not To Think Aloud: the issue of reactivity in SLA Research Methodology'. Studies in Second Langue Acquisition 26, 1, Mar, p.35-57
Macian, F.L. (1986). 'An analysis of state adopted foreign language textbooks used in first- and third-year high school Spanish classes'. Foreign Language Annals 19(2): 103-118. Patel, V. L. and Groen, G. J. (1991). 'The general and specific nature of medical expertise: a critical look' In Ericsson, K. A. & Smith, J. (Eds.) Towards a General Theory of Expertise. Cambridge : Cambridge University Press. Rivers, W. (1981). Teaching Foreign Language Skills Chicago : University of Chicago Press.
4 Supplementary bibliographies
4.1 Think-Aloud Protocol Bibliography
Arndt, V. (1987). Six writers in search of texts: A protocol-based study of L1 and L2 writing. ELT Journal, 41 (4), 257-267.
Brown, J.S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-42.
Cooper, M. & Holzman, M. (1983). Talking about protocols. College Composition and Communication, 34 (1), 284-293.
Ericsson, K.A. & Simon, H.A. (1980). Verbal reports as data. Psychological Review, 87 (3), 215-251.
Flower, L. & Hayes, J.R. (1981). A cognitive process theory of writing. College Composition and Communication, 32 , 365-387.
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook . Cambridge : Cambridge University Press.
Johnson, K. (2003). Designing language teaching tasks . Hampshire: Palgrave Macmillan.
Raimes, A. (1985). What unskilled ESL students do as they write: A classroom study of composing. TESOL Quarterly, 19 (2), 229-258.
Scribner, S. (1984). Studying working intelligence. In B. Rogoff & J. Lave (Eds.). Everyday cognition: It's development in social context (pp. 9-40). London : Harvard University Press.
Smagorinsky, P. (1989). The reliability and validity of protocol analysis. Written Communication, 6 (1), 463-479.
Swarts, H., Flower, L.S., & Hayes, J.R. (1984). Designing protocol studies of the writing process: An introduction. In R. Beach & L.S. Bridwell (Eds.). New directions in composition research (pp. 53-71). New York : The Guilford Press.
Woods, D. (1996). Teacher cognition in language teaching: Beliefs, decision-making and classroom practice . Cambridge : Cambridge University Press.
4.2 Expertise Bibliography
Bereiter, C., & Scardamalia, M. (1983). Does learning to write have to be so difficult? In Freeman, A., Pringle, I. , & Yalden, J. (Eds.). Learning to write: First language/second language (pp. 20-33). London : Longman.
Bereiter, C., & Scardamalia, M. (1993). Surpassing ourselves: An inquiry into the nature and implications of expertise . Chicago : Open Court.
Ericsson, K.A. & Smith, J. (Eds.). (1991). Toward a general theory of expertise . Cambridge : Cambridge University Press.
Hidi, S. & Klaiman, R. (1983). Notetaking by experts and novices: An attempt to identify teachable strategies. Curriculum Inquiry, 13 (4), 377-395
Johnson, K. (1996). Language teaching and skill learning . Oxford : Blackwell.
Tsui, A.B.M. (2003). Understanding expertise in teaching . Cambridge : Cambridge University Press.
Welker, R. (1991). Expertise and the teacher as expert: Rethinking a questionable metaphor. American Educational Research Journal, 28 (1), 19-35.
4.3 Textbook Evaluation Bibliography
Buckingham, T. (1978). Some basic considerations in the assessment of ESL materials. Mextesol Journal, 3 (2), 26-36.
Candlin, C. N., & Breen, M. P. (1979). Evaluating, adapting, and innovating languageteaching materials. In C. Yorio, K. Perkins & J. Schachter (Eds.), On TESOL '79: The learner in focus (pp. 86-108). Washington, D.C.: TESOL.
Candlin, C. N., & Widdowson, H. G. (Eds.). (1992). Evaluation . Oxford: Oxford University Press.
Cunningsworth, A. (1984). Evaluating and selecting EFL teaching materials . London: Heinemann Educational Books.
Cunningsworth, A. (1995). Choosing your coursebook . Oxford: Heinemann.
Cunningsworth, A., & Kusel, P. (1991). Evaluating teachers' guides. ELT Journal, 45 (2), 128-139.
Daoud, A.-M., & Celce-Murcia, M. (1979). Selecting and evaluating a textbook. In M. Celce-Murcia & L. McIntosh (Eds.), Teaching English as a second or foreign language (pp. 302-307). Rowley, Massachusetts: Newbury House Publishers, Inc.
Davison, W. F. (1976). Factors in evaluating and selecting texts for the foreign-language classroom. English Language Teaching Journal, 30 (4), 310-314.
Ellis, R. (1997). The empirical evaluation of language teaching materials. ELT Journal, 51 (1), 36-42.
Eraut, M., Goad, L., & Smith, G. (1975). The analysis of curriculum materials . Sussex, England: University of Sussex.
Johnson, R. K. (Ed.). (1989). The second language curriculum . Cambridge: Cambridge University Press.
Jones, F. R. (1993). Beyond the fringe: A framework for assessing teach-yourself materials for ab initio English-speaking learners. System, 21 (4), 453-469.
Lee, J. J. (1975). Choosing and using a textbook. English Teaching Forum, 13 (3/4), 364-365.
Littlejohn, A. (1998). The analysis of language teaching materials: Inside the trojan horse. In B. Tomlinson (Ed.), Materials development in language teaching (pp. 190-216). Cambridge: Cambridge University Press.
Low, G. (1987). The need for a multi-perspective approach to the evaluation of foreign language teaching materials. Evaluation and Research in Education, 1 (1), 19-29.
Mariani, L. (1980). Evaluating coursebooks. Modern English Teacher, 8 (1), 27-31.
Matthews, A. (1985). Choosing the best available textbook. In A. Matthews, M. Spratt & L. Dangerfield (Eds.), At the chalkface: Practical techniques in language teaching (pp. 202-206). London: Edward Arnold.
McDonough, J., & Shaw, C. (2000). Materials and methods in ELT: A teacher's guide . Oxford: Blackwell.
McGrath, I. (2002a). Choosing a coursebook 1: From analysis to first-glance evaluation. In Materials evaluation and design for language teaching (pp. 17-39). Edinburgh: Edinburgh University Press.
McGrath, I. (2002b). Choosing a coursebook 2: Close evaluation. In Materials evaluation and design for language teaching (pp. 40-56). Edinburgh: Edinburgh University Press.
McGrath, I. (2002c). A systematic approach to materials evaluation. In Materials evaluation and design for language teaching (pp. 7-16). Edinburgh: Edinburgh University Press.
Roberts, J. (1998). Evaluation of course books or materials evaluation. In K. Johnson & H. Johnson (Eds.), Encyclopedic dictionary of applied linguistics (pp. 119-125). Oxford: Blackwell.
Robertson, E. F. (1997). English for university studies: An evaluation of reading materials. Folio, 3 (2), 20-25.
Romero, R. M. (1975). What textbook shall we use. English Teaching Forum, 13 (3/4), 362-363.
Roxburgh, J. (1997). Procedures for the evaluation of in-house EAP textbooks. Folio, 4 (1), 15-18.
Rubdy, R. (2003). Selection of materials. In B. Tomlinson (Ed.), Developing materials for language teaching (pp. 37-57). London: Continuum.
Sheldon, L. E. (1988). Evaluating ELT textbooks and materials. ELT Journal, 42 (4), 237-246.
Sheldon, L. E. (Ed.). (1987). ELT textbooks and materials: Problems in evaluation and development . London: Modern English Publications and The British Council.
Skierso, A. (1991). Textbook selection and evaluation. In M. Celce-Murcia (Ed.), Teaching English as a second or foreign language (pp. 432-453). Boston: Heinle and Heinle.
Stevick, E. W. (1972). Evaluating and adapting language materials. In H. B. Allen & R. N. Campbell (Eds.), Teaching English as a second language (pp. 102-120). New York: McGraw-Hill.
Tomlinson, B. (2003). Materials evaluation. In B. Tomlinson (Ed.), Developing materials for language teaching (pp. 15-36). London: Continuum.
Troncoso, C. R. (2005). The matter of selecting and evaluating materials. Folio, 9 (2), 23-26.
Tucker, C. A. (1975). Evaluating beginning coursebooks. English Language Teaching Forum, 13 (3-4), 355-361.
Weir, C., & Roberts, J. (1994). Evaluation in ELT . Oxford: Blackwell.
Williams, D. (1983). Developing criteria for textbook evaluation. ELT Journal, 37 (3), 251-255.
Williams, R. (1981). A procedure for ESP textbook analysis and evaluation on teacher education courses. ESP Journal, 1 (2), 155-162.